system

Abstract

Provide a system. A means for converting voice data into text data in real time; means for periodically transmitting text data to a server; A means for analyzing the text data in the server and extracting important statements; A means for generating a summary based on the extracted utterances; a means for notifying a user of the summary; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] While it is important for parents to understand the content of their children's everyday speech and conversations, in modern society parents must be mindful of privacy issues and avoid excessive parental interference. At the same time, there is a need for systems that allow parents to communicate effectively with their children and respond quickly if a problem arises. However, existing systems require audio recording and storage, which poses privacy concerns. Furthermore, the lack of summaries and detailed information makes it difficult for parents to accurately obtain the information they need. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides a system that converts voice data into text data in real time and transmits the text data to a server. The server is provided with means for analyzing the received text data, extracting important comments, and generating a summary. The system also includes means for notifying the summary to a parent's smartphone, allowing the parent to input questions about points of concern after checking the summary. When a question is input, the system reanalyzes the text data to generate detailed information and provides it to the parent's smartphone, thereby realizing a system that allows the parent to accurately obtain the information they need while ensuring privacy.

[0006] "Audio data" is data that represents audio in digital form.

[0007] "Text data" is data that represents character information in digital form.

[0008] "Real-time" means that the results are reflected almost immediately after the processing or operation is performed.

[0009] A "server" is a computer system that sends, receives, and processes data over a network.

[0010] "Important statements" refer to specific statements or parts of conversations that parents are deemed to be aware of in their children's daily lives.

[0011] "Extraction" refers to the process of extracting specific elements from data.

[0012] A "summary" is a concise sentence that condenses important information extracted from text data.

[0013] "Notice" refers to the act of communicating information to a user by electronic means.

[0014] A "question" refers to an inquiry entered by a parent requesting additional information about a concern.

[0015] "Reanalysis" refers to the process of extracting additional information by reanalyzing existing text data.

[0016] "Detailed information" is information provided as a specific answer to a question. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to a system that enables parents to efficiently understand the content of their children's daily utterances and conversations. This system is designed to ensure that parents can obtain necessary information without interfering excessively. The following describes in detail an embodiment of the present invention.

[0039] First, the device worn by the child has the ability to convert voice data into text data in real time. The device uses a built-in microphone to capture the child's everyday speech and converts it into text using voice recognition software. This text data is then sent from the device to a server at regular intervals.

[0040] The server receives the text data sent from the device and stores it in a database. The server then analyzes the stored text data to extract important statements. This analysis uses natural language processing (NLP) algorithms to find important information based on specific keywords and context. The extracted information is then generated as a summary.

[0041] After the summary is generated, the server notifies the parent's smartphone. A dedicated app is installed on the parent's smartphone, and the parent can check the summary content through the smartphone. If the parent wants to know more details about the summary, they can input a question through the app. For example, "What did you do in art class today?"

[0042] When a question is entered, the server accepts the question and reanalyzes the related text data. The reanalyzed information is then sent back to the user's (parent's) smartphone as detailed information, allowing the parent to understand the specific situation and content.

[0043] Examples:

[0044] For example, suppose a device generates text from an utterance such as "Today's art class was fun" and sends it to a server. The server analyzes this text data and summarizes it as "I enjoyed art class." This summary is then sent to the parent's smartphone, who then types in a question such as "What did you do in art class?" The server then analyzes the text data again and generates more detailed information such as "I drew pictures during art class," which is then provided to the parent's smartphone.

[0045] As described above, this system goes through each important step to efficiently convey the contents of a child's statements and conversations to parents while protecting their privacy, allowing parents to communicate more smoothly and quickly obtain the information they need.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The device activates the microphone and captures the child's voice. The device is set to collect voice data at regular intervals.

[0049] Step 2:

[0050] The device converts the voice data it receives into text data in real time, using voice recognition software to convert the voice signal into text information.

[0051] Step 3:

[0052] The device then cleans up the converted text data and corrects it to make it grammatically meaningful, which makes subsequent parsing easier.

[0053] Step 4:

[0054] The device sends the organized text data to the server at regular intervals (e.g., every 30 minutes) using a secure communication protocol.

[0055] Step 5:

[0056] The server receives the text data sent from the device and stores it in a database. All received text is saved and can be used for later analysis.

[0057] Step 6:

[0058] The server applies natural language processing (NLP) algorithms to analyze the stored text data, extracting important statements based on specific keywords and context.

[0059] Step 7:

[0060] The server generates a summary based on the extracted important utterances. The summary is written in a concise and meaningful way for parents.

[0061] Step 8:

[0062] The server then sends the generated summary to the parent's smartphone via a dedicated app.

[0063] Step 9:

[0064] The user (parent) receives a notification on their smartphone, launches the app, and checks the summary, which is displayed in a format that is easy for the parent to understand.

[0065] Step 10:

[0066] If the user (parent) has any questions about the summary, they can enter them in the question form within the app. For example, "What did you do in art class?"

[0067] Step 11:

[0068] The server receives the parent's question, re-analyzes the relevant text data, and applies additional natural language processing algorithms depending on the question to extract more detailed information.

[0069] Step 12:

[0070] The server generates detailed information obtained through reanalysis and sends it to the user's (parent's) smartphone. The detailed information is also provided through a dedicated app.

[0071] Step 13:

[0072] The user (parent) receives the detailed information and checks it in the app, allowing the parent to understand the specific details of any concerns.

[0073] Example 1

[0074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0075] In recent years, while it has become increasingly important for parents to understand their children's daily events and emotions, there is also a growing need to avoid excessive interference. However, it is difficult to efficiently obtain the necessary information using conventional methods. The purpose of this invention is to enable parents to efficiently and appropriately understand their children's daily utterances and conversations.

[0076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0077] In this invention, the server includes: [means for converting voice data into text data in real time; [means for periodically transmitting the text data to the server; [means for analyzing the text data in the server and extracting important comments; [means for generating a summary based on the extracted comments; [means for notifying the user of the summary; [means for transmitting a question entered by the user to the server; [means for reanalyzing the text data based on the question and generating detailed information; and [means for notifying the user of the detailed information. This enables parents to efficiently keep track of their children's daily lives.

[0078] "Converting voice data into text data in real time" means instantly converting collected voice information into text information using voice recognition technology.

[0079] "Periodic transmission of text data to a server" means transferring the generated text information to a server via a network at regular time intervals.

[0080] "Analyzing text data on the server and extracting important statements" means analyzing the text information stored on the server using a natural language processing algorithm to extract meaningful information and keywords.

[0081] "Generating a summary based on extracted statements" means generating a concise sentence based on important information obtained from the analysis results.

[0082] "Notifying the user of the summary" means sending the generated summary to the user's device by means of push notification, email, or the like.

[0083] "Sending a question entered by a user to a server" means transferring a question entered by a user through a dedicated application to a server.

[0084] "Reanalyzing text data based on the question and generating detailed information" means reanalyzing the saved text information according to the content of the user's question and extracting related detailed information.

[0085] "Notifying the user of detailed information" means sending the reanalyzed results to the user's device and providing detailed information.

[0086] The present invention relates to a system that enables parents to efficiently understand their children's daily utterances and conversations. This system is designed to allow parents to reliably obtain necessary information while avoiding excessive interference. Specific embodiments for carrying out the present invention will be described below.

[0087] Hardware used

[0088] The device is a special wearable device that can be worn by children and has a built-in microphone that picks up surrounding sounds in real time.

[0089] The server used is a high-performance cloud-based database server, which can efficiently process and store large amounts of data.

[0090] The user device required is a smartphone for parents to receive information, on which a dedicated application is installed.

[0091] Software used

[0092] Speech recognition software: Speech recognition software such as Google® Speech-to-Text API is installed on the device, and converts voice data captured in real time into text data.

[0093] Natural language processing libraries: The server uses natural language processing algorithms such as SpaCy and NLTK to analyze the text data and extract important statements.

[0094] Database software: Data is stored in a database such as MySQL® or PostgreSQL.

[0095] Push notification service: Using services such as Firebase Cloud Messaging, the generated summary is sent to the parent's smartphone.

[0096] Specific examples of processing

[0097] A specific example of the process is as follows:

[0098] 1. Acquiring voice data: A child says, "I enjoyed art class today." The device's built-in microphone collects this voice data.

[0099] 2. Voice data to text conversion: The collected voice data is converted into text data in real time by the device's voice recognition software.

[0100] 3. Sending text data: The converted text data is sent to the server at regular intervals. Communication is carried out using the REST API.

[0101] 4. Analysis of text data and generation of summary sentence: The server analyzes the received text data and generates a summary sentence such as "I enjoyed art class" using a natural language processing library.

[0102] 5. Notification of summary: The generated summary is sent to the parent's smartphone via a push notification service.

[0103] Get more information

[0104] When a parent types a question into the app, such as "What did you do in art class?", the server receives the question, reanalyzes the text data, and finds the detailed information, such as "I drew a picture." This information is then sent back to the parent's smartphone.

[0105] Prompt Sentence Examples

[0106] An example of a prompt sentence when using a generative AI model is as follows:

[0107] A child says, "I enjoyed art class today." Generate a summary for the parent.

[0108] ---

[0109] Parents are asking, "What did you do in art class?" Please provide more information.

[0110] As described above, the present invention provides a means for parents to efficiently understand their children's daily lives by appropriately processing the information required at each step. This system facilitates smoother communication between parents and children, and enables them to quickly obtain the necessary information.

[0111] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0112] Step 1:

[0113] Acquiring audio data

[0114] Terminal

[0115] Example of operation: A child wears a dedicated device and begins everyday conversation. The device's built-in microphone collects voice data.

[0116] Input: Child's speech (audio data)

[0117] Data processing: The microphone captures the audio data.

[0118] Output: Audio data stored on the device

[0119] Step 2:

[0120] Converting audio data to text

[0121] Terminal

[0122] Example of operation: When a child says, "I enjoyed art class today," this speech is instantly converted into text data.

[0123] Input: Stored audio data

[0124] Data processing: Speech recognition software (e.g., Google Speech-to-Text API) recognizes the voice data and converts it into text.

[0125] Output: Converted text data

[0126] Step 3:

[0127] Sending text data

[0128] Terminal

[0129] Example of operation: The converted text data is sent to the server at regular intervals.

[0130] Input: Converted text data

[0131] Data transfer: Send text data to the server using the REST API.

[0132] Output: Text data sent to the server

[0133] Step 4:

[0134] Receiving and storing text data

[0135] server

[0136] Example of operation: The server saves the received text data in a database.

[0137] Input: Text data sent

[0138] Data processing: Save the text data in a database (e.g., MySQL).

[0139] Output: Text data stored in a database

[0140] Step 5:

[0141] Text data analysis

[0142] server

[0143] Example of how it works: Extracting important utterances from stored text data.

[0144] Input: Text data stored in a database

[0145] Data Computing: Use natural language processing algorithms (e.g., SpaCy, NLTK) to analyze important keywords and context.

[0146] Output: Important statements as a result of the analysis

[0147] Step 6:

[0148] Summary generation

[0149] server

[0150] Example of operation: Generate a summary sentence such as "Art class was fun" from the analysis results.

[0151] Input: Important statements as a result of the analysis

[0152] Data computation: Generative AI models and rule-based algorithms are used to create summaries based on key statements.

[0153] Output: Generated summary

[0154] Step 7:

[0155] Summary notification

[0156] server

[0157] Example of how it works: The summary text is sent to the parent's smartphone.

[0158] Input: Generated summary

[0159] Data transfer: A summary is sent to the parent's smartphone using a push notification service (e.g., Firebase Cloud Messaging).

[0160] Output: Summary displayed on parent's smartphone

[0161] Step 8:

[0162] Accepting user questions

[0163] User

[0164] Example of how it works: A parent types a question into a dedicated app, such as "What did you do in art class?", and sends it.

[0165] Input: Questions entered by the parent

[0166] Data transfer: Send your questions to the server through the app.

[0167] Output: The question sent to the server

[0168] Step 9:

[0169] Reanalysis of detailed information

[0170] server

[0171] Example of how it works: The server analyzes the text data again and finds the detailed information "I drew a picture."

[0172] Input: Submitted questions and saved text data

[0173] Data operations: Reanalyze relevant text data based on the question to extract more information.

[0174] Output: Reparsed details

[0175] Step 10:

[0176] Notification of detailed information

[0177] server

[0178] Example of how it works: The server sends detailed information to the parent's smartphone.

[0179] Input: Extracted details

[0180] Data transfer: Send detailed information to the parent's smartphone via push notification service.

[0181] Output: Detailed information displayed on the parent's smartphone

[0182] (Application example 1)

[0183] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0184] The present invention aims to provide a system that allows parents to efficiently understand their children's everyday comments and conversations and quickly and reliably obtain the necessary information. Another objective is to provide a sense of security for parents and improve their children's security by notifying them of important comments based on specific conditions while avoiding excessive interference. Another important objective is to be able to quickly respond to parents' requests for more information.

[0185] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0186] In this invention, the server includes: [means for converting voice data into text data in real time;] [means for periodically transmitting the text data to the server;] [means for analyzing the text data in the server and extracting important comments;] [means for generating a summary based on the extracted comments;] [means for notifying the user of the summary; and [means for notifying the user of the content of important comments on a display device based on specific conditions.] This allows parents to efficiently and quickly understand the content of their children's comments, giving them a sense of security and allowing them to quickly obtain the information they need.

[0187] "Audio data" means audio recorded in digital format and converted into text data in real time.

[0188] "Text data" refers to character information converted from voice data, and is sent to a server for analysis.

[0189] A "server" is a computer system connected via a network that is a central processing unit for receiving and analyzing text data.

[0190] "Analysis" is the process of understanding the text data transmitted and extracting important information based on specific criteria.

[0191] "Important comments" refer to comments that are worth notifying parents about, identified based on pre-set keywords and conditions.

[0192] A "summary" is a short document generated based on important statements extracted from text data, and is information that is notified to the user.

[0193] "Notification" is the act of transmitting information from a server to a user's display device, and is a means of providing information to a user in real time.

[0194] "Specific conditions" refer to rules such as keywords and contexts that are set as criteria for extracting important statements.

[0195] The "display device" is hardware for displaying notified information such as a summary, and is an interface through which the user can check the information.

[0196] In the present invention, the system is implemented based on the following configuration and procedures.

[0197] Hardware and Software Configuration

[0198] 1. Voice data collection terminal

[0199] The voice-collecting devices, such as smart bands and headsets worn by children, have built-in microphones and capture the child's everyday speech in real time.

[0200] 2. Speech Recognition Engine

[0201] The collected voice data is converted into text data in real time using a voice recognition engine, such as the Google Speech-to-Text API.

[0202] 3. Server

[0203] The converted text data is periodically sent to the server, which is responsible for the following:

[0204] Text Data Analysis

[0205] Use natural language processing (NLP) algorithms (e.g., the Transformers library) to analyze the incoming text data and extract key utterances.

[0206] Summary sentence generation

[0207] A summary is generated based on the extracted important statements.

[0208] notification

[0209] The summary sentence is sent to the parent's display device (e.g., smart glasses), and important utterances are instantly notified based on specific conditions.

[0210] 4. User Display Device

[0211] The parent's display device is a visual device such as smart glasses that displays the summary sent from the server and important comment notifications.

[0212] Data processing and calculation

[0213] 1. Audio data collection and conversion

[0214] Voice data is picked up in real time through a microphone and converted into text data by a voice recognition engine.

[0215] 2. Transmission and analysis of text data

[0216] The converted text data is sent to a server where it is analyzed using natural language processing algorithms, which extract important statements based on predefined keywords and context.

[0217] 3. Summary generation and notification

[0218] A summary is generated based on the extracted important comments and is displayed on the parent's display device. Furthermore, important comments based on specific conditions are immediately displayed.

[0219] Specific examples

[0220] For example, suppose a child says, "I enjoyed art class today." This voice data is converted into text data in real time and sent to the server. The server analyzes this text data, extracts the important statement, "I enjoyed art class," and summarizes it concisely. The parent's smart glasses are notified that "I enjoyed art class," and the parent can check detailed information from the smart glasses' interface. If the parent wants to know more, they can enter a question such as "What did you do in art class?" and the server will reanalyze the data and provide detailed information such as "I drew pictures during art class."

[0221] Prompt Sentence Examples

[0222] An example of a specific prompt from the user would be:

[0223] If you receive a user statement such as "I had a fight with a friend today," please determine whether parents should be notified and, if so, suggest the specific format in which they should be notified.

[0224] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0225] Step 1:

[0226] The child's device acquires voice data. The input is the child's everyday speech, which is collected using a microphone. The output is raw voice data.

[0227] Step 2:

[0228] The speech recognition engine converts raw speech data into text data. The input is captured speech data, which is converted into text information in real time using the Google Speech-to-Text API. The output is text data.

[0229] Step 3:

[0230] The terminal periodically sends the converted text data to the server. The input is the text data, which is sent to the server via the network. The output is the text data that arrives at the server.

[0231] Step 4:

[0232] The server analyzes the received text data using a natural language processing (NLP) algorithm. The input is the text data sent to the server, and an NLP model (e.g., the Transformers library) is used to extract important statements based on specific keywords or conditions. The output is a list of important statements.

[0233] Step 5:

[0234] The server generates a summary based on the extracted important comments. The input is a list of important comments, which is converted into a concise summary. The output is the summary.

[0235] Step 6:

[0236] The server notifies the user of the summary sentence on the display device. The input is the summary sentence, which is sent to the parent's smart glasses via the network. The output is the summary sentence notification displayed on the parent's smart glasses.

[0237] Step 7:

[0238] The server instantly notifies the parent of important comments based on specific conditions. The input is text data containing specific keywords, which triggers the immediate transmission of important comments to the parent's display device. The output is a notification of the important comment instantly displayed on the parent's smart glasses.

[0239] Step 8:

[0240] The user inputs a question requesting more information. The input is the question from the parent's smart glasses and is accepted by the server. The output is the question data.

[0241] Step 9:

[0242] The server re-analyzes the question and generates detailed information. The input is the user's question and related text data, and the NLP algorithm is applied again to generate detailed information. The output is detailed information.

[0243] Step 10:

[0244] The server sends the detailed information to the user's display device. The input is the detailed information, which is sent over the network to the parent's smart glasses. The output is the detailed information displayed on the parent's smart glasses.

[0245] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0246] The present invention combines a system that allows parents to efficiently understand the content of their children's daily utterances and conversations with an emotion engine that recognizes the user's emotions. This system is designed to ensure that parents can reliably obtain necessary information while avoiding excessive parental interference. The following describes in detail the embodiments of the present invention.

[0247] First, the device worn by the child has the ability to convert voice data into text data in real time. The device uses a built-in microphone to capture the child's everyday speech and converts it into text using voice recognition software. This text data is then sent from the device to a server at regular intervals.

[0248] The server receives the text data sent from the device and stores it in a database. The server then analyzes the stored text data to extract important statements. This analysis uses natural language processing (NLP) algorithms to find important information based on specific keywords and context. The extracted information is then generated as a summary.

[0249] Furthermore, the server is equipped with an emotion engine that recognizes the user's (parent's) emotions. When generating a summary, the emotion engine references the parent's emotion data and creates a summary in an appropriate format and with appropriate content. It also provides detailed information tailored to the parent's emotions as needed based on questions or requests from the parent.

[0250] After the summary is generated, the server notifies the parent's smartphone. A dedicated app is installed on the parent's smartphone, and the parent can check the summary content through the smartphone. If the parent wants to know more details about the summary, they can input a question through the app. For example, "What did you do in art class today?"

[0251] When a question is entered, the server accepts it and reanalyzes the related text data. The emotion engine also participates in this process, generating detailed information based on the parent's emotional state. The reanalyzed information is then sent back to the user's (parent's) smartphone as detailed information. This allows the parent to understand the specific situation and content.

[0252] Examples:

[0253] For example, suppose a device generates text from an utterance such as "Today's art class was fun" and sends it to a server. The server analyzes this text data and summarizes it as "I enjoyed art class." The emotion engine analyzes the parent's emotional state and adjusts the format and tone of the summary. This summary is notified to the parent's smartphone, and the parent types a question such as "What did you do in art class?" The server and emotion engine analyze the text data again, generating more detailed information such as "I drew pictures during art class," which is then provided to the parent's smartphone.

[0254] As described above, this system goes through each important step to efficiently convey the child's comments and conversations to parents while protecting their privacy. This system allows parents to communicate more smoothly on a daily basis and quickly obtain the information they need. In addition, the emotion engine enables the provision of information based on the parent's emotional state, enabling more personalized responses.

[0255] The processing flow will be explained below.

[0256] Step 1:

[0257] The device activates the microphone and captures the child's voice. The device is set to collect voice data at regular intervals.

[0258] Step 2:

[0259] The device converts the voice data it receives into text data in real time, using voice recognition software to convert the voice signal into text information.

[0260] Step 3:

[0261] The device organizes the converted text data to make it syntactically meaningful and formats it into a specific format.

[0262] Step 4:

[0263] The device sends the organized text data to the server at regular intervals (e.g., every 30 minutes) using a secure communication protocol.

[0264] Step 5:

[0265] The server receives the text data sent from the device and stores it in a database. All received text is saved and can be used for later analysis.

[0266] Step 6:

[0267] The server applies natural language processing (NLP) algorithms to analyze the stored text data, extracting important statements based on specific keywords and context.

[0268] Step 7:

[0269] The server generates a summary based on the extracted important utterances. The summary is written in a concise and meaningful way for parents.

[0270] Step 8:

[0271] The server activates the emotion engine and recognizes the user's (parent's) emotional state when generating a summary. The emotion engine refers to the emotional data obtained from the parent's smartphone app and adjusts the tone and content of the summary.

[0272] Step 9:

[0273] The server sends the generated summary to the parent's smartphone via a dedicated app.

[0274] Step 10:

[0275] The user (parent) receives a notification on their smartphone, launches the app, and checks the summary, which is displayed in a format that is easy for the parent to understand.

[0276] Step 11:

[0277] If the user (parent) has any questions about the summary, they can enter them in the question form within the app. For example, "What did you do in art class?"

[0278] Step 12:

[0279] The server receives the parent's question, re-analyzes the relevant text data, and applies additional natural language processing algorithms depending on the question to extract more detailed information.

[0280] Step 13:

[0281] The server then reactivates the emotion engine to generate detailed information based on the parent's emotional state, and the emotion engine adjusts the tone and content of the detailed information to match the parent's emotions.

[0282] Step 14:

[0283] The server generates detailed information obtained through reanalysis and sends it to the user's (parent's) smartphone. The detailed information is also provided through a dedicated app.

[0284] Step 15:

[0285] The user (parent) receives the detailed information and checks it in the app, allowing the parent to understand the specific details of any concerns.

[0286] Example 2

[0287] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0288] In the past, in order for parents to efficiently understand their children's everyday utterances and conversations, they had to directly observe and listen, which was a heavy burden. Furthermore, there was a lack of a system that could reliably obtain the necessary information while protecting privacy. Therefore, there is a need for a system that can efficiently understand the contents of children's everyday utterances and conversations while avoiding excessive parental interference.

[0289] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0290] In this invention, the server includes: [means for converting voice data into text data in real time;] [means for periodically transmitting the text data to a central processing unit;] [means for analyzing the text data in the central processing unit and extracting important comments;] [means for generating a summary based on the extracted comments;] [means for analyzing the parent's emotions using an emotion recognition module and adjusting the tone of the summary; and [means for notifying the user of the summary.] This enables parents to efficiently understand their child's daily comments and conversations without excessive interference, while still protecting their privacy.

[0291] "Means for converting voice data into text data in real time" refers to devices or software that instantly convert voice data acquired using voice recognition technology into text format.

[0292] The "means for periodically transmitting text data to a central processing unit" refers to a communication device or software for transmitting the converted text data to a remote server or data processing unit at regular time intervals.

[0293] "Means for analyzing text data in a central processing unit and extracting important statements" refers to devices or software that have the function of analyzing text data using natural language processing technology and extracting important keywords and phrases from it.

[0294] The "means for generating a summary based on extracted statements" refers to a device or software that aggregates the extracted important text information and compiles it into a short sentence format.

[0295] "Means for analyzing the parent's emotions using an emotion recognition module and adjusting the tone of the summary" refers to a device or software that recognizes and analyzes the parent's emotional state and adjusts the wording and expression of the summary based on the emotional information.

[0296] The "means for notifying the user of the summary" refers to a device or software that notifies the user of the generated summary to the user's terminal or application.

[0297] The "means for accepting questions from users" refers to devices or software that have the function of inputting and receiving questions from users requesting detailed information about the summary text.

[0298] "Means for reanalyzing text data based on a question and generating detailed information" refers to devices or software that reanalyze existing text data to extract and generate more specific information in response to a user's question.

[0299] MODE FOR CARRYING OUT THE INVENTION

[0300] This invention combines a system that allows parents to efficiently understand their children's daily utterances and conversations with an emotion engine that recognizes the user's emotions. Specific hardware and software configurations and data processing and calculation methods for implementing this invention will be described below.

[0301] 1. Acquiring children's speech and converting it into text

[0302] The device is equipped with a built-in microphone and voice recognition software. For example, a typical condenser microphone is used as the microphone, and Google Speech-to-Text or IBM Watson (registered trademark) is used as the voice recognition software. The device picks up the child's everyday speech using the microphone and collects it as voice data. The collected voice data is converted into text data in real time using the voice recognition software inside the device. For example, if a child says, "I played with my friends today," the speech is converted into text data.

[0303] 2. Sending text data

[0304] The converted text data is sent to the server at regular intervals (for example, every 5 minutes). Communication between the device and the server is via an internet connection, and Wi-Fi or mobile data communication can be used. When the device has accumulated text data to send, it automatically sends the data to the server.

[0305] 3. Saving text data

[0306] The server receives the text data sent from the device and stores it in a database. MySQL or MongoDB are suitable databases for this purpose. The server stores the received text data along with a timestamp. For example, data received on "October 1, 2023, 14:00:00" is stored along with the date and time information.

[0307] 4. Text Data Analysis and Summary Generation

[0308] The server analyzes the stored text data and extracts important statements. Natural language processing (NLP) algorithms are used for this analysis. Typical software includes SpaCy and Google's NL API. Using these, the server extracts information from the text data based on specific keywords and context, and generates a summary. For example, from the text data "I hung out with friends today," it generates the summary "I hung out with friends."

[0309] 5. Emotion engine recognizes parents' emotions

[0310] The server is equipped with an emotion engine that recognizes the user's (parent's) emotions. This emotion engine operates based on emotion data acquired when the parent accesses the server via their smartphone. For example, Azure® Emotion API recognizes and analyzes the parent's emotional state (satisfaction, anxiety, etc.). When generating a summary, the emotion engine refers to the parent's emotion data and adjusts the tone and format of the summary.

[0311] 6. Notification of Abstract

[0312] The server sends the generated summary to the user's smartphone using FireBase Cloud Messaging (FCM) or a similar push notification technology. The user receives these notifications through a dedicated app and can check the summary. For example, a summary such as "My child played with friends" may be sent to a parent's smartphone.

[0313] 7. User inquiries for more information

[0314] Users can use a dedicated app to enter questions about the summary text. For example, they can enter a question like, "What did you do with your friends?"

[0315] 8. Reanalysis and provision of detailed information

[0316] The server receives the user's question, analyzes the text data again, and generates detailed information. The natural language processing algorithm is applied again to the re-analysis, and the emotion engine is also activated to generate detailed information based on the parent's emotional state. For example, the generated detailed information might be "I played soccer with my friends." This detailed information is then sent to the user's smartphone again, allowing the parent to understand the specific details.

[0317] Examples of concrete examples and prompts

[0318] For example, if a child says, "I had fun in art class today," the device converts the speech into text and sends it to the server. The server analyzes the text data and generates a summary saying, "I had fun in art class." The emotion engine analyzes the parent's emotions, and a summary with adjusted tone is sent to the parent's smartphone. When the parent types a question into a dedicated app, such as, "What did you do in art class?", the server reanalyzes the data and generates and notifies the parent with more detailed information, such as, "I drew pictures in art class."

[0319] Example prompt sentence:

[0320] "Please summarize what your child said today."

[0321] "Tell me more about what you did in art class today."

[0322] This system allows parents to efficiently understand their children's daily comments and conversations and quickly obtain the information they need without excessive interference.In addition, the emotion engine enables personalized information provision according to the parent's emotional state.

[0323] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0324] Step 1:

[0325] Acquiring children's statements

[0326] The device's built-in microphone is used to capture the child's everyday speech in real time. For example, if a child says, "I played with my friends today," the voice is collected through the microphone.

[0327] Input: Child's speech (audio data)

[0328] Output: Collected audio data

[0329] Step 2:

[0330] Converting audio data to text

[0331] The device converts the collected voice data into text data in real time using voice recognition software (e.g., Google Speech-to-Text). For example, the text data generated is "I played with my friends today."

[0332] Input: Collected audio data

[0333] Output: Converted text data

[0334] Step 3:

[0335] Sending text data

[0336] The device sends the converted text data to the server at regular intervals (for example, every 5 minutes). This communication uses an internet connection (Wi-Fi or mobile data). For example, the text data sent might be "I played with friends today."

[0337] Input: Converted text data

[0338] Output: Text data sent to the server

[0339] Step 4:

[0340] Saving text data

[0341] The server receives the text data sent from the device and stores it in a database such as MySQL or MongoDB. For example, it is stored as data received on "October 1, 2023, 14:00:00."

[0342] Input: Received text data

[0343] Output: Text data stored in a database

[0344] Step 5:

[0345] Text data analysis and summary generation

[0346] The server analyzes the stored text data using a natural language processing (NLP) algorithm (e.g., SpaCy) to extract important statements. It then generates a summary based on the extracted information. For example, it generates a summary from "I played with my friends today" to "I played with my friends."

[0347] Input: Saved text data

[0348] Output: Summary

[0349] Step 6:

[0350] Parental emotion recognition using emotion engine

[0351] The server's emotion engine (e.g., Azure Emotion API) recognizes and analyzes the user's (parent's) emotions. For example, based on the emotional data collected when the parent uses a dedicated app, it analyzes whether the parent is "satisfied" or "anxious." Based on this analysis, the tone and format of the summary text are adjusted.

[0352] Input: Parent emotion data, summary sentence

[0353] Output: A summary with adjusted tone and format

[0354] Step 7:

[0355] Summary notification

[0356] The server sends the generated summary to the user's (parent's) smartphone via a dedicated app. For example, using Firebase Cloud Messaging (FCM), a summary such as "My child played with friends" is sent to the parent's smartphone.

[0357] Input: A summary with adjusted tone and format

[0358] Output: Summary sent to parent's smartphone

[0359] Step 8:

[0360] User inquiries for more information

[0361] The user (parent) can use a dedicated app to enter questions about the summary text, such as "What did you play with your friends?"

[0362] Input: Parent Question

[0363] Output: Questions accepted by the app

[0364] Step 9:

[0365] Reanalysis of detailed information

[0366] The server accepts questions from users and re-analyzes the related text data. A natural language processing algorithm is used for the re-analysis, and an emotion engine is also activated to generate detailed information based on the parent's emotional state. For example, the server might generate detailed information such as "I played soccer with my friends."

[0367] Input: Parent questions, initial text data

[0368] Output: Detailed information

[0369] Step 10:

[0370] Notification of detailed information

[0371] The server then sends the generated details to the parent's smartphone via a dedicated app. For example, the parent's smartphone may receive detailed information such as "My child played soccer with a friend."

[0372] Input: More information

[0373] Output: Detailed information sent to parent's smartphone

[0374] (Application example 2)

[0375] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0376] In virtual stores, there is a need for a system that can efficiently communicate between customers and sales staff (clerks) and respond quickly and appropriately to customer requests and questions. However, with current systems, it is difficult for sales staff to understand all customer conversations in real time and provide appropriate information each time. Furthermore, further technological improvements are needed to respond promptly to customer emotions and requests.

[0377] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data into text data in real time, means for periodically transmitting the text data to the server, means for analyzing the text data in the server and extracting important utterances, means for generating a summary based on the extracted utterances, means for notifying the user of the summary, means for recognizing the user's emotions and reflecting them in the summary, and means for use as a virtual customer service assistant that provides related information based on the user's questions. This enables customer service staff to efficiently respond to customers' questions and requests, thereby improving customer satisfaction.

[0378] "Means for converting voice data into text data in real time" refers to devices or technologies that instantly analyze voice and convert its content into text information.

[0379] "Means for periodically sending text data to a server" refers to a method or technique for transferring text data to a remote server at specific time intervals.

[0380] "Means for analyzing text data on a server and extracting important statements" refers to technology for processing text data on a server and identifying and extracting content that is considered particularly important.

[0381] "Means for generating a summary based on extracted statements" refers to a technology for creating a document that summarizes important statements.

[0382] "Means for notifying the user of the summary" refers to technology for notifying the user of the generated summary via a terminal or notification system.

[0383] "Means for recognizing the user's emotions and reflecting them in the summary" refers to technology for analyzing the user's emotional state and reflecting the results in the content and expression of the summary.

[0384] "Means for applying as a virtual customer service assistant that provides relevant information based on a user's question" refers to a virtual support system for processing questions from a user and providing appropriate information.

[0385] The present invention relates to a system for improving the efficiency of customer service in a virtual store. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS ... will be described in detail below with reference to the accompanying drawings.

[0386] First, a terminal equipped with a microphone is required to capture the conversation between the customer and the store clerk in order to convert the voice data into text data in real time. This terminal then uses voice recognition software to convert the voice data into text data, which is then sent to a server at specific time intervals.

[0387] The server receives the text data and analyzes it. This analysis is performed using a natural language processing (NLP) algorithm. The server extracts important statements from the analyzed data and generates a summary based on those statements. The generated summary is then notified to the user, but before that, the server uses an emotion recognition engine to analyze the user's emotions and reflects that information in the summary.

[0388] When a customer asks a store clerk a question, the system accepts the question, analyzes the text data again, and provides relevant information. This information is displayed on smart glasses or a head-mounted display, allowing the store clerk to respond immediately to the customer's question and provide appropriate information.

[0389] The hardware used includes smart glasses, a head-mounted display, and a microphone. The software uses speech recognition libraries (e.g., speech_recognition), natural language processing models (e.g., transformers), and data transmission libraries (e.g., requests). It also uses generative AI models for sentiment analysis.

[0390] Examples:

[0391] The device detects the customer's question, "Which product do you recommend most?" and converts the conversation into text data. The server analyzes the text data, extracts important information, generates a summary, and then displays an appropriate answer on the smart glasses, taking into account the emotional state of the salesperson. The following prompt sentence is used to generate this summary:

[0392] Example prompt sentence:

[0393] "Use sentiment analysis to generate the most appropriate answer for the customer's question, 'Which product do you recommend most?' If the customer's emotional state is positive, provide more information about the product; if it's negative, suggest an alternative."

[0394] As a result, the present invention can significantly improve the efficiency of customer service in a virtual store, thereby achieving increased customer satisfaction.

[0395] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0396] Step 1:

[0397] The terminal uses a microphone to capture the conversation between the customer and the store clerk, and converts the voice data into text data in real time using voice recognition software (e.g., the speech_recognition library). The input is voice data, and the output is text data.

[0398] Step 2:

[0399] The terminal sends the converted text data to the server at regular intervals. The input is text data, and the output is data transmission to the server. This transmission uses a data transmission library (e.g., requests).

[0400] Step 3:

[0401] The server stores the received text data and uses natural language processing (NLP) algorithms to analyze the data and extract key statements. The input is the text data, and the output is the analysis results containing the key statements. Specifically, the server processes the text data using NLP models (e.g., transformers).

[0402] Step 4:

[0403] The server generates a summary based on the extracted important statements. The input is the important statements, and the output is the summary. A text summarization algorithm is used to generate the summary.

[0404] Step 5:

[0405] The server uses an emotion recognition engine to analyze the user's emotions and reflects that information in the summary. The input is the summary and the user's emotion data, and the output is a summary based on the emotion. A generative AI model is used for emotion analysis.

[0406] Step 6:

[0407] The server sends the generated summary to the user's device. The input is a summary corresponding to the emotion, and the output is a notification to the user's device. This notification is done using a dedicated application.

[0408] Step 7:

[0409] The user (store clerk) checks the summary displayed on the smart glasses or head-mounted display and responds to the customer's questions and requests based on it. The input is the notified summary, and the output is a response to the customer.

[0410] Step 8:

[0411] When a user enters a question, the server re-analyzes the question and generates detailed information based on the related text data. The input is the question, and the output is detailed information. This re-analysis is performed using natural language processing algorithms.

[0412] Step 9:

[0413] The generated detailed information is again notified to the user, and the user (store clerk) provides a specific answer to the customer. The input is detailed information, and the output is a detailed answer to the customer.

[0414] Example prompt sentence:

[0415] "Use sentiment analysis to generate the most appropriate answer for the customer's question, 'Which product do you recommend most?' If the customer's emotional state is positive, provide more information about the product; if it's negative, suggest an alternative."

[0416] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0417] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0418] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0419] [Second embodiment]

[0420] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0421] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0422] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0423] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0424] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0425] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0426] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0427] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0428] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0429] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0430] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0431] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0432] The present invention relates to a system that enables parents to efficiently understand the content of their children's daily utterances and conversations. This system is designed to ensure that parents can obtain necessary information without interfering excessively. The following describes in detail an embodiment of the present invention.

[0433] First, the device worn by the child has the ability to convert voice data into text data in real time. The device uses a built-in microphone to capture the child's everyday speech and converts it into text using voice recognition software. This text data is then sent from the device to a server at regular intervals.

[0434] The server receives the text data sent from the device and stores it in a database. The server then analyzes the stored text data to extract important statements. This analysis uses natural language processing (NLP) algorithms to find important information based on specific keywords and context. The extracted information is then generated as a summary.

[0435] After the summary is generated, the server notifies the parent's smartphone. A dedicated app is installed on the parent's smartphone, and the parent can check the summary content through the smartphone. If the parent wants to know more details about the summary, they can input a question through the app. For example, "What did you do in art class today?"

[0436] When a question is entered, the server accepts the question and reanalyzes the related text data. The reanalyzed information is then sent back to the user's (parent's) smartphone as detailed information, allowing the parent to understand the specific situation and content.

[0437] Examples:

[0438] For example, suppose a device generates text from an utterance such as "Today's art class was fun" and sends it to a server. The server analyzes this text data and summarizes it as "I enjoyed art class." This summary is then sent to the parent's smartphone, who then types in a question such as "What did you do in art class?" The server then analyzes the text data again and generates more detailed information such as "I drew pictures during art class," which is then provided to the parent's smartphone.

[0439] As described above, this system goes through each important step to efficiently convey the contents of a child's statements and conversations to parents while protecting their privacy, allowing parents to communicate more smoothly and quickly obtain the information they need.

[0440] The processing flow will be explained below.

[0441] Step 1:

[0442] The device activates the microphone and captures the child's voice. The device is set to collect voice data at regular intervals.

[0443] Step 2:

[0444] The device converts the voice data it receives into text data in real time, using voice recognition software to convert the voice signal into text information.

[0445] Step 3:

[0446] The device then cleans up the converted text data and corrects it to make it grammatically meaningful, which makes subsequent parsing easier.

[0447] Step 4:

[0448] The device sends the organized text data to the server at regular intervals (e.g., every 30 minutes) using a secure communication protocol.

[0449] Step 5:

[0450] The server receives the text data sent from the device and stores it in a database. All received text is saved and can be used for later analysis.

[0451] Step 6:

[0452] The server applies natural language processing (NLP) algorithms to analyze the stored text data, extracting important statements based on specific keywords and context.

[0453] Step 7:

[0454] The server generates a summary based on the extracted important utterances. The summary is written in a concise and meaningful way for parents.

[0455] Step 8:

[0456] The server then sends the generated summary to the parent's smartphone via a dedicated app.

[0457] Step 9:

[0458] The user (parent) receives a notification on their smartphone, launches the app, and checks the summary, which is displayed in a format that is easy for the parent to understand.

[0459] Step 10:

[0460] If the user (parent) has any questions about the summary, they can enter them in the question form within the app. For example, "What did you do in art class?"

[0461] Step 11:

[0462] The server receives the parent's question, re-analyzes the relevant text data, and applies additional natural language processing algorithms depending on the question to extract more detailed information.

[0463] Step 12:

[0464] The server generates detailed information obtained through reanalysis and sends it to the user's (parent's) smartphone. The detailed information is also provided through a dedicated app.

[0465] Step 13:

[0466] The user (parent) receives the detailed information and checks it in the app, allowing the parent to understand the specific details of any concerns.

[0467] Example 1

[0468] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0469] In recent years, while it has become increasingly important for parents to understand their children's daily events and emotions, there is also a growing need to avoid excessive interference. However, it is difficult to efficiently obtain the necessary information using conventional methods. The purpose of this invention is to enable parents to efficiently and appropriately understand their children's daily utterances and conversations.

[0470] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0471] In this invention, the server includes: [means for converting voice data into text data in real time; [means for periodically transmitting the text data to the server; [means for analyzing the text data in the server and extracting important comments; [means for generating a summary based on the extracted comments; [means for notifying the user of the summary; [means for transmitting a question entered by the user to the server; [means for reanalyzing the text data based on the question and generating detailed information; and [means for notifying the user of the detailed information. This enables parents to efficiently keep track of their children's daily lives.

[0472] "Converting voice data into text data in real time" means instantly converting collected voice information into text information using voice recognition technology.

[0473] "Periodic transmission of text data to a server" means transferring the generated text information to a server via a network at regular time intervals.

[0474] "Analyzing text data on the server and extracting important statements" means analyzing the text information stored on the server using a natural language processing algorithm to extract meaningful information and keywords.

[0475] "Generating a summary based on extracted statements" means generating a concise sentence based on important information obtained from the analysis results.

[0476] "Notifying the user of the summary" means sending the generated summary to the user's device by means of push notification, email, or the like.

[0477] "Sending a question entered by a user to a server" means transferring a question entered by a user through a dedicated application to a server.

[0478] "Reanalyzing text data based on the question and generating detailed information" means reanalyzing the saved text information according to the content of the user's question and extracting related detailed information.

[0479] "Notifying the user of detailed information" means sending the reanalyzed results to the user's device and providing detailed information.

[0480] The present invention relates to a system that enables parents to efficiently understand their children's daily utterances and conversations. This system is designed to allow parents to reliably obtain necessary information while avoiding excessive interference. Specific embodiments for carrying out the present invention will be described below.

[0481] Hardware used

[0482] The device is a special wearable device that can be worn by children and has a built-in microphone that picks up surrounding sounds in real time.

[0483] The server used is a high-performance cloud-based database server, which can efficiently process and store large amounts of data.

[0484] The user device required is a smartphone for parents to receive information, on which a dedicated application is installed.

[0485] Software used

[0486] Speech recognition software: The device is installed with speech recognition software such as the Google Speech-to-Text API, which converts voice data captured in real time into text data.

[0487] Natural language processing libraries: The server uses natural language processing algorithms such as SpaCy and NLTK to analyze the text data and extract important statements.

[0488] Database software: Data is stored in a database such as MySQL or PostgreSQL.

[0489] Push notification service: Using services such as Firebase Cloud Messaging, the generated summary is sent to the parent's smartphone.

[0490] Specific examples of processing

[0491] A specific example of the process is as follows:

[0492] 1. Acquiring voice data: A child says, "I enjoyed art class today." The device's built-in microphone collects this voice data.

[0493] 2. Voice data to text conversion: The collected voice data is converted into text data in real time by the device's voice recognition software.

[0494] 3. Sending text data: The converted text data is sent to the server at regular intervals. Communication is carried out using the REST API.

[0495] 4. Analysis of text data and generation of summary sentence: The server analyzes the received text data and generates a summary sentence such as "I enjoyed art class" using a natural language processing library.

[0496] 5. Notification of summary: The generated summary is sent to the parent's smartphone via a push notification service.

[0497] Get more information

[0498] When a parent types a question into the app, such as "What did you do in art class?", the server receives the question, reanalyzes the text data, and finds the detailed information, such as "I drew a picture." This information is then sent back to the parent's smartphone.

[0499] Prompt Sentence Examples

[0500] An example of a prompt sentence when using a generative AI model is as follows:

[0501] A child says, "I enjoyed art class today." Generate a summary for the parent.

[0502] ---

[0503] Parents are asking, "What did you do in art class?" Please provide more information.

[0504] As described above, the present invention provides a means for parents to efficiently understand their children's daily lives by appropriately processing the information required at each step. This system facilitates smoother communication between parents and children, and enables them to quickly obtain the necessary information.

[0505] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0506] Step 1:

[0507] Acquiring audio data

[0508] Terminal

[0509] Example of operation: A child wears a dedicated device and begins everyday conversation. The device's built-in microphone collects voice data.

[0510] Input: Child's speech (audio data)

[0511] Data processing: The microphone captures the audio data.

[0512] Output: Audio data stored on the device

[0513] Step 2:

[0514] Converting audio data to text

[0515] Terminal

[0516] Example of operation: When a child says, "I enjoyed art class today," this speech is instantly converted into text data.

[0517] Input: Stored audio data

[0518] Data processing: Speech recognition software (e.g., Google Speech-to-Text API) recognizes the voice data and converts it into text.

[0519] Output: Converted text data

[0520] Step 3:

[0521] Sending text data

[0522] Terminal

[0523] Example of operation: The converted text data is sent to the server at regular intervals.

[0524] Input: Converted text data

[0525] Data transfer: Send text data to the server using the REST API.

[0526] Output: Text data sent to the server

[0527] Step 4:

[0528] Receiving and storing text data

[0529] server

[0530] Example of operation: The server saves the received text data in a database.

[0531] Input: Text data sent

[0532] Data processing: Save the text data in a database (e.g., MySQL).

[0533] Output: Text data stored in a database

[0534] Step 5:

[0535] Text data analysis

[0536] server

[0537] Example of how it works: Extracting important utterances from stored text data.

[0538] Input: Text data stored in a database

[0539] Data Computing: Use natural language processing algorithms (e.g., SpaCy, NLTK) to analyze important keywords and context.

[0540] Output: Important statements as a result of the analysis

[0541] Step 6:

[0542] Summary generation

[0543] server

[0544] Example of operation: Generate a summary sentence such as "Art class was fun" from the analysis results.

[0545] Input: Important statements as a result of the analysis

[0546] Data computation: Generative AI models and rule-based algorithms are used to create summaries based on key statements.

[0547] Output: Generated summary

[0548] Step 7:

[0549] Summary notification

[0550] server

[0551] Example of how it works: The summary text is sent to the parent's smartphone.

[0552] Input: Generated summary

[0553] Data transfer: A summary is sent to the parent's smartphone using a push notification service (e.g., Firebase Cloud Messaging).

[0554] Output: Summary displayed on parent's smartphone

[0555] Step 8:

[0556] Accepting user questions

[0557] User

[0558] Example of how it works: A parent types a question into a dedicated app, such as "What did you do in art class?", and sends it.

[0559] Input: Questions entered by the parent

[0560] Data transfer: Send your questions to the server through the app.

[0561] Output: The question sent to the server

[0562] Step 9:

[0563] Reanalysis of detailed information

[0564] server

[0565] Example of how it works: The server analyzes the text data again and finds the detailed information "I drew a picture."

[0566] Input: Submitted questions and saved text data

[0567] Data operations: Reanalyze relevant text data based on the question to extract more information.

[0568] Output: Reparsed details

[0569] Step 10:

[0570] Notification of detailed information

[0571] server

[0572] Example of how it works: The server sends detailed information to the parent's smartphone.

[0573] Input: Extracted details

[0574] Data transfer: Send detailed information to the parent's smartphone via push notification service.

[0575] Output: Detailed information displayed on the parent's smartphone

[0576] (Application example 1)

[0577] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0578] The present invention aims to provide a system that allows parents to efficiently understand their children's everyday comments and conversations and quickly and reliably obtain the necessary information. Another objective is to provide a sense of security for parents and improve their children's security by notifying them of important comments based on specific conditions while avoiding excessive interference. Another important objective is to be able to quickly respond to parents' requests for more information.

[0579] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0580] In this invention, the server includes: [means for converting voice data into text data in real time;] [means for periodically transmitting the text data to the server;] [means for analyzing the text data in the server and extracting important comments;] [means for generating a summary based on the extracted comments;] [means for notifying the user of the summary; and [means for notifying the user of the content of important comments on a display device based on specific conditions.] This allows parents to efficiently and quickly understand the content of their children's comments, giving them a sense of security and allowing them to quickly obtain the information they need.

[0581] "Audio data" means audio recorded in digital format and converted into text data in real time.

[0582] "Text data" refers to character information converted from voice data, and is sent to a server for analysis.

[0583] A "server" is a computer system connected via a network that is a central processing unit for receiving and analyzing text data.

[0584] "Analysis" is the process of understanding the text data transmitted and extracting important information based on specific criteria.

[0585] "Important comments" refer to comments that are worth notifying parents about, identified based on pre-set keywords and conditions.

[0586] A "summary" is a short document generated based on important statements extracted from text data, and is information that is notified to the user.

[0587] "Notification" is the act of transmitting information from a server to a user's display device, and is a means of providing information to a user in real time.

[0588] "Specific conditions" refer to rules such as keywords and contexts that are set as criteria for extracting important statements.

[0589] The "display device" is hardware for displaying notified information such as a summary, and is an interface through which the user can check the information.

[0590] In the present invention, the system is implemented based on the following configuration and procedures.

[0591] Hardware and Software Configuration

[0592] 1. Voice data collection terminal

[0593] The voice-collecting devices, such as smart bands and headsets worn by children, have built-in microphones and capture the child's everyday speech in real time.

[0594] 2. Speech Recognition Engine

[0595] The collected voice data is converted into text data in real time using a voice recognition engine, such as the Google Speech-to-Text API.

[0596] 3. Server

[0597] The converted text data is periodically sent to the server, which is responsible for the following:

[0598] Text Data Analysis

[0599] Use natural language processing (NLP) algorithms (e.g., the Transformers library) to analyze the incoming text data and extract key utterances.

[0600] Summary sentence generation

[0601] A summary is generated based on the extracted important statements.

[0602] notification

[0603] The summary sentence is sent to the parent's display device (e.g., smart glasses), and important utterances are instantly notified based on specific conditions.

[0604] 4. User Display Device

[0605] The parent's display device is a visual device such as smart glasses that displays the summary sent from the server and important comment notifications.

[0606] Data processing and calculation

[0607] 1. Audio data collection and conversion

[0608] Voice data is picked up in real time through a microphone and converted into text data by a voice recognition engine.

[0609] 2. Transmission and analysis of text data

[0610] The converted text data is sent to a server where it is analyzed using natural language processing algorithms, which extract important statements based on predefined keywords and context.

[0611] 3. Summary generation and notification

[0612] A summary is generated based on the extracted important comments and is displayed on the parent's display device. Furthermore, important comments based on specific conditions are immediately displayed.

[0613] Specific examples

[0614] For example, suppose a child says, "I enjoyed art class today." This voice data is converted into text data in real time and sent to the server. The server analyzes this text data, extracts the important statement, "I enjoyed art class," and summarizes it concisely. The parent's smart glasses are notified that "I enjoyed art class," and the parent can check detailed information from the smart glasses' interface. If the parent wants to know more, they can enter a question such as "What did you do in art class?" and the server will reanalyze the data and provide detailed information such as "I drew pictures during art class."

[0615] Prompt Sentence Examples

[0616] An example of a specific prompt from the user would be:

[0617] If you receive a user statement such as "I had a fight with a friend today," please determine whether parents should be notified and, if so, suggest the specific format in which they should be notified.

[0618] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0619] Step 1:

[0620] The child's device acquires voice data. The input is the child's everyday speech, which is collected using a microphone. The output is raw voice data.

[0621] Step 2:

[0622] The speech recognition engine converts raw speech data into text data. The input is captured speech data, which is converted into text information in real time using the Google Speech-to-Text API. The output is text data.

[0623] Step 3:

[0624] The terminal periodically sends the converted text data to the server. The input is the text data, which is sent to the server via the network. The output is the text data that arrives at the server.

[0625] Step 4:

[0626] The server analyzes the received text data using a natural language processing (NLP) algorithm. The input is the text data sent to the server, and an NLP model (e.g., the Transformers library) is used to extract important statements based on specific keywords or conditions. The output is a list of important statements.

[0627] Step 5:

[0628] The server generates a summary based on the extracted important comments. The input is a list of important comments, which is converted into a concise summary. The output is the summary.

[0629] Step 6:

[0630] The server notifies the user of the summary sentence on the display device. The input is the summary sentence, which is sent to the parent's smart glasses via the network. The output is the summary sentence notification displayed on the parent's smart glasses.

[0631] Step 7:

[0632] The server instantly notifies the parent of important comments based on specific conditions. The input is text data containing specific keywords, which triggers the immediate transmission of important comments to the parent's display device. The output is a notification of the important comment instantly displayed on the parent's smart glasses.

[0633] Step 8:

[0634] The user inputs a question requesting more information. The input is the question from the parent's smart glasses and is accepted by the server. The output is the question data.

[0635] Step 9:

[0636] The server re-analyzes the question and generates detailed information. The input is the user's question and related text data, and the NLP algorithm is applied again to generate detailed information. The output is detailed information.

[0637] Step 10:

[0638] The server sends the detailed information to the user's display device. The input is the detailed information, which is sent over the network to the parent's smart glasses. The output is the detailed information displayed on the parent's smart glasses.

[0639] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0640] The present invention combines a system that allows parents to efficiently understand the content of their children's daily utterances and conversations with an emotion engine that recognizes the user's emotions. This system is designed to ensure that parents can reliably obtain necessary information while avoiding excessive parental interference. The following describes in detail the embodiments of the present invention.

[0641] First, the device worn by the child has the ability to convert voice data into text data in real time. The device uses a built-in microphone to capture the child's everyday speech and converts it into text using voice recognition software. This text data is then sent from the device to a server at regular intervals.

[0642] The server receives the text data sent from the device and stores it in a database. The server then analyzes the stored text data to extract important statements. This analysis uses natural language processing (NLP) algorithms to find important information based on specific keywords and context. The extracted information is then generated as a summary.

[0643] Furthermore, the server is equipped with an emotion engine that recognizes the user's (parent's) emotions. When generating a summary, the emotion engine references the parent's emotion data and creates a summary in an appropriate format and with appropriate content. It also provides detailed information tailored to the parent's emotions as needed based on questions or requests from the parent.

[0644] After the summary is generated, the server notifies the parent's smartphone. A dedicated app is installed on the parent's smartphone, and the parent can check the summary content through the smartphone. If the parent wants to know more details about the summary, they can input a question through the app. For example, "What did you do in art class today?"

[0645] When a question is entered, the server accepts it and reanalyzes the related text data. The emotion engine also participates in this process, generating detailed information based on the parent's emotional state. The reanalyzed information is then sent back to the user's (parent's) smartphone as detailed information. This allows the parent to understand the specific situation and content.

[0646] Examples:

[0647] For example, suppose a device generates text from an utterance such as "Today's art class was fun" and sends it to a server. The server analyzes this text data and summarizes it as "I enjoyed art class." The emotion engine analyzes the parent's emotional state and adjusts the format and tone of the summary. This summary is notified to the parent's smartphone, and the parent types a question such as "What did you do in art class?" The server and emotion engine analyze the text data again, generating more detailed information such as "I drew pictures during art class," which is then provided to the parent's smartphone.

[0648] As described above, this system goes through each important step to efficiently convey the child's comments and conversations to parents while protecting their privacy. This system allows parents to communicate more smoothly on a daily basis and quickly obtain the information they need. In addition, the emotion engine enables the provision of information based on the parent's emotional state, enabling more personalized responses.

[0649] The processing flow will be explained below.

[0650] Step 1:

[0651] The device activates the microphone and captures the child's voice. The device is set to collect voice data at regular intervals.

[0652] Step 2:

[0653] The device converts the voice data it receives into text data in real time, using voice recognition software to convert the voice signal into text information.

[0654] Step 3:

[0655] The device organizes the converted text data to make it syntactically meaningful and formats it into a specific format.

[0656] Step 4:

[0657] The device sends the organized text data to the server at regular intervals (e.g., every 30 minutes) using a secure communication protocol.

[0658] Step 5:

[0659] The server receives the text data sent from the device and stores it in a database. All received text is saved and can be used for later analysis.

[0660] Step 6:

[0661] The server applies natural language processing (NLP) algorithms to analyze the stored text data, extracting important statements based on specific keywords and context.

[0662] Step 7:

[0663] The server generates a summary based on the extracted important utterances. The summary is written in a concise and meaningful way for parents.

[0664] Step 8:

[0665] The server activates the emotion engine and recognizes the user's (parent's) emotional state when generating a summary. The emotion engine refers to the emotional data obtained from the parent's smartphone app and adjusts the tone and content of the summary.

[0666] Step 9:

[0667] The server sends the generated summary to the parent's smartphone via a dedicated app.

[0668] Step 10:

[0669] The user (parent) receives a notification on their smartphone, launches the app, and checks the summary, which is displayed in a format that is easy for the parent to understand.

[0670] Step 11:

[0671] If the user (parent) has any questions about the summary, they can enter them in the question form within the app. For example, "What did you do in art class?"

[0672] Step 12:

[0673] The server receives the parent's question, re-analyzes the relevant text data, and applies additional natural language processing algorithms depending on the question to extract more detailed information.

[0674] Step 13:

[0675] The server then reactivates the emotion engine to generate detailed information based on the parent's emotional state, and the emotion engine adjusts the tone and content of the detailed information to match the parent's emotions.

[0676] Step 14:

[0677] The server generates detailed information obtained through reanalysis and sends it to the user's (parent's) smartphone. The detailed information is also provided through a dedicated app.

[0678] Step 15:

[0679] The user (parent) receives the detailed information and checks it in the app, allowing the parent to understand the specific details of any concerns.

[0680] Example 2

[0681] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0682] In the past, in order for parents to efficiently understand their children's everyday utterances and conversations, they had to directly observe and listen, which was a heavy burden. Furthermore, there was a lack of a system that could reliably obtain the necessary information while protecting privacy. Therefore, there is a need for a system that can efficiently understand the contents of children's everyday utterances and conversations while avoiding excessive parental interference.

[0683] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0684] In this invention, the server includes: [means for converting voice data into text data in real time;] [means for periodically transmitting the text data to a central processing unit;] [means for analyzing the text data in the central processing unit and extracting important comments;] [means for generating a summary based on the extracted comments;] [means for analyzing the parent's emotions using an emotion recognition module and adjusting the tone of the summary; and [means for notifying the user of the summary.] This enables parents to efficiently understand their child's daily comments and conversations without excessive interference, while still protecting their privacy.

[0685] "Means for converting voice data into text data in real time" refers to devices or software that instantly convert voice data acquired using voice recognition technology into text format.

[0686] The "means for periodically transmitting text data to a central processing unit" refers to a communication device or software for transmitting the converted text data to a remote server or data processing unit at regular time intervals.

[0687] "Means for analyzing text data in a central processing unit and extracting important statements" refers to devices or software that have the function of analyzing text data using natural language processing technology and extracting important keywords and phrases from it.

[0688] The "means for generating a summary based on extracted statements" refers to a device or software that aggregates the extracted important text information and compiles it into a short sentence format.

[0689] "Means for analyzing the parent's emotions using an emotion recognition module and adjusting the tone of the summary" refers to a device or software that recognizes and analyzes the parent's emotional state and adjusts the wording and expression of the summary based on the emotional information.

[0690] The "means for notifying the user of the summary" refers to a device or software that notifies the user of the generated summary to the user's terminal or application.

[0691] The "means for accepting questions from users" refers to devices or software that have the function of inputting and receiving questions from users requesting detailed information about the summary text.

[0692] "Means for reanalyzing text data based on a question and generating detailed information" refers to devices or software that reanalyze existing text data to extract and generate more specific information in response to a user's question.

[0693] MODE FOR CARRYING OUT THE INVENTION

[0694] This invention combines a system that allows parents to efficiently understand their children's daily utterances and conversations with an emotion engine that recognizes the user's emotions. Specific hardware and software configurations and data processing and calculation methods for implementing this invention will be described below.

[0695] 1. Acquiring children's speech and converting it into text

[0696] The device is equipped with a built-in microphone and voice recognition software. For example, a typical condenser microphone is used as the microphone, and Google Speech-to-Text or IBM Watson is used as the voice recognition software. The device picks up the child's everyday speech using the microphone and collects it as voice data. The collected voice data is converted into text data in real time using the voice recognition software inside the device. For example, if a child says, "I played with my friends today," the speech is converted into text data.

[0697] 2. Sending text data

[0698] The converted text data is sent to the server at regular intervals (for example, every 5 minutes). Communication between the device and the server is via an internet connection, and Wi-Fi or mobile data communication can be used. When the device has accumulated text data to send, it automatically sends the data to the server.

[0699] 3. Saving text data

[0700] The server receives the text data sent from the device and stores it in a database. MySQL or MongoDB are suitable databases for this purpose. The server stores the received text data along with a timestamp. For example, data received on "October 1, 2023, 14:00:00" is stored along with the date and time information.

[0701] 4. Text Data Analysis and Summary Generation

[0702] The server analyzes the stored text data and extracts important statements. Natural language processing (NLP) algorithms are used for this analysis. Typical software includes SpaCy and Google's NL API. Using these, the server extracts information from the text data based on specific keywords and context, and generates a summary. For example, from the text data "I hung out with friends today," it generates the summary "I hung out with friends."

[0703] 5. Emotion engine recognizes parents' emotions

[0704] The server is equipped with an emotion engine that recognizes the user's (parent's) emotions. This emotion engine operates based on emotion data acquired when the parent accesses the system via their smartphone. For example, Azure Emotion API recognizes and analyzes the parent's emotional state (satisfaction, anxiety, etc.). When generating a summary, the emotion engine refers to the parent's emotion data and adjusts the tone and format of the summary.

[0705] 6. Notification of Abstract

[0706] The server sends the generated summary to the user's smartphone using FireBase Cloud Messaging (FCM) or a similar push notification technology. The user receives these notifications through a dedicated app and can check the summary. For example, a summary such as "My child played with friends" may be sent to a parent's smartphone.

[0707] 7. User inquiries for more information

[0708] Users can use a dedicated app to enter questions about the summary text. For example, they can enter a question like, "What did you do with your friends?"

[0709] 8. Reanalysis and provision of detailed information

[0710] The server receives the user's question, analyzes the text data again, and generates detailed information. The natural language processing algorithm is applied again to the re-analysis, and the emotion engine is also activated to generate detailed information based on the parent's emotional state. For example, the generated detailed information might be "I played soccer with my friends." This detailed information is then sent to the user's smartphone again, allowing the parent to understand the specific details.

[0711] Examples of concrete examples and prompts

[0712] For example, if a child says, "I had fun in art class today," the device converts the speech into text and sends it to the server. The server analyzes the text data and generates a summary saying, "I had fun in art class." The emotion engine analyzes the parent's emotions, and a summary with adjusted tone is sent to the parent's smartphone. When the parent types a question into a dedicated app, such as, "What did you do in art class?", the server reanalyzes the data and generates and notifies the parent with more detailed information, such as, "I drew pictures in art class."

[0713] Example prompt sentence:

[0714] "Please summarize what your child said today."

[0715] "Tell me more about what you did in art class today."

[0716] This system allows parents to efficiently understand their children's daily comments and conversations and quickly obtain the information they need without excessive interference.In addition, the emotion engine enables personalized information provision according to the parent's emotional state.

[0717] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0718] Step 1:

[0719] Acquiring children's statements

[0720] The device's built-in microphone is used to capture the child's everyday speech in real time. For example, if a child says, "I played with my friends today," the voice is collected through the microphone.

[0721] Input: Child's speech (audio data)

[0722] Output: Collected audio data

[0723] Step 2:

[0724] Converting audio data to text

[0725] The device converts the collected voice data into text data in real time using voice recognition software (e.g., Google Speech-to-Text). For example, the text data generated is "I played with my friends today."

[0726] Input: Collected audio data

[0727] Output: Converted text data

[0728] Step 3:

[0729] Sending text data

[0730] The device sends the converted text data to the server at regular intervals (for example, every 5 minutes). This communication uses an internet connection (Wi-Fi or mobile data). For example, the text data sent might be "I played with friends today."

[0731] Input: Converted text data

[0732] Output: Text data sent to the server

[0733] Step 4:

[0734] Saving text data

[0735] The server receives the text data sent from the device and stores it in a database such as MySQL or MongoDB. For example, it is stored as data received on "October 1, 2023, 14:00:00."

[0736] Input: Received text data

[0737] Output: Text data stored in a database

[0738] Step 5:

[0739] Text data analysis and summary generation

[0740] The server analyzes the stored text data using a natural language processing (NLP) algorithm (e.g., SpaCy) to extract important statements. It then generates a summary based on the extracted information. For example, it generates a summary from "I played with my friends today" to "I played with my friends."

[0741] Input: Saved text data

[0742] Output: Summary

[0743] Step 6:

[0744] Parental emotion recognition using emotion engine

[0745] The server's emotion engine (e.g., Azure Emotion API) recognizes and analyzes the user's (parent's) emotions. For example, based on the emotional data collected when the parent uses a dedicated app, it analyzes whether the parent is "satisfied" or "anxious." Based on this analysis, the tone and format of the summary text are adjusted.

[0746] Input: Parent emotion data, summary sentence

[0747] Output: A summary with adjusted tone and format

[0748] Step 7:

[0749] Summary notification

[0750] The server sends the generated summary to the user's (parent's) smartphone via a dedicated app. For example, using Firebase Cloud Messaging (FCM), a summary such as "My child played with friends" is sent to the parent's smartphone.

[0751] Input: A summary with adjusted tone and format

[0752] Output: Summary sent to parent's smartphone

[0753] Step 8:

[0754] User inquiries for more information

[0755] The user (parent) can use a dedicated app to enter questions about the summary text, such as "What did you play with your friends?"

[0756] Input: Parent Question

[0757] Output: Questions accepted by the app

[0758] Step 9:

[0759] Reanalysis of detailed information

[0760] The server accepts questions from users and re-analyzes the related text data. A natural language processing algorithm is used for the re-analysis, and an emotion engine is also activated to generate detailed information based on the parent's emotional state. For example, the server might generate detailed information such as "I played soccer with my friends."

[0761] Input: Parent questions, initial text data

[0762] Output: Detailed information

[0763] Step 10:

[0764] Notification of detailed information

[0765] The server then sends the generated details to the parent's smartphone via a dedicated app. For example, the parent's smartphone may receive detailed information such as "My child played soccer with a friend."

[0766] Input: More information

[0767] Output: Detailed information sent to parent's smartphone

[0768] (Application example 2)

[0769] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0770] In virtual stores, there is a need for a system that can efficiently communicate between customers and sales staff (clerks) and respond quickly and appropriately to customer requests and questions. However, with current systems, it is difficult for sales staff to understand all customer conversations in real time and provide appropriate information each time. Furthermore, further technological improvements are needed to respond promptly to customer emotions and requests.

[0771] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data into text data in real time, means for periodically transmitting the text data to the server, means for analyzing the text data in the server and extracting important utterances, means for generating a summary based on the extracted utterances, means for notifying the user of the summary, means for recognizing the user's emotions and reflecting them in the summary, and means for use as a virtual customer service assistant that provides related information based on the user's questions. This enables customer service staff to efficiently respond to customers' questions and requests, thereby improving customer satisfaction.

[0772] "Means for converting voice data into text data in real time" refers to devices or technologies that instantly analyze voice and convert its content into text information.

[0773] "Means for periodically sending text data to a server" refers to a method or technique for transferring text data to a remote server at specific time intervals.

[0774] "Means for analyzing text data on a server and extracting important statements" refers to technology for processing text data on a server and identifying and extracting content that is considered particularly important.

[0775] "Means for generating a summary based on extracted statements" refers to a technology for creating a document that summarizes important statements.

[0776] "Means for notifying the user of the summary" refers to technology for notifying the user of the generated summary via a terminal or notification system.

[0777] "Means for recognizing the user's emotions and reflecting them in the summary" refers to technology for analyzing the user's emotional state and reflecting the results in the content and expression of the summary.

[0778] "Means for applying as a virtual customer service assistant that provides relevant information based on a user's question" refers to a virtual support system for processing questions from a user and providing appropriate information.

[0779] The present invention relates to a system for improving the efficiency of customer service in a virtual store. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS ... will be described in detail below with reference to the accompanying drawings.

[0780] First, a terminal equipped with a microphone is required to capture the conversation between the customer and the store clerk in order to convert the voice data into text data in real time. This terminal then uses voice recognition software to convert the voice data into text data, which is then sent to a server at specific time intervals.

[0781] The server receives the text data and analyzes it. This analysis is performed using a natural language processing (NLP) algorithm. The server extracts important statements from the analyzed data and generates a summary based on those statements. The generated summary is then notified to the user, but before that, the server uses an emotion recognition engine to analyze the user's emotions and reflects that information in the summary.

[0782] When a customer asks a store clerk a question, the system accepts the question, analyzes the text data again, and provides relevant information. This information is displayed on smart glasses or a head-mounted display, allowing the store clerk to respond immediately to the customer's question and provide appropriate information.

[0783] The hardware used includes smart glasses, a head-mounted display, and a microphone. The software uses speech recognition libraries (e.g., speech_recognition), natural language processing models (e.g., transformers), and data transmission libraries (e.g., requests). It also uses generative AI models for sentiment analysis.

[0784] Examples:

[0785] The device detects the customer's question, "Which product do you recommend most?" and converts the conversation into text data. The server analyzes the text data, extracts important information, generates a summary, and then displays an appropriate answer on the smart glasses, taking into account the emotional state of the salesperson. The following prompt sentence is used to generate this summary:

[0786] Example prompt sentence:

[0787] "Use sentiment analysis to generate the most appropriate answer for the customer's question, 'Which product do you recommend most?' If the customer's emotional state is positive, provide more information about the product; if it's negative, suggest an alternative."

[0788] As a result, the present invention can significantly improve the efficiency of customer service in a virtual store, thereby achieving increased customer satisfaction.

[0789] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0790] Step 1:

[0791] The terminal uses a microphone to capture the conversation between the customer and the store clerk, and converts the voice data into text data in real time using voice recognition software (e.g., the speech_recognition library). The input is voice data, and the output is text data.

[0792] Step 2:

[0793] The terminal sends the converted text data to the server at regular intervals. The input is text data, and the output is data transmission to the server. This transmission uses a data transmission library (e.g., requests).

[0794] Step 3:

[0795] The server stores the received text data and uses natural language processing (NLP) algorithms to analyze the data and extract key statements. The input is the text data, and the output is the analysis results containing the key statements. Specifically, the server processes the text data using NLP models (e.g., transformers).

[0796] Step 4:

[0797] The server generates a summary based on the extracted important statements. The input is the important statements, and the output is the summary. A text summarization algorithm is used to generate the summary.

[0798] Step 5:

[0799] The server uses an emotion recognition engine to analyze the user's emotions and reflects that information in the summary. The input is the summary and the user's emotion data, and the output is a summary based on the emotion. A generative AI model is used for emotion analysis.

[0800] Step 6:

[0801] The server sends the generated summary to the user's device. The input is a summary corresponding to the emotion, and the output is a notification to the user's device. This notification is done using a dedicated application.

[0802] Step 7:

[0803] The user (store clerk) checks the summary displayed on the smart glasses or head-mounted display and responds to the customer's questions and requests based on it. The input is the notified summary, and the output is a response to the customer.

[0804] Step 8:

[0805] When a user enters a question, the server re-analyzes the question and generates detailed information based on the related text data. The input is the question, and the output is detailed information. This re-analysis is performed using natural language processing algorithms.

[0806] Step 9:

[0807] The generated detailed information is again notified to the user, and the user (store clerk) provides a specific answer to the customer. The input is detailed information, and the output is a detailed answer to the customer.

[0808] Example prompt sentence:

[0809] "Use sentiment analysis to generate the most appropriate answer for the customer's question, 'Which product do you recommend most?' If the customer's emotional state is positive, provide more information about the product; if it's negative, suggest an alternative."

[0810] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0811] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0812] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0813] [Third embodiment]

[0814] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0815] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0816] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0817] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0818] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0819] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0820] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0821] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0822] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0823] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0824] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0825] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0826] The present invention relates to a system that enables parents to efficiently understand the content of their children's daily utterances and conversations. This system is designed to ensure that parents can obtain necessary information without interfering excessively. The following describes in detail an embodiment of the present invention.

[0827] First, the device worn by the child has the ability to convert voice data into text data in real time. The device uses a built-in microphone to capture the child's everyday speech and converts it into text using voice recognition software. This text data is then sent from the device to a server at regular intervals.

[0828] The server receives the text data sent from the device and stores it in a database. The server then analyzes the stored text data to extract important statements. This analysis uses natural language processing (NLP) algorithms to find important information based on specific keywords and context. The extracted information is then generated as a summary.

[0829] After the summary is generated, the server notifies the parent's smartphone. A dedicated app is installed on the parent's smartphone, and the parent can check the summary content through the smartphone. If the parent wants to know more details about the summary, they can input a question through the app. For example, "What did you do in art class today?"

[0830] When a question is entered, the server accepts the question and reanalyzes the related text data. The reanalyzed information is then sent back to the user's (parent's) smartphone as detailed information, allowing the parent to understand the specific situation and content.

[0831] Examples:

[0832] For example, suppose a device generates text from an utterance such as "Today's art class was fun" and sends it to a server. The server analyzes this text data and summarizes it as "I enjoyed art class." This summary is then sent to the parent's smartphone, who then types in a question such as "What did you do in art class?" The server then analyzes the text data again and generates more detailed information such as "I drew pictures during art class," which is then provided to the parent's smartphone.

[0833] As described above, this system goes through each important step to efficiently convey the contents of a child's statements and conversations to parents while protecting their privacy, allowing parents to communicate more smoothly and quickly obtain the information they need.

[0834] The processing flow will be explained below.

[0835] Step 1:

[0836] The device activates the microphone and captures the child's voice. The device is set to collect voice data at regular intervals.

[0837] Step 2:

[0838] The device converts the voice data it receives into text data in real time, using voice recognition software to convert the voice signal into text information.

[0839] Step 3:

[0840] The device then cleans up the converted text data and corrects it to make it grammatically meaningful, which makes subsequent parsing easier.

[0841] Step 4:

[0842] The device sends the organized text data to the server at regular intervals (e.g., every 30 minutes) using a secure communication protocol.

[0843] Step 5:

[0844] The server receives the text data sent from the device and stores it in a database. All received text is saved and can be used for later analysis.

[0845] Step 6:

[0846] The server applies natural language processing (NLP) algorithms to analyze the stored text data, extracting important statements based on specific keywords and context.

[0847] Step 7:

[0848] The server generates a summary based on the extracted important utterances. The summary is written in a concise and meaningful way for parents.

[0849] Step 8:

[0850] The server then sends the generated summary to the parent's smartphone via a dedicated app.

[0851] Step 9:

[0852] The user (parent) receives a notification on their smartphone, launches the app, and checks the summary, which is displayed in a format that is easy for the parent to understand.

[0853] Step 10:

[0854] If the user (parent) has any questions about the summary, they can enter them in the question form within the app. For example, "What did you do in art class?"

[0855] Step 11:

[0856] The server receives the parent's question, re-analyzes the relevant text data, and applies additional natural language processing algorithms depending on the question to extract more detailed information.

[0857] Step 12:

[0858] The server generates detailed information obtained through reanalysis and sends it to the user's (parent's) smartphone. The detailed information is also provided through a dedicated app.

[0859] Step 13:

[0860] The user (parent) receives the detailed information and checks it in the app, allowing the parent to understand the specific details of any concerns.

[0861] Example 1

[0862] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0863] In recent years, while it has become increasingly important for parents to understand their children's daily events and emotions, there is also a growing need to avoid excessive interference. However, it is difficult to efficiently obtain the necessary information using conventional methods. The purpose of this invention is to enable parents to efficiently and appropriately understand their children's daily utterances and conversations.

[0864] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0865] In this invention, the server includes: [means for converting voice data into text data in real time; [means for periodically transmitting the text data to the server; [means for analyzing the text data in the server and extracting important comments; [means for generating a summary based on the extracted comments; [means for notifying the user of the summary; [means for transmitting a question entered by the user to the server; [means for reanalyzing the text data based on the question and generating detailed information; and [means for notifying the user of the detailed information. This enables parents to efficiently keep track of their children's daily lives.

[0866] "Converting voice data into text data in real time" means instantly converting collected voice information into text information using voice recognition technology.

[0867] "Periodic transmission of text data to a server" means transferring the generated text information to a server via a network at regular time intervals.

[0868] "Analyzing text data on the server and extracting important statements" means analyzing the text information stored on the server using a natural language processing algorithm to extract meaningful information and keywords.

[0869] "Generating a summary based on extracted statements" means generating a concise sentence based on important information obtained from the analysis results.

[0870] "Notifying the user of the summary" means sending the generated summary to the user's device by means of push notification, email, or the like.

[0871] "Sending a question entered by a user to a server" means transferring a question entered by a user through a dedicated application to a server.

[0872] "Reanalyzing text data based on the question and generating detailed information" means reanalyzing the saved text information according to the content of the user's question and extracting related detailed information.

[0873] "Notifying the user of detailed information" means sending the reanalyzed results to the user's device and providing detailed information.

[0874] The present invention relates to a system that enables parents to efficiently understand their children's daily utterances and conversations. This system is designed to allow parents to reliably obtain necessary information while avoiding excessive interference. Specific embodiments for carrying out the present invention will be described below.

[0875] Hardware used

[0876] The device is a special wearable device that can be worn by children and has a built-in microphone that picks up surrounding sounds in real time.

[0877] The server used is a high-performance cloud-based database server, which can efficiently process and store large amounts of data.

[0878] The user device required is a smartphone for parents to receive information, on which a dedicated application is installed.

[0879] Software used

[0880] Speech recognition software: The device is installed with speech recognition software such as the Google Speech-to-Text API, which converts voice data captured in real time into text data.

[0881] Natural language processing libraries: The server uses natural language processing algorithms such as SpaCy and NLTK to analyze the text data and extract important statements.

[0882] Database software: Data is stored in a database such as MySQL or PostgreSQL.

[0883] Push notification service: Using services such as Firebase Cloud Messaging, the generated summary is sent to the parent's smartphone.

[0884] Specific examples of processing

[0885] A specific example of the process is as follows:

[0886] 1. Acquiring voice data: A child says, "I enjoyed art class today." The device's built-in microphone collects this voice data.

[0887] 2. Voice data to text conversion: The collected voice data is converted into text data in real time by the device's voice recognition software.

[0888] 3. Sending text data: The converted text data is sent to the server at regular intervals. Communication is carried out using the REST API.

[0889] 4. Analysis of text data and generation of summary sentence: The server analyzes the received text data and generates a summary sentence such as "I enjoyed art class" using a natural language processing library.

[0890] 5. Notification of summary: The generated summary is sent to the parent's smartphone via a push notification service.

[0891] Get more information

[0892] When a parent types a question into the app, such as "What did you do in art class?", the server receives the question, reanalyzes the text data, and finds the detailed information, such as "I drew a picture." This information is then sent back to the parent's smartphone.

[0893] Prompt Sentence Examples

[0894] An example of a prompt sentence when using a generative AI model is as follows:

[0895] A child says, "I enjoyed art class today." Generate a summary for the parent.

[0896] ---

[0897] Parents are asking, "What did you do in art class?" Please provide more information.

[0898] As described above, the present invention provides a means for parents to efficiently understand their children's daily lives by appropriately processing the information required at each step. This system facilitates smoother communication between parents and children, and enables them to quickly obtain the necessary information.

[0899] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0900] Step 1:

[0901] Acquiring audio data

[0902] Terminal

[0903] Example of operation: A child wears a dedicated device and begins everyday conversation. The device's built-in microphone collects voice data.

[0904] Input: Child's speech (audio data)

[0905] Data processing: The microphone captures the audio data.

[0906] Output: Audio data stored on the device

[0907] Step 2:

[0908] Converting audio data to text

[0909] Terminal

[0910] Example of operation: When a child says, "I enjoyed art class today," this speech is instantly converted into text data.

[0911] Input: Stored audio data

[0912] Data processing: Speech recognition software (e.g., Google Speech-to-Text API) recognizes the voice data and converts it into text.

[0913] Output: Converted text data

[0914] Step 3:

[0915] Sending text data

[0916] Terminal

[0917] Example of operation: The converted text data is sent to the server at regular intervals.

[0918] Input: Converted text data

[0919] Data transfer: Send text data to the server using the REST API.

[0920] Output: Text data sent to the server

[0921] Step 4:

[0922] Receiving and storing text data

[0923] server

[0924] Example of operation: The server saves the received text data in a database.

[0925] Input: Text data sent

[0926] Data processing: Save the text data in a database (e.g., MySQL).

[0927] Output: Text data stored in a database

[0928] Step 5:

[0929] Text data analysis

[0930] server

[0931] Example of how it works: Extracting important utterances from stored text data.

[0932] Input: Text data stored in a database

[0933] Data Computing: Use natural language processing algorithms (e.g., SpaCy, NLTK) to analyze important keywords and context.

[0934] Output: Important statements as a result of the analysis

[0935] Step 6:

[0936] Summary generation

[0937] server

[0938] Example of operation: Generate a summary sentence such as "Art class was fun" from the analysis results.

[0939] Input: Important statements as a result of the analysis

[0940] Data computation: Generative AI models and rule-based algorithms are used to create summaries based on key statements.

[0941] Output: Generated summary

[0942] Step 7:

[0943] Summary notification

[0944] server

[0945] Example of how it works: The summary text is sent to the parent's smartphone.

[0946] Input: Generated summary

[0947] Data transfer: A summary is sent to the parent's smartphone using a push notification service (e.g., Firebase Cloud Messaging).

[0948] Output: Summary displayed on parent's smartphone

[0949] Step 8:

[0950] Accepting user questions

[0951] User

[0952] Example of how it works: A parent types a question into a dedicated app, such as "What did you do in art class?", and sends it.

[0953] Input: Questions entered by the parent

[0954] Data transfer: Send your questions to the server through the app.

[0955] Output: The question sent to the server

[0956] Step 9:

[0957] Reanalysis of detailed information

[0958] server

[0959] Example of how it works: The server analyzes the text data again and finds the detailed information "I drew a picture."

[0960] Input: Submitted questions and saved text data

[0961] Data operations: Reanalyze relevant text data based on the question to extract more information.

[0962] Output: Reparsed details

[0963] Step 10:

[0964] Notification of detailed information

[0965] server

[0966] Example of how it works: The server sends detailed information to the parent's smartphone.

[0967] Input: Extracted details

[0968] Data transfer: Send detailed information to the parent's smartphone via push notification service.

[0969] Output: Detailed information displayed on the parent's smartphone

[0970] (Application example 1)

[0971] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0972] The present invention aims to provide a system that allows parents to efficiently understand their children's everyday comments and conversations and quickly and reliably obtain the necessary information. Another objective is to provide a sense of security for parents and improve their children's security by notifying them of important comments based on specific conditions while avoiding excessive interference. Another important objective is to be able to quickly respond to parents' requests for more information.

[0973] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0974] In this invention, the server includes: [means for converting voice data into text data in real time;] [means for periodically transmitting the text data to the server;] [means for analyzing the text data in the server and extracting important comments;] [means for generating a summary based on the extracted comments;] [means for notifying the user of the summary; and [means for notifying the user of the content of important comments on a display device based on specific conditions.] This allows parents to efficiently and quickly understand the content of their children's comments, giving them a sense of security and allowing them to quickly obtain the information they need.

[0975] "Audio data" means audio recorded in digital format and converted into text data in real time.

[0976] "Text data" refers to character information converted from voice data, and is sent to a server for analysis.

[0977] A "server" is a computer system connected via a network that is a central processing unit for receiving and analyzing text data.

[0978] "Analysis" is the process of understanding the text data transmitted and extracting important information based on specific criteria.

[0979] "Important comments" refer to comments that are worth notifying parents about, identified based on pre-set keywords and conditions.

[0980] A "summary" is a short document generated based on important statements extracted from text data, and is information that is notified to the user.

[0981] "Notification" is the act of transmitting information from a server to a user's display device, and is a means of providing information to a user in real time.

[0982] "Specific conditions" refer to rules such as keywords and contexts that are set as criteria for extracting important statements.

[0983] The "display device" is hardware for displaying notified information such as a summary, and is an interface through which the user can check the information.

[0984] In the present invention, the system is implemented based on the following configuration and procedures.

[0985] Hardware and Software Configuration

[0986] 1. Voice data collection terminal

[0987] The voice-collecting devices, such as smart bands and headsets worn by children, have built-in microphones and capture the child's everyday speech in real time.

[0988] 2. Speech Recognition Engine

[0989] The collected voice data is converted into text data in real time using a voice recognition engine, such as the Google Speech-to-Text API.

[0990] 3. Server

[0991] The converted text data is periodically sent to the server, which is responsible for the following:

[0992] Text Data Analysis

[0993] Use natural language processing (NLP) algorithms (e.g., the Transformers library) to analyze the incoming text data and extract key utterances.

[0994] Summary sentence generation

[0995] A summary is generated based on the extracted important statements.

[0996] notification

[0997] The summary sentence is sent to the parent's display device (e.g., smart glasses), and important utterances are instantly notified based on specific conditions.

[0998] 4. User Display Device

[0999] The parent's display device is a visual device such as smart glasses that displays the summary sent from the server and important comment notifications.

[1000] Data processing and calculation

[1001] 1. Audio data collection and conversion

[1002] Voice data is picked up in real time through a microphone and converted into text data by a voice recognition engine.

[1003] 2. Transmission and analysis of text data

[1004] The converted text data is sent to a server where it is analyzed using natural language processing algorithms, which extract important statements based on predefined keywords and context.

[1005] 3. Summary generation and notification

[1006] A summary is generated based on the extracted important comments and is displayed on the parent's display device. Furthermore, important comments based on specific conditions are immediately displayed.

[1007] Specific examples

[1008] For example, suppose a child says, "I enjoyed art class today." This voice data is converted into text data in real time and sent to the server. The server analyzes this text data, extracts the important statement, "I enjoyed art class," and summarizes it concisely. The parent's smart glasses are notified that "I enjoyed art class," and the parent can check detailed information from the smart glasses' interface. If the parent wants to know more, they can enter a question such as "What did you do in art class?" and the server will reanalyze the data and provide detailed information such as "I drew pictures during art class."

[1009] Prompt Sentence Examples

[1010] An example of a specific prompt from the user would be:

[1011] If you receive a user statement such as "I had a fight with a friend today," please determine whether parents should be notified and, if so, suggest the specific format in which they should be notified.

[1012] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1013] Step 1:

[1014] The child's device acquires voice data. The input is the child's everyday speech, which is collected using a microphone. The output is raw voice data.

[1015] Step 2:

[1016] The speech recognition engine converts raw speech data into text data. The input is captured speech data, which is converted into text information in real time using the Google Speech-to-Text API. The output is text data.

[1017] Step 3:

[1018] The terminal periodically sends the converted text data to the server. The input is the text data, which is sent to the server via the network. The output is the text data that arrives at the server.

[1019] Step 4:

[1020] The server analyzes the received text data using a natural language processing (NLP) algorithm. The input is the text data sent to the server, and an NLP model (e.g., the Transformers library) is used to extract important statements based on specific keywords or conditions. The output is a list of important statements.

[1021] Step 5:

[1022] The server generates a summary based on the extracted important comments. The input is a list of important comments, which is converted into a concise summary. The output is the summary.

[1023] Step 6:

[1024] The server notifies the user of the summary sentence on the display device. The input is the summary sentence, which is sent to the parent's smart glasses via the network. The output is the summary sentence notification displayed on the parent's smart glasses.

[1025] Step 7:

[1026] The server instantly notifies the parent of important comments based on specific conditions. The input is text data containing specific keywords, which triggers the immediate transmission of important comments to the parent's display device. The output is a notification of the important comment instantly displayed on the parent's smart glasses.

[1027] Step 8:

[1028] The user inputs a question requesting more information. The input is the question from the parent's smart glasses and is accepted by the server. The output is the question data.

[1029] Step 9:

[1030] The server re-analyzes the question and generates detailed information. The input is the user's question and related text data, and the NLP algorithm is applied again to generate detailed information. The output is detailed information.

[1031] Step 10:

[1032] The server sends the detailed information to the user's display device. The input is the detailed information, which is sent over the network to the parent's smart glasses. The output is the detailed information displayed on the parent's smart glasses.

[1033] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1034] The present invention combines a system that allows parents to efficiently understand the content of their children's daily utterances and conversations with an emotion engine that recognizes the user's emotions. This system is designed to ensure that parents can reliably obtain necessary information while avoiding excessive parental interference. The following describes in detail the embodiments of the present invention.

[1035] First, the device worn by the child has the ability to convert voice data into text data in real time. The device uses a built-in microphone to capture the child's everyday speech and converts it into text using voice recognition software. This text data is then sent from the device to a server at regular intervals.

[1036] The server receives the text data sent from the device and stores it in a database. The server then analyzes the stored text data to extract important statements. This analysis uses natural language processing (NLP) algorithms to find important information based on specific keywords and context. The extracted information is then generated as a summary.

[1037] Furthermore, the server is equipped with an emotion engine that recognizes the user's (parent's) emotions. When generating a summary, the emotion engine references the parent's emotion data and creates a summary in an appropriate format and with appropriate content. It also provides detailed information tailored to the parent's emotions as needed based on questions or requests from the parent.

[1038] After the summary is generated, the server notifies the parent's smartphone. A dedicated app is installed on the parent's smartphone, and the parent can check the summary content through the smartphone. If the parent wants to know more details about the summary, they can input a question through the app. For example, "What did you do in art class today?"

[1039] When a question is entered, the server accepts it and reanalyzes the related text data. The emotion engine also participates in this process, generating detailed information based on the parent's emotional state. The reanalyzed information is then sent back to the user's (parent's) smartphone as detailed information. This allows the parent to understand the specific situation and content.

[1040] Examples:

[1041] For example, suppose a device generates text from an utterance such as "Today's art class was fun" and sends it to a server. The server analyzes this text data and summarizes it as "I enjoyed art class." The emotion engine analyzes the parent's emotional state and adjusts the format and tone of the summary. This summary is notified to the parent's smartphone, and the parent types a question such as "What did you do in art class?" The server and emotion engine analyze the text data again, generating more detailed information such as "I drew pictures during art class," which is then provided to the parent's smartphone.

[1042] As described above, this system goes through each important step to efficiently convey the child's comments and conversations to parents while protecting their privacy. This system allows parents to communicate more smoothly on a daily basis and quickly obtain the information they need. In addition, the emotion engine enables the provision of information based on the parent's emotional state, enabling more personalized responses.

[1043] The processing flow will be explained below.

[1044] Step 1:

[1045] The device activates the microphone and captures the child's voice. The device is set to collect voice data at regular intervals.

[1046] Step 2:

[1047] The device converts the voice data it receives into text data in real time, using voice recognition software to convert the voice signal into text information.

[1048] Step 3:

[1049] The device organizes the converted text data to make it syntactically meaningful and formats it into a specific format.

[1050] Step 4:

[1051] The device sends the organized text data to the server at regular intervals (e.g., every 30 minutes) using a secure communication protocol.

[1052] Step 5:

[1053] The server receives the text data sent from the device and stores it in a database. All received text is saved and can be used for later analysis.

[1054] Step 6:

[1055] The server applies natural language processing (NLP) algorithms to analyze the stored text data, extracting important statements based on specific keywords and context.

[1056] Step 7:

[1057] The server generates a summary based on the extracted important utterances. The summary is written in a concise and meaningful way for parents.

[1058] Step 8:

[1059] The server activates the emotion engine and recognizes the user's (parent's) emotional state when generating a summary. The emotion engine refers to the emotional data obtained from the parent's smartphone app and adjusts the tone and content of the summary.

[1060] Step 9:

[1061] The server sends the generated summary to the parent's smartphone via a dedicated app.

[1062] Step 10:

[1063] The user (parent) receives a notification on their smartphone, launches the app, and checks the summary, which is displayed in a format that is easy for the parent to understand.

[1064] Step 11:

[1065] If the user (parent) has any questions about the summary, they can enter them in the question form within the app. For example, "What did you do in art class?"

[1066] Step 12:

[1067] The server receives the parent's question, re-analyzes the relevant text data, and applies additional natural language processing algorithms depending on the question to extract more detailed information.

[1068] Step 13:

[1069] The server then reactivates the emotion engine to generate detailed information based on the parent's emotional state, and the emotion engine adjusts the tone and content of the detailed information to match the parent's emotions.

[1070] Step 14:

[1071] The server generates detailed information obtained through reanalysis and sends it to the user's (parent's) smartphone. The detailed information is also provided through a dedicated app.

[1072] Step 15:

[1073] The user (parent) receives the detailed information and checks it in the app, allowing the parent to understand the specific details of any concerns.

[1074] Example 2

[1075] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1076] In the past, in order for parents to efficiently understand their children's everyday utterances and conversations, they had to directly observe and listen, which was a heavy burden. Furthermore, there was a lack of a system that could reliably obtain the necessary information while protecting privacy. Therefore, there is a need for a system that can efficiently understand the contents of children's everyday utterances and conversations while avoiding excessive parental interference.

[1077] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1078] In this invention, the server includes: [means for converting voice data into text data in real time;] [means for periodically transmitting the text data to a central processing unit;] [means for analyzing the text data in the central processing unit and extracting important comments;] [means for generating a summary based on the extracted comments;] [means for analyzing the parent's emotions using an emotion recognition module and adjusting the tone of the summary; and [means for notifying the user of the summary.] This enables parents to efficiently understand their child's daily comments and conversations without excessive interference, while still protecting their privacy.

[1079] "Means for converting voice data into text data in real time" refers to devices or software that instantly convert voice data acquired using voice recognition technology into text format.

[1080] The "means for periodically transmitting text data to a central processing unit" refers to a communication device or software for transmitting the converted text data to a remote server or data processing unit at regular time intervals.

[1081] "Means for analyzing text data in a central processing unit and extracting important statements" refers to devices or software that have the function of analyzing text data using natural language processing technology and extracting important keywords and phrases from it.

[1082] The "means for generating a summary based on extracted statements" refers to a device or software that aggregates the extracted important text information and compiles it into a short sentence format.

[1083] "Means for analyzing the parent's emotions using an emotion recognition module and adjusting the tone of the summary" refers to a device or software that recognizes and analyzes the parent's emotional state and adjusts the wording and expression of the summary based on the emotional information.

[1084] The "means for notifying the user of the summary" refers to a device or software that notifies the user of the generated summary to the user's terminal or application.

[1085] The "means for accepting questions from users" refers to devices or software that have the function of inputting and receiving questions from users requesting detailed information about the summary text.

[1086] "Means for reanalyzing text data based on a question and generating detailed information" refers to devices or software that reanalyze existing text data to extract and generate more specific information in response to a user's question.

[1087] MODE FOR CARRYING OUT THE INVENTION

[1088] This invention combines a system that allows parents to efficiently understand their children's daily utterances and conversations with an emotion engine that recognizes the user's emotions. Specific hardware and software configurations and data processing and calculation methods for implementing this invention will be described below.

[1089] 1. Acquiring children's speech and converting it into text

[1090] The device is equipped with a built-in microphone and voice recognition software. For example, a typical condenser microphone is used as the microphone, and Google Speech-to-Text or IBM Watson is used as the voice recognition software. The device picks up the child's everyday speech using the microphone and collects it as voice data. The collected voice data is converted into text data in real time using the voice recognition software inside the device. For example, if a child says, "I played with my friends today," the speech is converted into text data.

[1091] 2. Sending text data

[1092] The converted text data is sent to the server at regular intervals (for example, every 5 minutes). Communication between the device and the server is via an internet connection, and Wi-Fi or mobile data communication can be used. When the device has accumulated text data to send, it automatically sends the data to the server.

[1093] 3. Saving text data

[1094] The server receives the text data sent from the device and stores it in a database. MySQL or MongoDB are suitable databases for this purpose. The server stores the received text data along with a timestamp. For example, data received on "October 1, 2023, 14:00:00" is stored along with the date and time information.

[1095] 4. Text Data Analysis and Summary Generation

[1096] The server analyzes the stored text data and extracts important statements. Natural language processing (NLP) algorithms are used for this analysis. Typical software includes SpaCy and Google's NL API. Using these, the server extracts information from the text data based on specific keywords and context, and generates a summary. For example, from the text data "I hung out with friends today," it generates the summary "I hung out with friends."

[1097] 5. Emotion engine recognizes parents' emotions

[1098] The server is equipped with an emotion engine that recognizes the user's (parent's) emotions. This emotion engine operates based on emotion data acquired when the parent accesses the system via their smartphone. For example, Azure Emotion API recognizes and analyzes the parent's emotional state (satisfaction, anxiety, etc.). When generating a summary, the emotion engine refers to the parent's emotion data and adjusts the tone and format of the summary.

[1099] 6. Notification of Abstract

[1100] The server sends the generated summary to the user's smartphone using FireBase Cloud Messaging (FCM) or a similar push notification technology. The user receives these notifications through a dedicated app and can check the summary. For example, a summary such as "My child played with friends" may be sent to a parent's smartphone.

[1101] 7. User inquiries for more information

[1102] Users can use a dedicated app to enter questions about the summary text. For example, they can enter a question like, "What did you do with your friends?"

[1103] 8. Reanalysis and provision of detailed information

[1104] The server receives the user's question, analyzes the text data again, and generates detailed information. The natural language processing algorithm is applied again to the re-analysis, and the emotion engine is also activated to generate detailed information based on the parent's emotional state. For example, the generated detailed information might be "I played soccer with my friends." This detailed information is then sent to the user's smartphone again, allowing the parent to understand the specific details.

[1105] Examples of concrete examples and prompts

[1106] For example, if a child says, "I had fun in art class today," the device converts the speech into text and sends it to the server. The server analyzes the text data and generates a summary saying, "I had fun in art class." The emotion engine analyzes the parent's emotions, and a summary with adjusted tone is sent to the parent's smartphone. When the parent types a question into a dedicated app, such as, "What did you do in art class?", the server reanalyzes the data and generates and notifies the parent with more detailed information, such as, "I drew pictures in art class."

[1107] Example prompt sentence:

[1108] "Please summarize what your child said today."

[1109] "Tell me more about what you did in art class today."

[1110] This system allows parents to efficiently understand their children's daily comments and conversations and quickly obtain the information they need without excessive interference.In addition, the emotion engine enables personalized information provision according to the parent's emotional state.

[1111] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1112] Step 1:

[1113] Acquiring children's statements

[1114] The device's built-in microphone is used to capture the child's everyday speech in real time. For example, if a child says, "I played with my friends today," the voice is collected through the microphone.

[1115] Input: Child's speech (audio data)

[1116] Output: Collected audio data

[1117] Step 2:

[1118] Converting audio data to text

[1119] The device converts the collected voice data into text data in real time using voice recognition software (e.g., Google Speech-to-Text). For example, the text data generated is "I played with my friends today."

[1120] Input: Collected audio data

[1121] Output: Converted text data

[1122] Step 3:

[1123] Sending text data

[1124] The device sends the converted text data to the server at regular intervals (for example, every 5 minutes). This communication uses an internet connection (Wi-Fi or mobile data). For example, the text data sent might be "I played with friends today."

[1125] Input: Converted text data

[1126] Output: Text data sent to the server

[1127] Step 4:

[1128] Saving text data

[1129] The server receives the text data sent from the device and stores it in a database such as MySQL or MongoDB. For example, it is stored as data received on "October 1, 2023, 14:00:00."

[1130] Input: Received text data

[1131] Output: Text data stored in a database

[1132] Step 5:

[1133] Text data analysis and summary generation

[1134] The server analyzes the stored text data using a natural language processing (NLP) algorithm (e.g., SpaCy) to extract important statements. It then generates a summary based on the extracted information. For example, it generates a summary from "I played with my friends today" to "I played with my friends."

[1135] Input: Saved text data

[1136] Output: Summary

[1137] Step 6:

[1138] Parental emotion recognition using emotion engine

[1139] The server's emotion engine (e.g., Azure Emotion API) recognizes and analyzes the user's (parent's) emotions. For example, based on the emotional data collected when the parent uses a dedicated app, it analyzes whether the parent is "satisfied" or "anxious." Based on this analysis, the tone and format of the summary text are adjusted.

[1140] Input: Parent emotion data, summary sentence

[1141] Output: A summary with adjusted tone and format

[1142] Step 7:

[1143] Summary notification

[1144] The server sends the generated summary to the user's (parent's) smartphone via a dedicated app. For example, using Firebase Cloud Messaging (FCM), a summary such as "My child played with friends" is sent to the parent's smartphone.

[1145] Input: A summary with adjusted tone and format

[1146] Output: Summary sent to parent's smartphone

[1147] Step 8:

[1148] User inquiries for more information

[1149] The user (parent) can use a dedicated app to enter questions about the summary text, such as "What did you play with your friends?"

[1150] Input: Parent Question

[1151] Output: Questions accepted by the app

[1152] Step 9:

[1153] Reanalysis of detailed information

[1154] The server accepts questions from users and re-analyzes the related text data. A natural language processing algorithm is used for the re-analysis, and an emotion engine is also activated to generate detailed information based on the parent's emotional state. For example, the server might generate detailed information such as "I played soccer with my friends."

[1155] Input: Parent questions, initial text data

[1156] Output: Detailed information

[1157] Step 10:

[1158] Notification of detailed information

[1159] The server then sends the generated details to the parent's smartphone via a dedicated app. For example, the parent's smartphone may receive detailed information such as "My child played soccer with a friend."

[1160] Input: More information

[1161] Output: Detailed information sent to parent's smartphone

[1162] (Application example 2)

[1163] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1164] In virtual stores, there is a need for a system that can efficiently communicate between customers and sales staff (clerks) and respond quickly and appropriately to customer requests and questions. However, with current systems, it is difficult for sales staff to understand all customer conversations in real time and provide appropriate information each time. Furthermore, further technological improvements are needed to respond promptly to customer emotions and requests.

[1165] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data into text data in real time, means for periodically transmitting the text data to the server, means for analyzing the text data in the server and extracting important utterances, means for generating a summary based on the extracted utterances, means for notifying the user of the summary, means for recognizing the user's emotions and reflecting them in the summary, and means for use as a virtual customer service assistant that provides related information based on the user's questions. This enables customer service staff to efficiently respond to customers' questions and requests, thereby improving customer satisfaction.

[1166] "Means for converting voice data into text data in real time" refers to devices or technologies that instantly analyze voice and convert its content into text information.

[1167] "Means for periodically sending text data to a server" refers to a method or technique for transferring text data to a remote server at specific time intervals.

[1168] "Means for analyzing text data on a server and extracting important statements" refers to technology for processing text data on a server and identifying and extracting content that is considered particularly important.

[1169] "Means for generating a summary based on extracted statements" refers to a technology for creating a document that summarizes important statements.

[1170] "Means for notifying the user of the summary" refers to technology for notifying the user of the generated summary via a terminal or notification system.

[1171] "Means for recognizing the user's emotions and reflecting them in the summary" refers to technology for analyzing the user's emotional state and reflecting the results in the content and expression of the summary.

[1172] "Means for applying as a virtual customer service assistant that provides relevant information based on a user's question" refers to a virtual support system for processing questions from a user and providing appropriate information.

[1173] The present invention relates to a system for improving the efficiency of customer service in a virtual store. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS ... will be described in detail below with reference to the accompanying drawings.

[1174] First, a terminal equipped with a microphone is required to capture the conversation between the customer and the store clerk in order to convert the voice data into text data in real time. This terminal then uses voice recognition software to convert the voice data into text data, which is then sent to a server at specific time intervals.

[1175] The server receives the text data and analyzes it. This analysis is performed using a natural language processing (NLP) algorithm. The server extracts important statements from the analyzed data and generates a summary based on those statements. The generated summary is then notified to the user, but before that, the server uses an emotion recognition engine to analyze the user's emotions and reflects that information in the summary.

[1176] When a customer asks a store clerk a question, the system accepts the question, analyzes the text data again, and provides relevant information. This information is displayed on smart glasses or a head-mounted display, allowing the store clerk to respond immediately to the customer's question and provide appropriate information.

[1177] The hardware used includes smart glasses, a head-mounted display, and a microphone. The software uses speech recognition libraries (e.g., speech_recognition), natural language processing models (e.g., transformers), and data transmission libraries (e.g., requests). It also uses generative AI models for sentiment analysis.

[1178] Examples:

[1179] The device detects the customer's question, "Which product do you recommend most?" and converts the conversation into text data. The server analyzes the text data, extracts important information, generates a summary, and then displays an appropriate answer on the smart glasses, taking into account the emotional state of the salesperson. The following prompt sentence is used to generate this summary:

[1180] Example prompt sentence:

[1181] "Use sentiment analysis to generate the most appropriate answer for the customer's question, 'Which product do you recommend most?' If the customer's emotional state is positive, provide more information about the product; if it's negative, suggest an alternative."

[1182] As a result, the present invention can significantly improve the efficiency of customer service in a virtual store, thereby achieving increased customer satisfaction.

[1183] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1184] Step 1:

[1185] The terminal uses a microphone to capture the conversation between the customer and the store clerk, and converts the voice data into text data in real time using voice recognition software (e.g., the speech_recognition library). The input is voice data, and the output is text data.

[1186] Step 2:

[1187] The terminal sends the converted text data to the server at regular intervals. The input is text data, and the output is data transmission to the server. This transmission uses a data transmission library (e.g., requests).

[1188] Step 3:

[1189] The server stores the received text data and uses natural language processing (NLP) algorithms to analyze the data and extract key statements. The input is the text data, and the output is the analysis results containing the key statements. Specifically, the server processes the text data using NLP models (e.g., transformers).

[1190] Step 4:

[1191] The server generates a summary based on the extracted important statements. The input is the important statements, and the output is the summary. A text summarization algorithm is used to generate the summary.

[1192] Step 5:

[1193] The server uses an emotion recognition engine to analyze the user's emotions and reflects that information in the summary. The input is the summary and the user's emotion data, and the output is a summary based on the emotion. A generative AI model is used for emotion analysis.

[1194] Step 6:

[1195] The server sends the generated summary to the user's device. The input is a summary corresponding to the emotion, and the output is a notification to the user's device. This notification is done using a dedicated application.

[1196] Step 7:

[1197] The user (store clerk) checks the summary displayed on the smart glasses or head-mounted display and responds to the customer's questions and requests based on it. The input is the notified summary, and the output is a response to the customer.

[1198] Step 8:

[1199] When a user enters a question, the server re-analyzes the question and generates detailed information based on the related text data. The input is the question, and the output is detailed information. This re-analysis is performed using natural language processing algorithms.

[1200] Step 9:

[1201] The generated detailed information is again notified to the user, and the user (store clerk) provides a specific answer to the customer. The input is detailed information, and the output is a detailed answer to the customer.

[1202] Example prompt sentence:

[1203] "Use sentiment analysis to generate the most appropriate answer for the customer's question, 'Which product do you recommend most?' If the customer's emotional state is positive, provide more information about the product; if it's negative, suggest an alternative."

[1204] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1205] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1206] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1207] [Fourth embodiment]

[1208] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1209] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1210] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1211] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1212] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1213] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1214] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1215] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1216] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1217] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1218] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1219] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1220] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1221] The present invention relates to a system that enables parents to efficiently understand the content of their children's daily utterances and conversations. This system is designed to ensure that parents can obtain necessary information without interfering excessively. The following describes in detail an embodiment of the present invention.

[1222] First, the device worn by the child has the ability to convert voice data into text data in real time. The device uses a built-in microphone to capture the child's everyday speech and converts it into text using voice recognition software. This text data is then sent from the device to a server at regular intervals.

[1223] The server receives the text data sent from the device and stores it in a database. The server then analyzes the stored text data to extract important statements. This analysis uses natural language processing (NLP) algorithms to find important information based on specific keywords and context. The extracted information is then generated as a summary.

[1224] After the summary is generated, the server notifies the parent's smartphone. A dedicated app is installed on the parent's smartphone, and the parent can check the summary content through the smartphone. If the parent wants to know more details about the summary, they can input a question through the app. For example, "What did you do in art class today?"

[1225] When a question is entered, the server accepts the question and reanalyzes the related text data. The reanalyzed information is then sent back to the user's (parent's) smartphone as detailed information, allowing the parent to understand the specific situation and content.

[1226] Examples:

[1227] For example, suppose a device generates text from an utterance such as "Today's art class was fun" and sends it to a server. The server analyzes this text data and summarizes it as "I enjoyed art class." This summary is then sent to the parent's smartphone, who then types in a question such as "What did you do in art class?" The server then analyzes the text data again and generates more detailed information such as "I drew pictures during art class," which is then provided to the parent's smartphone.

[1228] As described above, this system goes through each important step to efficiently convey the contents of a child's statements and conversations to parents while protecting their privacy, allowing parents to communicate more smoothly and quickly obtain the information they need.

[1229] The processing flow will be explained below.

[1230] Step 1:

[1231] The device activates the microphone and captures the child's voice. The device is set to collect voice data at regular intervals.

[1232] Step 2:

[1233] The device converts the voice data it receives into text data in real time, using voice recognition software to convert the voice signal into text information.

[1234] Step 3:

[1235] The device then cleans up the converted text data and corrects it to make it grammatically meaningful, which makes subsequent parsing easier.

[1236] Step 4:

[1237] The device sends the organized text data to the server at regular intervals (e.g., every 30 minutes) using a secure communication protocol.

[1238] Step 5:

[1239] The server receives the text data sent from the device and stores it in a database. All received text is saved and can be used for later analysis.

[1240] Step 6:

[1241] The server applies natural language processing (NLP) algorithms to analyze the stored text data, extracting important statements based on specific keywords and context.

[1242] Step 7:

[1243] The server generates a summary based on the extracted important utterances. The summary is written in a concise and meaningful way for parents.

[1244] Step 8:

[1245] The server then sends the generated summary to the parent's smartphone via a dedicated app.

[1246] Step 9:

[1247] The user (parent) receives a notification on their smartphone, launches the app, and checks the summary, which is displayed in a format that is easy for the parent to understand.

[1248] Step 10:

[1249] If the user (parent) has any questions about the summary, they can enter them in the question form within the app. For example, "What did you do in art class?"

[1250] Step 11:

[1251] The server receives the parent's question, re-analyzes the relevant text data, and applies additional natural language processing algorithms depending on the question to extract more detailed information.

[1252] Step 12:

[1253] The server generates detailed information obtained through reanalysis and sends it to the user's (parent's) smartphone. The detailed information is also provided through a dedicated app.

[1254] Step 13:

[1255] The user (parent) receives the detailed information and checks it in the app, allowing the parent to understand the specific details of any concerns.

[1256] Example 1

[1257] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1258] In recent years, while it has become increasingly important for parents to understand their children's daily events and emotions, there is also a growing need to avoid excessive interference. However, it is difficult to efficiently obtain the necessary information using conventional methods. The purpose of this invention is to enable parents to efficiently and appropriately understand their children's daily utterances and conversations.

[1259] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1260] In this invention, the server includes: [means for converting voice data into text data in real time; [means for periodically transmitting the text data to the server; [means for analyzing the text data in the server and extracting important comments; [means for generating a summary based on the extracted comments; [means for notifying the user of the summary; [means for transmitting a question entered by the user to the server; [means for reanalyzing the text data based on the question and generating detailed information; and [means for notifying the user of the detailed information. This enables parents to efficiently keep track of their children's daily lives.

[1261] "Converting voice data into text data in real time" means instantly converting collected voice information into text information using voice recognition technology.

[1262] "Periodic transmission of text data to a server" means transferring the generated text information to a server via a network at regular time intervals.

[1263] "Analyzing text data on the server and extracting important statements" means analyzing the text information stored on the server using a natural language processing algorithm to extract meaningful information and keywords.

[1264] "Generating a summary based on extracted statements" means generating a concise sentence based on important information obtained from the analysis results.

[1265] "Notifying the user of the summary" means sending the generated summary to the user's device by means of push notification, email, or the like.

[1266] "Sending a question entered by a user to a server" means transferring a question entered by a user through a dedicated application to a server.

[1267] "Reanalyzing text data based on the question and generating detailed information" means reanalyzing the saved text information according to the content of the user's question and extracting related detailed information.

[1268] "Notifying the user of detailed information" means sending the reanalyzed results to the user's device and providing detailed information.

[1269] The present invention relates to a system that enables parents to efficiently understand their children's daily utterances and conversations. This system is designed to allow parents to reliably obtain necessary information while avoiding excessive interference. Specific embodiments for carrying out the present invention will be described below.

[1270] Hardware used

[1271] The device is a special wearable device that can be worn by children and has a built-in microphone that picks up surrounding sounds in real time.

[1272] The server used is a high-performance cloud-based database server, which can efficiently process and store large amounts of data.

[1273] The user device required is a smartphone for parents to receive information, on which a dedicated application is installed.

[1274] Software used

[1275] Speech recognition software: The device is installed with speech recognition software such as the Google Speech-to-Text API, which converts voice data captured in real time into text data.

[1276] Natural language processing libraries: The server uses natural language processing algorithms such as SpaCy and NLTK to analyze the text data and extract important statements.

[1277] Database software: Data is stored in a database such as MySQL or PostgreSQL.

[1278] Push notification service: Using services such as Firebase Cloud Messaging, the generated summary is sent to the parent's smartphone.

[1279] Specific examples of processing

[1280] A specific example of the process is as follows:

[1281] 1. Acquiring voice data: A child says, "I enjoyed art class today." The device's built-in microphone collects this voice data.

[1282] 2. Voice data to text conversion: The collected voice data is converted into text data in real time by the device's voice recognition software.

[1283] 3. Sending text data: The converted text data is sent to the server at regular intervals. Communication is carried out using the REST API.

[1284] 4. Analysis of text data and generation of summary sentence: The server analyzes the received text data and generates a summary sentence such as "I enjoyed art class" using a natural language processing library.

[1285] 5. Notification of summary: The generated summary is sent to the parent's smartphone via a push notification service.

[1286] Get more information

[1287] When a parent types a question into the app, such as "What did you do in art class?", the server receives the question, reanalyzes the text data, and finds the detailed information, such as "I drew a picture." This information is then sent back to the parent's smartphone.

[1288] Prompt Sentence Examples

[1289] An example of a prompt sentence when using a generative AI model is as follows:

[1290] A child says, "I enjoyed art class today." Generate a summary for the parent.

[1291] ---

[1292] Parents are asking, "What did you do in art class?" Please provide more information.

[1293] As described above, the present invention provides a means for parents to efficiently understand their children's daily lives by appropriately processing the information required at each step. This system facilitates smoother communication between parents and children, and enables them to quickly obtain the necessary information.

[1294] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1295] Step 1:

[1296] Acquiring audio data

[1297] Terminal

[1298] Example of operation: A child wears a dedicated device and begins everyday conversation. The device's built-in microphone collects voice data.

[1299] Input: Child's speech (audio data)

[1300] Data processing: The microphone captures the audio data.

[1301] Output: Audio data stored on the device

[1302] Step 2:

[1303] Converting audio data to text

[1304] Terminal

[1305] Example of operation: When a child says, "I enjoyed art class today," this speech is instantly converted into text data.

[1306] Input: Stored audio data

[1307] Data processing: Speech recognition software (e.g., Google Speech-to-Text API) recognizes the voice data and converts it into text.

[1308] Output: Converted text data

[1309] Step 3:

[1310] Sending text data

[1311] Terminal

[1312] Example of operation: The converted text data is sent to the server at regular intervals.

[1313] Input: Converted text data

[1314] Data transfer: Send text data to the server using the REST API.

[1315] Output: Text data sent to the server

[1316] Step 4:

[1317] Receiving and storing text data

[1318] server

[1319] Example of operation: The server saves the received text data in a database.

[1320] Input: Text data sent

[1321] Data processing: Save the text data in a database (e.g., MySQL).

[1322] Output: Text data stored in a database

[1323] Step 5:

[1324] Text data analysis

[1325] server

[1326] Example of how it works: Extracting important utterances from stored text data.

[1327] Input: Text data stored in a database

[1328] Data Computing: Use natural language processing algorithms (e.g., SpaCy, NLTK) to analyze important keywords and context.

[1329] Output: Important statements as a result of the analysis

[1330] Step 6:

[1331] Summary generation

[1332] server

[1333] Example of operation: Generate a summary sentence such as "Art class was fun" from the analysis results.

[1334] Input: Important statements as a result of the analysis

[1335] Data computation: Generative AI models and rule-based algorithms are used to create summaries based on key statements.

[1336] Output: Generated summary

[1337] Step 7:

[1338] Summary notification

[1339] server

[1340] Example of how it works: The summary text is sent to the parent's smartphone.

[1341] Input: Generated summary

[1342] Data transfer: A summary is sent to the parent's smartphone using a push notification service (e.g., Firebase Cloud Messaging).

[1343] Output: Summary displayed on parent's smartphone

[1344] Step 8:

[1345] Accepting user questions

[1346] User

[1347] Example of how it works: A parent types a question into a dedicated app, such as "What did you do in art class?", and sends it.

[1348] Input: Questions entered by the parent

[1349] Data transfer: Send your questions to the server through the app.

[1350] Output: The question sent to the server

[1351] Step 9:

[1352] Reanalysis of detailed information

[1353] server

[1354] Example of how it works: The server analyzes the text data again and finds the detailed information "I drew a picture."

[1355] Input: Submitted questions and saved text data

[1356] Data operations: Reanalyze relevant text data based on the question to extract more information.

[1357] Output: Reparsed details

[1358] Step 10:

[1359] Notification of detailed information

[1360] server

[1361] Example of how it works: The server sends detailed information to the parent's smartphone.

[1362] Input: Extracted details

[1363] Data transfer: Send detailed information to the parent's smartphone via push notification service.

[1364] Output: Detailed information displayed on the parent's smartphone

[1365] (Application example 1)

[1366] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1367] The present invention aims to provide a system that allows parents to efficiently understand their children's everyday comments and conversations and quickly and reliably obtain the necessary information. Another objective is to provide a sense of security for parents and improve their children's security by notifying them of important comments based on specific conditions while avoiding excessive interference. Another important objective is to be able to quickly respond to parents' requests for more information.

[1368] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1369] In this invention, the server includes: [means for converting voice data into text data in real time;] [means for periodically transmitting the text data to the server;] [means for analyzing the text data in the server and extracting important comments;] [means for generating a summary based on the extracted comments;] [means for notifying the user of the summary; and [means for notifying the user of the content of important comments on a display device based on specific conditions.] This allows parents to efficiently and quickly understand the content of their children's comments, giving them a sense of security and allowing them to quickly obtain the information they need.

[1370] "Audio data" means audio recorded in digital format and converted into text data in real time.

[1371] "Text data" refers to character information converted from voice data, and is sent to a server for analysis.

[1372] A "server" is a computer system connected via a network that is a central processing unit for receiving and analyzing text data.

[1373] "Analysis" is the process of understanding the text data transmitted and extracting important information based on specific criteria.

[1374] "Important comments" refer to comments that are worth notifying parents about, identified based on pre-set keywords and conditions.

[1375] A "summary" is a short document generated based on important statements extracted from text data, and is information that is notified to the user.

[1376] "Notification" is the act of transmitting information from a server to a user's display device, and is a means of providing information to a user in real time.

[1377] "Specific conditions" refer to rules such as keywords and contexts that are set as criteria for extracting important statements.

[1378] The "display device" is hardware for displaying notified information such as a summary, and is an interface through which the user can check the information.

[1379] In the present invention, the system is implemented based on the following configuration and procedures.

[1380] Hardware and Software Configuration

[1381] 1. Voice data collection terminal

[1382] The voice-collecting devices, such as smart bands and headsets worn by children, have built-in microphones and capture the child's everyday speech in real time.

[1383] 2. Speech Recognition Engine

[1384] The collected voice data is converted into text data in real time using a voice recognition engine, such as the Google Speech-to-Text API.

[1385] 3. Server

[1386] The converted text data is periodically sent to the server, which is responsible for the following:

[1387] Text Data Analysis

[1388] Use natural language processing (NLP) algorithms (e.g., the Transformers library) to analyze the incoming text data and extract key utterances.

[1389] Summary sentence generation

[1390] A summary is generated based on the extracted important statements.

[1391] notification

[1392] The summary sentence is sent to the parent's display device (e.g., smart glasses), and important utterances are instantly notified based on specific conditions.

[1393] 4. User Display Device

[1394] The parent's display device is a visual device such as smart glasses that displays the summary sent from the server and important comment notifications.

[1395] Data processing and calculation

[1396] 1. Audio data collection and conversion

[1397] Voice data is picked up in real time through a microphone and converted into text data by a voice recognition engine.

[1398] 2. Transmission and analysis of text data

[1399] The converted text data is sent to a server where it is analyzed using natural language processing algorithms, which extract important statements based on predefined keywords and context.

[1400] 3. Summary generation and notification

[1401] A summary is generated based on the extracted important comments and is displayed on the parent's display device. Furthermore, important comments based on specific conditions are immediately displayed.

[1402] Specific examples

[1403] For example, suppose a child says, "I enjoyed art class today." This voice data is converted into text data in real time and sent to the server. The server analyzes this text data, extracts the important statement, "I enjoyed art class," and summarizes it concisely. The parent's smart glasses are notified that "I enjoyed art class," and the parent can check detailed information from the smart glasses' interface. If the parent wants to know more, they can enter a question such as "What did you do in art class?" and the server will reanalyze the data and provide detailed information such as "I drew pictures during art class."

[1404] Prompt Sentence Examples

[1405] An example of a specific prompt from the user would be:

[1406] If you receive a user statement such as "I had a fight with a friend today," please determine whether parents should be notified and, if so, suggest the specific format in which they should be notified.

[1407] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1408] Step 1:

[1409] The child's device acquires voice data. The input is the child's everyday speech, which is collected using a microphone. The output is raw voice data.

[1410] Step 2:

[1411] The speech recognition engine converts raw speech data into text data. The input is captured speech data, which is converted into text information in real time using the Google Speech-to-Text API. The output is text data.

[1412] Step 3:

[1413] The terminal periodically sends the converted text data to the server. The input is the text data, which is sent to the server via the network. The output is the text data that arrives at the server.

[1414] Step 4:

[1415] The server analyzes the received text data using a natural language processing (NLP) algorithm. The input is the text data sent to the server, and an NLP model (e.g., the Transformers library) is used to extract important statements based on specific keywords or conditions. The output is a list of important statements.

[1416] Step 5:

[1417] The server generates a summary based on the extracted important comments. The input is a list of important comments, which is converted into a concise summary. The output is the summary.

[1418] Step 6:

[1419] The server notifies the user of the summary sentence on the display device. The input is the summary sentence, which is sent to the parent's smart glasses via the network. The output is the summary sentence notification displayed on the parent's smart glasses.

[1420] Step 7:

[1421] The server instantly notifies the parent of important comments based on specific conditions. The input is text data containing specific keywords, which triggers the immediate transmission of important comments to the parent's display device. The output is a notification of the important comment instantly displayed on the parent's smart glasses.

[1422] Step 8:

[1423] The user inputs a question requesting more information. The input is the question from the parent's smart glasses and is accepted by the server. The output is the question data.

[1424] Step 9:

[1425] The server re-analyzes the question and generates detailed information. The input is the user's question and related text data, and the NLP algorithm is applied again to generate detailed information. The output is detailed information.

[1426] Step 10:

[1427] The server sends the detailed information to the user's display device. The input is the detailed information, which is sent over the network to the parent's smart glasses. The output is the detailed information displayed on the parent's smart glasses.

[1428] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1429] The present invention combines a system that allows parents to efficiently understand the content of their children's daily utterances and conversations with an emotion engine that recognizes the user's emotions. This system is designed to ensure that parents can reliably obtain necessary information while avoiding excessive parental interference. The following describes in detail the embodiments of the present invention.

[1430] First, the device worn by the child has the ability to convert voice data into text data in real time. The device uses a built-in microphone to capture the child's everyday speech and converts it into text using voice recognition software. This text data is then sent from the device to a server at regular intervals.

[1431] The server receives the text data sent from the device and stores it in a database. The server then analyzes the stored text data to extract important statements. This analysis uses natural language processing (NLP) algorithms to find important information based on specific keywords and context. The extracted information is then generated as a summary.

[1432] Furthermore, the server is equipped with an emotion engine that recognizes the user's (parent's) emotions. When generating a summary, the emotion engine references the parent's emotion data and creates a summary in an appropriate format and with appropriate content. It also provides detailed information tailored to the parent's emotions as needed based on questions or requests from the parent.

[1433] After the summary is generated, the server notifies the parent's smartphone. A dedicated app is installed on the parent's smartphone, and the parent can check the summary content through the smartphone. If the parent wants to know more details about the summary, they can input a question through the app. For example, "What did you do in art class today?"

[1434] When a question is entered, the server accepts it and reanalyzes the related text data. The emotion engine also participates in this process, generating detailed information based on the parent's emotional state. The reanalyzed information is then sent back to the user's (parent's) smartphone as detailed information. This allows the parent to understand the specific situation and content.

[1435] Examples:

[1436] For example, suppose a device generates text from an utterance such as "Today's art class was fun" and sends it to a server. The server analyzes this text data and summarizes it as "I enjoyed art class." The emotion engine analyzes the parent's emotional state and adjusts the format and tone of the summary. This summary is notified to the parent's smartphone, and the parent types a question such as "What did you do in art class?" The server and emotion engine analyze the text data again, generating more detailed information such as "I drew pictures during art class," which is then provided to the parent's smartphone.

[1437] As described above, this system goes through each important step to efficiently convey the child's comments and conversations to parents while protecting their privacy. This system allows parents to communicate more smoothly on a daily basis and quickly obtain the information they need. In addition, the emotion engine enables the provision of information based on the parent's emotional state, enabling more personalized responses.

[1438] The processing flow will be explained below.

[1439] Step 1:

[1440] The device activates the microphone and captures the child's voice. The device is set to collect voice data at regular intervals.

[1441] Step 2:

[1442] The device converts the voice data it receives into text data in real time, using voice recognition software to convert the voice signal into text information.

[1443] Step 3:

[1444] The device organizes the converted text data to make it syntactically meaningful and formats it into a specific format.

[1445] Step 4:

[1446] The device sends the organized text data to the server at regular intervals (e.g., every 30 minutes) using a secure communication protocol.

[1447] Step 5:

[1448] The server receives the text data sent from the device and stores it in a database. All received text is saved and can be used for later analysis.

[1449] Step 6:

[1450] The server applies natural language processing (NLP) algorithms to analyze the stored text data, extracting important statements based on specific keywords and context.

[1451] Step 7:

[1452] The server generates a summary based on the extracted important utterances. The summary is written in a concise and meaningful way for parents.

[1453] Step 8:

[1454] The server activates the emotion engine and recognizes the user's (parent's) emotional state when generating a summary. The emotion engine refers to the emotional data obtained from the parent's smartphone app and adjusts the tone and content of the summary.

[1455] Step 9:

[1456] The server sends the generated summary to the parent's smartphone via a dedicated app.

[1457] Step 10:

[1458] The user (parent) receives a notification on their smartphone, launches the app, and checks the summary, which is displayed in a format that is easy for the parent to understand.

[1459] Step 11:

[1460] If the user (parent) has any questions about the summary, they can enter them in the question form within the app. For example, "What did you do in art class?"

[1461] Step 12:

[1462] The server receives the parent's question, re-analyzes the relevant text data, and applies additional natural language processing algorithms depending on the question to extract more detailed information.

[1463] Step 13:

[1464] The server then reactivates the emotion engine to generate detailed information based on the parent's emotional state, and the emotion engine adjusts the tone and content of the detailed information to match the parent's emotions.

[1465] Step 14:

[1466] The server generates detailed information obtained through reanalysis and sends it to the user's (parent's) smartphone. The detailed information is also provided through a dedicated app.

[1467] Step 15:

[1468] The user (parent) receives the detailed information and checks it in the app, allowing the parent to understand the specific details of any concerns.

[1469] Example 2

[1470] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1471] In the past, in order for parents to efficiently understand their children's everyday utterances and conversations, they had to directly observe and listen, which was a heavy burden. Furthermore, there was a lack of a system that could reliably obtain the necessary information while protecting privacy. Therefore, there is a need for a system that can efficiently understand the contents of children's everyday utterances and conversations while avoiding excessive parental interference.

[1472] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1473] In this invention, the server includes: [means for converting voice data into text data in real time;] [means for periodically transmitting the text data to a central processing unit;] [means for analyzing the text data in the central processing unit and extracting important comments;] [means for generating a summary based on the extracted comments;] [means for analyzing the parent's emotions using an emotion recognition module and adjusting the tone of the summary; and [means for notifying the user of the summary.] This enables parents to efficiently understand their child's daily comments and conversations without excessive interference, while still protecting their privacy.

[1474] "Means for converting voice data into text data in real time" refers to devices or software that instantly convert voice data acquired using voice recognition technology into text format.

[1475] The "means for periodically transmitting text data to a central processing unit" refers to a communication device or software for transmitting the converted text data to a remote server or data processing unit at regular time intervals.

[1476] "Means for analyzing text data in a central processing unit and extracting important statements" refers to devices or software that have the function of analyzing text data using natural language processing technology and extracting important keywords and phrases from it.

[1477] The "means for generating a summary based on extracted statements" refers to a device or software that aggregates the extracted important text information and compiles it into a short sentence format.

[1478] "Means for analyzing the parent's emotions using an emotion recognition module and adjusting the tone of the summary" refers to a device or software that recognizes and analyzes the parent's emotional state and adjusts the wording and expression of the summary based on the emotional information.

[1479] The "means for notifying the user of the summary" refers to a device or software that notifies the user of the generated summary to the user's terminal or application.

[1480] The "means for accepting questions from users" refers to devices or software that have the function of inputting and receiving questions from users requesting detailed information about the summary text.

[1481] "Means for reanalyzing text data based on a question and generating detailed information" refers to devices or software that reanalyze existing text data to extract and generate more specific information in response to a user's question.

[1482] MODE FOR CARRYING OUT THE INVENTION

[1483] This invention combines a system that allows parents to efficiently understand their children's daily utterances and conversations with an emotion engine that recognizes the user's emotions. Specific hardware and software configurations and data processing and calculation methods for implementing this invention will be described below.

[1484] 1. Acquiring children's speech and converting it into text

[1485] The device is equipped with a built-in microphone and voice recognition software. For example, a typical condenser microphone is used as the microphone, and Google Speech-to-Text or IBM Watson is used as the voice recognition software. The device picks up the child's everyday speech using the microphone and collects it as voice data. The collected voice data is converted into text data in real time using the voice recognition software inside the device. For example, if a child says, "I played with my friends today," the speech is converted into text data.

[1486] 2. Sending text data

[1487] The converted text data is sent to the server at regular intervals (for example, every 5 minutes). Communication between the device and the server is via an internet connection, and Wi-Fi or mobile data communication can be used. When the device has accumulated text data to send, it automatically sends the data to the server.

[1488] 3. Saving text data

[1489] The server receives the text data sent from the device and stores it in a database. MySQL or MongoDB are suitable databases for this purpose. The server stores the received text data along with a timestamp. For example, data received on "October 1, 2023, 14:00:00" is stored along with the date and time information.

[1490] 4. Text Data Analysis and Summary Generation

[1491] The server analyzes the stored text data and extracts important statements. Natural language processing (NLP) algorithms are used for this analysis. Typical software includes SpaCy and Google's NL API. Using these, the server extracts information from the text data based on specific keywords and context, and generates a summary. For example, from the text data "I hung out with friends today," it generates the summary "I hung out with friends."

[1492] 5. Emotion engine recognizes parents' emotions

[1493] The server is equipped with an emotion engine that recognizes the user's (parent's) emotions. This emotion engine operates based on emotion data acquired when the parent accesses the system via their smartphone. For example, Azure Emotion API recognizes and analyzes the parent's emotional state (satisfaction, anxiety, etc.). When generating a summary, the emotion engine refers to the parent's emotion data and adjusts the tone and format of the summary.

[1494] 6. Notification of Abstract

[1495] The server sends the generated summary to the user's smartphone using FireBase Cloud Messaging (FCM) or a similar push notification technology. The user receives these notifications through a dedicated app and can check the summary. For example, a summary such as "My child played with friends" may be sent to a parent's smartphone.

[1496] 7. User inquiries for more information

[1497] Users can use a dedicated app to enter questions about the summary text. For example, they can enter a question like, "What did you do with your friends?"

[1498] 8. Reanalysis and provision of detailed information

[1499] The server receives the user's question, analyzes the text data again, and generates detailed information. The natural language processing algorithm is applied again to the re-analysis, and the emotion engine is also activated to generate detailed information based on the parent's emotional state. For example, the generated detailed information might be "I played soccer with my friends." This detailed information is then sent to the user's smartphone again, allowing the parent to understand the specific details.

[1500] Examples of concrete examples and prompts

[1501] For example, if a child says, "I had fun in art class today," the device converts the speech into text and sends it to the server. The server analyzes the text data and generates a summary saying, "I had fun in art class." The emotion engine analyzes the parent's emotions, and a summary with adjusted tone is sent to the parent's smartphone. When the parent types a question into a dedicated app, such as, "What did you do in art class?", the server reanalyzes the data and generates and notifies the parent with more detailed information, such as, "I drew pictures in art class."

[1502] Example prompt sentence:

[1503] "Please summarize what your child said today."

[1504] "Tell me more about what you did in art class today."

[1505] This system allows parents to efficiently understand their children's daily comments and conversations and quickly obtain the information they need without excessive interference.In addition, the emotion engine enables personalized information provision according to the parent's emotional state.

[1506] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1507] Step 1:

[1508] Acquiring children's statements

[1509] The device's built-in microphone is used to capture the child's everyday speech in real time. For example, if a child says, "I played with my friends today," the voice is collected through the microphone.

[1510] Input: Child's speech (audio data)

[1511] Output: Collected audio data

[1512] Step 2:

[1513] Converting audio data to text

[1514] The device converts the collected voice data into text data in real time using voice recognition software (e.g., Google Speech-to-Text). For example, the text data generated is "I played with my friends today."

[1515] Input: Collected audio data

[1516] Output: Converted text data

[1517] Step 3:

[1518] Sending text data

[1519] The device sends the converted text data to the server at regular intervals (for example, every 5 minutes). This communication uses an internet connection (Wi-Fi or mobile data). For example, the text data sent might be "I played with friends today."

[1520] Input: Converted text data

[1521] Output: Text data sent to the server

[1522] Step 4:

[1523] Saving text data

[1524] The server receives the text data sent from the device and stores it in a database such as MySQL or MongoDB. For example, it is stored as data received on "October 1, 2023, 14:00:00."

[1525] Input: Received text data

[1526] Output: Text data stored in a database

[1527] Step 5:

[1528] Text data analysis and summary generation

[1529] The server analyzes the stored text data using a natural language processing (NLP) algorithm (e.g., SpaCy) to extract important statements. It then generates a summary based on the extracted information. For example, it generates a summary from "I played with my friends today" to "I played with my friends."

[1530] Input: Saved text data

[1531] Output: Summary

[1532] Step 6:

[1533] Parental emotion recognition using emotion engine

[1534] The server's emotion engine (e.g., Azure Emotion API) recognizes and analyzes the user's (parent's) emotions. For example, based on the emotional data collected when the parent uses a dedicated app, it analyzes whether the parent is "satisfied" or "anxious." Based on this analysis, the tone and format of the summary text are adjusted.

[1535] Input: Parent emotion data, summary sentence

[1536] Output: A summary with adjusted tone and format

[1537] Step 7:

[1538] Summary notification

[1539] The server sends the generated summary to the user's (parent's) smartphone via a dedicated app. For example, using Firebase Cloud Messaging (FCM), a summary such as "My child played with friends" is sent to the parent's smartphone.

[1540] Input: A summary with adjusted tone and format

[1541] Output: Summary sent to parent's smartphone

[1542] Step 8:

[1543] User inquiries for more information

[1544] The user (parent) can use a dedicated app to enter questions about the summary text, such as "What did you play with your friends?"

[1545] Input: Parent Question

[1546] Output: Questions accepted by the app

[1547] Step 9:

[1548] Reanalysis of detailed information

[1549] The server accepts questions from users and re-analyzes the related text data. A natural language processing algorithm is used for the re-analysis, and an emotion engine is also activated to generate detailed information based on the parent's emotional state. For example, the server might generate detailed information such as "I played soccer with my friends."

[1550] Input: Parent questions, initial text data

[1551] Output: Detailed information

[1552] Step 10:

[1553] Notification of detailed information

[1554] The server then sends the generated details to the parent's smartphone via a dedicated app. For example, the parent's smartphone may receive detailed information such as "My child played soccer with a friend."

[1555] Input: More information

[1556] Output: Detailed information sent to parent's smartphone

[1557] (Application example 2)

[1558] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1559] In virtual stores, there is a need for a system that can efficiently communicate between customers and sales staff (clerks) and respond quickly and appropriately to customer requests and questions. However, with current systems, it is difficult for sales staff to understand all customer conversations in real time and provide appropriate information each time. Furthermore, further technological improvements are needed to respond promptly to customer emotions and requests.

[1560] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data into text data in real time, means for periodically transmitting the text data to the server, means for analyzing the text data in the server and extracting important utterances, means for generating a summary based on the extracted utterances, means for notifying the user of the summary, means for recognizing the user's emotions and reflecting them in the summary, and means for use as a virtual customer service assistant that provides related information based on the user's questions. This enables customer service staff to efficiently respond to customers' questions and requests, thereby improving customer satisfaction.

[1561] "Means for converting voice data into text data in real time" refers to devices or technologies that instantly analyze voice and convert its content into text information.

[1562] "Means for periodically sending text data to a server" refers to a method or technique for transferring text data to a remote server at specific time intervals.

[1563] "Means for analyzing text data on a server and extracting important statements" refers to technology for processing text data on a server and identifying and extracting content that is considered particularly important.

[1564] "Means for generating a summary based on extracted statements" refers to a technology for creating a document that summarizes important statements.

[1565] "Means for notifying the user of the summary" refers to technology for notifying the user of the generated summary via a terminal or notification system.

[1566] "Means for recognizing the user's emotions and reflecting them in the summary" refers to technology for analyzing the user's emotional state and reflecting the results in the content and expression of the summary.

[1567] "Means for applying as a virtual customer service assistant that provides relevant information based on a user's question" refers to a virtual support system for processing questions from a user and providing appropriate information.

[1568] The present invention relates to a system for improving the efficiency of customer service in a virtual store. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS ... will be described in detail below with reference to the accompanying drawings.

[1569] First, a terminal equipped with a microphone is required to capture the conversation between the customer and the store clerk in order to convert the voice data into text data in real time. This terminal then uses voice recognition software to convert the voice data into text data, which is then sent to a server at specific time intervals.

[1570] The server receives the text data and analyzes it. This analysis is performed using a natural language processing (NLP) algorithm. The server extracts important statements from the analyzed data and generates a summary based on those statements. The generated summary is then notified to the user, but before that, the server uses an emotion recognition engine to analyze the user's emotions and reflects that information in the summary.

[1571] When a customer asks a store clerk a question, the system accepts the question, analyzes the text data again, and provides relevant information. This information is displayed on smart glasses or a head-mounted display, allowing the store clerk to respond immediately to the customer's question and provide appropriate information.

[1572] The hardware used includes smart glasses, a head-mounted display, and a microphone. The software uses speech recognition libraries (e.g., speech_recognition), natural language processing models (e.g., transformers), and data transmission libraries (e.g., requests). It also uses generative AI models for sentiment analysis.

[1573] Examples:

[1574] The device detects the customer's question, "Which product do you recommend most?" and converts the conversation into text data. The server analyzes the text data, extracts important information, generates a summary, and then displays an appropriate answer on the smart glasses, taking into account the emotional state of the salesperson. The following prompt sentence is used to generate this summary:

[1575] Example prompt sentence:

[1576] "Use sentiment analysis to generate the most appropriate answer for the customer's question, 'Which product do you recommend most?' If the customer's emotional state is positive, provide more information about the product; if it's negative, suggest an alternative."

[1577] As a result, the present invention can significantly improve the efficiency of customer service in a virtual store, thereby achieving increased customer satisfaction.

[1578] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1579] Step 1:

[1580] The terminal uses a microphone to capture the conversation between the customer and the store clerk, and converts the voice data into text data in real time using voice recognition software (e.g., the speech_recognition library). The input is voice data, and the output is text data.

[1581] Step 2:

[1582] The terminal sends the converted text data to the server at regular intervals. The input is text data, and the output is data transmission to the server. This transmission uses a data transmission library (e.g., requests).

[1583] Step 3:

[1584] The server stores the received text data and uses natural language processing (NLP) algorithms to analyze the data and extract key statements. The input is the text data, and the output is the analysis results containing the key statements. Specifically, the server processes the text data using NLP models (e.g., transformers).

[1585] Step 4:

[1586] The server generates a summary based on the extracted important statements. The input is the important statements, and the output is the summary. A text summarization algorithm is used to generate the summary.

[1587] Step 5:

[1588] The server uses an emotion recognition engine to analyze the user's emotions and reflects that information in the summary. The input is the summary and the user's emotion data, and the output is a summary based on the emotion. A generative AI model is used for emotion analysis.

[1589] Step 6:

[1590] The server sends the generated summary to the user's device. The input is a summary corresponding to the emotion, and the output is a notification to the user's device. This notification is done using a dedicated application.

[1591] Step 7:

[1592] The user (store clerk) checks the summary displayed on the smart glasses or head-mounted display and responds to the customer's questions and requests based on it. The input is the notified summary, and the output is a response to the customer.

[1593] Step 8:

[1594] When a user enters a question, the server re-analyzes the question and generates detailed information based on the related text data. The input is the question, and the output is detailed information. This re-analysis is performed using natural language processing algorithms.

[1595] Step 9:

[1596] The generated detailed information is again notified to the user, and the user (store clerk) provides a specific answer to the customer. The input is detailed information, and the output is a detailed answer to the customer.

[1597] Example prompt sentence:

[1598] "Use sentiment analysis to generate the most appropriate answer for the customer's question, 'Which product do you recommend most?' If the customer's emotional state is positive, provide more information about the product; if it's negative, suggest an alternative."

[1599] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1600] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1601] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1602] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1603] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1604] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1605] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1606] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1607] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1608] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1609] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1610] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1611] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1612] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1613] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1614] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1615] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1616] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1617] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1618] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1619] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1620] The following is further disclosed regarding the above embodiment.

[1621] (Claim 1)

[1622] [Means for converting voice data into text data in real time;

[1623] [Means for periodically sending text data to a server;

[1624] [Means for analyzing text data on a server and extracting important statements;

[1625] [Means for generating a summary based on extracted utterances;

[1626] [Means for notifying the user of the summary;

[1627] A system including:

[1628] (Claim 2)

[1629] [The system according to claim 1, which accepts questions from users.

[1630] (Claim 3)

[1631] [The system of claim 1 reanalyzes text data based on the question and generates detailed information.

[1632] "Example 1"

[1633] (Claim 1)

[1634] [Means for converting voice data into text data in real time;

[1635] [Means for periodically sending text data to a server;

[1636] [Means for analyzing text data on a server and extracting important statements;

[1637] [Means for generating a summary based on extracted utterances;

[1638] [Means for notifying the user of the summary;

[1639] [Means for sending a question input by a user to a server;

[1640] [Means for reanalyzing text data and generating detailed information based on questions;

[1641] [Means of informing the user of further information;

[1642] A system including:

[1643] (Claim 2)

[1644] [The system of claim 1, including a user device for receiving a notification based on the abstract sentence.

[1645] (Claim 3)

[1646] [The system of claim 1, wherein the analysis and reanalysis process uses natural language processing algorithms.

[1647] "Application Example 1"

[1648] (Claim 1)

[1649] [Means for converting voice data into text data in real time;

[1650] [Means for periodically sending text data to a server;

[1651] [Means for analyzing text data on a server and extracting important statements;

[1652] [Means for generating a summary based on extracted utterances;

[1653] [Means for notifying the user of the summary;

[1654] [Means for notifying important comments to a user's display device based on specific conditions;

[1655] A system including:

[1656] (Claim 2)

[1657] [The system according to claim 1, which accepts questions from users.

[1658] (Claim 3)

[1659] [The system of claim 1 reanalyzes text data based on the question and generates detailed information.

[1660] "Example 2: Combining Emotion Engines"

[1661] (Claim 1)

[1662] [Means for converting voice data into text data in real time;

[1663] [means for periodically transmitting text data to a central processing unit;

[1664] [Means for analyzing text data and extracting important statements in a central processing unit;

[1665] [Means for generating a summary based on extracted utterances;

[1666] [Means for analyzing the parent's emotions through an emotion recognition module and adjusting the tone of the summary sentence;

[1667] [Means for notifying the user of the summary;

[1668] A system including:

[1669] (Claim 2)

[1670] [The system according to claim 1, which accepts questions from users.

[1671] (Claim 3)

[1672] [The system of claim 1 reanalyzes text data based on the question and generates detailed information.

[1673] "Application example 2 when combining emotion engines"

[1674] (Claim 1)

[1675] [Means for converting voice data into text data in real time;

[1676] [Means for periodically sending text data to a server;

[1677] [Means for analyzing text data on a server and extracting important statements;

[1678] [Means for generating a summary based on extracted utterances;

[1679] [Means for notifying the user of the summary;

[1680] [Means for recognizing user emotions and reflecting them in summaries;

[1681] [Means for application as a virtual customer service assistant that provides relevant information based on user questions;

[1682] A system including:

[1683] (Claim 2)

[1684] [The system according to claim 1, which accepts questions from users.

[1685] (Claim 3)

[1686] [The system of claim 1 reanalyzes text data based on the question and generates detailed information. [Explanation of symbols]

[1687] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for converting voice data into text data in real time; means for periodically transmitting text data to a server; A means for analyzing the text data in the server and extracting important statements; A means for generating a summary based on the extracted utterances; a means for notifying a user of the summary; A system including:

2. The system of claim 1, wherein the system accepts questions from a user.

3. The system according to claim 1, wherein the system reanalyzes the text data based on the query to generate detailed information.

Images