System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring voice uttered by a parent; means for converting the acquired voice into digital data; means for transmitting the digital data to a server on a cloud; means for analyzing the digital data stored in the server on the cloud and generating advice related to childcare consultation; means for returning the generated advice to a playback terminal as voice and text; and means for playing back the voice and displaying the text on the playback terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there has been an increase in cases where parents are raising their children without being able to consult with those around them, and the worries and stress associated with raising children have become a serious problem. This has led to concerns that the parent-child bond may be weakened and anxiety about raising children may increase. It is also difficult for parents to find time to relax, which ultimately leads to issues such as increased mental and physical strain on parents. [Means for solving the problem]

[0005] The present invention provides a system that strengthens communication between parents and children and reduces parental anxiety by acquiring voices uttered by parents, analyzing the voices, providing advice related to child-rearing consultations, and playing fairy tales and lullabies recorded in the parents' voices to the children. Specifically, the system includes the following means:

[0006] 1. A means of acquiring parental vocalizations;

[0007] 2. Means for converting the captured audio into digital data;

[0008] 3. A means of transmitting digital data to a server on the cloud;

[0009] 4. A means for analyzing digital data stored on a cloud server and generating advice related to child-rearing consultations;

[0010] 5. Means for returning the generated advice to the playback terminal as voice and text;

[0011] 6. Means for playing the audio and displaying the text on a playback device;

[0012] 7. A means for recording the parent's vocalizations;

[0013] 8. A means of storing the recorded audio as digital data on a cloud server;

[0014] 9. A means for searching digital data stored on a cloud server and transmitting it to a playback device at a specified timing;

[0015] 10. Means for playing the audio on a playback device;

[0016] 11. A means for selecting appropriate fairy tales and lullabies based on the age of the child;

[0017] 12. A means for transmitting the selected fairy tales and lullabies from a server on the cloud to a playback device;

[0018] 13. A means of playing fairy tales and lullabies on a playback device.

[0019] By combining these methods, we can create a system that provides an environment where parents can seek child-rearing advice 24 hours a day, 365 days a year, strengthens communication with their children through fairy tale readings and lullabies in their own voices, and reduces the burden on parents by providing relaxing music.

[0020] A "parent" is an adult who has the responsibility for raising a child and is involved in the child's development and well-being on a daily basis.

[0021] "Speech" refers to natural language sounds made by a parent, including parenting instructions and questions.

[0022] The "means for capturing" is a combination of hardware and software that physically captures the parent's voice and stores it as digital data.

[0023] "Digital data" refers to data that has been converted from audio into a digital format and is used for communication and analysis.

[0024] A "cloud server" is a server accessible via the Internet that stores digital data, analyzes it, and returns the results.

[0025] The "means of analyzing and generating advice" refers to the process in which a cloud-based server uses artificial intelligence technology to analyze digital data and generate appropriate answers and advice to parents' child-rearing questions.

[0026] A "playback device" is a device for playing audio and text, such as a smart speaker or a display device.

[0027] The "means for returning the advice to the playback terminal as voice and text" is a process in which the server on the cloud sends the generated advice to the playback terminal as voice data and text data.

[0028] "Recording means" refers to a combination of equipment and software that records the voice of a parent, such as a fairy tale or lullaby, and stores it as digital data.

[0029] The "specified timing" refers to the time conditions for the server on the cloud to transmit audio data to the playback terminal, and refers to a time requested by the user or scheduled.

[0030] "Means for selecting appropriate fairy tales and lullabies based on the age of the child" refers to a process in which a cloud-based server uses artificial intelligence technology to select appropriate fairy tales and lullabies based on the age of the child.

[0031] "Means for transmitting selected fairy tales and lullabies from a server on the cloud to a playback device" refers to the process of transmitting audio data of the selected fairy tales and lullabies from a server on the cloud to a playback device via a network.

[0032] "Means for providing relaxing music" refers to a system that sends music for parents to relax from a cloud server to a playback device and plays it back. [Brief explanation of the drawings]

[0033] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0034] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0035] First, the terms used in the following description will be explained.

[0036] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0037] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0038] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0039] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0040] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0041] [First embodiment]

[0042] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0043] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0044] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0045] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0046] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0047] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0048] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0049] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0050] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0051] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0052] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0053] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0054] The present invention is a system that inputs the voice of a parent to provide advice on child-rearing, analyzes the acquired voice, and provides appropriate advice. It also has a function that stores the voice data recorded by the parent on the cloud and plays it back to the child as a fairy tale or lullaby at a specified time. Specific embodiments for implementing this system are described below.

[0055] The system is broadly composed of three main elements: terminals, servers, and users.

[0056] Terminal

[0057] The device is a smart speaker or display device that captures the parent's voice. The device has the following features:

[0058] Voice input function: Parental voice is captured through a microphone.

[0059] Recording function: Saves captured audio as digital data.

[0060] Audio playback function: Plays audio data received from a server on the cloud.

[0061] Text display function: Displays text data received from a server on the cloud.

[0062] server

[0063] A server is a computing unit that runs on the cloud and has the following main functions:

[0064] Voice analysis function: Converts voice data sent from the device into text data.

[0065] Advice generation function: Generates appropriate parenting advice from analyzed voice data.

[0066] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[0067] Text generation function: Generates text data of advice and sends it to the terminal.

[0068] Fairy tale and lullaby selection function: Selects the most appropriate fairy tale or lullaby based on the child's age and sends it to the device.

[0069] User

[0070] Users, primarily parents, perform the following actions:

[0071] Voice input: Record parenting advice, fairy tales, and lullabies.

[0072] Audio playback request: Request playback of a fairy tale or lullaby that you have recorded yourself.

[0073] Receiving advice: Receive advice provided by the server in voice and text format.

[0074] Specific examples

[0075] Specific examples of child-rearing consultation

[0076] 1. The user speaks to the device and asks for advice on child-rearing. For example, "My child has been crying a lot lately. What should I do?"

[0077] 2. The device captures the audio and sends the digital data to the server.

[0078] 3. The server converts the speech into text and performs analysis.

[0079] 4. The server generates appropriate advice based on the analysis results and sends the advice to the device as text and audio data.

[0080] 5. The device plays the transmitted audio data and displays the text data on the display.

[0081] 6. Users receive advice and apply it to solve their parenting problems.

[0082] Examples of fairy tales and lullabies

[0083] 1. The user sits down at the device and starts recording a fairy tale. For example, say, "I'm going to read you the fairy tale of Momotaro."

[0084] 2. The device enters recording mode and records the user's voice.

[0085] 3. After the device finishes recording, it sends the digital data to the server and stores it.

[0086] 4. The user later requests, "Play the fairy tale of Momotaro."

[0087] 5. The server sends the saved audio data to the playback device.

[0088] 6. The device plays the recorded story and reads it to the child in the user's voice.

[0089] This system allows parents to receive advice 24 hours a day, 365 days a year to resolve their child-rearing concerns, and also deepens the bond with their children by having them listen to fairy tales and lullabies recorded in their own voices.

[0090] The processing flow will be explained below.

[0091] Childcare consultation process flow

[0092] Step 1:

[0093] The user can ask for parenting advice by voice, for example, "My child has been crying at night recently. What should I do?"

[0094] Step 2:

[0095] The device records the user's voice. It captures the input voice through a microphone and stores it as digital data.

[0096] Step 3:

[0097] The device sends the voice data to a server on the cloud, and the voice data is uploaded to the server via network communication.

[0098] Step 4:

[0099] The server converts the voice data into text data using a voice recognition API.

[0100] Step 5:

[0101] The server analyzes the text data and uses natural language processing (NLP) to understand the keywords and context within the text and analyze the content of the parenting consultation.

[0102] Step 6:

[0103] The server generates advice based on the analysis results. Using an AI model, it generates appropriate advice in text format. For example, "We recommend creating a relaxing environment and establishing a regular routine before bed."

[0104] Step 7:

[0105] The server converts the text data into audio data using a text-to-speech API.

[0106] Step 8:

[0107] The server transmits the generated voice data and text data to the terminal via network communication.

[0108] Step 9:

[0109] The device plays the audio and displays the text: it plays the audio data through the speaker and displays the text data on the display.

[0110] Processing flow for recording and playing back fairy tales in a parent's voice

[0111] Step 1:

[0112] The user gives a voice command to the terminal to "start recording a fairy tale." For example, the user says, "I will record the fairy tale of Momotaro."

[0113] Step 2:

[0114] The device switches to recording mode, starts the recording function, and prepares to record the user's voice.

[0115] Step 3:

[0116] The user reads a fairy tale aloud and the device records the audio, for example, "Once upon a time, there lived an old man and an old woman."

[0117] Step 4:

[0118] The terminal ends the recording. The recording stops due to a user instruction or a timeout.

[0119] Step 5:

[0120] The device sends the recorded data to a server on the cloud, and the audio data is uploaded to the server via network communication.

[0121] Step 6:

[0122] The server saves the recorded data and performs the necessary processing. The audio data is stored in storage and the data format is adjusted.

[0123] Step 7:

[0124] The user gives a voice command to the terminal to "start playing a fairy tale." For example, the user says, "Please play the fairy tale of Momotaro."

[0125] Step 8:

[0126] The server sends the recorded voice data to the terminal, and the saved voice data is provided to the terminal via the network.

[0127] Step 9:

[0128] The device plays the audio data, and the recorded story is played through the speaker for the child to listen to.

[0129] Relaxing music provision process flow

[0130] Step 1:

[0131] The user vocally requests relaxing music, for example, "Play some relaxing music."

[0132] Step 2:

[0133] The device records the audio and sends it to a server in the cloud. The audio data is captured and uploaded to the server via the network.

[0134] Step 3:

[0135] The server converts the voice into text data, and then uses a voice recognition API to convert the voice data into text data.

[0136] Step 4:

[0137] The server searches for appropriate relaxing music, analyzes the text data, and selects appropriate songs from the relaxing music library.

[0138] Step 5:

[0139] The server sends the selected music data to the device, and the music file is uploaded to the device via the network.

[0140] Step 6:

[0141] The device will play music, and the speaker will play the selected relaxing music.

[0142] These processing steps allow users to easily receive parenting advice, play recorded fairy tales and lullabies in their own voice, or request relaxing music to help reduce stress.

[0143] Example 1

[0144] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0145] Modern parents face many challenges and worries regarding child-rearing, but there is a lack of means to quickly and accurately resolve these issues. While it is useful for busy parents to record fairy tales and lullabies in their own voices for their children, the recording and playback methods are inconvenient. Furthermore, there is a lack of systems that can select and effectively play appropriate fairy tales and lullabies based on the child's age.

[0146] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0147] In this invention, the server includes means for acquiring voice uttered by a parent, means for converting the acquired voice into digital data, means for transmitting the digital data to a cloud-based computer system, means for analyzing the digital data stored in the cloud-based computer system and generating advice related to child-rearing consultations, means for returning the generated advice as voice and text to a playback terminal, means for playing the voice and displaying the text on the playback terminal, and means for the user to request audio playback from the cloud-based computer system. This allows parents to easily receive child-rearing consultations at any time and effectively use fairy tales and lullabies recorded in their own voices.

[0148] The "means for acquiring the voice uttered by the parent" refers to a device or apparatus that collects the voice uttered by the parent using a voice input device such as a microphone.

[0149] "Means for converting captured audio into digital data" refers to devices or software that convert analog audio signals into digital format.

[0150] "Means for transmitting digital data to a computer system on the cloud" refers to a communication method for transferring digital data from a terminal to a cloud server via the Internet.

[0151] "Means for analyzing digital data stored in a cloud-based computer system and generating advice related to childcare consultations" refers to a function that uses artificial intelligence and analytical algorithms to analyze digital data stored on a cloud server and generate advice related to childcare.

[0152] "Means for returning the generated advice to the playback terminal as audio and text" refers to a communication method for transmitting the advice generated by the cloud server to the playback device in the form of an audio file and a text file.

[0153] "Means for playing back audio and displaying text on a playback terminal" refers to the function of playing back audio data received in a playback device through a speaker and displaying text data on a display.

[0154] "Means for a user to make a request to a cloud computer system to play audio" refers to a communication means for a user to send an instruction to a cloud computer system to request audio playback.

[0155] "Means for recording sounds uttered by a parent" refers to an apparatus or method for recording sounds uttered by a parent using a recording device.

[0156] "Means for storing recorded audio as digital data in a computer system on the cloud" refers to a function for storing recorded audio data in digital format on a cloud server.

[0157] "Means for searching for digital data stored in a computer system on the cloud and transmitting it to a playback terminal at a specified time" refers to a function for searching for audio data stored on a cloud server and transferring it to a playback device at a time specified by the user.

[0158] "Means for selecting appropriate stories and lullabies based on the age of the child" refers to filtering and selection algorithms used to select content appropriate for the child's age.

[0159] "Means for transmitting selected stories and lullabies from a cloud-based computer system to a playback device" refers to a communication method for transferring selected audio data from a cloud server to a playback device.

[0160] "Means for playing back a story or lullaby on a playback terminal" refers to a function for playing back audio data of a story or lullaby received by a playback device on a speaker.

[0161] This system transmits the voices of parents to a cloud-based computer system, analyzes the voices, generates advice related to childcare consultations, and plays fairy tales and lullabies recorded by the user at appropriate times. This system is mainly composed of three main elements: a terminal, a server, and a user.

[0162] Hardware and software used

[0163] Device: The device includes a voice input function for capturing the voice spoken by the parent, a recording function, a voice playback function, and a text display function. Specific examples of this device include a voice input device and a device with a display. Specifically, this applies to smart speakers and smart devices with a display.

[0164] Server: The server is located in the cloud and has functions for voice analysis, advice generation, data storage, text generation, and selection of fairy tales and lullabies. Specific examples include platforms that provide cloud computing services (e.g., AWS (registered trademark) and Google (registered trademark) Cloud Platform). Amazon Transcribe and Google Cloud Speech-to-Text are used as voice analysis software, and generative AI models (e.g., GPT-3 (registered trademark) and ChatGPT (registered trademark)) are used for text generation.

[0165] Details of data processing and calculation

[0166] 1. Audio capture and conversion:

[0167] Device: The device captures the user's voice and converts the analog voice signal into digital data. Specifically, smart speakers and smart devices with displays use voice input functionality to capture the user's spoken voice through a microphone and convert the voice into a digital format.

[0168] 2. Data transmission and analysis:

[0169] Terminal: The converted digital data is sent to a computer system in the cloud, using communication means via the Internet.

[0170] Server: The server converts the received digital data into text data using speech analysis software, for example, Amazon Transcribe or Google Cloud Speech-to-Text.

[0171] 3. Advice Generation:

[0172] Server: Based on the analyzed text data, a generative AI model (e.g., GPT-3 or ChatGPT) is used to generate appropriate advice related to childcare consultations.

[0173] 4. Data transmission and display:

[0174] Server: Sends the generated advice to the device as voice and text data.

[0175] Terminal: The terminal plays the received audio data and displays the text data on the display.

[0176] 5. Record and play fairy tales and lullabies:

[0177] Device: The user starts recording a fairy tale or lullaby, and the device goes into recording mode, collects the audio, and temporarily stores it as digital data.

[0178] Server: After recording is complete, the device sends the data to a cloud-based computer system, where it is securely stored. When a user requests playback, the server sends the stored data to the device, which then plays the audio data.

[0179] Specific use cases

[0180] Specific examples of child-rearing consultation

[0181] 1. The user speaks into the device and says, "My child has been crying at night lately. What should I do?"

[0182] 2. The device captures the audio and sends the digital data to the server.

[0183] 3. The server converts the speech into text and performs analysis.

[0184] 4. Based on the analysis results, the server generates advice such as, "To help your crying child relax at night, it would be a good idea to create a quiet environment and stick to a consistent bedtime."

[0185] 5. The device plays the transmitted audio data and displays the text data on the display.

[0186] Examples of fairy tales and lullabies

[0187] 1. The user approaches the device and says, "I'm going to read the fairy tale of Momotaro," and begins recording.

[0188] 2. The device enters recording mode and records the user's voice.

[0189] 3. After the device finishes recording, it sends the digital data to the server, where it is stored.

[0190] 4. The user later requests, "Play the fairy tale of Momotaro."

[0191] 5. The server sends the saved audio data to the device.

[0192] 6. The device plays the recorded story and reads it to the child in the user's voice.

[0193] Examples of prompt statements

[0194] "Recently, my child has been crying at night and it's bothering me. What should I do?"

[0195] "I want to record the Momotaro fairy tale so I can read it to you."

[0196] "Please give me some advice on how to get my child to sleep early."

[0197] This system allows parents to easily consult with their children at any time, and also makes it possible to effectively use fairy tales and lullabies recorded in their own voices.

[0198] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0199] Step 1:

[0200] The user's voice is input into the device. For example, the user might say, "My child has been crying at night recently and it's bothering me. What should I do?"

[0201] Input: User utterance

[0202] Output: Audio data captured by a microphone

[0203] Step 2:

[0204] The device converts the user's voice into digital data. The smart speaker uses its voice input function to convert analog voice into digital data.

[0205] Input: Captured audio data

[0206] Output: Digitized audio data

[0207] Step 3:

[0208] The device sends digital audio data to a server in the cloud, where it is sent as encrypted data over the internet.

[0209] Input: Digitized audio data

[0210] Output: Digital data sent to the server

[0211] Step 4:

[0212] The server converts the audio data into text using speech analysis software, such as Amazon Transcribe or Google Cloud Speech-to-Text.

[0213] Input: Digital data sent to the server

[0214] Output: Text data

[0215] Step 5:

[0216] The server analyzes the text data using a generative AI model and generates advice for the user's childcare consultation. The analysis uses a database of childcare-related information and a generative AI model (e.g., GPT-3 or ChatGPT).

[0217] Input: Text data

[0218] Output: Advice text and audio data

[0219] Step 6:

[0220] The server transmits the generated advice to the terminal via the Internet as voice data and text data.

[0221] Input: Advice text and audio data

[0222] Output: Advice data sent to the terminal

[0223] Step 7:

[0224] The device plays the advice audio data and displays the text data, while the smart speaker or device with a display plays the audio and displays the text on the display.

[0225] Input: Advice data sent to the terminal

[0226] Output: Played audio and displayed text

[0227] Step 8:

[0228] The user gives the command to start recording. For example, they say, "I'm going to read you the fairy tale of Momotaro."

[0229] Input: User utterance

[0230] Output: Start recording command

[0231] Step 9:

[0232] The device enters recording mode, recording the user's speech and temporarily saving it as digital data.

[0233] Input: Start recording

[0234] Output: Digitized recording data

[0235] Step 10:

[0236] After the device finishes recording, it sends the recorded data to a cloud server and stores it there. The server then securely stores the received data.

[0237] Input: Digitized recording data

[0238] Output: Data stored on a cloud server

[0239] Step 11:

[0240] The user makes a playback request, for example, "Play the fairy tale of Momotaro."

[0241] Input: User's playback request

[0242] Output: Playback request data

[0243] Step 12:

[0244] The server retrieves the stored voice data and sends it to the device, which then transfers the data over the Internet.

[0245] Input: Playback request data

[0246] Output: Audio data sent to the device

[0247] Step 13:

[0248] The device plays the audio data, and the recorded story is played on the speaker in the user's voice for the child to hear.

[0249] Input: Audio data sent to the device

[0250] Output: Played fairy tale audio

[0251] This will allow parents to easily record and play back childcare advice and fairy tales.

[0252] (Application example 1)

[0253] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0254] Conventional parenting support systems focus solely on analyzing parents' voice input and providing appropriate advice. However, there are no systems that can respond to security-related consultations or inquiries using similar voice recognition technology. There is a need for a system that can provide advice on both parenting and security in response to parents' spoken inquiries and manage them centrally. In addition, the generated advice must be provided in both voice and text format, allowing for flexible responses tailored to each user's usage situation.

[0255] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0256] In this invention, the server includes means for acquiring voice uttered by parents, means for converting the acquired voice into digital data, means for transmitting the digital data to a cloud server, means for analyzing the digital data stored in the cloud server and generating advice on child-rearing consultations and security, means for returning the generated advice to a playback terminal as voice and text, and means for playing the voice and displaying the text on the playback terminal, thereby making it possible to provide appropriate advice in a unified manner in response to questions from parents about child-rearing consultations and security.

[0257] The "means for acquiring parental vocalizations" is a device or system for collecting parental vocalizations in digital form.

[0258] The "means for converting captured audio into digital data" refers to a technique or device for converting audio signals into digital form.

[0259] The "means for transmitting digital data to a server on the cloud" refers to a technology or device for transmitting the converted digital data to a cloud server via the Internet.

[0260] "Means for analyzing digital data stored on a server on the cloud and generating advice on child-rearing consultations and security" refers to a technology or system that analyzes digital data stored on a cloud server and generates advice related to child-rearing and security based on the results.

[0261] The "means for returning the generated advice to the playback terminal as voice and text" refers to a technique and device for converting the generated advice into voice and text format and transmitting it to the playback terminal.

[0262] "Means for playing audio and displaying text at a playback terminal" refers to a device or system for playing received audio and displaying text.

[0263] A "means for recording parental vocalizations" is a device or technology for recording parental vocalizations.

[0264] "Means for storing recorded audio as digital data on a server on the cloud" refers to a method or system for storing recorded audio in a digital format on a cloud server.

[0265] "Means for searching for digital data stored on a server on the cloud and transmitting it to a playback terminal at a specified time" refers to a function or device for searching for data stored on a cloud server under specific conditions or at specific times and transmitting it to a playback terminal.

[0266] The "means for selecting appropriate fairy tales and lullabies based on the age of the child" refers to an algorithm or device that selects the most appropriate fairy tales and lullabies based on the age information of the child.

[0267] "Means for transmitting selected fairy tales and lullabies from a server on the cloud to a playback terminal" refers to means for transmitting data of selected fairy tales and lullabies from a cloud server to a playback terminal.

[0268] "Means for analyzing captured audio and generating security advice" refers to a system or software capable of analyzing audio data and generating appropriate security-related advice.

[0269] The "means for returning the generated security advice to the playback terminal as voice and text" refers to a technique or device that converts the generated security advice into voice and text format and sends it to the playback terminal.

[0270] To implement this invention, a system is required that captures the voices spoken by parents, transmits them to a cloud server, and generates appropriate advice on child-rearing and security. The system is broadly composed of three main elements: a terminal, a server, and a user.

[0271] Terminal

[0272] The device is a smart speaker or display device that captures the parent's voice. The device has the following features:

[0273] Voice input function: Parental voice is captured through a microphone.

[0274] Recording function: Saves captured audio as digital data.

[0275] Audio playback function: Plays audio data received from a server on the cloud.

[0276] Text display function: Displays text data received from a server on the cloud.

[0277] server

[0278] A server is a computing unit that runs on the cloud and has the following main functions:

[0279] Speech analysis function: Converts voice data sent from the device into text data. Uses the speech_recognition library.

[0280] Advice generation function: Generates appropriate parenting and security advice from the analyzed voice data. It uses a generative AI model with the transformers library.

[0281] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[0282] Text generation function: Generates text data for advice and sends it to the terminal. This uses the tflow library.

[0283] Fairy tale and lullaby selection function: Selects the most appropriate fairy tale or lullaby based on the child's age and sends it to the device.

[0284] User

[0285] Users, primarily parents, perform the following actions:

[0286] Voice input: Record parenting advice, security advice, fairy tales, lullabies, etc. For example, ask, "There's been a suspicious person hanging around recently. What should I do?"

[0287] Audio playback request: Request playback of a fairy tale or lullaby that you have recorded yourself.

[0288] Receive advice: Receive parenting and security advice provided by the server via voice and text.

[0289] Specific examples

[0290] Specific examples of childcare and security consultations

[0291] 1. A user speaks to a device and asks a question about parenting or security. For example, "My child has been crying a lot lately. What should I do?"

[0292] 2. The device captures the audio and sends the digital data to the server.

[0293] 3. The server converts the speech to text and performs the analysis. The speech_recognition library is used here.

[0294] 4. The server generates appropriate advice based on the analysis results and sends the advice to the device as text and audio data. This is where a generative AI model using the transformers library comes into play.

[0295] 5. The device plays the transmitted audio data and displays the text data on the display.

[0296] 6. The user receives the advice and applies it to solve the problem.

[0297] Prompt Sentence Examples

[0298] An example of a prompt to be input to the generative AI model is as follows:

[0299] "Generate appropriate security-related advice based on spoken questions. For example, what should I do if my neighbors are bothering me with noise?"

[0300] In this way, parents can quickly receive appropriate advice on child-rearing and security-related questions, creating a multi-functional support system that meets the needs of both parties.

[0301] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0302] Step 1:

[0303] The user speaks to the device and makes a voice input about childcare or security. For example, "My child has been crying at night recently. What should I do?" This voice is captured as a digital signal through the microphone. This is the state in which the input voice data has been captured.

[0304] Step 2:

[0305] The device picks up the voice and sends the digital data to a server in the cloud. Software on the device converts the voice data into packets and sends them to the server via the Internet. The input voice data is sent to the server.

[0306] Step 3:

[0307] The server converts the speech to text and performs analysis. The speech_recognition library is used to convert the speech data to text data. The input is speech data and the output is text data. The speech recognition engine analyzes the speech signal and converts it to text.

[0308] Step 4:

[0309] The server generates appropriate advice using a generative AI model based on the analysis results. The server uses the transformers library to input the analyzed text and the prompt sentence into the generative AI model, and outputs the optimal advice. The input is text data, and the output is text data of the advice.

[0310] Step 5:

[0311] The server returns the generated advice to the playback device as audio and text data. The generated advice text data is converted into audio data using the gTTS (Google Text to Speech) library and sent to the device via the Internet. The input is the advice text data, and the output is audio data and text data.

[0312] Step 6:

[0313] The terminal plays the transmitted voice data and displays the text data on the display. The built-in speaker of the terminal plays the voice data and the display displays the text data. The input is voice data and text data, and the output is voice and displayed text. This allows the user to receive advice.

[0314] These are the processing steps of the system that realizes this application example. The generated advice is useful for solving problems related to child rearing and security, and users can receive specific instructions and suggestions.

[0315] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0316] The present invention is a system that inputs the user's voice, analyzes the acquired voice, and provides appropriate advice, and also combines it with an emotion engine that recognizes the user's emotions and generates a response according to their emotional state. This enables more personalized and appropriate advice, improving the effectiveness of comprehensive child-rearing support.

[0317] Key Components

[0318] The system is broadly divided into four main components: the terminal, the server, the emotion engine, and the user.

[0319] Terminal

[0320] The device is a smart speaker or a device with a display that receives user voice and has the following functions:

[0321] Voice input function: Parental voice is captured through a microphone.

[0322] Recording function: Saves captured audio as digital data.

[0323] Audio playback function: Plays audio data received from a server on the cloud.

[0324] Text display function: Displays text data received from a server on the cloud.

[0325] server

[0326] A server is a computing unit that runs on the cloud and has the following main functions:

[0327] Voice analysis function: Converts voice data sent from the device into text data.

[0328] Advice generation function: Generates appropriate parenting advice from analyzed voice data.

[0329] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[0330] Text generation function: Generates text data of advice and sends it to the terminal.

[0331] Emotion data processing function: Analyzes the emotional data sent from the emotion engine and adjusts the advice content.

[0332] Emotion Engine

[0333] The emotion engine is a software component that recognizes emotions from user voice data and analyzes that information. It has the following functions:

[0334] Emotion recognition function: Recognizes emotional states from voice data.

[0335] Emotion data transmission function: Recognized emotion data is transmitted to the server.

[0336] Emotional feedback function: Tailor appropriate advice based on emotions.

[0337] User

[0338] Users, primarily parents, perform the following actions:

[0339] Voice input: Record parenting advice, fairy tales, and lullabies.

[0340] Audio playback request: Request playback of a fairy tale or lullaby that you have recorded yourself.

[0341] Receiving advice: Receive advice provided by the server in voice and text format.

[0342] Specific examples

[0343] Specific examples of child-rearing consultation

[0344] Example 1: A system that recognizes the user's emotional state and provides advice

[0345] 1. The user speaks to the device and asks for advice on child-rearing. For example, "My child has been crying a lot lately. What should I do?"

[0346] 2. The device captures the voice and sends the digital data to the emotion engine.

[0347] 3. The emotion engine analyzes the voice data to recognize the user's emotional state, for example, recognizing that the user is feeling stressed.

[0348] 4. The emotion engine sends the emotion data to the server.

[0349] 5. The server analyzes the emotional and voice data and generates appropriate advice based on the child's emotional state. For example, it might say, "To reduce stress factors for your child, we recommend creating a relaxing bedtime routine."

[0350] 6. The server converts the generated advice into voice and text data and sends it to the device.

[0351] 7. The device plays the audio and displays the text.

[0352] 8. Users receive advice and apply it to solve their parenting problems.

[0353] Examples of fairy tales and lullabies

[0354] Example 2: Reading fairy tales using an emotion engine

[0355] 1. The user sits down at the device and starts recording a fairy tale. For example, say, "I'm going to read you the fairy tale of Momotaro."

[0356] 2. The device enters recording mode and records the user's voice.

[0357] 3. After the device finishes recording, it sends the digital data to the server and stores it.

[0358] 4. The emotion engine analyzes the recording data and attaches emotional information.

[0359] 5. The user later requests, "Play the fairy tale of Momotaro."

[0360] 6. The server selects the recorded voice data taking into consideration the attached emotional information and sends it to the terminal.

[0361] 7. The device plays the audio data and plays it to the child in the user's voice.

[0362] The system can recognize the user's emotions and provide appropriate advice based on those emotions, enabling more personalized and effective parenting support. It can also deepen the bond between parents and their children by playing fairy tales and lullabies recorded in their own voices.

[0363] The processing flow will be explained below.

[0364] A process flow for parenting consultations that combines an emotion engine

[0365] Step 1:

[0366] The user can ask for parenting advice by voice, for example, "My child has been crying at night recently. What should I do?"

[0367] Step 2:

[0368] The device records the user's voice. It captures the input voice through a microphone and stores it as digital data.

[0369] Step 3:

[0370] The device sends the recorded voice data to the emotion engine, which then uploads it to the emotion engine via network communication.

[0371] Step 4:

[0372] The emotion engine analyzes the voice data and recognizes the user's emotional state. For example, it may recognize that the user is feeling stressed.

[0373] Step 5:

[0374] The emotion engine sends the recognized emotion data to the server, where the voice data and emotion data are combined.

[0375] Step 6:

[0376] The server converts the voice data into text data using a voice recognition API.

[0377] Step 7:

[0378] The server analyzes the text data and emotional data, using natural language processing (NLP) to understand the relationship between keywords and context in the text and emotions, and analyzes the content of the parenting consultation.

[0379] Step 8:

[0380] The server generates advice based on the analysis results. Using an AI model that reflects emotional data, appropriate advice is generated in text format. For example, it might say, "To reduce the factors that cause your child stress, we recommend creating a relaxing bedtime routine."

[0381] Step 9:

[0382] The server converts the text data into audio data using a text-to-speech API.

[0383] Step 10:

[0384] The server transmits the generated voice data and text data to the terminal via network communication.

[0385] Step 11:

[0386] The device plays the audio and displays the text: it plays the audio data through the speaker and displays the text data on the display.

[0387] Processing flow for recording and playing fairy tales using the emotion engine

[0388] Step 1:

[0389] The user approaches the device and begins recording a fairy tale. For example, they say, "I'm going to read you the fairy tale of Momotaro."

[0390] Step 2:

[0391] The device switches to recording mode, starts the recording function, and prepares to record the user's voice.

[0392] Step 3:

[0393] The user reads a fairy tale aloud and the device records the audio, for example, "Once upon a time, there lived an old man and an old woman."

[0394] Step 4:

[0395] The terminal ends the recording. The recording stops due to a user instruction or a timeout.

[0396] Step 5:

[0397] The device sends the recorded voice data to the emotion engine, which then uploads it to the emotion engine via network communication.

[0398] Step 6:

[0399] The emotion engine analyzes the audio data and recognizes the emotional state of the recording.

[0400] Step 7:

[0401] The emotion engine sends the recognized emotion data to the server, where the voice data and emotion data are combined.

[0402] Step 8:

[0403] The server stores the recording data and stores it safely in the cloud.

[0404] Step 9:

[0405] The user later gives a voice command such as "Play the fairy tale of Momotaro."

[0406] Step 10:

[0407] The server searches for the stored voice data and emotion data and sends them to the device at the specified time.

[0408] Step 11:

[0409] The device plays back the recorded fairy tale audio data in the user's voice, and the audio is played back through the speaker so that the child can listen to it in the user's voice.

[0410] Processing flow for providing relaxing music using an emotion engine

[0411] Step 1:

[0412] The user vocally requests relaxing music, for example, "Play some relaxing music."

[0413] Step 2:

[0414] The device records the voice and sends it to the emotion engine, which then uploads it to the emotion engine via network communication.

[0415] Step 3:

[0416] The emotion engine analyzes the voice data and recognizes the user's emotional state.

[0417] Step 4:

[0418] The emotion engine sends the recognized emotion data to the server, where the voice data and emotion data are combined.

[0419] Step 5:

[0420] The server converts the speech into text and selects appropriate relaxing music taking into account emotional data.

[0421] Step 6:

[0422] The server sends the selected music data to the device, which then uploads the music data to the device via the network.

[0423] Step 7:

[0424] The device will play music, and the speaker will play the selected relaxing music.

[0425] These processing steps allow users to easily receive parenting advice, play fairy tales or lullabies recorded in their own voice, or request relaxing music to help reduce stress. The combination of an emotion engine provides more personalized assistance based on the user's emotional state.

[0426] Example 2

[0427] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0428] In today's world, many parents face the burdens of work and family life, requiring them to quickly receive appropriate parenting advice. However, conventional parenting support systems have struggled to accurately recognize parents' emotional states and provide personalized advice based on those states. In particular, when parents are stressed or tired, general advice that ignores their emotional state has limited effectiveness and does not contribute to caring for the parents' physical and mental health. A system that solves this problem and provides more effective and compassionate parenting support is needed.

[0429] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for transmitting digital data to a cloud server, a means for analyzing the digital data stored in the cloud server and recognizing the emotional state of the user, and a means for generating advice related to child-rearing consultation according to the emotional state. This makes it possible to provide personalized advice according to the emotional state of the parent.

[0430] "Voice spoken by the parent" refers to the voice spoken by the parent into the device for conversation, consultation, or recording.

[0431] "Digital data" means information such as voice or text that has been converted into a form that can be processed, stored, or transmitted electronically.

[0432] A "cloud server" is an online data processing and storage system provided via the Internet by computers located in remote locations.

[0433] An "emotion engine" is a software component that analyzes and recognizes a user's emotions and psychological state from their voice and text data.

[0434] A "voice analysis engine" is software or hardware that has the function of converting voice data into text data.

[0435] An "advice generation engine" is software or hardware that has the function of generating appropriate advice based on input data.

[0436] A "playback terminal" is a device for playing and displaying audio and text data sent from a server on the cloud.

[0437] "Users" are those who use the system to receive consultations, recordings, and advice on child-rearing, and are primarily parents.

[0438] "Childcare consultation" refers to the act of parents seeking professional advice regarding questions or problems they have regarding childcare.

[0439] "Emotional state" refers to the psychological state or emotion expressed by the user through voice or text, and includes, for example, stress, joy, anger, etc.

[0440] The "audio playback function" is a function that plays back audio data so that the user can hear it.

[0441] "Text display function" refers to the function of displaying text data on a display or other display device.

[0442] The present invention is a system that inputs the user's voice, analyzes the acquired voice, and provides appropriate advice, and also combines it with an emotion engine that recognizes the user's emotions and generates a response according to their emotional state. This system enables more personalized and appropriate advice, improving the effectiveness of comprehensive child-rearing support.

[0443] Key Components

[0444] The system is broadly divided into four main components: the terminal, the server, the emotion engine, and the user.

[0445] Terminal

[0446] The device is a smart speaker or a device with a display that receives user voice and has the following functions:

[0447] Voice input function: Captures the user's voice through a microphone.

[0448] Recording function: Saves captured audio as digital data.

[0449] Audio playback function: Plays audio data received from a server on the cloud.

[0450] Text display function: Displays text data received from a server on the cloud.

[0451] server

[0452] A server is a computing unit that runs on the cloud and has the following main functions:

[0453] Voice analysis function: Converts voice data sent from the device into text data.

[0454] Emotion recognition function: Analyze the user's emotional state through the emotion engine.

[0455] Advice generation function: Generates appropriate parenting advice based on analyzed voice data and emotional data.

[0456] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[0457] Text generation function: Generates text data of advice and sends it to the terminal.

[0458] Emotion Engine

[0459] The emotion engine is a software component that recognizes emotions from user voice data and analyzes that information. It has the following functions:

[0460] Emotion Recognition: Recognizes emotional states from speech data. It uses natural language processing techniques and applies a BERT-based sentiment analysis model.

[0461] Emotion data transmission function: Recognized emotion data is transmitted to the server.

[0462] User

[0463] Users, primarily parents, perform the following actions:

[0464] Voice input: Record parenting advice, fairy tales, and lullabies.

[0465] Audio playback request: Requests playback of a recorded fairy tale or lullaby.

[0466] Receiving advice: Receive advice provided by the server in voice and text format.

[0467] Specific examples

[0468] Specific examples of child-rearing consultation

[0469] 1. The user speaks to the device and asks for advice on child-rearing. For example, "My child has been crying a lot lately. What should I do?"

[0470] 2. The device captures the audio and sends the digital data to the server.

[0471] 3. The server uses a speech analysis engine to analyze the voice data and convert it into text data.

[0472] 4. The server recognizes the user's emotional state through an emotion engine and identifies that the user is feeling stressed.

[0473] 5. The server generates appropriate advice based on the emotion and voice data. For example, "To reduce stress factors for your child, we recommend creating a relaxing bedtime routine."

[0474] 6. The server converts the generated advice into voice and text data and sends it to the device.

[0475] 7. The device plays the audio and displays the text.

[0476] Examples of fairy tales and lullabies

[0477] 1. The user sits down at the device and starts recording a fairy tale. For example, say, "I'm going to read you the fairy tale of Momotaro."

[0478] 2. The device enters recording mode and records the user's voice.

[0479] 3. After the device finishes recording, it sends the digital data to the server and stores it.

[0480] 4. The emotion engine analyzes the recording data and attaches emotional information.

[0481] 5. The user later requests, "Play the fairy tale of Momotaro."

[0482] 6. The server selects the recorded voice data taking into consideration the attached emotional information and sends it to the terminal.

[0483] 7. The device plays the audio data and plays it to the child in the user's voice.

[0484] Example prompt sentence:

[0485] My child has been crying at night recently, what should I do?

[0486] "Play the fairy tale of Momotaro"

[0487] This system can provide appropriate advice based on the user's emotional state, enabling more personalized and effective parenting support. It can also deepen the bond between parents and their children by playing fairy tales and lullabies recorded in their own voices.

[0488] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0489] Step 1: Voice Input

[0490] The user speaks into the device to ask questions or ask for advice about child-rearing. For example, "My child has been crying at night recently. What should I do?"

[0491] The input is the user's voice data.

[0492] The device picks up audio using a built-in microphone, and performs noise reduction and echo cancellation to capture clear audio data.

[0493] The output is audio data converted into digital form.

[0494] Step 2: Sending audio data

[0495] The device converts the captured audio data into a digital format, specifically, converting the audio into 16-bit PCM format, compressing the data, and packetizing it.

[0496] The input is the converted digital audio data.

[0497] The device then sends the converted digital audio data to a cloud server using HTTPS to ensure data security.

[0498] The output is the audio data received by the server on the cloud.

[0499] Step 3: Analyzing the audio data

[0500] The server receives the voice data sent from the terminal.

[0501] The input is voice data sent to a server on the cloud.

[0502] The server uses a speech analysis engine to convert the audio data into text data, using the Google Cloud Speech-to-Text API.

[0503] The output is text data converted from the audio data.

[0504] Step 4: Recognize emotions

[0505] The server passes the converted text data to the emotion engine.

[0506] The input is text data.

[0507] The emotion engine uses natural language processing technology to analyze the user's emotional state from text data. It uses a BERT-based sentiment analysis model to recognize users' stress, joy, anger, etc.

[0508] The output is the recognized emotion data.

[0509] Step 5: Generating Advice

[0510] The server integrates the emotion data sent from the emotion engine with the text data.

[0511] The input is emotion data and text data.

[0512] The server uses an advice generation engine to generate appropriate advice using a GPT-based generative AI model, such as "To reduce stress factors for your child, we recommend creating a relaxing bedtime routine."

[0513] The output is the generated advice.

[0514] Step 6: Send and display advice data

[0515] The server converts the generated advice into both text and audio data.

[0516] The input is the generated advice.

[0517] The server transmits the converted data to the terminal.

[0518] The device analyzes the received data, plays back the voice data, and displays the text data on the display, using a voice playback engine and a text-to-speech (TTS) engine.

[0519] The output is a spoken advice and a displayed text advice.

[0520] Step 7: Take and apply advice

[0521] The user receives the advice provided by the device and applies it in real life.

[0522] The inputs are spoken advice and displayed text advice.

[0523] The hope is that by following the advice, users will find their parenting problems alleviated.

[0524] The output is the effect of the problem mitigation.

[0525] (Application example 2)

[0526] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0527] Conventional advertising delivery systems deliver advertisements uniformly without considering the user's emotional state, making it difficult to provide a personalized advertising experience that reflects the user's interests and preferences. Similarly, parenting support systems provide advice without understanding the parent's emotional state, resulting in an inability to provide effective support.

[0528] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0529] In this invention, the server includes means for recognizing an emotional state from acquired voice data and facial expression data, means for adjusting the content of advice based on the recognized emotional state, and means for selecting and displaying advertisements in accordance with the emotional state recognized from the acquired voice data and facial expression data, thereby making it possible to analyze the user's emotional state in real time and provide optimal advertisements and advice.

[0530] "Voice data" is a digital representation of a user's voice.

[0531] "Facial expression data" refers to facial expressions of a user captured using a camera and converted into an analyzable format.

[0532] The "emotional state" refers to the user's emotions analyzed based on voice data and facial expression data. Specifically, emotions include joy, sadness, anger, fear, surprise, and excitement.

[0533] "Means for recognizing" includes software and hardware configurations for analyzing voice data and facial expression data to identify the emotional state of the user.

[0534] The "means for adjusting the content of advice" includes algorithms and programs for generating or modifying appropriate advice for a user based on the perceived emotional state.

[0535] "Means for selecting advertisements" includes software and hardware configurations for selecting the most appropriate advertisements for a user based on the perceived emotional state and other relevant information.

[0536] The term "reproducing means" includes a device and software configuration for reproducing audio or video data received from a server on a reproducing terminal.

[0537] "Storage means" includes software and hardware configurations for safely storing acquired voice data and facial expression data in digital format on a server on the cloud.

[0538] This invention is a system that acquires and analyzes a user's voice and facial expression data to provide appropriate advertisements and advice. The system is mainly composed of four main elements: a terminal, a server, an emotion recognition engine, and the user.

[0539] Terminal

[0540] The terminal is a mobile device such as a smartphone or smart glasses that has the following capabilities:

[0541] Voice and facial expression input function: Uses a microphone and camera to capture the user's voice and facial expression data.

[0542] Recording function: Saves captured audio as digital data.

[0543] Audio playback and display function: Plays back and displays data sent from the server on the cloud.

[0544] server

[0545] A server is a computing unit that runs on the cloud and has the following main functions:

[0546] Voice and facial expression analysis function: Analyzes voice and facial expression data sent from the device and converts it into text data.

[0547] Advertisement and advice generation function: Recognizes the user's emotional state from the analyzed data and generates appropriate advertisements and advice.

[0548] Data storage function: Recorded voice data and facial expression data are safely stored.

[0549] Data transmission function: Sends generated advertisements and advice to the device.

[0550] Emotion Recognition Engine

[0551] The emotion recognition engine is a software component that recognizes emotions from the user's voice data and facial expression data and provides that information. It has the following functions:

[0552] Emotion recognition function: Recognizes emotional states from voice data and facial expression data.

[0553] Emotion data transmission function: Recognized emotion data is transmitted to the server.

[0554] Emotional feedback: Tailor appropriate ads and advice based on emotions.

[0555] User

[0556] Users are primarily consumers and parents who receive advertising and perform the following actions:

[0557] Voice and facial expression input: Voice and facial expression data are input through questions and conversations.

[0558] Receiving advertisements and advice: Receive advertisements and advice provided by the server via voice and text.

[0559] Specific processing flow

[0560] 1. The user speaks into the device and shows their facial expressions to the camera.

[0561] 2. The device acquires voice and facial expression data and sends it as digital data to a cloud server.

[0562] 3. The server receives the acquired data, converts it into voice and analyzes facial expressions, and sends it to the emotion recognition engine.

[0563] 4. The emotion recognition engine recognizes the emotional state and sends the data back to the server.

[0564] 5. The server generates appropriate advertisements or advice based on the emotional state and sends them back to the device.

[0565] 6. The device provides the user with the sent advertisements and advice via voice and text.

[0566] Use cases and input prompts for generative AI models

[0567] A specific example of use could be to display advertisements for luxury goods when the user looks satisfied, and to display soothing advertisements when the user is feeling stressed.

[0568] Example prompts to input to a generative AI model:

[0569] Create an application for smartphones or smart glasses that recognizes emotions from a user's voice and facial expression data, and selects and displays the most appropriate advertisement based on that data. Convert the voice data into text and analyze the emotions. Select an advertisement based on the analyzed emotional data and display it on the user's display.

[0570] This allows users to receive advertisements and advice that match their emotions at the time, making it possible to provide more personalized and effective services.

[0571] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0572] Step 1:

[0573] The user inputs voice into the terminal and shows facial expressions to the camera.

[0574] Input: User's voice and facial expressions.

[0575] Processing: Capture voice and facial expression data using the device's microphone and camera.

[0576] Output: Acquired voice and facial expression data.

[0577] Step 2:

[0578] The voice data and facial expression data acquired by the device are converted into digital format and sent to a server on the cloud.

[0579] Input: Analog voice data and facial expression data acquired on the device.

[0580] Processing: Analog audio is converted to digital format through signal processing, and facial expression data is converted to digital format through image processing. The data is then sent to a cloud server.

[0581] Output: Digital voice and facial expression data in a format usable by the cloud server.

[0582] Step 3:

[0583] The server converts the received voice data into text data and analyzes the facial expression data to recognize the emotional state.

[0584] Input: Digital voice data and facial expression data received by the cloud server.

[0585] Processing: A speech recognition algorithm is used to convert the voice data into text data, and an emotion recognition engine is used to analyze emotions from facial expression data.

[0586] Output: Text data and emotional state data.

[0587] Step 4:

[0588] The server generates appropriate advertisements or advice based on the analysis results.

[0589] Input: Text data and emotional state data.

[0590] Processing: Based on the sentiment data, an advice generation algorithm or ad selection engine is used to select the best ad or advice.

[0591] Output: Selected ads or generated advice.

[0592] Step 5:

[0593] The server sends the generated advertisement or advice to the terminal, which plays and displays it.

[0594] Input: Selected Advertisement or Generated Advice Data.

[0595] Processing: Converting the data into the appropriate format and sending it to the device, where it plays the audio and displays the text.

[0596] Output: Audio and text feedback to the user.

[0597] Step 6:

[0598] The user checks the advertisements and advice provided and re-enters them if necessary.

[0599] Input: Advertisements or advice provided by the device.

[0600] Action: Check the content of the advertisement or advice, and provide feedback or re-enter it.

[0601] Output: Return to step 1 to acquire the user's voice and facial expressions again.

[0602] This will enable the provision of personalized advertisements and advice in real time according to the user's emotional state. Each step uses hardware such as smartphones and smart glasses, and software such as voice recognition algorithms, emotion recognition engines, and ad selection engines.

[0603] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0604] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0605] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0606] [Second embodiment]

[0607] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0608] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0609] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0610] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0611] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0612] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0613] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0614] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0615] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0616] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0617] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0618] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0619] The present invention is a system that inputs the voice of a parent to provide advice on child-rearing, analyzes the acquired voice, and provides appropriate advice. It also has a function that stores the voice data recorded by the parent on the cloud and plays it back to the child as a fairy tale or lullaby at a specified time. Specific embodiments for implementing this system are described below.

[0620] The system is broadly composed of three main elements: terminals, servers, and users.

[0621] Terminal

[0622] The device is a smart speaker or display device that captures the parent's voice. The device has the following features:

[0623] Voice input function: Parental voice is captured through a microphone.

[0624] Recording function: Saves captured audio as digital data.

[0625] Audio playback function: Plays audio data received from a server on the cloud.

[0626] Text display function: Displays text data received from a server on the cloud.

[0627] server

[0628] A server is a computing unit that runs on the cloud and has the following main functions:

[0629] Voice analysis function: Converts voice data sent from the device into text data.

[0630] Advice generation function: Generates appropriate parenting advice from analyzed voice data.

[0631] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[0632] Text generation function: Generates text data of advice and sends it to the terminal.

[0633] Fairy tale and lullaby selection function: Selects the most appropriate fairy tale or lullaby based on the child's age and sends it to the device.

[0634] User

[0635] Users, primarily parents, perform the following actions:

[0636] Voice input: Record parenting advice, fairy tales, and lullabies.

[0637] Audio playback request: Request playback of a fairy tale or lullaby that you have recorded yourself.

[0638] Receiving advice: Receive advice provided by the server in voice and text format.

[0639] Specific examples

[0640] Specific examples of child-rearing consultation

[0641] 1. The user speaks to the device and asks for advice on child-rearing. For example, "My child has been crying a lot lately. What should I do?"

[0642] 2. The device captures the audio and sends the digital data to the server.

[0643] 3. The server converts the speech into text and performs analysis.

[0644] 4. The server generates appropriate advice based on the analysis results and sends the advice to the device as text and audio data.

[0645] 5. The device plays the transmitted audio data and displays the text data on the display.

[0646] 6. Users receive advice and apply it to solve their parenting problems.

[0647] Examples of fairy tales and lullabies

[0648] 1. The user sits down at the device and starts recording a fairy tale. For example, say, "I'm going to read you the fairy tale of Momotaro."

[0649] 2. The device enters recording mode and records the user's voice.

[0650] 3. After the device finishes recording, it sends the digital data to the server and stores it.

[0651] 4. The user later requests, "Play the fairy tale of Momotaro."

[0652] 5. The server sends the saved audio data to the playback device.

[0653] 6. The device plays the recorded story and reads it to the child in the user's voice.

[0654] This system allows parents to receive advice 24 hours a day, 365 days a year to resolve their child-rearing concerns, and also deepens the bond with their children by having them listen to fairy tales and lullabies recorded in their own voices.

[0655] The processing flow will be explained below.

[0656] Childcare consultation process flow

[0657] Step 1:

[0658] The user can ask for parenting advice by voice, for example, "My child has been crying at night recently. What should I do?"

[0659] Step 2:

[0660] The device records the user's voice. It captures the input voice through a microphone and stores it as digital data.

[0661] Step 3:

[0662] The device sends the voice data to a server on the cloud, and the voice data is uploaded to the server via network communication.

[0663] Step 4:

[0664] The server converts the voice data into text data using a voice recognition API.

[0665] Step 5:

[0666] The server analyzes the text data and uses natural language processing (NLP) to understand the keywords and context within the text and analyze the content of the parenting consultation.

[0667] Step 6:

[0668] The server generates advice based on the analysis results. Using an AI model, it generates appropriate advice in text format. For example, "We recommend creating a relaxing environment and establishing a regular routine before bed."

[0669] Step 7:

[0670] The server converts the text data into audio data using a text-to-speech API.

[0671] Step 8:

[0672] The server transmits the generated voice data and text data to the terminal via network communication.

[0673] Step 9:

[0674] The device plays the audio and displays the text: it plays the audio data through the speaker and displays the text data on the display.

[0675] Processing flow for recording and playing back fairy tales in a parent's voice

[0676] Step 1:

[0677] The user gives a voice command to the terminal to "start recording a fairy tale." For example, the user says, "I will record the fairy tale of Momotaro."

[0678] Step 2:

[0679] The device switches to recording mode, starts the recording function, and prepares to record the user's voice.

[0680] Step 3:

[0681] The user reads a fairy tale aloud and the device records the audio, for example, "Once upon a time, there lived an old man and an old woman."

[0682] Step 4:

[0683] The terminal ends the recording. The recording stops due to a user instruction or a timeout.

[0684] Step 5:

[0685] The device sends the recorded data to a server on the cloud, and the audio data is uploaded to the server via network communication.

[0686] Step 6:

[0687] The server saves the recorded data and performs the necessary processing. The audio data is stored in storage and the data format is adjusted.

[0688] Step 7:

[0689] The user gives a voice command to the terminal to "start playing a fairy tale." For example, the user says, "Please play the fairy tale of Momotaro."

[0690] Step 8:

[0691] The server sends the recorded voice data to the terminal, and the saved voice data is provided to the terminal via the network.

[0692] Step 9:

[0693] The device plays the audio data, and the recorded story is played through the speaker for the child to listen to.

[0694] Relaxing music provision process flow

[0695] Step 1:

[0696] The user vocally requests relaxing music, for example, "Play some relaxing music."

[0697] Step 2:

[0698] The device records the audio and sends it to a server in the cloud. The audio data is captured and uploaded to the server via the network.

[0699] Step 3:

[0700] The server converts the voice into text data, and then uses a voice recognition API to convert the voice data into text data.

[0701] Step 4:

[0702] The server searches for appropriate relaxing music, analyzes the text data, and selects appropriate songs from the relaxing music library.

[0703] Step 5:

[0704] The server sends the selected music data to the device, and the music file is uploaded to the device via the network.

[0705] Step 6:

[0706] The device will play music, and the speaker will play the selected relaxing music.

[0707] These processing steps allow users to easily receive parenting advice, play recorded fairy tales and lullabies in their own voice, or request relaxing music to help reduce stress.

[0708] Example 1

[0709] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0710] Modern parents face many challenges and worries regarding child-rearing, but there is a lack of means to quickly and accurately resolve these issues. While it is useful for busy parents to record fairy tales and lullabies in their own voices for their children, the recording and playback methods are inconvenient. Furthermore, there is a lack of systems that can select and effectively play appropriate fairy tales and lullabies based on the child's age.

[0711] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0712] In this invention, the server includes means for acquiring voice uttered by a parent, means for converting the acquired voice into digital data, means for transmitting the digital data to a cloud-based computer system, means for analyzing the digital data stored in the cloud-based computer system and generating advice related to child-rearing consultations, means for returning the generated advice as voice and text to a playback terminal, means for playing the voice and displaying the text on the playback terminal, and means for the user to request audio playback from the cloud-based computer system. This allows parents to easily receive child-rearing consultations at any time and effectively use fairy tales and lullabies recorded in their own voices.

[0713] The "means for acquiring the voice uttered by the parent" refers to a device or apparatus that collects the voice uttered by the parent using a voice input device such as a microphone.

[0714] "Means for converting captured audio into digital data" refers to devices or software that convert analog audio signals into digital format.

[0715] "Means for transmitting digital data to a computer system on the cloud" refers to a communication method for transferring digital data from a terminal to a cloud server via the Internet.

[0716] "Means for analyzing digital data stored in a cloud-based computer system and generating advice related to childcare consultations" refers to a function that uses artificial intelligence and analytical algorithms to analyze digital data stored on a cloud server and generate advice related to childcare.

[0717] "Means for returning the generated advice to the playback terminal as audio and text" refers to a communication method for transmitting the advice generated by the cloud server to the playback device in the form of an audio file and a text file.

[0718] "Means for playing back audio and displaying text on a playback terminal" refers to the function of playing back audio data received in a playback device through a speaker and displaying text data on a display.

[0719] "Means for a user to make a request to a cloud computer system to play audio" refers to a communication means for a user to send an instruction to a cloud computer system to request audio playback.

[0720] "Means for recording sounds uttered by a parent" refers to an apparatus or method for recording sounds uttered by a parent using a recording device.

[0721] "Means for storing recorded audio as digital data in a computer system on the cloud" refers to a function for storing recorded audio data in digital format on a cloud server.

[0722] "Means for searching for digital data stored in a computer system on the cloud and transmitting it to a playback terminal at a specified time" refers to a function for searching for audio data stored on a cloud server and transferring it to a playback device at a time specified by the user.

[0723] "Means for selecting appropriate stories and lullabies based on the age of the child" refers to filtering and selection algorithms used to select content appropriate for the child's age.

[0724] "Means for transmitting selected stories and lullabies from a cloud-based computer system to a playback device" refers to a communication method for transferring selected audio data from a cloud server to a playback device.

[0725] "Means for playing back a story or lullaby on a playback terminal" refers to a function for playing back audio data of a story or lullaby received by a playback device on a speaker.

[0726] This system transmits the voices of parents to a cloud-based computer system, analyzes the voices, generates advice related to childcare consultations, and plays fairy tales and lullabies recorded by the user at appropriate times. This system is mainly composed of three main elements: a terminal, a server, and a user.

[0727] Hardware and software used

[0728] Device: The device includes a voice input function for capturing the voice spoken by the parent, a recording function, a voice playback function, and a text display function. Specific examples of this device include a voice input device and a device with a display. Specifically, this applies to smart speakers and smart devices with a display.

[0729] Server: The server is located in the cloud and has functions for voice analysis, advice generation, data storage, text generation, and selection of fairy tales and lullabies. Specific examples include platforms that provide cloud computing services (such as AWS and Google Cloud Platform). Amazon Transcribe and Google Cloud Speech-to-Text are used for voice analysis, and generative AI models (such as GPT-3 and ChatGPT) are used for text generation.

[0730] Details of data processing and calculation

[0731] 1. Audio capture and conversion:

[0732] Device: The device captures the user's voice and converts the analog voice signal into digital data. Specifically, smart speakers and smart devices with displays use voice input functionality to capture the user's spoken voice through a microphone and convert the voice into a digital format.

[0733] 2. Data transmission and analysis:

[0734] Terminal: The converted digital data is sent to a computer system in the cloud, using communication means via the Internet.

[0735] Server: The server converts the received digital data into text data using speech analysis software, for example, Amazon Transcribe or Google Cloud Speech-to-Text.

[0736] 3. Advice Generation:

[0737] Server: Based on the analyzed text data, a generative AI model (e.g., GPT-3 or ChatGPT) is used to generate appropriate advice related to childcare consultations.

[0738] 4. Data transmission and display:

[0739] Server: Sends the generated advice to the device as voice and text data.

[0740] Terminal: The terminal plays the received audio data and displays the text data on the display.

[0741] 5. Record and play fairy tales and lullabies:

[0742] Device: The user starts recording a fairy tale or lullaby, and the device goes into recording mode, collects the audio, and temporarily stores it as digital data.

[0743] Server: After recording is complete, the device sends the data to a cloud-based computer system, where it is securely stored. When a user requests playback, the server sends the stored data to the device, which then plays the audio data.

[0744] Specific use cases

[0745] Specific examples of child-rearing consultation

[0746] 1. The user speaks into the device and says, "My child has been crying at night lately. What should I do?"

[0747] 2. The device captures the audio and sends the digital data to the server.

[0748] 3. The server converts the speech into text and performs analysis.

[0749] 4. Based on the analysis results, the server generates advice such as, "To help your crying child relax at night, it would be a good idea to create a quiet environment and stick to a consistent bedtime."

[0750] 5. The device plays the transmitted audio data and displays the text data on the display.

[0751] Examples of fairy tales and lullabies

[0752] 1. The user approaches the device and says, "I'm going to read the fairy tale of Momotaro," and begins recording.

[0753] 2. The device enters recording mode and records the user's voice.

[0754] 3. After the device finishes recording, it sends the digital data to the server, where it is stored.

[0755] 4. The user later requests, "Play the fairy tale of Momotaro."

[0756] 5. The server sends the saved audio data to the device.

[0757] 6. The device plays the recorded story and reads it to the child in the user's voice.

[0758] Examples of prompt statements

[0759] "Recently, my child has been crying at night and it's bothering me. What should I do?"

[0760] "I want to record the Momotaro fairy tale so I can read it to you."

[0761] "Please give me some advice on how to get my child to sleep early."

[0762] This system allows parents to easily consult with their children at any time, and also makes it possible to effectively use fairy tales and lullabies recorded in their own voices.

[0763] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0764] Step 1:

[0765] The user's voice is input into the device. For example, the user might say, "My child has been crying at night recently and it's bothering me. What should I do?"

[0766] Input: User utterance

[0767] Output: Audio data captured by a microphone

[0768] Step 2:

[0769] The device converts the user's voice into digital data. The smart speaker uses its voice input function to convert analog voice into digital data.

[0770] Input: Captured audio data

[0771] Output: Digitized audio data

[0772] Step 3:

[0773] The device sends digital audio data to a server in the cloud, where it is sent as encrypted data over the internet.

[0774] Input: Digitized audio data

[0775] Output: Digital data sent to the server

[0776] Step 4:

[0777] The server converts the audio data into text using speech analysis software, such as Amazon Transcribe or Google Cloud Speech-to-Text.

[0778] Input: Digital data sent to the server

[0779] Output: Text data

[0780] Step 5:

[0781] The server analyzes the text data using a generative AI model and generates advice for the user's childcare consultation. The analysis uses a database of childcare-related information and a generative AI model (e.g., GPT-3 or ChatGPT).

[0782] Input: Text data

[0783] Output: Advice text and audio data

[0784] Step 6:

[0785] The server transmits the generated advice to the terminal via the Internet as voice data and text data.

[0786] Input: Advice text and audio data

[0787] Output: Advice data sent to the terminal

[0788] Step 7:

[0789] The device plays the advice audio data and displays the text data, while the smart speaker or device with a display plays the audio and displays the text on the display.

[0790] Input: Advice data sent to the terminal

[0791] Output: Played audio and displayed text

[0792] Step 8:

[0793] The user gives the command to start recording. For example, they say, "I'm going to read you the fairy tale of Momotaro."

[0794] Input: User utterance

[0795] Output: Start recording command

[0796] Step 9:

[0797] The device enters recording mode, recording the user's speech and temporarily saving it as digital data.

[0798] Input: Start recording

[0799] Output: Digitized recording data

[0800] Step 10:

[0801] After the device finishes recording, it sends the recorded data to a cloud server and stores it there. The server then securely stores the received data.

[0802] Input: Digitized recording data

[0803] Output: Data stored on a cloud server

[0804] Step 11:

[0805] The user makes a playback request, for example, "Play the fairy tale of Momotaro."

[0806] Input: User's playback request

[0807] Output: Playback request data

[0808] Step 12:

[0809] The server retrieves the stored voice data and sends it to the device, which then transfers the data over the Internet.

[0810] Input: Playback request data

[0811] Output: Audio data sent to the device

[0812] Step 13:

[0813] The device plays the audio data, and the recorded story is played on the speaker in the user's voice for the child to hear.

[0814] Input: Audio data sent to the device

[0815] Output: Played fairy tale audio

[0816] This will allow parents to easily record and play back childcare advice and fairy tales.

[0817] (Application example 1)

[0818] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0819] Conventional parenting support systems focus solely on analyzing parents' voice input and providing appropriate advice. However, there are no systems that can respond to security-related consultations or inquiries using similar voice recognition technology. There is a need for a system that can provide advice on both parenting and security in response to parents' spoken inquiries and manage them centrally. In addition, the generated advice must be provided in both voice and text format, allowing for flexible responses tailored to each user's usage situation.

[0820] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0821] In this invention, the server includes means for acquiring voice uttered by parents, means for converting the acquired voice into digital data, means for transmitting the digital data to a cloud server, means for analyzing the digital data stored in the cloud server and generating advice on child-rearing consultations and security, means for returning the generated advice to a playback terminal as voice and text, and means for playing the voice and displaying the text on the playback terminal, thereby making it possible to provide appropriate advice in a unified manner in response to questions from parents about child-rearing consultations and security.

[0822] The "means for acquiring parental vocalizations" is a device or system for collecting parental vocalizations in digital form.

[0823] The "means for converting captured audio into digital data" refers to a technique or device for converting audio signals into digital form.

[0824] The "means for transmitting digital data to a server on the cloud" refers to a technology or device for transmitting the converted digital data to a cloud server via the Internet.

[0825] "Means for analyzing digital data stored on a server on the cloud and generating advice on child-rearing consultations and security" refers to a technology or system that analyzes digital data stored on a cloud server and generates advice related to child-rearing and security based on the results.

[0826] The "means for returning the generated advice to the playback terminal as voice and text" refers to a technique and device for converting the generated advice into voice and text format and transmitting it to the playback terminal.

[0827] "Means for playing audio and displaying text at a playback terminal" refers to a device or system for playing received audio and displaying text.

[0828] A "means for recording parental vocalizations" is a device or technology for recording parental vocalizations.

[0829] "Means for storing recorded audio as digital data on a server on the cloud" refers to a method or system for storing recorded audio in a digital format on a cloud server.

[0830] "Means for searching for digital data stored on a server on the cloud and transmitting it to a playback terminal at a specified time" refers to a function or device for searching for data stored on a cloud server under specific conditions or at specific times and transmitting it to a playback terminal.

[0831] The "means for selecting appropriate fairy tales and lullabies based on the age of the child" refers to an algorithm or device that selects the most appropriate fairy tales and lullabies based on the age information of the child.

[0832] "Means for transmitting selected fairy tales and lullabies from a server on the cloud to a playback terminal" refers to means for transmitting data of selected fairy tales and lullabies from a cloud server to a playback terminal.

[0833] "Means for analyzing captured audio and generating security advice" refers to a system or software capable of analyzing audio data and generating appropriate security-related advice.

[0834] The "means for returning the generated security advice to the playback terminal as voice and text" refers to a technique or device that converts the generated security advice into voice and text format and sends it to the playback terminal.

[0835] To implement this invention, a system is required that captures the voices spoken by parents, transmits them to a cloud server, and generates appropriate advice on child-rearing and security. The system is broadly composed of three main elements: a terminal, a server, and a user.

[0836] Terminal

[0837] The device is a smart speaker or display device that captures the parent's voice. The device has the following features:

[0838] Voice input function: Parental voice is captured through a microphone.

[0839] Recording function: Saves captured audio as digital data.

[0840] Audio playback function: Plays audio data received from a server on the cloud.

[0841] Text display function: Displays text data received from a server on the cloud.

[0842] server

[0843] A server is a computing unit that runs on the cloud and has the following main functions:

[0844] Speech analysis function: Converts voice data sent from the device into text data. Uses the speech_recognition library.

[0845] Advice generation function: Generates appropriate parenting and security advice from the analyzed voice data. It uses a generative AI model with the transformers library.

[0846] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[0847] Text generation function: Generates text data for advice and sends it to the terminal. This uses the tflow library.

[0848] Fairy tale and lullaby selection function: Selects the most appropriate fairy tale or lullaby based on the child's age and sends it to the device.

[0849] User

[0850] Users, primarily parents, perform the following actions:

[0851] Voice input: Record parenting advice, security advice, fairy tales, lullabies, etc. For example, ask, "There's been a suspicious person hanging around recently. What should I do?"

[0852] Audio playback request: Request playback of a fairy tale or lullaby that you have recorded yourself.

[0853] Receive advice: Receive parenting and security advice provided by the server via voice and text.

[0854] Specific examples

[0855] Specific examples of childcare and security consultations

[0856] 1. A user speaks to a device and asks a question about parenting or security. For example, "My child has been crying a lot lately. What should I do?"

[0857] 2. The device captures the audio and sends the digital data to the server.

[0858] 3. The server converts the speech to text and performs the analysis. The speech_recognition library is used here.

[0859] 4. The server generates appropriate advice based on the analysis results and sends the advice to the device as text and audio data. This is where a generative AI model using the transformers library comes into play.

[0860] 5. The device plays the transmitted audio data and displays the text data on the display.

[0861] 6. The user receives the advice and applies it to solve the problem.

[0862] Prompt Sentence Examples

[0863] An example of a prompt to be input to the generative AI model is as follows:

[0864] "Generate appropriate security-related advice based on spoken questions. For example, what should I do if my neighbors are bothering me with noise?"

[0865] In this way, parents can quickly receive appropriate advice on child-rearing and security-related questions, creating a multi-functional support system that meets the needs of both parties.

[0866] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0867] Step 1:

[0868] The user speaks to the device and makes a voice input about childcare or security. For example, "My child has been crying at night recently. What should I do?" This voice is captured as a digital signal through the microphone. This is the state in which the input voice data has been captured.

[0869] Step 2:

[0870] The device picks up the voice and sends the digital data to a server in the cloud. Software on the device converts the voice data into packets and sends them to the server via the Internet. The input voice data is sent to the server.

[0871] Step 3:

[0872] The server converts the speech to text and performs analysis. The speech_recognition library is used to convert the speech data to text data. The input is speech data and the output is text data. The speech recognition engine analyzes the speech signal and converts it to text.

[0873] Step 4:

[0874] The server generates appropriate advice using a generative AI model based on the analysis results. The server uses the transformers library to input the analyzed text and the prompt sentence into the generative AI model, and outputs the optimal advice. The input is text data, and the output is text data of the advice.

[0875] Step 5:

[0876] The server returns the generated advice to the playback device as audio and text data. The generated advice text data is converted into audio data using the gTTS (Google Text to Speech) library and sent to the device via the Internet. The input is the advice text data, and the output is audio data and text data.

[0877] Step 6:

[0878] The terminal plays the transmitted voice data and displays the text data on the display. The built-in speaker of the terminal plays the voice data and the display displays the text data. The input is voice data and text data, and the output is voice and displayed text. This allows the user to receive advice.

[0879] These are the processing steps of the system that realizes this application example. The generated advice is useful for solving problems related to child rearing and security, and users can receive specific instructions and suggestions.

[0880] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0881] The present invention is a system that inputs the user's voice, analyzes the acquired voice, and provides appropriate advice, and also combines it with an emotion engine that recognizes the user's emotions and generates a response according to their emotional state. This enables more personalized and appropriate advice, improving the effectiveness of comprehensive child-rearing support.

[0882] Key Components

[0883] The system is broadly divided into four main components: the terminal, the server, the emotion engine, and the user.

[0884] Terminal

[0885] The device is a smart speaker or a device with a display that receives user voice and has the following functions:

[0886] Voice input function: Parental voice is captured through a microphone.

[0887] Recording function: Saves captured audio as digital data.

[0888] Audio playback function: Plays audio data received from a server on the cloud.

[0889] Text display function: Displays text data received from a server on the cloud.

[0890] server

[0891] A server is a computing unit that runs on the cloud and has the following main functions:

[0892] Voice analysis function: Converts voice data sent from the device into text data.

[0893] Advice generation function: Generates appropriate parenting advice from analyzed voice data.

[0894] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[0895] Text generation function: Generates text data of advice and sends it to the terminal.

[0896] Emotion data processing function: Analyzes the emotional data sent from the emotion engine and adjusts the advice content.

[0897] Emotion Engine

[0898] The emotion engine is a software component that recognizes emotions from user voice data and analyzes that information. It has the following functions:

[0899] Emotion recognition function: Recognizes emotional states from voice data.

[0900] Emotion data transmission function: Recognized emotion data is transmitted to the server.

[0901] Emotional feedback function: Tailor appropriate advice based on emotions.

[0902] User

[0903] Users, primarily parents, perform the following actions:

[0904] Voice input: Record parenting advice, fairy tales, and lullabies.

[0905] Audio playback request: Request playback of a fairy tale or lullaby that you have recorded yourself.

[0906] Receiving advice: Receive advice provided by the server in voice and text format.

[0907] Specific examples

[0908] Specific examples of child-rearing consultation

[0909] Example 1: A system that recognizes the user's emotional state and provides advice

[0910] 1. The user speaks to the device and asks for advice on child-rearing. For example, "My child has been crying a lot lately. What should I do?"

[0911] 2. The device captures the voice and sends the digital data to the emotion engine.

[0912] 3. The emotion engine analyzes the voice data to recognize the user's emotional state, for example, recognizing that the user is feeling stressed.

[0913] 4. The emotion engine sends the emotion data to the server.

[0914] 5. The server analyzes the emotional and voice data and generates appropriate advice based on the child's emotional state. For example, it might say, "To reduce stress factors for your child, we recommend creating a relaxing bedtime routine."

[0915] 6. The server converts the generated advice into voice and text data and sends it to the device.

[0916] 7. The device plays the audio and displays the text.

[0917] 8. Users receive advice and apply it to solve their parenting problems.

[0918] Examples of fairy tales and lullabies

[0919] Example 2: Reading fairy tales using an emotion engine

[0920] 1. The user sits down at the device and starts recording a fairy tale. For example, say, "I'm going to read you the fairy tale of Momotaro."

[0921] 2. The device enters recording mode and records the user's voice.

[0922] 3. After the device finishes recording, it sends the digital data to the server and stores it.

[0923] 4. The emotion engine analyzes the recording data and attaches emotional information.

[0924] 5. The user later requests, "Play the fairy tale of Momotaro."

[0925] 6. The server selects the recorded voice data taking into consideration the attached emotional information and sends it to the terminal.

[0926] 7. The device plays the audio data and plays it to the child in the user's voice.

[0927] The system can recognize the user's emotions and provide appropriate advice based on those emotions, enabling more personalized and effective parenting support. It can also deepen the bond between parents and their children by playing fairy tales and lullabies recorded in their own voices.

[0928] The processing flow will be explained below.

[0929] A process flow for parenting consultations that combines an emotion engine

[0930] Step 1:

[0931] The user can ask for parenting advice by voice, for example, "My child has been crying at night recently. What should I do?"

[0932] Step 2:

[0933] The device records the user's voice. It captures the input voice through a microphone and stores it as digital data.

[0934] Step 3:

[0935] The device sends the recorded voice data to the emotion engine, which then uploads it to the emotion engine via network communication.

[0936] Step 4:

[0937] The emotion engine analyzes the voice data and recognizes the user's emotional state. For example, it may recognize that the user is feeling stressed.

[0938] Step 5:

[0939] The emotion engine sends the recognized emotion data to the server, where the voice data and emotion data are combined.

[0940] Step 6:

[0941] The server converts the voice data into text data using a voice recognition API.

[0942] Step 7:

[0943] The server analyzes the text data and emotional data, using natural language processing (NLP) to understand the relationship between keywords and context in the text and emotions, and analyzes the content of the parenting consultation.

[0944] Step 8:

[0945] The server generates advice based on the analysis results. Using an AI model that reflects emotional data, appropriate advice is generated in text format. For example, it might say, "To reduce the factors that cause your child stress, we recommend creating a relaxing bedtime routine."

[0946] Step 9:

[0947] The server converts the text data into audio data using a text-to-speech API.

[0948] Step 10:

[0949] The server transmits the generated voice data and text data to the terminal via network communication.

[0950] Step 11:

[0951] The device plays the audio and displays the text: it plays the audio data through the speaker and displays the text data on the display.

[0952] Processing flow for recording and playing fairy tales using the emotion engine

[0953] Step 1:

[0954] The user approaches the device and begins recording a fairy tale. For example, they say, "I'm going to read you the fairy tale of Momotaro."

[0955] Step 2:

[0956] The device switches to recording mode, starts the recording function, and prepares to record the user's voice.

[0957] Step 3:

[0958] The user reads a fairy tale aloud and the device records the audio, for example, "Once upon a time, there lived an old man and an old woman."

[0959] Step 4:

[0960] The terminal ends the recording. The recording stops due to a user instruction or a timeout.

[0961] Step 5:

[0962] The device sends the recorded voice data to the emotion engine, which then uploads it to the emotion engine via network communication.

[0963] Step 6:

[0964] The emotion engine analyzes the audio data and recognizes the emotional state of the recording.

[0965] Step 7:

[0966] The emotion engine sends the recognized emotion data to the server, where the voice data and emotion data are combined.

[0967] Step 8:

[0968] The server stores the recording data and stores it safely in the cloud.

[0969] Step 9:

[0970] The user later gives a voice command such as "Play the fairy tale of Momotaro."

[0971] Step 10:

[0972] The server searches for the stored voice data and emotion data and sends them to the device at the specified time.

[0973] Step 11:

[0974] The device plays back the recorded fairy tale audio data in the user's voice, and the audio is played back through the speaker so that the child can listen to it in the user's voice.

[0975] Processing flow for providing relaxing music using an emotion engine

[0976] Step 1:

[0977] The user vocally requests relaxing music, for example, "Play some relaxing music."

[0978] Step 2:

[0979] The device records the voice and sends it to the emotion engine, which then uploads it to the emotion engine via network communication.

[0980] Step 3:

[0981] The emotion engine analyzes the voice data and recognizes the user's emotional state.

[0982] Step 4:

[0983] The emotion engine sends the recognized emotion data to the server, where the voice data and emotion data are combined.

[0984] Step 5:

[0985] The server converts the speech into text and selects appropriate relaxing music taking into account emotional data.

[0986] Step 6:

[0987] The server sends the selected music data to the device, which then uploads the music data to the device via the network.

[0988] Step 7:

[0989] The device will play music, and the speaker will play the selected relaxing music.

[0990] These processing steps allow users to easily receive parenting advice, play fairy tales or lullabies recorded in their own voice, or request relaxing music to help reduce stress. The combination of an emotion engine provides more personalized assistance based on the user's emotional state.

[0991] Example 2

[0992] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0993] In today's world, many parents face the burdens of work and family life, requiring them to quickly receive appropriate parenting advice. However, conventional parenting support systems have struggled to accurately recognize parents' emotional states and provide personalized advice based on those states. In particular, when parents are stressed or tired, general advice that ignores their emotional state has limited effectiveness and does not contribute to caring for the parents' physical and mental health. A system that solves this problem and provides more effective and compassionate parenting support is needed.

[0994] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for transmitting digital data to a cloud server, a means for analyzing the digital data stored in the cloud server and recognizing the emotional state of the user, and a means for generating advice related to child-rearing consultation according to the emotional state. This makes it possible to provide personalized advice according to the emotional state of the parent.

[0995] "Voice spoken by the parent" refers to the voice spoken by the parent into the device for conversation, consultation, or recording.

[0996] "Digital data" means information such as voice or text that has been converted into a form that can be processed, stored, or transmitted electronically.

[0997] A "cloud server" is an online data processing and storage system provided via the Internet by computers located in remote locations.

[0998] An "emotion engine" is a software component that analyzes and recognizes a user's emotions and psychological state from their voice and text data.

[0999] A "voice analysis engine" is software or hardware that has the function of converting voice data into text data.

[1000] An "advice generation engine" is software or hardware that has the function of generating appropriate advice based on input data.

[1001] A "playback terminal" is a device for playing and displaying audio and text data sent from a server on the cloud.

[1002] "Users" are those who use the system to receive consultations, recordings, and advice on child-rearing, and are primarily parents.

[1003] "Childcare consultation" refers to the act of parents seeking professional advice regarding questions or problems they have regarding childcare.

[1004] "Emotional state" refers to the psychological state or emotion expressed by the user through voice or text, and includes, for example, stress, joy, anger, etc.

[1005] The "audio playback function" is a function that plays back audio data so that the user can hear it.

[1006] "Text display function" refers to the function of displaying text data on a display or other display device.

[1007] The present invention is a system that inputs the user's voice, analyzes the acquired voice, and provides appropriate advice, and also combines it with an emotion engine that recognizes the user's emotions and generates a response according to their emotional state. This system enables more personalized and appropriate advice, improving the effectiveness of comprehensive child-rearing support.

[1008] Key Components

[1009] The system is broadly divided into four main components: the terminal, the server, the emotion engine, and the user.

[1010] Terminal

[1011] The device is a smart speaker or a device with a display that receives user voice and has the following functions:

[1012] Voice input function: Captures the user's voice through a microphone.

[1013] Recording function: Saves captured audio as digital data.

[1014] Audio playback function: Plays audio data received from a server on the cloud.

[1015] Text display function: Displays text data received from a server on the cloud.

[1016] server

[1017] A server is a computing unit that runs on the cloud and has the following main functions:

[1018] Voice analysis function: Converts voice data sent from the device into text data.

[1019] Emotion recognition function: Analyze the user's emotional state through the emotion engine.

[1020] Advice generation function: Generates appropriate parenting advice based on analyzed voice data and emotional data.

[1021] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[1022] Text generation function: Generates text data of advice and sends it to the terminal.

[1023] Emotion Engine

[1024] The emotion engine is a software component that recognizes emotions from user voice data and analyzes that information. It has the following functions:

[1025] Emotion Recognition: Recognizes emotional states from speech data. It uses natural language processing techniques and applies a BERT-based sentiment analysis model.

[1026] Emotion data transmission function: Recognized emotion data is transmitted to the server.

[1027] User

[1028] Users, primarily parents, perform the following actions:

[1029] Voice input: Record parenting advice, fairy tales, and lullabies.

[1030] Audio playback request: Requests playback of a recorded fairy tale or lullaby.

[1031] Receiving advice: Receive advice provided by the server in voice and text format.

[1032] Specific examples

[1033] Specific examples of child-rearing consultation

[1034] 1. The user speaks to the device and asks for advice on child-rearing. For example, "My child has been crying a lot lately. What should I do?"

[1035] 2. The device captures the audio and sends the digital data to the server.

[1036] 3. The server uses a speech analysis engine to analyze the voice data and convert it into text data.

[1037] 4. The server recognizes the user's emotional state through an emotion engine and identifies that the user is feeling stressed.

[1038] 5. The server generates appropriate advice based on the emotion and voice data. For example, "To reduce stress factors for your child, we recommend creating a relaxing bedtime routine."

[1039] 6. The server converts the generated advice into voice and text data and sends it to the device.

[1040] 7. The device plays the audio and displays the text.

[1041] Examples of fairy tales and lullabies

[1042] 1. The user sits down at the device and starts recording a fairy tale. For example, say, "I'm going to read you the fairy tale of Momotaro."

[1043] 2. The device enters recording mode and records the user's voice.

[1044] 3. After the device finishes recording, it sends the digital data to the server and stores it.

[1045] 4. The emotion engine analyzes the recording data and attaches emotional information.

[1046] 5. The user later requests, "Play the fairy tale of Momotaro."

[1047] 6. The server selects the recorded voice data taking into consideration the attached emotional information and sends it to the terminal.

[1048] 7. The device plays the audio data and plays it to the child in the user's voice.

[1049] Example prompt sentence:

[1050] My child has been crying at night recently, what should I do?

[1051] "Play the fairy tale of Momotaro"

[1052] This system can provide appropriate advice based on the user's emotional state, enabling more personalized and effective parenting support. It can also deepen the bond between parents and their children by playing fairy tales and lullabies recorded in their own voices.

[1053] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1054] Step 1: Voice Input

[1055] The user speaks into the device to ask questions or ask for advice about child-rearing. For example, "My child has been crying at night recently. What should I do?"

[1056] The input is the user's voice data.

[1057] The device picks up audio using a built-in microphone, and performs noise reduction and echo cancellation to capture clear audio data.

[1058] The output is audio data converted into digital form.

[1059] Step 2: Sending audio data

[1060] The device converts the captured audio data into a digital format, specifically, converting the audio into 16-bit PCM format, compressing the data, and packetizing it.

[1061] The input is the converted digital audio data.

[1062] The device then sends the converted digital audio data to a cloud server using HTTPS to ensure data security.

[1063] The output is the audio data received by the server on the cloud.

[1064] Step 3: Analyzing the audio data

[1065] The server receives the voice data sent from the terminal.

[1066] The input is voice data sent to a server on the cloud.

[1067] The server uses a speech analysis engine to convert the audio data into text data, using the Google Cloud Speech-to-Text API.

[1068] The output is text data converted from the audio data.

[1069] Step 4: Recognize emotions

[1070] The server passes the converted text data to the emotion engine.

[1071] The input is text data.

[1072] The emotion engine uses natural language processing technology to analyze the user's emotional state from text data. It uses a BERT-based sentiment analysis model to recognize users' stress, joy, anger, etc.

[1073] The output is the recognized emotion data.

[1074] Step 5: Generating Advice

[1075] The server integrates the emotion data sent from the emotion engine with the text data.

[1076] The input is emotion data and text data.

[1077] The server uses an advice generation engine to generate appropriate advice using a GPT-based generative AI model, such as "To reduce stress factors for your child, we recommend creating a relaxing bedtime routine."

[1078] The output is the generated advice.

[1079] Step 6: Send and display advice data

[1080] The server converts the generated advice into both text and audio data.

[1081] The input is the generated advice.

[1082] The server transmits the converted data to the terminal.

[1083] The device analyzes the received data, plays back the voice data, and displays the text data on the display, using a voice playback engine and a text-to-speech (TTS) engine.

[1084] The output is a spoken advice and a displayed text advice.

[1085] Step 7: Take and apply advice

[1086] The user receives the advice provided by the device and applies it in real life.

[1087] The inputs are spoken advice and displayed text advice.

[1088] The hope is that by following the advice, users will find their parenting problems alleviated.

[1089] The output is the effect of the problem mitigation.

[1090] (Application example 2)

[1091] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1092] Conventional advertising delivery systems deliver advertisements uniformly without considering the user's emotional state, making it difficult to provide a personalized advertising experience that reflects the user's interests and preferences. Similarly, parenting support systems provide advice without understanding the parent's emotional state, resulting in an inability to provide effective support.

[1093] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1094] In this invention, the server includes means for recognizing an emotional state from acquired voice data and facial expression data, means for adjusting the content of advice based on the recognized emotional state, and means for selecting and displaying advertisements in accordance with the emotional state recognized from the acquired voice data and facial expression data, thereby making it possible to analyze the user's emotional state in real time and provide optimal advertisements and advice.

[1095] "Voice data" is a digital representation of a user's voice.

[1096] "Facial expression data" refers to facial expressions of a user captured using a camera and converted into an analyzable format.

[1097] The "emotional state" refers to the user's emotions analyzed based on voice data and facial expression data. Specifically, emotions include joy, sadness, anger, fear, surprise, and excitement.

[1098] "Means for recognizing" includes software and hardware configurations for analyzing voice data and facial expression data to identify the emotional state of the user.

[1099] The "means for adjusting the content of advice" includes algorithms and programs for generating or modifying appropriate advice for a user based on the perceived emotional state.

[1100] "Means for selecting advertisements" includes software and hardware configurations for selecting the most appropriate advertisements for a user based on the perceived emotional state and other relevant information.

[1101] The term "reproducing means" includes a device and software configuration for reproducing audio or video data received from a server on a reproducing terminal.

[1102] "Storage means" includes software and hardware configurations for safely storing acquired voice data and facial expression data in digital format on a server on the cloud.

[1103] This invention is a system that acquires and analyzes a user's voice and facial expression data to provide appropriate advertisements and advice. The system is mainly composed of four main elements: a terminal, a server, an emotion recognition engine, and the user.

[1104] Terminal

[1105] The terminal is a mobile device such as a smartphone or smart glasses that has the following capabilities:

[1106] Voice and facial expression input function: Uses a microphone and camera to capture the user's voice and facial expression data.

[1107] Recording function: Saves captured audio as digital data.

[1108] Audio playback and display function: Plays back and displays data sent from the server on the cloud.

[1109] server

[1110] A server is a computing unit that runs on the cloud and has the following main functions:

[1111] Voice and facial expression analysis function: Analyzes voice and facial expression data sent from the device and converts it into text data.

[1112] Advertisement and advice generation function: Recognizes the user's emotional state from the analyzed data and generates appropriate advertisements and advice.

[1113] Data storage function: Recorded voice data and facial expression data are safely stored.

[1114] Data transmission function: Sends generated advertisements and advice to the device.

[1115] Emotion Recognition Engine

[1116] The emotion recognition engine is a software component that recognizes emotions from the user's voice data and facial expression data and provides that information. It has the following functions:

[1117] Emotion recognition function: Recognizes emotional states from voice data and facial expression data.

[1118] Emotion data transmission function: Recognized emotion data is transmitted to the server.

[1119] Emotional feedback: Tailor appropriate ads and advice based on emotions.

[1120] User

[1121] Users are primarily consumers and parents who receive advertising and perform the following actions:

[1122] Voice and facial expression input: Voice and facial expression data are input through questions and conversations.

[1123] Receiving advertisements and advice: Receive advertisements and advice provided by the server via voice and text.

[1124] Specific processing flow

[1125] 1. The user speaks into the device and shows their facial expressions to the camera.

[1126] 2. The device acquires voice and facial expression data and sends it as digital data to a cloud server.

[1127] 3. The server receives the acquired data, converts it into voice and analyzes facial expressions, and sends it to the emotion recognition engine.

[1128] 4. The emotion recognition engine recognizes the emotional state and sends the data back to the server.

[1129] 5. The server generates appropriate advertisements or advice based on the emotional state and sends them back to the device.

[1130] 6. The device provides the user with the sent advertisements and advice via voice and text.

[1131] Use cases and input prompts for generative AI models

[1132] A specific example of use could be to display advertisements for luxury goods when the user looks satisfied, and to display soothing advertisements when the user is feeling stressed.

[1133] Example prompts to input to a generative AI model:

[1134] Create an application for smartphones or smart glasses that recognizes emotions from a user's voice and facial expression data, and selects and displays the most appropriate advertisement based on that data. Convert the voice data into text and analyze the emotions. Select an advertisement based on the analyzed emotional data and display it on the user's display.

[1135] This allows users to receive advertisements and advice that match their emotions at the time, making it possible to provide more personalized and effective services.

[1136] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1137] Step 1:

[1138] The user inputs voice into the terminal and shows facial expressions to the camera.

[1139] Input: User's voice and facial expressions.

[1140] Processing: Capture voice and facial expression data using the device's microphone and camera.

[1141] Output: Acquired voice and facial expression data.

[1142] Step 2:

[1143] The voice data and facial expression data acquired by the device are converted into digital format and sent to a server on the cloud.

[1144] Input: Analog voice data and facial expression data acquired on the device.

[1145] Processing: Analog audio is converted to digital format through signal processing, and facial expression data is converted to digital format through image processing. The data is then sent to a cloud server.

[1146] Output: Digital voice and facial expression data in a format usable by the cloud server.

[1147] Step 3:

[1148] The server converts the received voice data into text data and analyzes the facial expression data to recognize the emotional state.

[1149] Input: Digital voice data and facial expression data received by the cloud server.

[1150] Processing: A speech recognition algorithm is used to convert the voice data into text data, and an emotion recognition engine is used to analyze emotions from facial expression data.

[1151] Output: Text data and emotional state data.

[1152] Step 4:

[1153] The server generates appropriate advertisements or advice based on the analysis results.

[1154] Input: Text data and emotional state data.

[1155] Processing: Based on the sentiment data, an advice generation algorithm or ad selection engine is used to select the best ad or advice.

[1156] Output: Selected ads or generated advice.

[1157] Step 5:

[1158] The server sends the generated advertisement or advice to the terminal, which plays and displays it.

[1159] Input: Selected Advertisement or Generated Advice Data.

[1160] Processing: Converting the data into the appropriate format and sending it to the device, where it plays the audio and displays the text.

[1161] Output: Audio and text feedback to the user.

[1162] Step 6:

[1163] The user checks the advertisements and advice provided and re-enters them if necessary.

[1164] Input: Advertisements or advice provided by the device.

[1165] Action: Check the content of the advertisement or advice, and provide feedback or re-enter it.

[1166] Output: Return to step 1 to acquire the user's voice and facial expressions again.

[1167] This will enable the provision of personalized advertisements and advice in real time according to the user's emotional state. Each step uses hardware such as smartphones and smart glasses, and software such as voice recognition algorithms, emotion recognition engines, and ad selection engines.

[1168] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1169] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1170] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1171] [Third embodiment]

[1172] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1173] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1174] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1175] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1176] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1177] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1178] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1179] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1180] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1181] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1182] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1183] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1184] The present invention is a system that inputs the voice of a parent to provide advice on child-rearing, analyzes the acquired voice, and provides appropriate advice. It also has a function that stores the voice data recorded by the parent on the cloud and plays it back to the child as a fairy tale or lullaby at a specified time. Specific embodiments for implementing this system are described below.

[1185] The system is broadly composed of three main elements: terminals, servers, and users.

[1186] Terminal

[1187] The device is a smart speaker or display device that captures the parent's voice. The device has the following features:

[1188] Voice input function: Parental voice is captured through a microphone.

[1189] Recording function: Saves captured audio as digital data.

[1190] Audio playback function: Plays audio data received from a server on the cloud.

[1191] Text display function: Displays text data received from a server on the cloud.

[1192] server

[1193] A server is a computing unit that runs on the cloud and has the following main functions:

[1194] Voice analysis function: Converts voice data sent from the device into text data.

[1195] Advice generation function: Generates appropriate parenting advice from analyzed voice data.

[1196] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[1197] Text generation function: Generates text data of advice and sends it to the terminal.

[1198] Fairy tale and lullaby selection function: Selects the most appropriate fairy tale or lullaby based on the child's age and sends it to the device.

[1199] User

[1200] Users, primarily parents, perform the following actions:

[1201] Voice input: Record parenting advice, fairy tales, and lullabies.

[1202] Audio playback request: Request playback of a fairy tale or lullaby that you have recorded yourself.

[1203] Receiving advice: Receive advice provided by the server in voice and text format.

[1204] Specific examples

[1205] Specific examples of child-rearing consultation

[1206] 1. The user speaks to the device and asks for advice on child-rearing. For example, "My child has been crying a lot lately. What should I do?"

[1207] 2. The device captures the audio and sends the digital data to the server.

[1208] 3. The server converts the speech into text and performs analysis.

[1209] 4. The server generates appropriate advice based on the analysis results and sends the advice to the device as text and audio data.

[1210] 5. The device plays the transmitted audio data and displays the text data on the display.

[1211] 6. Users receive advice and apply it to solve their parenting problems.

[1212] Examples of fairy tales and lullabies

[1213] 1. The user sits down at the device and starts recording a fairy tale. For example, say, "I'm going to read you the fairy tale of Momotaro."

[1214] 2. The device enters recording mode and records the user's voice.

[1215] 3. After the device finishes recording, it sends the digital data to the server and stores it.

[1216] 4. The user later requests, "Play the fairy tale of Momotaro."

[1217] 5. The server sends the saved audio data to the playback device.

[1218] 6. The device plays the recorded story and reads it to the child in the user's voice.

[1219] This system allows parents to receive advice 24 hours a day, 365 days a year to resolve their child-rearing concerns, and also deepens the bond with their children by having them listen to fairy tales and lullabies recorded in their own voices.

[1220] The processing flow will be explained below.

[1221] Childcare consultation process flow

[1222] Step 1:

[1223] The user can ask for parenting advice by voice, for example, "My child has been crying at night recently. What should I do?"

[1224] Step 2:

[1225] The device records the user's voice. It captures the input voice through a microphone and stores it as digital data.

[1226] Step 3:

[1227] The device sends the voice data to a server on the cloud, and the voice data is uploaded to the server via network communication.

[1228] Step 4:

[1229] The server converts the voice data into text data using a voice recognition API.

[1230] Step 5:

[1231] The server analyzes the text data and uses natural language processing (NLP) to understand the keywords and context within the text and analyze the content of the parenting consultation.

[1232] Step 6:

[1233] The server generates advice based on the analysis results. Using an AI model, it generates appropriate advice in text format. For example, "We recommend creating a relaxing environment and establishing a regular routine before bed."

[1234] Step 7:

[1235] The server converts the text data into audio data using a text-to-speech API.

[1236] Step 8:

[1237] The server transmits the generated voice data and text data to the terminal via network communication.

[1238] Step 9:

[1239] The device plays the audio and displays the text: it plays the audio data through the speaker and displays the text data on the display.

[1240] Processing flow for recording and playing back fairy tales in a parent's voice

[1241] Step 1:

[1242] The user gives a voice command to the terminal to "start recording a fairy tale." For example, the user says, "I will record the fairy tale of Momotaro."

[1243] Step 2:

[1244] The device switches to recording mode, starts the recording function, and prepares to record the user's voice.

[1245] Step 3:

[1246] The user reads a fairy tale aloud and the device records the audio, for example, "Once upon a time, there lived an old man and an old woman."

[1247] Step 4:

[1248] The terminal ends the recording. The recording stops due to a user instruction or a timeout.

[1249] Step 5:

[1250] The device sends the recorded data to a server on the cloud, and the audio data is uploaded to the server via network communication.

[1251] Step 6:

[1252] The server saves the recorded data and performs the necessary processing. The audio data is stored in storage and the data format is adjusted.

[1253] Step 7:

[1254] The user gives a voice command to the terminal to "start playing a fairy tale." For example, the user says, "Please play the fairy tale of Momotaro."

[1255] Step 8:

[1256] The server sends the recorded voice data to the terminal, and the saved voice data is provided to the terminal via the network.

[1257] Step 9:

[1258] The device plays the audio data, and the recorded story is played through the speaker for the child to listen to.

[1259] Relaxing music provision process flow

[1260] Step 1:

[1261] The user vocally requests relaxing music, for example, "Play some relaxing music."

[1262] Step 2:

[1263] The device records the audio and sends it to a server in the cloud. The audio data is captured and uploaded to the server via the network.

[1264] Step 3:

[1265] The server converts the voice into text data, and then uses a voice recognition API to convert the voice data into text data.

[1266] Step 4:

[1267] The server searches for appropriate relaxing music, analyzes the text data, and selects appropriate songs from the relaxing music library.

[1268] Step 5:

[1269] The server sends the selected music data to the device, and the music file is uploaded to the device via the network.

[1270] Step 6:

[1271] The device will play music, and the speaker will play the selected relaxing music.

[1272] These processing steps allow users to easily receive parenting advice, play recorded fairy tales and lullabies in their own voice, or request relaxing music to help reduce stress.

[1273] Example 1

[1274] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1275] Modern parents face many challenges and worries regarding child-rearing, but there is a lack of means to quickly and accurately resolve these issues. While it is useful for busy parents to record fairy tales and lullabies in their own voices for their children, the recording and playback methods are inconvenient. Furthermore, there is a lack of systems that can select and effectively play appropriate fairy tales and lullabies based on the child's age.

[1276] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1277] In this invention, the server includes means for acquiring voice uttered by a parent, means for converting the acquired voice into digital data, means for transmitting the digital data to a cloud-based computer system, means for analyzing the digital data stored in the cloud-based computer system and generating advice related to child-rearing consultations, means for returning the generated advice as voice and text to a playback terminal, means for playing the voice and displaying the text on the playback terminal, and means for the user to request audio playback from the cloud-based computer system. This allows parents to easily receive child-rearing consultations at any time and effectively use fairy tales and lullabies recorded in their own voices.

[1278] The "means for acquiring the voice uttered by the parent" refers to a device or apparatus that collects the voice uttered by the parent using a voice input device such as a microphone.

[1279] "Means for converting captured audio into digital data" refers to devices or software that convert analog audio signals into digital format.

[1280] "Means for transmitting digital data to a computer system on the cloud" refers to a communication method for transferring digital data from a terminal to a cloud server via the Internet.

[1281] "Means for analyzing digital data stored in a cloud-based computer system and generating advice related to childcare consultations" refers to a function that uses artificial intelligence and analytical algorithms to analyze digital data stored on a cloud server and generate advice related to childcare.

[1282] "Means for returning the generated advice to the playback terminal as audio and text" refers to a communication method for transmitting the advice generated by the cloud server to the playback device in the form of an audio file and a text file.

[1283] "Means for playing back audio and displaying text on a playback terminal" refers to the function of playing back audio data received in a playback device through a speaker and displaying text data on a display.

[1284] "Means for a user to make a request to a cloud computer system to play audio" refers to a communication means for a user to send an instruction to a cloud computer system to request audio playback.

[1285] "Means for recording sounds uttered by a parent" refers to an apparatus or method for recording sounds uttered by a parent using a recording device.

[1286] "Means for storing recorded audio as digital data in a computer system on the cloud" refers to a function for storing recorded audio data in digital format on a cloud server.

[1287] "Means for searching for digital data stored in a computer system on the cloud and transmitting it to a playback terminal at a specified time" refers to a function for searching for audio data stored on a cloud server and transferring it to a playback device at a time specified by the user.

[1288] "Means for selecting appropriate stories and lullabies based on the age of the child" refers to filtering and selection algorithms used to select content appropriate for the child's age.

[1289] "Means for transmitting selected stories and lullabies from a cloud-based computer system to a playback device" refers to a communication method for transferring selected audio data from a cloud server to a playback device.

[1290] "Means for playing back a story or lullaby on a playback terminal" refers to a function for playing back audio data of a story or lullaby received by a playback device on a speaker.

[1291] This system transmits the voices of parents to a cloud-based computer system, analyzes the voices, generates advice related to childcare consultations, and plays fairy tales and lullabies recorded by the user at appropriate times. This system is mainly composed of three main elements: a terminal, a server, and a user.

[1292] Hardware and software used

[1293] Device: The device includes a voice input function for capturing the voice spoken by the parent, a recording function, a voice playback function, and a text display function. Specific examples of this device include a voice input device and a device with a display. Specifically, this applies to smart speakers and smart devices with a display.

[1294] Server: The server is located in the cloud and has functions for voice analysis, advice generation, data storage, text generation, and selection of fairy tales and lullabies. Specific examples include platforms that provide cloud computing services (such as AWS and Google Cloud Platform). Amazon Transcribe and Google Cloud Speech-to-Text are used for voice analysis, and generative AI models (such as GPT-3 and ChatGPT) are used for text generation.

[1295] Details of data processing and calculation

[1296] 1. Audio capture and conversion:

[1297] Device: The device captures the user's voice and converts the analog voice signal into digital data. Specifically, smart speakers and smart devices with displays use voice input functionality to capture the user's spoken voice through a microphone and convert the voice into a digital format.

[1298] 2. Data transmission and analysis:

[1299] Terminal: The converted digital data is sent to a computer system in the cloud, using communication means via the Internet.

[1300] Server: The server converts the received digital data into text data using speech analysis software, for example, Amazon Transcribe or Google Cloud Speech-to-Text.

[1301] 3. Advice Generation:

[1302] Server: Based on the analyzed text data, a generative AI model (e.g., GPT-3 or ChatGPT) is used to generate appropriate advice related to childcare consultations.

[1303] 4. Data transmission and display:

[1304] Server: Sends the generated advice to the device as voice and text data.

[1305] Terminal: The terminal plays the received audio data and displays the text data on the display.

[1306] 5. Record and play fairy tales and lullabies:

[1307] Device: The user starts recording a fairy tale or lullaby, and the device goes into recording mode, collects the audio, and temporarily stores it as digital data.

[1308] Server: After recording is complete, the device sends the data to a cloud-based computer system, where it is securely stored. When a user requests playback, the server sends the stored data to the device, which then plays the audio data.

[1309] Specific use cases

[1310] Specific examples of child-rearing consultation

[1311] 1. The user speaks into the device and says, "My child has been crying at night lately. What should I do?"

[1312] 2. The device captures the audio and sends the digital data to the server.

[1313] 3. The server converts the speech into text and performs analysis.

[1314] 4. Based on the analysis results, the server generates advice such as, "To help your crying child relax at night, it would be a good idea to create a quiet environment and stick to a consistent bedtime."

[1315] 5. The device plays the transmitted audio data and displays the text data on the display.

[1316] Examples of fairy tales and lullabies

[1317] 1. The user approaches the device and says, "I'm going to read the fairy tale of Momotaro," and begins recording.

[1318] 2. The device enters recording mode and records the user's voice.

[1319] 3. After the device finishes recording, it sends the digital data to the server, where it is stored.

[1320] 4. The user later requests, "Play the fairy tale of Momotaro."

[1321] 5. The server sends the saved audio data to the device.

[1322] 6. The device plays the recorded story and reads it to the child in the user's voice.

[1323] Examples of prompt statements

[1324] "Recently, my child has been crying at night and it's bothering me. What should I do?"

[1325] "I want to record the Momotaro fairy tale so I can read it to you."

[1326] "Please give me some advice on how to get my child to sleep early."

[1327] This system allows parents to easily consult with their children at any time, and also makes it possible to effectively use fairy tales and lullabies recorded in their own voices.

[1328] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1329] Step 1:

[1330] The user's voice is input into the device. For example, the user might say, "My child has been crying at night recently and it's bothering me. What should I do?"

[1331] Input: User utterance

[1332] Output: Audio data captured by a microphone

[1333] Step 2:

[1334] The device converts the user's voice into digital data. The smart speaker uses its voice input function to convert analog voice into digital data.

[1335] Input: Captured audio data

[1336] Output: Digitized audio data

[1337] Step 3:

[1338] The device sends digital audio data to a server in the cloud, where it is sent as encrypted data over the internet.

[1339] Input: Digitized audio data

[1340] Output: Digital data sent to the server

[1341] Step 4:

[1342] The server converts the audio data into text using speech analysis software, such as Amazon Transcribe or Google Cloud Speech-to-Text.

[1343] Input: Digital data sent to the server

[1344] Output: Text data

[1345] Step 5:

[1346] The server analyzes the text data using a generative AI model and generates advice for the user's childcare consultation. The analysis uses a database of childcare-related information and a generative AI model (e.g., GPT-3 or ChatGPT).

[1347] Input: Text data

[1348] Output: Advice text and audio data

[1349] Step 6:

[1350] The server transmits the generated advice to the terminal via the Internet as voice data and text data.

[1351] Input: Advice text and audio data

[1352] Output: Advice data sent to the terminal

[1353] Step 7:

[1354] The device plays the advice audio data and displays the text data, while the smart speaker or device with a display plays the audio and displays the text on the display.

[1355] Input: Advice data sent to the terminal

[1356] Output: Played audio and displayed text

[1357] Step 8:

[1358] The user gives the command to start recording. For example, they say, "I'm going to read you the fairy tale of Momotaro."

[1359] Input: User utterance

[1360] Output: Start recording command

[1361] Step 9:

[1362] The device enters recording mode, recording the user's speech and temporarily saving it as digital data.

[1363] Input: Start recording

[1364] Output: Digitized recording data

[1365] Step 10:

[1366] After the device finishes recording, it sends the recorded data to a cloud server and stores it there. The server then securely stores the received data.

[1367] Input: Digitized recording data

[1368] Output: Data stored on a cloud server

[1369] Step 11:

[1370] The user makes a playback request, for example, "Play the fairy tale of Momotaro."

[1371] Input: User's playback request

[1372] Output: Playback request data

[1373] Step 12:

[1374] The server retrieves the stored voice data and sends it to the device, which then transfers the data over the Internet.

[1375] Input: Playback request data

[1376] Output: Audio data sent to the device

[1377] Step 13:

[1378] The device plays the audio data, and the recorded story is played on the speaker in the user's voice for the child to hear.

[1379] Input: Audio data sent to the device

[1380] Output: Played fairy tale audio

[1381] This will allow parents to easily record and play back childcare advice and fairy tales.

[1382] (Application example 1)

[1383] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1384] Conventional parenting support systems focus solely on analyzing parents' voice input and providing appropriate advice. However, there are no systems that can respond to security-related consultations or inquiries using similar voice recognition technology. There is a need for a system that can provide advice on both parenting and security in response to parents' spoken inquiries and manage them centrally. In addition, the generated advice must be provided in both voice and text format, allowing for flexible responses tailored to each user's usage situation.

[1385] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1386] In this invention, the server includes means for acquiring voice uttered by parents, means for converting the acquired voice into digital data, means for transmitting the digital data to a cloud server, means for analyzing the digital data stored in the cloud server and generating advice on child-rearing consultations and security, means for returning the generated advice to a playback terminal as voice and text, and means for playing the voice and displaying the text on the playback terminal, thereby making it possible to provide appropriate advice in a unified manner in response to questions from parents about child-rearing consultations and security.

[1387] The "means for acquiring parental vocalizations" is a device or system for collecting parental vocalizations in digital form.

[1388] The "means for converting captured audio into digital data" refers to a technique or device for converting audio signals into digital form.

[1389] The "means for transmitting digital data to a server on the cloud" refers to a technology or device for transmitting the converted digital data to a cloud server via the Internet.

[1390] "Means for analyzing digital data stored on a server on the cloud and generating advice on child-rearing consultations and security" refers to a technology or system that analyzes digital data stored on a cloud server and generates advice related to child-rearing and security based on the results.

[1391] The "means for returning the generated advice to the playback terminal as voice and text" refers to a technique and device for converting the generated advice into voice and text format and transmitting it to the playback terminal.

[1392] "Means for playing audio and displaying text at a playback terminal" refers to a device or system for playing received audio and displaying text.

[1393] A "means for recording parental vocalizations" is a device or technology for recording parental vocalizations.

[1394] "Means for storing recorded audio as digital data on a server on the cloud" refers to a method or system for storing recorded audio in a digital format on a cloud server.

[1395] "Means for searching for digital data stored on a server on the cloud and transmitting it to a playback terminal at a specified time" refers to a function or device for searching for data stored on a cloud server under specific conditions or at specific times and transmitting it to a playback terminal.

[1396] The "means for selecting appropriate fairy tales and lullabies based on the age of the child" refers to an algorithm or device that selects the most appropriate fairy tales and lullabies based on the age information of the child.

[1397] "Means for transmitting selected fairy tales and lullabies from a server on the cloud to a playback terminal" refers to means for transmitting data of selected fairy tales and lullabies from a cloud server to a playback terminal.

[1398] "Means for analyzing captured audio and generating security advice" refers to a system or software capable of analyzing audio data and generating appropriate security-related advice.

[1399] The "means for returning the generated security advice to the playback terminal as voice and text" refers to a technique or device that converts the generated security advice into voice and text format and sends it to the playback terminal.

[1400] To implement this invention, a system is required that captures the voices spoken by parents, transmits them to a cloud server, and generates appropriate advice on child-rearing and security. The system is broadly composed of three main elements: a terminal, a server, and a user.

[1401] Terminal

[1402] The device is a smart speaker or display device that captures the parent's voice. The device has the following features:

[1403] Voice input function: Parental voice is captured through a microphone.

[1404] Recording function: Saves captured audio as digital data.

[1405] Audio playback function: Plays audio data received from a server on the cloud.

[1406] Text display function: Displays text data received from a server on the cloud.

[1407] server

[1408] A server is a computing unit that runs on the cloud and has the following main functions:

[1409] Speech analysis function: Converts voice data sent from the device into text data. Uses the speech_recognition library.

[1410] Advice generation function: Generates appropriate parenting and security advice from the analyzed voice data. It uses a generative AI model with the transformers library.

[1411] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[1412] Text generation function: Generates text data for advice and sends it to the terminal. This uses the tflow library.

[1413] Fairy tale and lullaby selection function: Selects the most appropriate fairy tale or lullaby based on the child's age and sends it to the device.

[1414] User

[1415] Users, primarily parents, perform the following actions:

[1416] Voice input: Record parenting advice, security advice, fairy tales, lullabies, etc. For example, ask, "There's been a suspicious person hanging around recently. What should I do?"

[1417] Audio playback request: Request playback of a fairy tale or lullaby that you have recorded yourself.

[1418] Receive advice: Receive parenting and security advice provided by the server via voice and text.

[1419] Specific examples

[1420] Specific examples of childcare and security consultations

[1421] 1. A user speaks to a device and asks a question about parenting or security. For example, "My child has been crying a lot lately. What should I do?"

[1422] 2. The device captures the audio and sends the digital data to the server.

[1423] 3. The server converts the speech to text and performs the analysis. The speech_recognition library is used here.

[1424] 4. The server generates appropriate advice based on the analysis results and sends the advice to the device as text and audio data. This is where a generative AI model using the transformers library comes into play.

[1425] 5. The device plays the transmitted audio data and displays the text data on the display.

[1426] 6. The user receives the advice and applies it to solve the problem.

[1427] Prompt Sentence Examples

[1428] An example of a prompt to be input to the generative AI model is as follows:

[1429] "Generate appropriate security-related advice based on spoken questions. For example, what should I do if my neighbors are bothering me with noise?"

[1430] In this way, parents can quickly receive appropriate advice on child-rearing and security-related questions, creating a multi-functional support system that meets the needs of both parties.

[1431] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1432] Step 1:

[1433] The user speaks to the device and makes a voice input about childcare or security. For example, "My child has been crying at night recently. What should I do?" This voice is captured as a digital signal through the microphone. This is the state in which the input voice data has been captured.

[1434] Step 2:

[1435] The device picks up the voice and sends the digital data to a server in the cloud. Software on the device converts the voice data into packets and sends them to the server via the Internet. The input voice data is sent to the server.

[1436] Step 3:

[1437] The server converts the speech to text and performs analysis. The speech_recognition library is used to convert the speech data to text data. The input is speech data and the output is text data. The speech recognition engine analyzes the speech signal and converts it to text.

[1438] Step 4:

[1439] The server generates appropriate advice using a generative AI model based on the analysis results. The server uses the transformers library to input the analyzed text and the prompt sentence into the generative AI model, and outputs the optimal advice. The input is text data, and the output is text data of the advice.

[1440] Step 5:

[1441] The server returns the generated advice to the playback device as audio and text data. The generated advice text data is converted into audio data using the gTTS (Google Text to Speech) library and sent to the device via the Internet. The input is the advice text data, and the output is audio data and text data.

[1442] Step 6:

[1443] The terminal plays the transmitted voice data and displays the text data on the display. The built-in speaker of the terminal plays the voice data and the display displays the text data. The input is voice data and text data, and the output is voice and displayed text. This allows the user to receive advice.

[1444] These are the processing steps of the system that realizes this application example. The generated advice is useful for solving problems related to child rearing and security, and users can receive specific instructions and suggestions.

[1445] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1446] The present invention is a system that inputs the user's voice, analyzes the acquired voice, and provides appropriate advice, and also combines it with an emotion engine that recognizes the user's emotions and generates a response according to their emotional state. This enables more personalized and appropriate advice, improving the effectiveness of comprehensive child-rearing support.

[1447] Key Components

[1448] The system is broadly divided into four main components: the terminal, the server, the emotion engine, and the user.

[1449] Terminal

[1450] The device is a smart speaker or a device with a display that receives user voice and has the following functions:

[1451] Voice input function: Parental voice is captured through a microphone.

[1452] Recording function: Saves captured audio as digital data.

[1453] Audio playback function: Plays audio data received from a server on the cloud.

[1454] Text display function: Displays text data received from a server on the cloud.

[1455] server

[1456] A server is a computing unit that runs on the cloud and has the following main functions:

[1457] Voice analysis function: Converts voice data sent from the device into text data.

[1458] Advice generation function: Generates appropriate parenting advice from analyzed voice data.

[1459] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[1460] Text generation function: Generates text data of advice and sends it to the terminal.

[1461] Emotion data processing function: Analyzes the emotional data sent from the emotion engine and adjusts the advice content.

[1462] Emotion Engine

[1463] The emotion engine is a software component that recognizes emotions from user voice data and analyzes that information. It has the following functions:

[1464] Emotion recognition function: Recognizes emotional states from voice data.

[1465] Emotion data transmission function: Recognized emotion data is transmitted to the server.

[1466] Emotional feedback function: Tailor appropriate advice based on emotions.

[1467] User

[1468] Users, primarily parents, perform the following actions:

[1469] Voice input: Record parenting advice, fairy tales, and lullabies.

[1470] Audio playback request: Request playback of a fairy tale or lullaby that you have recorded yourself.

[1471] Receiving advice: Receive advice provided by the server in voice and text format.

[1472] Specific examples

[1473] Specific examples of child-rearing consultation

[1474] Example 1: A system that recognizes the user's emotional state and provides advice

[1475] 1. The user speaks to the device and asks for advice on child-rearing. For example, "My child has been crying a lot lately. What should I do?"

[1476] 2. The device captures the voice and sends the digital data to the emotion engine.

[1477] 3. The emotion engine analyzes the voice data to recognize the user's emotional state, for example, recognizing that the user is feeling stressed.

[1478] 4. The emotion engine sends the emotion data to the server.

[1479] 5. The server analyzes the emotional and voice data and generates appropriate advice based on the child's emotional state. For example, it might say, "To reduce stress factors for your child, we recommend creating a relaxing bedtime routine."

[1480] 6. The server converts the generated advice into voice and text data and sends it to the device.

[1481] 7. The device plays the audio and displays the text.

[1482] 8. Users receive advice and apply it to solve their parenting problems.

[1483] Examples of fairy tales and lullabies

[1484] Example 2: Reading fairy tales using an emotion engine

[1485] 1. The user sits down at the device and starts recording a fairy tale. For example, say, "I'm going to read you the fairy tale of Momotaro."

[1486] 2. The device enters recording mode and records the user's voice.

[1487] 3. After the device finishes recording, it sends the digital data to the server and stores it.

[1488] 4. The emotion engine analyzes the recording data and attaches emotional information.

[1489] 5. The user later requests, "Play the fairy tale of Momotaro."

[1490] 6. The server selects the recorded voice data taking into consideration the attached emotional information and sends it to the terminal.

[1491] 7. The device plays the audio data and plays it to the child in the user's voice.

[1492] The system can recognize the user's emotions and provide appropriate advice based on those emotions, enabling more personalized and effective parenting support. It can also deepen the bond between parents and their children by playing fairy tales and lullabies recorded in their own voices.

[1493] The processing flow will be explained below.

[1494] A process flow for parenting consultations that combines an emotion engine

[1495] Step 1:

[1496] The user can ask for parenting advice by voice, for example, "My child has been crying at night recently. What should I do?"

[1497] Step 2:

[1498] The device records the user's voice. It captures the input voice through a microphone and stores it as digital data.

[1499] Step 3:

[1500] The device sends the recorded voice data to the emotion engine, which then uploads it to the emotion engine via network communication.

[1501] Step 4:

[1502] The emotion engine analyzes the voice data and recognizes the user's emotional state. For example, it may recognize that the user is feeling stressed.

[1503] Step 5:

[1504] The emotion engine sends the recognized emotion data to the server, where the voice data and emotion data are combined.

[1505] Step 6:

[1506] The server converts the voice data into text data using a voice recognition API.

[1507] Step 7:

[1508] The server analyzes the text data and emotional data, using natural language processing (NLP) to understand the relationship between keywords and context in the text and emotions, and analyzes the content of the parenting consultation.

[1509] Step 8:

[1510] The server generates advice based on the analysis results. Using an AI model that reflects emotional data, appropriate advice is generated in text format. For example, it might say, "To reduce the factors that cause your child stress, we recommend creating a relaxing bedtime routine."

[1511] Step 9:

[1512] The server converts the text data into audio data using a text-to-speech API.

[1513] Step 10:

[1514] The server transmits the generated voice data and text data to the terminal via network communication.

[1515] Step 11:

[1516] The device plays the audio and displays the text: it plays the audio data through the speaker and displays the text data on the display.

[1517] Processing flow for recording and playing fairy tales using the emotion engine

[1518] Step 1:

[1519] The user approaches the device and begins recording a fairy tale. For example, they say, "I'm going to read you the fairy tale of Momotaro."

[1520] Step 2:

[1521] The device switches to recording mode, starts the recording function, and prepares to record the user's voice.

[1522] Step 3:

[1523] The user reads a fairy tale aloud and the device records the audio, for example, "Once upon a time, there lived an old man and an old woman."

[1524] Step 4:

[1525] The terminal ends the recording. The recording stops due to a user instruction or a timeout.

[1526] Step 5:

[1527] The device sends the recorded voice data to the emotion engine, which then uploads it to the emotion engine via network communication.

[1528] Step 6:

[1529] The emotion engine analyzes the audio data and recognizes the emotional state of the recording.

[1530] Step 7:

[1531] The emotion engine sends the recognized emotion data to the server, where the voice data and emotion data are combined.

[1532] Step 8:

[1533] The server stores the recording data and stores it safely in the cloud.

[1534] Step 9:

[1535] The user later gives a voice command such as "Play the fairy tale of Momotaro."

[1536] Step 10:

[1537] The server searches for the stored voice data and emotion data and sends them to the device at the specified time.

[1538] Step 11:

[1539] The device plays back the recorded fairy tale audio data in the user's voice, and the audio is played back through the speaker so that the child can listen to it in the user's voice.

[1540] Processing flow for providing relaxing music using an emotion engine

[1541] Step 1:

[1542] The user vocally requests relaxing music, for example, "Play some relaxing music."

[1543] Step 2:

[1544] The device records the voice and sends it to the emotion engine, which then uploads it to the emotion engine via network communication.

[1545] Step 3:

[1546] The emotion engine analyzes the voice data and recognizes the user's emotional state.

[1547] Step 4:

[1548] The emotion engine sends the recognized emotion data to the server, where the voice data and emotion data are combined.

[1549] Step 5:

[1550] The server converts the speech into text and selects appropriate relaxing music taking into account emotional data.

[1551] Step 6:

[1552] The server sends the selected music data to the device, which then uploads the music data to the device via the network.

[1553] Step 7:

[1554] The device will play music, and the speaker will play the selected relaxing music.

[1555] These processing steps allow users to easily receive parenting advice, play fairy tales or lullabies recorded in their own voice, or request relaxing music to help reduce stress. The combination of an emotion engine provides more personalized assistance based on the user's emotional state.

[1556] Example 2

[1557] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1558] In today's world, many parents face the burdens of work and family life, requiring them to quickly receive appropriate parenting advice. However, conventional parenting support systems have struggled to accurately recognize parents' emotional states and provide personalized advice based on those states. In particular, when parents are stressed or tired, general advice that ignores their emotional state has limited effectiveness and does not contribute to caring for the parents' physical and mental health. A system that solves this problem and provides more effective and compassionate parenting support is needed.

[1559] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for transmitting digital data to a cloud server, a means for analyzing the digital data stored in the cloud server and recognizing the emotional state of the user, and a means for generating advice related to child-rearing consultation according to the emotional state. This makes it possible to provide personalized advice according to the emotional state of the parent.

[1560] "Voice spoken by the parent" refers to the voice spoken by the parent into the device for conversation, consultation, or recording.

[1561] "Digital data" means information such as voice or text that has been converted into a form that can be processed, stored, or transmitted electronically.

[1562] A "cloud server" is an online data processing and storage system provided via the Internet by computers located in remote locations.

[1563] An "emotion engine" is a software component that analyzes and recognizes a user's emotions and psychological state from their voice and text data.

[1564] A "voice analysis engine" is software or hardware that has the function of converting voice data into text data.

[1565] An "advice generation engine" is software or hardware that has the function of generating appropriate advice based on input data.

[1566] A "playback terminal" is a device for playing and displaying audio and text data sent from a server on the cloud.

[1567] "Users" are those who use the system to receive consultations, recordings, and advice on child-rearing, and are primarily parents.

[1568] "Childcare consultation" refers to the act of parents seeking professional advice regarding questions or problems they have regarding childcare.

[1569] "Emotional state" refers to the psychological state or emotion expressed by the user through voice or text, and includes, for example, stress, joy, anger, etc.

[1570] The "audio playback function" is a function that plays back audio data so that the user can hear it.

[1571] "Text display function" refers to the function of displaying text data on a display or other display device.

[1572] The present invention is a system that inputs the user's voice, analyzes the acquired voice, and provides appropriate advice, and also combines it with an emotion engine that recognizes the user's emotions and generates a response according to their emotional state. This system enables more personalized and appropriate advice, improving the effectiveness of comprehensive child-rearing support.

[1573] Key Components

[1574] The system is broadly divided into four main components: the terminal, the server, the emotion engine, and the user.

[1575] Terminal

[1576] The device is a smart speaker or a device with a display that receives user voice and has the following functions:

[1577] Voice input function: Captures the user's voice through a microphone.

[1578] Recording function: Saves captured audio as digital data.

[1579] Audio playback function: Plays audio data received from a server on the cloud.

[1580] Text display function: Displays text data received from a server on the cloud.

[1581] server

[1582] A server is a computing unit that runs on the cloud and has the following main functions:

[1583] Voice analysis function: Converts voice data sent from the device into text data.

[1584] Emotion recognition function: Analyze the user's emotional state through the emotion engine.

[1585] Advice generation function: Generates appropriate parenting advice based on analyzed voice data and emotional data.

[1586] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[1587] Text generation function: Generates text data of advice and sends it to the terminal.

[1588] Emotion Engine

[1589] The emotion engine is a software component that recognizes emotions from user voice data and analyzes that information. It has the following functions:

[1590] Emotion Recognition: Recognizes emotional states from speech data. It uses natural language processing techniques and applies a BERT-based sentiment analysis model.

[1591] Emotion data transmission function: Recognized emotion data is transmitted to the server.

[1592] User

[1593] Users, primarily parents, perform the following actions:

[1594] Voice input: Record parenting advice, fairy tales, and lullabies.

[1595] Audio playback request: Requests playback of a recorded fairy tale or lullaby.

[1596] Receiving advice: Receive advice provided by the server in voice and text format.

[1597] Specific examples

[1598] Specific examples of child-rearing consultation

[1599] 1. The user speaks to the device and asks for advice on child-rearing. For example, "My child has been crying a lot lately. What should I do?"

[1600] 2. The device captures the audio and sends the digital data to the server.

[1601] 3. The server uses a speech analysis engine to analyze the voice data and convert it into text data.

[1602] 4. The server recognizes the user's emotional state through an emotion engine and identifies that the user is feeling stressed.

[1603] 5. The server generates appropriate advice based on the emotion and voice data. For example, "To reduce stress factors for your child, we recommend creating a relaxing bedtime routine."

[1604] 6. The server converts the generated advice into voice and text data and sends it to the device.

[1605] 7. The device plays the audio and displays the text.

[1606] Examples of fairy tales and lullabies

[1607] 1. The user sits down at the device and starts recording a fairy tale. For example, say, "I'm going to read you the fairy tale of Momotaro."

[1608] 2. The device enters recording mode and records the user's voice.

[1609] 3. After the device finishes recording, it sends the digital data to the server and stores it.

[1610] 4. The emotion engine analyzes the recording data and attaches emotional information.

[1611] 5. The user later requests, "Play the fairy tale of Momotaro."

[1612] 6. The server selects the recorded voice data taking into consideration the attached emotional information and sends it to the terminal.

[1613] 7. The device plays the audio data and plays it to the child in the user's voice.

[1614] Example prompt sentence:

[1615] My child has been crying at night recently, what should I do?

[1616] "Play the fairy tale of Momotaro"

[1617] This system can provide appropriate advice based on the user's emotional state, enabling more personalized and effective parenting support. It can also deepen the bond between parents and their children by playing fairy tales and lullabies recorded in their own voices.

[1618] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1619] Step 1: Voice Input

[1620] The user speaks into the device to ask questions or ask for advice about child-rearing. For example, "My child has been crying at night recently. What should I do?"

[1621] The input is the user's voice data.

[1622] The device picks up audio using a built-in microphone, and performs noise reduction and echo cancellation to capture clear audio data.

[1623] The output is audio data converted into digital form.

[1624] Step 2: Sending audio data

[1625] The device converts the captured audio data into a digital format, specifically, converting the audio into 16-bit PCM format, compressing the data, and packetizing it.

[1626] The input is the converted digital audio data.

[1627] The device then sends the converted digital audio data to a cloud server using HTTPS to ensure data security.

[1628] The output is the audio data received by the server on the cloud.

[1629] Step 3: Analyzing the audio data

[1630] The server receives the voice data sent from the terminal.

[1631] The input is voice data sent to a server on the cloud.

[1632] The server uses a speech analysis engine to convert the audio data into text data, using the Google Cloud Speech-to-Text API.

[1633] The output is text data converted from the audio data.

[1634] Step 4: Recognize emotions

[1635] The server passes the converted text data to the emotion engine.

[1636] The input is text data.

[1637] The emotion engine uses natural language processing technology to analyze the user's emotional state from text data. It uses a BERT-based sentiment analysis model to recognize users' stress, joy, anger, etc.

[1638] The output is the recognized emotion data.

[1639] Step 5: Generating Advice

[1640] The server integrates the emotion data sent from the emotion engine with the text data.

[1641] The input is emotion data and text data.

[1642] The server uses an advice generation engine to generate appropriate advice using a GPT-based generative AI model, such as "To reduce stress factors for your child, we recommend creating a relaxing bedtime routine."

[1643] The output is the generated advice.

[1644] Step 6: Send and display advice data

[1645] The server converts the generated advice into both text and audio data.

[1646] The input is the generated advice.

[1647] The server transmits the converted data to the terminal.

[1648] The device analyzes the received data, plays back the voice data, and displays the text data on the display, using a voice playback engine and a text-to-speech (TTS) engine.

[1649] The output is a spoken advice and a displayed text advice.

[1650] Step 7: Take and apply advice

[1651] The user receives the advice provided by the device and applies it in real life.

[1652] The inputs are spoken advice and displayed text advice.

[1653] The hope is that by following the advice, users will find their parenting problems alleviated.

[1654] The output is the effect of the problem mitigation.

[1655] (Application example 2)

[1656] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1657] Conventional advertising delivery systems deliver advertisements uniformly without considering the user's emotional state, making it difficult to provide a personalized advertising experience that reflects the user's interests and preferences. Similarly, parenting support systems provide advice without understanding the parent's emotional state, resulting in an inability to provide effective support.

[1658] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1659] In this invention, the server includes means for recognizing an emotional state from acquired voice data and facial expression data, means for adjusting the content of advice based on the recognized emotional state, and means for selecting and displaying advertisements in accordance with the emotional state recognized from the acquired voice data and facial expression data, thereby making it possible to analyze the user's emotional state in real time and provide optimal advertisements and advice.

[1660] "Voice data" is a digital representation of a user's voice.

[1661] "Facial expression data" refers to facial expressions of a user captured using a camera and converted into an analyzable format.

[1662] The "emotional state" refers to the user's emotions analyzed based on voice data and facial expression data. Specifically, emotions include joy, sadness, anger, fear, surprise, and excitement.

[1663] "Means for recognizing" includes software and hardware configurations for analyzing voice data and facial expression data to identify the emotional state of the user.

[1664] The "means for adjusting the content of advice" includes algorithms and programs for generating or modifying appropriate advice for a user based on the perceived emotional state.

[1665] "Means for selecting advertisements" includes software and hardware configurations for selecting the most appropriate advertisements for a user based on the perceived emotional state and other relevant information.

[1666] The term "reproducing means" includes a device and software configuration for reproducing audio or video data received from a server on a reproducing terminal.

[1667] "Storage means" includes software and hardware configurations for safely storing acquired voice data and facial expression data in digital format on a server on the cloud.

[1668] This invention is a system that acquires and analyzes a user's voice and facial expression data to provide appropriate advertisements and advice. The system is mainly composed of four main elements: a terminal, a server, an emotion recognition engine, and the user.

[1669] Terminal

[1670] The terminal is a mobile device such as a smartphone or smart glasses that has the following capabilities:

[1671] Voice and facial expression input function: Uses a microphone and camera to capture the user's voice and facial expression data.

[1672] Recording function: Saves captured audio as digital data.

[1673] Audio playback and display function: Plays back and displays data sent from the server on the cloud.

[1674] server

[1675] A server is a computing unit that runs on the cloud and has the following main functions:

[1676] Voice and facial expression analysis function: Analyzes voice and facial expression data sent from the device and converts it into text data.

[1677] Advertisement and advice generation function: Recognizes the user's emotional state from the analyzed data and generates appropriate advertisements and advice.

[1678] Data storage function: Recorded voice data and facial expression data are safely stored.

[1679] Data transmission function: Sends generated advertisements and advice to the device.

[1680] Emotion Recognition Engine

[1681] The emotion recognition engine is a software component that recognizes emotions from the user's voice data and facial expression data and provides that information. It has the following functions:

[1682] Emotion recognition function: Recognizes emotional states from voice data and facial expression data.

[1683] Emotion data transmission function: Recognized emotion data is transmitted to the server.

[1684] Emotional feedback: Tailor appropriate ads and advice based on emotions.

[1685] User

[1686] Users are primarily consumers and parents who receive advertising and perform the following actions:

[1687] Voice and facial expression input: Voice and facial expression data are input through questions and conversations.

[1688] Receiving advertisements and advice: Receive advertisements and advice provided by the server via voice and text.

[1689] Specific processing flow

[1690] 1. The user speaks into the device and shows their facial expressions to the camera.

[1691] 2. The device acquires voice and facial expression data and sends it as digital data to a cloud server.

[1692] 3. The server receives the acquired data, converts it into voice and analyzes facial expressions, and sends it to the emotion recognition engine.

[1693] 4. The emotion recognition engine recognizes the emotional state and sends the data back to the server.

[1694] 5. The server generates appropriate advertisements or advice based on the emotional state and sends them back to the device.

[1695] 6. The device provides the user with the sent advertisements and advice via voice and text.

[1696] Use cases and input prompts for generative AI models

[1697] A specific example of use could be to display advertisements for luxury goods when the user looks satisfied, and to display soothing advertisements when the user is feeling stressed.

[1698] Example prompts to input to a generative AI model:

[1699] Create an application for smartphones or smart glasses that recognizes emotions from a user's voice and facial expression data, and selects and displays the most appropriate advertisement based on that data. Convert the voice data into text and analyze the emotions. Select an advertisement based on the analyzed emotional data and display it on the user's display.

[1700] This allows users to receive advertisements and advice that match their emotions at the time, making it possible to provide more personalized and effective services.

[1701] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1702] Step 1:

[1703] The user inputs voice into the terminal and shows facial expressions to the camera.

[1704] Input: User's voice and facial expressions.

[1705] Processing: Capture voice and facial expression data using the device's microphone and camera.

[1706] Output: Acquired voice and facial expression data.

[1707] Step 2:

[1708] The voice data and facial expression data acquired by the device are converted into digital format and sent to a server on the cloud.

[1709] Input: Analog voice data and facial expression data acquired on the device.

[1710] Processing: Analog audio is converted to digital format through signal processing, and facial expression data is converted to digital format through image processing. The data is then sent to a cloud server.

[1711] Output: Digital voice and facial expression data in a format usable by the cloud server.

[1712] Step 3:

[1713] The server converts the received voice data into text data and analyzes the facial expression data to recognize the emotional state.

[1714] Input: Digital voice data and facial expression data received by the cloud server.

[1715] Processing: A speech recognition algorithm is used to convert the voice data into text data, and an emotion recognition engine is used to analyze emotions from facial expression data.

[1716] Output: Text data and emotional state data.

[1717] Step 4:

[1718] The server generates appropriate advertisements or advice based on the analysis results.

[1719] Input: Text data and emotional state data.

[1720] Processing: Based on the sentiment data, an advice generation algorithm or ad selection engine is used to select the best ad or advice.

[1721] Output: Selected ads or generated advice.

[1722] Step 5:

[1723] The server sends the generated advertisement or advice to the terminal, which plays and displays it.

[1724] Input: Selected Advertisement or Generated Advice Data.

[1725] Processing: Converting the data into the appropriate format and sending it to the device, where it plays the audio and displays the text.

[1726] Output: Audio and text feedback to the user.

[1727] Step 6:

[1728] The user checks the advertisements and advice provided and re-enters them if necessary.

[1729] Input: Advertisements or advice provided by the device.

[1730] Action: Check the content of the advertisement or advice, and provide feedback or re-enter it.

[1731] Output: Return to step 1 to acquire the user's voice and facial expressions again.

[1732] This will enable the provision of personalized advertisements and advice in real time according to the user's emotional state. Each step uses hardware such as smartphones and smart glasses, and software such as voice recognition algorithms, emotion recognition engines, and ad selection engines.

[1733] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1734] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1735] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1736] [Fourth embodiment]

[1737] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1738] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1739] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1740] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1741] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1742] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1743] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1744] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1745] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1746] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1747] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1748] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1749] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1750] The present invention is a system that inputs the voice of a parent to provide advice on child-rearing, analyzes the acquired voice, and provides appropriate advice. It also has a function that stores the voice data recorded by the parent on the cloud and plays it back to the child as a fairy tale or lullaby at a specified time. Specific embodiments for implementing this system are described below.

[1751] The system is broadly composed of three main elements: terminals, servers, and users.

[1752] Terminal

[1753] The device is a smart speaker or display device that captures the parent's voice. The device has the following features:

[1754] Voice input function: Parental voice is captured through a microphone.

[1755] Recording function: Saves captured audio as digital data.

[1756] Audio playback function: Plays audio data received from a server on the cloud.

[1757] Text display function: Displays text data received from a server on the cloud.

[1758] server

[1759] A server is a computing unit that runs on the cloud and has the following main functions:

[1760] Voice analysis function: Converts voice data sent from the device into text data.

[1761] Advice generation function: Generates appropriate parenting advice from analyzed voice data.

[1762] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[1763] Text generation function: Generates text data of advice and sends it to the terminal.

[1764] Fairy tale and lullaby selection function: Selects the most appropriate fairy tale or lullaby based on the child's age and sends it to the device.

[1765] User

[1766] Users, primarily parents, perform the following actions:

[1767] Voice input: Record parenting advice, fairy tales, and lullabies.

[1768] Audio playback request: Request playback of a fairy tale or lullaby that you have recorded yourself.

[1769] Receiving advice: Receive advice provided by the server in voice and text format.

[1770] Specific examples

[1771] Specific examples of child-rearing consultation

[1772] 1. The user speaks to the device and asks for advice on child-rearing. For example, "My child has been crying a lot lately. What should I do?"

[1773] 2. The device captures the audio and sends the digital data to the server.

[1774] 3. The server converts the speech into text and performs analysis.

[1775] 4. The server generates appropriate advice based on the analysis results and sends the advice to the device as text and audio data.

[1776] 5. The device plays the transmitted audio data and displays the text data on the display.

[1777] 6. Users receive advice and apply it to solve their parenting problems.

[1778] Examples of fairy tales and lullabies

[1779] 1. The user sits down at the device and starts recording a fairy tale. For example, say, "I'm going to read you the fairy tale of Momotaro."

[1780] 2. The device enters recording mode and records the user's voice.

[1781] 3. After the device finishes recording, it sends the digital data to the server and stores it.

[1782] 4. The user later requests, "Play the fairy tale of Momotaro."

[1783] 5. The server sends the saved audio data to the playback device.

[1784] 6. The device plays the recorded story and reads it to the child in the user's voice.

[1785] This system allows parents to receive advice 24 hours a day, 365 days a year to resolve their child-rearing concerns, and also deepens the bond with their children by having them listen to fairy tales and lullabies recorded in their own voices.

[1786] The processing flow will be explained below.

[1787] Childcare consultation process flow

[1788] Step 1:

[1789] The user can ask for parenting advice by voice, for example, "My child has been crying at night recently. What should I do?"

[1790] Step 2:

[1791] The device records the user's voice. It captures the input voice through a microphone and stores it as digital data.

[1792] Step 3:

[1793] The device sends the voice data to a server on the cloud, and the voice data is uploaded to the server via network communication.

[1794] Step 4:

[1795] The server converts the voice data into text data using a voice recognition API.

[1796] Step 5:

[1797] The server analyzes the text data and uses natural language processing (NLP) to understand the keywords and context within the text and analyze the content of the parenting consultation.

[1798] Step 6:

[1799] The server generates advice based on the analysis results. Using an AI model, it generates appropriate advice in text format. For example, "We recommend creating a relaxing environment and establishing a regular routine before bed."

[1800] Step 7:

[1801] The server converts the text data into audio data using a text-to-speech API.

[1802] Step 8:

[1803] The server transmits the generated voice data and text data to the terminal via network communication.

[1804] Step 9:

[1805] The device plays the audio and displays the text: it plays the audio data through the speaker and displays the text data on the display.

[1806] Processing flow for recording and playing back fairy tales in a parent's voice

[1807] Step 1:

[1808] The user gives a voice command to the terminal to "start recording a fairy tale." For example, the user says, "I will record the fairy tale of Momotaro."

[1809] Step 2:

[1810] The device switches to recording mode, starts the recording function, and prepares to record the user's voice.

[1811] Step 3:

[1812] The user reads a fairy tale aloud and the device records the audio, for example, "Once upon a time, there lived an old man and an old woman."

[1813] Step 4:

[1814] The terminal ends the recording. The recording stops due to a user instruction or a timeout.

[1815] Step 5:

[1816] The device sends the recorded data to a server on the cloud, and the audio data is uploaded to the server via network communication.

[1817] Step 6:

[1818] The server saves the recorded data and performs the necessary processing. The audio data is stored in storage and the data format is adjusted.

[1819] Step 7:

[1820] The user gives a voice command to the terminal to "start playing a fairy tale." For example, the user says, "Please play the fairy tale of Momotaro."

[1821] Step 8:

[1822] The server sends the recorded voice data to the terminal, and the saved voice data is provided to the terminal via the network.

[1823] Step 9:

[1824] The device plays the audio data, and the recorded story is played through the speaker for the child to listen to.

[1825] Relaxing music provision process flow

[1826] Step 1:

[1827] The user vocally requests relaxing music, for example, "Play some relaxing music."

[1828] Step 2:

[1829] The device records the audio and sends it to a server in the cloud. The audio data is captured and uploaded to the server via the network.

[1830] Step 3:

[1831] The server converts the voice into text data, and then uses a voice recognition API to convert the voice data into text data.

[1832] Step 4:

[1833] The server searches for appropriate relaxing music, analyzes the text data, and selects appropriate songs from the relaxing music library.

[1834] Step 5:

[1835] The server sends the selected music data to the device, and the music file is uploaded to the device via the network.

[1836] Step 6:

[1837] The device will play music, and the speaker will play the selected relaxing music.

[1838] These processing steps allow users to easily receive parenting advice, play recorded fairy tales and lullabies in their own voice, or request relaxing music to help reduce stress.

[1839] Example 1

[1840] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1841] Modern parents face many challenges and worries regarding child-rearing, but there is a lack of means to quickly and accurately resolve these issues. While it is useful for busy parents to record fairy tales and lullabies in their own voices for their children, the recording and playback methods are inconvenient. Furthermore, there is a lack of systems that can select and effectively play appropriate fairy tales and lullabies based on the child's age.

[1842] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1843] In this invention, the server includes means for acquiring voice uttered by a parent, means for converting the acquired voice into digital data, means for transmitting the digital data to a cloud-based computer system, means for analyzing the digital data stored in the cloud-based computer system and generating advice related to child-rearing consultations, means for returning the generated advice as voice and text to a playback terminal, means for playing the voice and displaying the text on the playback terminal, and means for the user to request audio playback from the cloud-based computer system. This allows parents to easily receive child-rearing consultations at any time and effectively use fairy tales and lullabies recorded in their own voices.

[1844] The "means for acquiring the voice uttered by the parent" refers to a device or apparatus that collects the voice uttered by the parent using a voice input device such as a microphone.

[1845] "Means for converting captured audio into digital data" refers to devices or software that convert analog audio signals into digital format.

[1846] "Means for transmitting digital data to a computer system on the cloud" refers to a communication method for transferring digital data from a terminal to a cloud server via the Internet.

[1847] "Means for analyzing digital data stored in a cloud-based computer system and generating advice related to childcare consultations" refers to a function that uses artificial intelligence and analytical algorithms to analyze digital data stored on a cloud server and generate advice related to childcare.

[1848] "Means for returning the generated advice to the playback terminal as audio and text" refers to a communication method for transmitting the advice generated by the cloud server to the playback device in the form of an audio file and a text file.

[1849] "Means for playing back audio and displaying text on a playback terminal" refers to the function of playing back audio data received in a playback device through a speaker and displaying text data on a display.

[1850] "Means for a user to make a request to a cloud computer system to play audio" refers to a communication means for a user to send an instruction to a cloud computer system to request audio playback.

[1851] "Means for recording sounds uttered by a parent" refers to an apparatus or method for recording sounds uttered by a parent using a recording device.

[1852] "Means for storing recorded audio as digital data in a computer system on the cloud" refers to a function for storing recorded audio data in digital format on a cloud server.

[1853] "Means for searching for digital data stored in a computer system on the cloud and transmitting it to a playback terminal at a specified time" refers to a function for searching for audio data stored on a cloud server and transferring it to a playback device at a time specified by the user.

[1854] "Means for selecting appropriate stories and lullabies based on the age of the child" refers to filtering and selection algorithms used to select content appropriate for the child's age.

[1855] "Means for transmitting selected stories and lullabies from a cloud-based computer system to a playback device" refers to a communication method for transferring selected audio data from a cloud server to a playback device.

[1856] "Means for playing back a story or lullaby on a playback terminal" refers to a function for playing back audio data of a story or lullaby received by a playback device on a speaker.

[1857] This system transmits the voices of parents to a cloud-based computer system, analyzes the voices, generates advice related to childcare consultations, and plays fairy tales and lullabies recorded by the user at appropriate times. This system is mainly composed of three main elements: a terminal, a server, and a user.

[1858] Hardware and software used

[1859] Device: The device includes a voice input function for capturing the voice spoken by the parent, a recording function, a voice playback function, and a text display function. Specific examples of this device include a voice input device and a device with a display. Specifically, this applies to smart speakers and smart devices with a display.

[1860] Server: The server is located in the cloud and has functions for voice analysis, advice generation, data storage, text generation, and selection of fairy tales and lullabies. Specific examples include platforms that provide cloud computing services (such as AWS and Google Cloud Platform). Amazon Transcribe and Google Cloud Speech-to-Text are used for voice analysis, and generative AI models (such as GPT-3 and ChatGPT) are used for text generation.

[1861] Details of data processing and calculation

[1862] 1. Audio capture and conversion:

[1863] Device: The device captures the user's voice and converts the analog voice signal into digital data. Specifically, smart speakers and smart devices with displays use voice input functionality to capture the user's spoken voice through a microphone and convert the voice into a digital format.

[1864] 2. Data transmission and analysis:

[1865] Terminal: The converted digital data is sent to a computer system in the cloud, using communication means via the Internet.

[1866] Server: The server converts the received digital data into text data using speech analysis software, for example, Amazon Transcribe or Google Cloud Speech-to-Text.

[1867] 3. Advice Generation:

[1868] Server: Based on the analyzed text data, a generative AI model (e.g., GPT-3 or ChatGPT) is used to generate appropriate advice related to childcare consultations.

[1869] 4. Data transmission and display:

[1870] Server: Sends the generated advice to the device as voice and text data.

[1871] Terminal: The terminal plays the received audio data and displays the text data on the display.

[1872] 5. Record and play fairy tales and lullabies:

[1873] Device: The user starts recording a fairy tale or lullaby, and the device goes into recording mode, collects the audio, and temporarily stores it as digital data.

[1874] Server: After recording is complete, the device sends the data to a cloud-based computer system, where it is securely stored. When a user requests playback, the server sends the stored data to the device, which then plays the audio data.

[1875] Specific use cases

[1876] Specific examples of child-rearing consultation

[1877] 1. The user speaks into the device and says, "My child has been crying at night lately. What should I do?"

[1878] 2. The device captures the audio and sends the digital data to the server.

[1879] 3. The server converts the speech into text and performs analysis.

[1880] 4. Based on the analysis results, the server generates advice such as, "To help your crying child relax at night, it would be a good idea to create a quiet environment and stick to a consistent bedtime."

[1881] 5. The device plays the transmitted audio data and displays the text data on the display.

[1882] Examples of fairy tales and lullabies

[1883] 1. The user approaches the device and says, "I'm going to read the fairy tale of Momotaro," and begins recording.

[1884] 2. The device enters recording mode and records the user's voice.

[1885] 3. After the device finishes recording, it sends the digital data to the server, where it is stored.

[1886] 4. The user later requests, "Play the fairy tale of Momotaro."

[1887] 5. The server sends the saved audio data to the device.

[1888] 6. The device plays the recorded story and reads it to the child in the user's voice.

[1889] Examples of prompt statements

[1890] "Recently, my child has been crying at night and it's bothering me. What should I do?"

[1891] "I want to record the Momotaro fairy tale so I can read it to you."

[1892] "Please give me some advice on how to get my child to sleep early."

[1893] This system allows parents to easily consult with their children at any time, and also makes it possible to effectively use fairy tales and lullabies recorded in their own voices.

[1894] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1895] Step 1:

[1896] The user's voice is input into the device. For example, the user might say, "My child has been crying at night recently and it's bothering me. What should I do?"

[1897] Input: User utterance

[1898] Output: Audio data captured by a microphone

[1899] Step 2:

[1900] The device converts the user's voice into digital data. The smart speaker uses its voice input function to convert analog voice into digital data.

[1901] Input: Captured audio data

[1902] Output: Digitized audio data

[1903] Step 3:

[1904] The device sends digital audio data to a server in the cloud, where it is sent as encrypted data over the internet.

[1905] Input: Digitized audio data

[1906] Output: Digital data sent to the server

[1907] Step 4:

[1908] The server converts the audio data into text using speech analysis software, such as Amazon Transcribe or Google Cloud Speech-to-Text.

[1909] Input: Digital data sent to the server

[1910] Output: Text data

[1911] Step 5:

[1912] The server analyzes the text data using a generative AI model and generates advice for the user's childcare consultation. The analysis uses a database of childcare-related information and a generative AI model (e.g., GPT-3 or ChatGPT).

[1913] Input: Text data

[1914] Output: Advice text and audio data

[1915] Step 6:

[1916] The server transmits the generated advice to the terminal via the Internet as voice data and text data.

[1917] Input: Advice text and audio data

[1918] Output: Advice data sent to the terminal

[1919] Step 7:

[1920] The device plays the advice audio data and displays the text data, while the smart speaker or device with a display plays the audio and displays the text on the display.

[1921] Input: Advice data sent to the terminal

[1922] Output: Played audio and displayed text

[1923] Step 8:

[1924] The user gives the command to start recording. For example, they say, "I'm going to read you the fairy tale of Momotaro."

[1925] Input: User utterance

[1926] Output: Start recording command

[1927] Step 9:

[1928] The device enters recording mode, recording the user's speech and temporarily saving it as digital data.

[1929] Input: Start recording

[1930] Output: Digitized recording data

[1931] Step 10:

[1932] After the device finishes recording, it sends the recorded data to a cloud server and stores it there. The server then securely stores the received data.

[1933] Input: Digitized recording data

[1934] Output: Data stored on a cloud server

[1935] Step 11:

[1936] The user makes a playback request, for example, "Play the fairy tale of Momotaro."

[1937] Input: User's playback request

[1938] Output: Playback request data

[1939] Step 12:

[1940] The server retrieves the stored voice data and sends it to the device, which then transfers the data over the Internet.

[1941] Input: Playback request data

[1942] Output: Audio data sent to the device

[1943] Step 13:

[1944] The device plays the audio data, and the recorded story is played on the speaker in the user's voice for the child to hear.

[1945] Input: Audio data sent to the device

[1946] Output: Played fairy tale audio

[1947] This will allow parents to easily record and play back childcare advice and fairy tales.

[1948] (Application example 1)

[1949] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1950] Conventional parenting support systems focus solely on analyzing parents' voice input and providing appropriate advice. However, there are no systems that can respond to security-related consultations or inquiries using similar voice recognition technology. There is a need for a system that can provide advice on both parenting and security in response to parents' spoken inquiries and manage them centrally. In addition, the generated advice must be provided in both voice and text format, allowing for flexible responses tailored to each user's usage situation.

[1951] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1952] In this invention, the server includes means for acquiring voice uttered by parents, means for converting the acquired voice into digital data, means for transmitting the digital data to a cloud server, means for analyzing the digital data stored in the cloud server and generating advice on child-rearing consultations and security, means for returning the generated advice to a playback terminal as voice and text, and means for playing the voice and displaying the text on the playback terminal, thereby making it possible to provide appropriate advice in a unified manner in response to questions from parents about child-rearing consultations and security.

[1953] The "means for acquiring parental vocalizations" is a device or system for collecting parental vocalizations in digital form.

[1954] The "means for converting captured audio into digital data" refers to a technique or device for converting audio signals into digital form.

[1955] The "means for transmitting digital data to a server on the cloud" refers to a technology or device for transmitting the converted digital data to a cloud server via the Internet.

[1956] "Means for analyzing digital data stored on a server on the cloud and generating advice on child-rearing consultations and security" refers to a technology or system that analyzes digital data stored on a cloud server and generates advice related to child-rearing and security based on the results.

[1957] The "means for returning the generated advice to the playback terminal as voice and text" refers to a technique and device for converting the generated advice into voice and text format and transmitting it to the playback terminal.

[1958] "Means for playing audio and displaying text at a playback terminal" refers to a device or system for playing received audio and displaying text.

[1959] A "means for recording parental vocalizations" is a device or technology for recording parental vocalizations.

[1960] "Means for storing recorded audio as digital data on a server on the cloud" refers to a method or system for storing recorded audio in a digital format on a cloud server.

[1961] "Means for searching for digital data stored on a server on the cloud and transmitting it to a playback terminal at a specified time" refers to a function or device for searching for data stored on a cloud server under specific conditions or at specific times and transmitting it to a playback terminal.

[1962] The "means for selecting appropriate fairy tales and lullabies based on the age of the child" refers to an algorithm or device that selects the most appropriate fairy tales and lullabies based on the age information of the child.

[1963] "Means for transmitting selected fairy tales and lullabies from a server on the cloud to a playback terminal" refers to means for transmitting data of selected fairy tales and lullabies from a cloud server to a playback terminal.

[1964] "Means for analyzing captured audio and generating security advice" refers to a system or software capable of analyzing audio data and generating appropriate security-related advice.

[1965] The "means for returning the generated security advice to the playback terminal as voice and text" refers to a technique or device that converts the generated security advice into voice and text format and sends it to the playback terminal.

[1966] To implement this invention, a system is required that captures the voices spoken by parents, transmits them to a cloud server, and generates appropriate advice on child-rearing and security. The system is broadly composed of three main elements: a terminal, a server, and a user.

[1967] Terminal

[1968] The device is a smart speaker or display device that captures the parent's voice. The device has the following features:

[1969] Voice input function: Parental voice is captured through a microphone.

[1970] Recording function: Saves captured audio as digital data.

[1971] Audio playback function: Plays audio data received from a server on the cloud.

[1972] Text display function: Displays text data received from a server on the cloud.

[1973] server

[1974] A server is a computing unit that runs on the cloud and has the following main functions:

[1975] Speech analysis function: Converts voice data sent from the device into text data. Uses the speech_recognition library.

[1976] Advice generation function: Generates appropriate parenting and security advice from the analyzed voice data. It uses a generative AI model with the transformers library.

[1977] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[1978] Text generation function: Generates text data for advice and sends it to the terminal. This uses the tflow library.

[1979] Fairy tale and lullaby selection function: Selects the most appropriate fairy tale or lullaby based on the child's age and sends it to the device.

[1980] User

[1981] Users, primarily parents, perform the following actions:

[1982] Voice input: Record parenting advice, security advice, fairy tales, lullabies, etc. For example, ask, "There's been a suspicious person hanging around recently. What should I do?"

[1983] Audio playback request: Request playback of a fairy tale or lullaby that you have recorded yourself.

[1984] Receive advice: Receive parenting and security advice provided by the server via voice and text.

[1985] Specific examples

[1986] Specific examples of childcare and security consultations

[1987] 1. A user speaks to a device and asks a question about parenting or security. For example, "My child has been crying a lot lately. What should I do?"

[1988] 2. The device captures the audio and sends the digital data to the server.

[1989] 3. The server converts the speech to text and performs the analysis. The speech_recognition library is used here.

[1990] 4. The server generates appropriate advice based on the analysis results and sends the advice to the device as text and audio data. This is where a generative AI model using the transformers library comes into play.

[1991] 5. The device plays the transmitted audio data and displays the text data on the display.

[1992] 6. The user receives the advice and applies it to solve the problem.

[1993] Prompt Sentence Examples

[1994] An example of a prompt to be input to the generative AI model is as follows:

[1995] "Generate appropriate security-related advice based on spoken questions. For example, what should I do if my neighbors are bothering me with noise?"

[1996] In this way, parents can quickly receive appropriate advice on child-rearing and security-related questions, creating a multi-functional support system that meets the needs of both parties.

[1997] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1998] Step 1:

[1999] The user speaks to the device and makes a voice input about childcare or security. For example, "My child has been crying at night recently. What should I do?" This voice is captured as a digital signal through the microphone. This is the state in which the input voice data has been captured.

[2000] Step 2:

[2001] The device picks up the voice and sends the digital data to a server in the cloud. Software on the device converts the voice data into packets and sends them to the server via the Internet. The input voice data is sent to the server.

[2002] Step 3:

[2003] The server converts the speech to text and performs analysis. The speech_recognition library is used to convert the speech data to text data. The input is speech data and the output is text data. The speech recognition engine analyzes the speech signal and converts it to text.

[2004] Step 4:

[2005] The server generates appropriate advice using a generative AI model based on the analysis results. The server uses the transformers library to input the analyzed text and the prompt sentence into the generative AI model, and outputs the optimal advice. The input is text data, and the output is text data of the advice.

[2006] Step 5:

[2007] The server returns the generated advice to the playback device as audio and text data. The generated advice text data is converted into audio data using the gTTS (Google Text to Speech) library and sent to the device via the Internet. The input is the advice text data, and the output is audio data and text data.

[2008] Step 6:

[2009] The terminal plays the transmitted voice data and displays the text data on the display. The built-in speaker of the terminal plays the voice data and the display displays the text data. The input is voice data and text data, and the output is voice and displayed text. This allows the user to receive advice.

[2010] These are the processing steps of the system that realizes this application example. The generated advice is useful for solving problems related to child rearing and security, and users can receive specific instructions and suggestions.

[2011] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2012] The present invention is a system that inputs the user's voice, analyzes the acquired voice, and provides appropriate advice, and also combines it with an emotion engine that recognizes the user's emotions and generates a response according to their emotional state. This enables more personalized and appropriate advice, improving the effectiveness of comprehensive child-rearing support.

[2013] Key Components

[2014] The system is broadly divided into four main components: the terminal, the server, the emotion engine, and the user.

[2015] Terminal

[2016] The device is a smart speaker or a device with a display that receives user voice and has the following functions:

[2017] Voice input function: Parental voice is captured through a microphone.

[2018] Recording function: Saves captured audio as digital data.

[2019] Audio playback function: Plays audio data received from a server on the cloud.

[2020] Text display function: Displays text data received from a server on the cloud.

[2021] server

[2022] A server is a computing unit that runs on the cloud and has the following main functions:

[2023] Voice analysis function: Converts voice data sent from the device into text data.

[2024] Advice generation function: Generates appropriate parenting advice from analyzed voice data.

[2025] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[2026] Text generation function: Generates text data of advice and sends it to the terminal.

[2027] Emotion data processing function: Analyzes the emotional data sent from the emotion engine and adjusts the advice content.

[2028] Emotion Engine

[2029] The emotion engine is a software component that recognizes emotions from user voice data and analyzes that information. It has the following functions:

[2030] Emotion recognition function: Recognizes emotional states from voice data.

[2031] Emotion data transmission function: Recognized emotion data is transmitted to the server.

[2032] Emotional feedback function: Tailor appropriate advice based on emotions.

[2033] User

[2034] Users, primarily parents, perform the following actions:

[2035] Voice input: Record parenting advice, fairy tales, and lullabies.

[2036] Audio playback request: Request playback of a fairy tale or lullaby that you have recorded yourself.

[2037] Receiving advice: Receive advice provided by the server in voice and text format.

[2038] Specific examples

[2039] Specific examples of child-rearing consultation

[2040] Example 1: A system that recognizes the user's emotional state and provides advice

[2041] 1. The user speaks to the device and asks for advice on child-rearing. For example, "My child has been crying a lot lately. What should I do?"

[2042] 2. The device captures the voice and sends the digital data to the emotion engine.

[2043] 3. The emotion engine analyzes the voice data to recognize the user's emotional state, for example, recognizing that the user is feeling stressed.

[2044] 4. The emotion engine sends the emotion data to the server.

[2045] 5. The server analyzes the emotional and voice data and generates appropriate advice based on the child's emotional state. For example, it might say, "To reduce stress factors for your child, we recommend creating a relaxing bedtime routine."

[2046] 6. The server converts the generated advice into voice and text data and sends it to the device.

[2047] 7. The device plays the audio and displays the text.

[2048] 8. Users receive advice and apply it to solve their parenting problems.

[2049] Examples of fairy tales and lullabies

[2050] Example 2: Reading fairy tales using an emotion engine

[2051] 1. The user sits down at the device and starts recording a fairy tale. For example, say, "I'm going to read you the fairy tale of Momotaro."

[2052] 2. The device enters recording mode and records the user's voice.

[2053] 3. After the device finishes recording, it sends the digital data to the server and stores it.

[2054] 4. The emotion engine analyzes the recording data and attaches emotional information.

[2055] 5. The user later requests, "Play the fairy tale of Momotaro."

[2056] 6. The server selects the recorded voice data taking into consideration the attached emotional information and sends it to the terminal.

[2057] 7. The device plays the audio data and plays it to the child in the user's voice.

[2058] The system can recognize the user's emotions and provide appropriate advice based on those emotions, enabling more personalized and effective parenting support. It can also deepen the bond between parents and their children by playing fairy tales and lullabies recorded in their own voices.

[2059] The processing flow will be explained below.

[2060] A process flow for parenting consultations that combines an emotion engine

[2061] Step 1:

[2062] The user can ask for parenting advice by voice, for example, "My child has been crying at night recently. What should I do?"

[2063] Step 2:

[2064] The device records the user's voice. It captures the input voice through a microphone and stores it as digital data.

[2065] Step 3:

[2066] The device sends the recorded voice data to the emotion engine, which then uploads it to the emotion engine via network communication.

[2067] Step 4:

[2068] The emotion engine analyzes the voice data and recognizes the user's emotional state. For example, it may recognize that the user is feeling stressed.

[2069] Step 5:

[2070] The emotion engine sends the recognized emotion data to the server, where the voice data and emotion data are combined.

[2071] Step 6:

[2072] The server converts the voice data into text data using a voice recognition API.

[2073] Step 7:

[2074] The server analyzes the text data and emotional data, using natural language processing (NLP) to understand the relationship between keywords and context in the text and emotions, and analyzes the content of the parenting consultation.

[2075] Step 8:

[2076] The server generates advice based on the analysis results. Using an AI model that reflects emotional data, appropriate advice is generated in text format. For example, it might say, "To reduce the factors that cause your child stress, we recommend creating a relaxing bedtime routine."

[2077] Step 9:

[2078] The server converts the text data into audio data using a text-to-speech API.

[2079] Step 10:

[2080] The server transmits the generated voice data and text data to the terminal via network communication.

[2081] Step 11:

[2082] The device plays the audio and displays the text: it plays the audio data through the speaker and displays the text data on the display.

[2083] Processing flow for recording and playing fairy tales using the emotion engine

[2084] Step 1:

[2085] The user approaches the device and begins recording a fairy tale. For example, they say, "I'm going to read you the fairy tale of Momotaro."

[2086] Step 2:

[2087] The device switches to recording mode, starts the recording function, and prepares to record the user's voice.

[2088] Step 3:

[2089] The user reads a fairy tale aloud and the device records the audio, for example, "Once upon a time, there lived an old man and an old woman."

[2090] Step 4:

[2091] The terminal ends the recording. The recording stops due to a user instruction or a timeout.

[2092] Step 5:

[2093] The device sends the recorded voice data to the emotion engine, which then uploads it to the emotion engine via network communication.

[2094] Step 6:

[2095] The emotion engine analyzes the audio data and recognizes the emotional state of the recording.

[2096] Step 7:

[2097] The emotion engine sends the recognized emotion data to the server, where the voice data and emotion data are combined.

[2098] Step 8:

[2099] The server stores the recording data and stores it safely in the cloud.

[2100] Step 9:

[2101] The user later gives a voice command such as "Play the fairy tale of Momotaro."

[2102] Step 10:

[2103] The server searches for the stored voice data and emotion data and sends them to the device at the specified time.

[2104] Step 11:

[2105] The device plays back the recorded fairy tale audio data in the user's voice, and the audio is played back through the speaker so that the child can listen to it in the user's voice.

[2106] Processing flow for providing relaxing music using an emotion engine

[2107] Step 1:

[2108] The user vocally requests relaxing music, for example, "Play some relaxing music."

[2109] Step 2:

[2110] The device records the voice and sends it to the emotion engine, which then uploads it to the emotion engine via network communication.

[2111] Step 3:

[2112] The emotion engine analyzes the voice data and recognizes the user's emotional state.

[2113] Step 4:

[2114] The emotion engine sends the recognized emotion data to the server, where the voice data and emotion data are combined.

[2115] Step 5:

[2116] The server converts the speech into text and selects appropriate relaxing music taking into account emotional data.

[2117] Step 6:

[2118] The server sends the selected music data to the device, which then uploads the music data to the device via the network.

[2119] Step 7:

[2120] The device will play music, and the speaker will play the selected relaxing music.

[2121] These processing steps allow users to easily receive parenting advice, play fairy tales or lullabies recorded in their own voice, or request relaxing music to help reduce stress. The combination of an emotion engine provides more personalized assistance based on the user's emotional state.

[2122] Example 2

[2123] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2124] In today's world, many parents face the burdens of work and family life, requiring them to quickly receive appropriate parenting advice. However, conventional parenting support systems have struggled to accurately recognize parents' emotional states and provide personalized advice based on those states. In particular, when parents are stressed or tired, general advice that ignores their emotional state has limited effectiveness and does not contribute to caring for the parents' physical and mental health. A system that solves this problem and provides more effective and compassionate parenting support is needed.

[2125] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for transmitting digital data to a cloud server, a means for analyzing the digital data stored in the cloud server and recognizing the emotional state of the user, and a means for generating advice related to child-rearing consultation according to the emotional state. This makes it possible to provide personalized advice according to the emotional state of the parent.

[2126] "Voice spoken by the parent" refers to the voice spoken by the parent into the device for conversation, consultation, or recording.

[2127] "Digital data" means information such as voice or text that has been converted into a form that can be processed, stored, or transmitted electronically.

[2128] A "cloud server" is an online data processing and storage system provided via the Internet by computers located in remote locations.

[2129] An "emotion engine" is a software component that analyzes and recognizes a user's emotions and psychological state from their voice and text data.

[2130] A "voice analysis engine" is software or hardware that has the function of converting voice data into text data.

[2131] An "advice generation engine" is software or hardware that has the function of generating appropriate advice based on input data.

[2132] A "playback terminal" is a device for playing and displaying audio and text data sent from a server on the cloud.

[2133] "Users" are those who use the system to receive consultations, recordings, and advice on child-rearing, and are primarily parents.

[2134] "Childcare consultation" refers to the act of parents seeking professional advice regarding questions or problems they have regarding childcare.

[2135] "Emotional state" refers to the psychological state or emotion expressed by the user through voice or text, and includes, for example, stress, joy, anger, etc.

[2136] The "audio playback function" is a function that plays back audio data so that the user can hear it.

[2137] "Text display function" refers to the function of displaying text data on a display or other display device.

[2138] The present invention is a system that inputs the user's voice, analyzes the acquired voice, and provides appropriate advice, and also combines it with an emotion engine that recognizes the user's emotions and generates a response according to their emotional state. This system enables more personalized and appropriate advice, improving the effectiveness of comprehensive child-rearing support.

[2139] Key Components

[2140] The system is broadly divided into four main components: the terminal, the server, the emotion engine, and the user.

[2141] Terminal

[2142] The device is a smart speaker or a device with a display that receives user voice and has the following functions:

[2143] Voice input function: Captures the user's voice through a microphone.

[2144] Recording function: Saves captured audio as digital data.

[2145] Audio playback function: Plays audio data received from a server on the cloud.

[2146] Text display function: Displays text data received from a server on the cloud.

[2147] server

[2148] A server is a computing unit that runs on the cloud and has the following main functions:

[2149] Voice analysis function: Converts voice data sent from the device into text data.

[2150] Emotion recognition function: Analyze the user's emotional state through the emotion engine.

[2151] Advice generation function: Generates appropriate parenting advice based on analyzed voice data and emotional data.

[2152] Data storage function: Recorded audio data is safely stored and can be accessed at specified times.

[2153] Text generation function: Generates text data of advice and sends it to the terminal.

[2154] Emotion Engine

[2155] The emotion engine is a software component that recognizes emotions from user voice data and analyzes that information. It has the following functions:

[2156] Emotion Recognition: Recognizes emotional states from speech data. It uses natural language processing techniques and applies a BERT-based sentiment analysis model.

[2157] Emotion data transmission function: Recognized emotion data is transmitted to the server.

[2158] User

[2159] Users, primarily parents, perform the following actions:

[2160] Voice input: Record parenting advice, fairy tales, and lullabies.

[2161] Audio playback request: Requests playback of a recorded fairy tale or lullaby.

[2162] Receiving advice: Receive advice provided by the server in voice and text format.

[2163] Specific examples

[2164] Specific examples of child-rearing consultation

[2165] 1. The user speaks to the device and asks for advice on child-rearing. For example, "My child has been crying a lot lately. What should I do?"

[2166] 2. The device captures the audio and sends the digital data to the server.

[2167] 3. The server uses a speech analysis engine to analyze the voice data and convert it into text data.

[2168] 4. The server recognizes the user's emotional state through an emotion engine and identifies that the user is feeling stressed.

[2169] 5. The server generates appropriate advice based on the emotion and voice data. For example, "To reduce stress factors for your child, we recommend creating a relaxing bedtime routine."

[2170] 6. The server converts the generated advice into voice and text data and sends it to the device.

[2171] 7. The device plays the audio and displays the text.

[2172] Examples of fairy tales and lullabies

[2173] 1. The user sits down at the device and starts recording a fairy tale. For example, say, "I'm going to read you the fairy tale of Momotaro."

[2174] 2. The device enters recording mode and records the user's voice.

[2175] 3. After the device finishes recording, it sends the digital data to the server and stores it.

[2176] 4. The emotion engine analyzes the recording data and attaches emotional information.

[2177] 5. The user later requests, "Play the fairy tale of Momotaro."

[2178] 6. The server selects the recorded voice data taking into consideration the attached emotional information and sends it to the terminal.

[2179] 7. The device plays the audio data and plays it to the child in the user's voice.

[2180] Example prompt sentence:

[2181] My child has been crying at night recently, what should I do?

[2182] "Play the fairy tale of Momotaro"

[2183] This system can provide appropriate advice based on the user's emotional state, enabling more personalized and effective parenting support. It can also deepen the bond between parents and their children by playing fairy tales and lullabies recorded in their own voices.

[2184] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2185] Step 1: Voice Input

[2186] The user speaks into the device to ask questions or ask for advice about child-rearing. For example, "My child has been crying at night recently. What should I do?"

[2187] The input is the user's voice data.

[2188] The device picks up audio using a built-in microphone, and performs noise reduction and echo cancellation to capture clear audio data.

[2189] The output is audio data converted into digital form.

[2190] Step 2: Sending audio data

[2191] The device converts the captured audio data into a digital format, specifically, converting the audio into 16-bit PCM format, compressing the data, and packetizing it.

[2192] The input is the converted digital audio data.

[2193] The device then sends the converted digital audio data to a cloud server using HTTPS to ensure data security.

[2194] The output is the audio data received by the server on the cloud.

[2195] Step 3: Analyzing the audio data

[2196] The server receives the voice data sent from the terminal.

[2197] The input is voice data sent to a server on the cloud.

[2198] The server uses a speech analysis engine to convert the audio data into text data, using the Google Cloud Speech-to-Text API.

[2199] The output is text data converted from the audio data.

[2200] Step 4: Recognize emotions

[2201] The server passes the converted text data to the emotion engine.

[2202] The input is text data.

[2203] The emotion engine uses natural language processing technology to analyze the user's emotional state from text data. It uses a BERT-based sentiment analysis model to recognize users' stress, joy, anger, etc.

[2204] The output is the recognized emotion data.

[2205] Step 5: Generating Advice

[2206] The server integrates the emotion data sent from the emotion engine with the text data.

[2207] The input is emotion data and text data.

[2208] The server uses an advice generation engine to generate appropriate advice using a GPT-based generative AI model, such as "To reduce stress factors for your child, we recommend creating a relaxing bedtime routine."

[2209] The output is the generated advice.

[2210] Step 6: Send and display advice data

[2211] The server converts the generated advice into both text and audio data.

[2212] The input is the generated advice.

[2213] The server transmits the converted data to the terminal.

[2214] The device analyzes the received data, plays back the voice data, and displays the text data on the display, using a voice playback engine and a text-to-speech (TTS) engine.

[2215] The output is a spoken advice and a displayed text advice.

[2216] Step 7: Take and apply advice

[2217] The user receives the advice provided by the device and applies it in real life.

[2218] The inputs are spoken advice and displayed text advice.

[2219] The hope is that by following the advice, users will find their parenting problems alleviated.

[2220] The output is the effect of the problem mitigation.

[2221] (Application example 2)

[2222] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2223] Conventional advertising delivery systems deliver advertisements uniformly without considering the user's emotional state, making it difficult to provide a personalized advertising experience that reflects the user's interests and preferences. Similarly, parenting support systems provide advice without understanding the parent's emotional state, resulting in an inability to provide effective support.

[2224] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2225] In this invention, the server includes means for recognizing an emotional state from acquired voice data and facial expression data, means for adjusting the content of advice based on the recognized emotional state, and means for selecting and displaying advertisements in accordance with the emotional state recognized from the acquired voice data and facial expression data, thereby making it possible to analyze the user's emotional state in real time and provide optimal advertisements and advice.

[2226] "Voice data" is a digital representation of a user's voice.

[2227] "Facial expression data" refers to facial expressions of a user captured using a camera and converted into an analyzable format.

[2228] The "emotional state" refers to the user's emotions analyzed based on voice data and facial expression data. Specifically, emotions include joy, sadness, anger, fear, surprise, and excitement.

[2229] "Means for recognizing" includes software and hardware configurations for analyzing voice data and facial expression data to identify the emotional state of the user.

[2230] The "means for adjusting the content of advice" includes algorithms and programs for generating or modifying appropriate advice for a user based on the perceived emotional state.

[2231] "Means for selecting advertisements" includes software and hardware configurations for selecting the most appropriate advertisements for a user based on the perceived emotional state and other relevant information.

[2232] The term "reproducing means" includes a device and software configuration for reproducing audio or video data received from a server on a reproducing terminal.

[2233] "Storage means" includes software and hardware configurations for safely storing acquired voice data and facial expression data in digital format on a server on the cloud.

[2234] This invention is a system that acquires and analyzes a user's voice and facial expression data to provide appropriate advertisements and advice. The system is mainly composed of four main elements: a terminal, a server, an emotion recognition engine, and the user.

[2235] Terminal

[2236] The terminal is a mobile device such as a smartphone or smart glasses that has the following capabilities:

[2237] Voice and facial expression input function: Uses a microphone and camera to capture the user's voice and facial expression data.

[2238] Recording function: Saves captured audio as digital data.

[2239] Audio playback and display function: Plays back and displays data sent from the server on the cloud.

[2240] server

[2241] A server is a computing unit that runs on the cloud and has the following main functions:

[2242] Voice and facial expression analysis function: Analyzes voice and facial expression data sent from the device and converts it into text data.

[2243] Advertisement and advice generation function: Recognizes the user's emotional state from the analyzed data and generates appropriate advertisements and advice.

[2244] Data storage function: Recorded voice data and facial expression data are safely stored.

[2245] Data transmission function: Sends generated advertisements and advice to the device.

[2246] Emotion Recognition Engine

[2247] The emotion recognition engine is a software component that recognizes emotions from the user's voice data and facial expression data and provides that information. It has the following functions:

[2248] Emotion recognition function: Recognizes emotional states from voice data and facial expression data.

[2249] Emotion data transmission function: Recognized emotion data is transmitted to the server.

[2250] Emotional feedback: Tailor appropriate ads and advice based on emotions.

[2251] User

[2252] Users are primarily consumers and parents who receive advertising and perform the following actions:

[2253] Voice and facial expression input: Voice and facial expression data are input through questions and conversations.

[2254] Receiving advertisements and advice: Receive advertisements and advice provided by the server via voice and text.

[2255] Specific processing flow

[2256] 1. The user speaks into the device and shows their facial expressions to the camera.

[2257] 2. The device acquires voice and facial expression data and sends it as digital data to a cloud server.

[2258] 3. The server receives the acquired data, converts it into voice and analyzes facial expressions, and sends it to the emotion recognition engine.

[2259] 4. The emotion recognition engine recognizes the emotional state and sends the data back to the server.

[2260] 5. The server generates appropriate advertisements or advice based on the emotional state and sends them back to the device.

[2261] 6. The device provides the user with the sent advertisements and advice via voice and text.

[2262] Use cases and input prompts for generative AI models

[2263] A specific example of use could be to display advertisements for luxury goods when the user looks satisfied, and to display soothing advertisements when the user is feeling stressed.

[2264] Example prompts to input to a generative AI model:

[2265] Create an application for smartphones or smart glasses that recognizes emotions from a user's voice and facial expression data, and selects and displays the most appropriate advertisement based on that data. Convert the voice data into text and analyze the emotions. Select an advertisement based on the analyzed emotional data and display it on the user's display.

[2266] This allows users to receive advertisements and advice that match their emotions at the time, making it possible to provide more personalized and effective services.

[2267] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2268] Step 1:

[2269] The user inputs voice into the terminal and shows facial expressions to the camera.

[2270] Input: User's voice and facial expressions.

[2271] Processing: Capture voice and facial expression data using the device's microphone and camera.

[2272] Output: Acquired voice and facial expression data.

[2273] Step 2:

[2274] The voice data and facial expression data acquired by the device are converted into digital format and sent to a server on the cloud.

[2275] Input: Analog voice data and facial expression data acquired on the device.

[2276] Processing: Analog audio is converted to digital format through signal processing, and facial expression data is converted to digital format through image processing. The data is then sent to a cloud server.

[2277] Output: Digital voice and facial expression data in a format usable by the cloud server.

[2278] Step 3:

[2279] The server converts the received voice data into text data and analyzes the facial expression data to recognize the emotional state.

[2280] Input: Digital voice data and facial expression data received by the cloud server.

[2281] Processing: A speech recognition algorithm is used to convert the voice data into text data, and an emotion recognition engine is used to analyze emotions from facial expression data.

[2282] Output: Text data and emotional state data.

[2283] Step 4:

[2284] The server generates appropriate advertisements or advice based on the analysis results.

[2285] Input: Text data and emotional state data.

[2286] Processing: Based on the sentiment data, an advice generation algorithm or ad selection engine is used to select the best ad or advice.

[2287] Output: Selected ads or generated advice.

[2288] Step 5:

[2289] The server sends the generated advertisement or advice to the terminal, which plays and displays it.

[2290] Input: Selected Advertisement or Generated Advice Data.

[2291] Processing: Converting the data into the appropriate format and sending it to the device, where it plays the audio and displays the text.

[2292] Output: Audio and text feedback to the user.

[2293] Step 6:

[2294] The user checks the advertisements and advice provided and re-enters them if necessary.

[2295] Input: Advertisements or advice provided by the device.

[2296] Action: Check the content of the advertisement or advice, and provide feedback or re-enter it.

[2297] Output: Return to step 1 to acquire the user's voice and facial expressions again.

[2298] This will enable the provision of personalized advertisements and advice in real time according to the user's emotional state. Each step uses hardware such as smartphones and smart glasses, and software such as voice recognition algorithms, emotion recognition engines, and ad selection engines.

[2299] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2300] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2301] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2302] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2303] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2304] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2305] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2306] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2307] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2308] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2309] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2310] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2311] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2312] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2313] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2314] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2315] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2316] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2317] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2318] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2319] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2320] The following is further disclosed regarding the above embodiment.

[2321] (Claim 1)

[2322] a means for acquiring audio uttered by the parent;

[2323] means for converting the captured audio into digital data;

[2324] A means for transmitting digital data to a server on the cloud;

[2325] A means for analyzing digital data stored on a server on the cloud and generating advice related to child-rearing consultations;

[2326] means for returning the generated advice to the playback terminal as voice and text;

[2327] means for playing the audio and displaying the text on a playback device;

[2328] A system including:

[2329] (Claim 2)

[2330] a means for recording the parent's vocalizations;

[2331] A means of storing the recorded audio as digital data on a cloud server;

[2332] A means for searching for digital data stored on a server on the cloud and transmitting the data to a playback terminal at a specified timing;

[2333] a means for playing the audio on a playback device;

[2334] 10. The system of claim 1, comprising:

[2335] (Claim 3)

[2336] a means for selecting appropriate fairy tales and lullabies based on the age of the child;

[2337] A means for transmitting selected fairy tales and lullabies from a cloud serv...

Claims

1. a means for acquiring audio uttered by the parent; means for converting the captured audio into digital data; A means for transmitting digital data to a server on the cloud; A means for analyzing digital data stored on a server on the cloud and generating advice related to child-rearing consultations; means for returning the generated advice to the playback terminal as voice and text; means for playing the audio and displaying the text on a playback device; A system including:

2. a means for recording the parent's vocalizations; A means of storing the recorded audio as digital data on a cloud server; A means for searching for digital data stored on a server on the cloud and transmitting the data to a playback terminal at a specified timing; a means for playing the audio on a playback device; The system of claim 1 , comprising:

3. a means for selecting appropriate fairy tales and lullabies based on the age of the child; A means for transmitting selected fairy tales and lullabies from a cloud server to a playback device; A means to play fairy tales and lullabies on a playback device, The system of claim 1 , comprising:

Images