System

Abstract

A system is provided.SOLUTION: A system comprising: sensor means for sensing a baby's speech, facial expressions, and body movements; communication means for transmitting the sensed data; receiving means for receiving and processing the transmitted data; emotion determination means for analyzing the received data and determining the baby's emotion; suggestion generation means for generating an appropriate action based on the determined emotion; and notification means for notifying the user of the generated suggestion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the past, it has been difficult for parents, especially those raising children for the first time or those with little support, to understand a baby's emotions from their vocalizations, facial expressions, and body movements. This has led to problems such as being unable to accurately grasp a baby's emotions and respond appropriately, which increases the anxiety and burden of childcare. The present invention aims to solve these problems by providing a system that allows parents to easily understand a baby's emotions and respond appropriately. [Means for solving the problem]

[0005] The present invention solves the conventional problems with a system including a sensor means for reading a baby's vocalizations, facial expressions, and body movements, a communication means for transmitting the read data, a receiving means for receiving and processing the transmitted data, an emotion determination means for analyzing the received data and determining the baby's emotion, a proposal generation means for generating appropriate actions based on the determined emotion, and a notification means for notifying a user of the generated proposal. In particular, the emotion determination means achieves highly accurate emotion determination by analyzing the vocalizations, facial expressions, and body movements using a machine learning model. Furthermore, the proposal generation means converts the generated proposals into specific actions that the user can execute, making it easier for parents to take appropriate actions for their baby.

[0006] "Vocalization" refers to the sounds that babies make with their mouths and is one way of expressing emotions.

[0007] "Facial expressions" are emotions and moods expressed through the movement of a baby's facial muscles.

[0008] "Body movements" are movements and changes in the movements of a baby's limbs and whole body, and are one way of expressing emotions and needs.

[0009] "Sensor means" refers to a device or apparatus used to read a baby's vocalizations, facial expressions, and body movements.

[0010] "Communication means" refers to a function or device for transmitting the read data to another device or server.

[0011] The "receiving means" is a function or device for receiving transmitted data and performing analysis and processing.

[0012] The "emotion determination means" is a function or device for analyzing the received data and determining the baby's emotion.

[0013] The "suggestion generation means" is a function or device for generating specific actions or responses that the parent can take based on the determined emotion.

[0014] "Notification means" is a function or device for communicating generated suggestions to a parent or caregiver.

[0015] A "machine learning model" is an algorithm for analyzing data, specifically a statistical method or artificial intelligence technique used to determine emotions. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system that analyzes a baby's emotions from their vocalizations, facial expressions, and body movements, and suggests appropriate responses to parents. This system is mainly composed of three components: a terminal (including a smartphone), a server, and a user. Specific embodiments of the present invention will be described below.

[0038] Device data collection:

[0039] The device uses a camera and microphone to read the baby's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it within the device. For example, the device captures the baby's smiling and crying faces with the camera, and records crying and cooing (gentle voices) with the microphone.

[0040] Sending data from device to server:

[0041] The collected data is encrypted and then sent to the server using a secure protocol (e.g., HTTPS). This ensures the safety of the data. As a specific example, the baby's crying voice data and facial expression data are encrypted and then sent to a specific endpoint on the server.

[0042] Data analysis by server:

[0043] The server decodes the received data and performs processing to determine the baby's emotions. The emotion determination means analyzes the data using a machine learning model. For example, the server analyzes the specific patterns of the baby's crying and the characteristics of the crying face to determine that the baby is "sad."

[0044] Server suggested actions:

[0045] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model generates specific responses by referring to the baby's emotions and past data. For example, if the baby is determined to be "sad," the server will suggest "hold the baby and talk to him / her gently."

[0046] Device suggestion notification:

[0047] The server sends the generated suggestions to the device, which then notifies the user. Possible notification methods include a pop-up message or voice guidance. The user checks the notification and performs the suggested action. For example, an app might display a notification saying, "The baby feels sad. Hold the baby and talk to him gently."

[0048] This system allows users (parents and caregivers) to quickly and accurately grasp a baby's emotions and respond appropriately. This facilitates communication between parents and children and helps reduce the burden of childcare. The system we propose will contribute to the healthy growth of babies and reduce the stress of childcare for parents.

[0049] The processing flow will be explained below.

[0050] Step 1: Collect data (device)

[0051] The device activates a microphone and camera to capture the baby's vocalizations, facial expressions, and body movements. The microphone records the baby's vocalizations (crying and cooing), and the camera captures the baby's facial expressions and body movements (such as limb movements). The collected audio and video data is stored in temporary storage within the device.

[0052] Step 2: Sending data (terminal)

[0053] The collected audio and video data is encrypted and sent to a server using a secure protocol (e.g., HTTPS), which reduces the risk of the data being stolen or tampered with during transmission.

[0054] Step 3: Receiving and analyzing data (server)

[0055] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data is input to the emotion determination means. The emotion determination means analyzes the data using a machine learning model and determines the baby's emotion. For example, it detects specific crying patterns and facial features and determines the emotion as "sad."

[0056] Step 4: Proposing Actions (Server)

[0057] The server uses a generative AI model to suggest appropriate actions based on the emotion data determined by the emotion determination means. The generative AI model uses past databases and existing knowledge to suggest how the user should respond to the baby. For example, if the emotion is determined to be "sad," a specific action such as "hold the baby and talk to him / her gently" is generated.

[0058] Step 5: Proposal notification (device)

[0059] The device receives the suggestions sent from the server and notifies the user. The notification is done via a pop-up message or voice guidance, helping the user to respond quickly. For example, a message such as "The baby is feeling sad. Please hold the baby and talk to him / her gently" may be displayed.

[0060] Step 6: Execute Action (User)

[0061] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user can take action by holding the baby and talking to it gently. This series of actions allows the user to understand the baby's emotions and respond appropriately.

[0062] In this way, this system, which determines a baby's emotions from their vocalizations, facial expressions, and body movements and suggests appropriate responses, helps to reduce anxiety in child-rearing and facilitates smooth communication between parents and children.

[0063] Example 1

[0064] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0065] In today's childcare environment, it is important for parents and caregivers to quickly and accurately grasp a baby's emotions and needs and respond appropriately, but this can be difficult, especially for first-time parents. The present invention aims to solve this problem by providing a system that analyzes a baby's emotions and suggests appropriate responses.

[0066] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0067] In this invention, the server includes a sensor means for reading the baby's vocalizations, facial expressions, and body movements, a storage means for temporarily storing the read data, an encryption means for encrypting the stored data, a communication means for transmitting the encrypted data, a receiving means for receiving and decrypting the transmitted data, an emotion determination means for analyzing the decrypted data and determining the baby's emotion, a suggestion generation means using a generative AI model for generating appropriate actions based on the determined emotion, and a notification means for notifying the user of the generated suggestion. This enables parents and caregivers to quickly and accurately grasp the baby's emotion and take appropriate action.

[0068] "Sensor means" refers to hardware that reads the baby's vocalizations, facial expressions, and body movements, as well as software that controls its operation.

[0069] The "storage means" is a memory or storage device for temporarily storing data read by the sensor means.

[0070] "Encryption means" refers to software and its execution environment for encrypting data using an encryption algorithm to protect the stored data.

[0071] A "communication means" is software that implements a network interface and communication protocol for transmitting encrypted data.

[0072] The "receiving means" refers to hardware and software for receiving and decoding data transmitted by the transmitting means.

[0073] The "emotion determination means" is a machine learning model and its execution environment for analyzing the data decoded by the receiving means and determining the baby's emotions.

[0074] The "proposal generation means" refers to software and its execution environment for generating appropriate actions using a generative AI model based on the determined emotions.

[0075] The "notification means" is a part of the application for notifying the user of the generated proposal, and hardware that realizes the notification function.

[0076] This system analyzes a baby's emotions based on their vocalizations, facial expressions, and body movements, and suggests appropriate responses to parents. This system is primarily composed of three components: a terminal, a server, and a user.

[0077] Data collection by terminal

[0078] The device uses a camera and microphone as sensors to read the baby's vocalizations, facial expressions, and body movements. These sensors capture the baby's smiling or crying face with the camera, and record the baby's cries and cooing (gentle voices) with the microphone. For example, the camera takes a picture of the baby's face every second, while the microphone simultaneously collects audio data.

[0079] Data Retention and Encryption

[0080] The device temporarily stores the captured data in its internal memory, which ensures data integrity in real time. The stored data is encrypted using the AES encryption algorithm to ensure security.

[0081] Sending data from the device to the server

[0082] The encrypted data is sent to the server via a communication means. Specifically, the encrypted data is sent to the server via a POST request using HTTPS, a secure communication protocol.

[0083] Data analysis by server

[0084] The server receives the transmitted data and decodes it using a decoding method. The decoded data contains the baby's voice and video, which are then analyzed using a machine learning model. To determine emotions, models such as CNN (convolutional neural network) and RNN (recurrent neural network) are used to extract and analyze voice frequency patterns and facial expression features. For example, if a baby's crying has a specific pattern that indicates "sadness," the server will determine that as the emotion.

[0085] Server-based action suggestions

[0086] The server uses a generative AI model to suggest appropriate actions based on the emotional information determined from the analysis. Examples of generative AI models that can be used include GPT-3. Prompts are sent to the model, which suggests specific actions that parents or caregivers can take. For example, by sending the prompt "What should you do when your baby is sad?", the server generates the specific response of "hold the baby and talk to him / her gently."

[0087] Proposal notification on device

[0088] The generated suggestion is sent back to the terminal and notified to the user via a notification means. Possible notification methods include a pop-up message or voice guidance. For example, a notification message such as "The baby is feeling sad. Please hold the baby and talk to him / her gently" may be displayed.

[0089] Specific examples

[0090] When a baby starts crying, the device's camera captures the crying face and the microphone records the sound. This data is encrypted and sent to a server, where a machine learning model analyzes the emotion as "sad." The generative AI model then generates a suggestion to "hold the baby and talk to him / her gently" and notifies the device.

[0091] Example prompts for generative AI models

[0092] "Analyze the crying data and crying face image data to determine the baby's emotions and suggest appropriate ways to respond."

[0093] This system allows users to understand their baby's emotions in real time and respond appropriately, which is expected to improve communication between parents and children and reduce the burden of childcare.

[0094] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0095] Step 1:

[0096] The device initializes the camera and microphone at system startup, specifically by setting the camera resolution and frame rate and the microphone sampling rate. The input to this process is the camera and microphone hardware, and the output is the ready-to-use sensor device.

[0097] Step 2:

[0098] The device reads and captures the baby's vocalizations, facial expressions, and body movements at regular intervals. The input is real-time video and audio, and the output is the image frame and audio data at that time. Specifically, the device captures an image frame from the camera every second and simultaneously records audio data from the microphone.

[0099] Step 3:

[0100] The device temporarily stores the captured data in its internal memory. The input is the captured image and audio data, and the output is the temporarily stored data. Specifically, this data is organized in chronological order and stored in a buffer.

[0101] Step 4:

[0102] The terminal encrypts the stored data. The input is the temporarily stored data, and the output is the encrypted data. Specifically, the data is encrypted using the AES encryption algorithm.

[0103] Step 5:

[0104] The device sends encrypted data to the server. The input is the encrypted data, and the output is a notification to the server that the data has been sent. Specifically, the device composes an HTTPS POST request and sends the data to the server's endpoint.

[0105] Step 6:

[0106] The server receives the transmitted data and decrypts it. The input is the encrypted data, and the output is the original decrypted data. Specifically, the server decrypts the data using the AES encryption key used on the terminal.

[0107] Step 7:

[0108] The server analyzes the decoded data and determines the baby's emotions. The input is the decoded voice data and image data, and the output is the emotion determination result. Specifically, it uses a machine learning model (e.g., CNN or RNN) to analyze the frequency patterns of the voice and facial features.

[0109] Step 8:

[0110] The server uses a generative AI model to suggest appropriate actions based on the determined emotion information. The input is the emotion determination result, and the output is a specific action suggestion. As a specific operation, the server sends a prompt sentence to the generative AI model (e.g., GPT-3) saying, "What should you do when a baby is sad?"

[0111] Step 9:

[0112] The server sends the generated action proposal to the terminal. The input is the specific action proposal, and the output is a notification to the terminal that the action proposal has been sent.

[0113] Step 10:

[0114] The device notifies the user of the received action suggestion. The input is the action suggestion sent from the server, and the output is the notification displayed to the user. Specific actions include using a pop-up message or voice guidance to display a notification such as "The baby is feeling sad. Please hold the baby and talk to him gently."

[0115] Through these steps, the system can analyze the baby's emotions and suggest appropriate responses to parents or caregivers in real time.

[0116] (Application example 1)

[0117] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0118] Conventional baby emotion analysis systems only address the specific situations of babies, making it difficult to apply them to other purposes. Furthermore, when dealing with customers in brick-and-mortar stores, it is difficult to quickly and accurately grasp their emotions and provide appropriate service. To improve this situation, a system is needed that can analyze the emotions of babies and customers and propose appropriate responses.

[0119] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0120] In this invention, the server includes a sensor means for reading the vocalizations, facial expressions, and body movements of the baby or customer, a communication means for transmitting the read data, a receiving means for receiving and processing the transmitted data, an emotion determination means for analyzing the received data and determining the emotion of the baby or customer, a proposal generation means for generating an appropriate action based on the determined emotion, and a notification means for notifying the user of the generated proposal, thereby enabling an analysis of the emotion of the customer or baby and a proposal of a countermeasure.

[0121] "Sensor means" refers to a device for reading the vocalizations, facial expressions, and body movements of the baby or customer.

[0122] "Communication means" refers to the device or technology used to transmit the scanned data to another device or server.

[0123] A "receiving means" is a device or module for receiving and processing transmitted data.

[0124] The "emotion determination means" is a device or algorithm for analyzing the received data and determining the emotion of the baby or customer.

[0125] The "suggestion generator" is a device or algorithm for generating appropriate actions based on the determined emotion.

[0126] The "notification means" is a device or interface for notifying the user of the generated proposal.

[0127] A "machine learning model" is an algorithm or mathematical model that analyzes data and finds patterns.

[0128] A "generative AI model" is an artificial intelligence model that generates appropriate output based on specific input.

[0129] This invention is a system that analyzes the emotions of a baby or customer based on their vocalizations, facial expressions, and body movements, and proposes appropriate responses. This system is mainly composed of three entities: a terminal, a server, and a user.

[0130] Device data collection:

[0131] The device (such as a smartphone or a robot in a brick-and-mortar store) uses a camera and microphone to read the baby's or customer's vocalizations, facial expressions, and body movements. The data collected by these sensors is temporarily stored on the device. Specific examples include a camera capturing the customer's facial expressions and a microphone recording the tone of voice.

[0132] Sending data from device to server:

[0133] The collected data is encrypted and sent to the server using a secure communication protocol (e.g., HTTPS). This ensures the security of the data. For example, customer voice and facial expression data are encrypted and sent to the server endpoint.

[0134] Data analysis by server:

[0135] The server decodes the received data and analyzes it using an emotion determination method, which uses a machine learning model. For example, the server analyzes specific patterns in the customer's voice and facial features to determine that the customer is "tired."

[0136] Server suggested actions:

[0137] Based on the analysis results, a generative AI model is used to suggest appropriate actions. The generative AI model compares the results with past data to generate specific responses. For example, if the user is determined to be "tired," the server will suggest "please offer me a drink."

[0138] Device suggestion notification:

[0139] The server sends the generated suggestions to the device, which then notifies the user. Possible notification methods include a pop-up message or voice guidance. For example, the app might display a notification saying, "You seem tired. Let's offer you a drink."

[0140] This allows users (store staff and parents) to quickly and accurately grasp the emotions of customers and babies and respond appropriately, which contributes to improving customer satisfaction and facilitating communication between parents and children, and improving the overall service experience.

[0141] Example prompt sentence:

[0142] "If you determine from a customer's facial expression and voice that they are tired, please suggest what kind of hospitality they should provide."

[0143] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0144] Step 1: Data collection

[0145] The device (a smartphone or a robot in a brick-and-mortar store) uses a camera and microphone to read the baby's or customer's vocalizations, facial expressions, and body movements in real time. Specifically, the camera captures facial expressions and the microphone records voice. The input for this step is the baby's or customer's real-time voice and facial expressions, and the output is these audio and video data.

[0146] Step 2: Send data

[0147] The device encrypts the collected data and sends it securely to the server using the HTTPS protocol, ensuring the safety of the data. Specifically, the device encodes and encrypts the audio and video data and sends it to a specific endpoint on the server. The input to this step is the collected audio and video data, and the output is the encrypted data.

[0148] Step 3: Receiving and Decrypting Data

[0149] The server receives and decrypts the data sent from the terminal. Specifically, it decodes the received data and converts it back to the original audio and video data. The input of this step is the encrypted data, and the output is the decrypted audio and video data.

[0150] Step 4: Sentiment Analysis

[0151] The server uses emotion determination means to analyze the received data and determine the baby's or customer's emotion. Using a machine learning model, it analyzes specific patterns of voice tone and facial expressions to determine specific emotions (e.g., "tired," "sad," etc.). The input of this step is the decoded audio and video data, and the output is the emotion determination result.

[0152] Step 5: Action proposal generation

[0153] The server uses a generative AI model to suggest an appropriate action based on the determined emotion. For example, if the customer is determined to be "tired," the server generates the suggestion "Please offer me a drink." Specifically, the emotion determination result is input into the AI ​​model, and an appropriate response prompt is generated. The input for this step is the emotion determination result, and the output is the response prompt.

[0154] Step 6: Proposal Notification

[0155] The server sends the generated suggestion to the terminal, which then notifies the user (parent or store staff). Notification methods include a pop-up message on the terminal or voice guidance. Specifically, the suggestion is sent from the server to the terminal, and the terminal displays the suggestion on the screen or provides voice guidance. The input to this step is a prompt for a solution, and the output is the notification to the user.

[0156] Through the above steps, the system is able to quickly and accurately analyze the emotions of the baby or customer and propose appropriate countermeasures.

[0157] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0158] The present invention is a system that analyzes a baby's emotions from its vocalizations, facial expressions, and body movements, and suggests appropriate responses based on the results.The present invention also includes a system that analyzes a user's vocalizations, facial expressions, and body movements to determine the user's emotions, and suggests additional actions for the baby based on the determined emotions.The present invention is mainly composed of three entities: a terminal (including a smartphone), a server, and a user.Specific embodiments of the present invention will be described below.

[0159] Device data collection:

[0160] The device uses a camera and microphone to capture the baby's and the user's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it within the device. For example, the device captures the baby's smiling and crying faces and the user's facial expressions with the camera, and records the baby's crying and cooing and the user's vocalizations with the microphone.

[0161] Sending data from device to server:

[0162] The collected audio and video data is encrypted and sent to the server using a secure protocol (e.g., HTTPS). Because the data is encrypted before being sent, the risk of the data being stolen or tampered with during transmission is reduced. As a specific example, the baby's crying voice data and facial expression data, and the user's voice and facial expression data are encrypted before being sent to a specific endpoint on the server.

[0163] Data analysis by server:

[0164] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data is input to the emotion determination means. The emotion determination means analyzes the data using a machine learning model to determine the emotions of the baby and the user. For example, the server analyzes the specific pattern of the baby's crying and the characteristics of the crying face to determine the emotion as "sad." It also analyzes the user's vocalizations and facial expressions to determine that the user is "feeling stressed."

[0165] Server suggested actions:

[0166] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model uses the baby's and user's emotions, as well as past databases and known knowledge, to generate specific responses. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will generate a suggestion to "hold the baby and talk to him gently" to calm the baby, and also suggest actions to the user such as "take a deep breath and relax."

[0167] Device suggestion notification:

[0168] The device receives the suggestions sent from the server and notifies the user. Possible notification methods include a pop-up message or voice guidance. The user checks the notification and performs the suggested action. For example, the app displays the message, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0169] User performs the action:

[0170] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user will take action such as holding the baby and talking to it gently. Also, if the notification suggests ways for the user to relax, the user will take action such as deep breathing or stretching.

[0171] In this way, this system, which determines the emotions of the baby and the user from their vocalizations, facial expressions, and physical movements and suggests appropriate responses, helps to reduce anxiety in child-rearing and facilitates communication between parents and children. Furthermore, by taking the user's emotions into consideration, it also contributes to reducing the stress of child-rearing.

[0172] The processing flow will be explained below.

[0173] Step 1: Collect data (device)

[0174] The device activates a microphone and camera to capture the baby's and the user's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it within the device. For example, the device captures the baby's smiling and crying face with the camera and records the baby's crying and cooing with the microphone. For the user, the device captures facial expressions with the camera and records the tone of voice with the microphone.

[0175] Step 2: Sending data (terminal)

[0176] The collected audio and video data is encrypted and sent to the server using a secure protocol (e.g., HTTPS). Because the data is encrypted before being sent, the risk of the data being stolen or tampered with during transmission is reduced. Specifically, the baby's crying data, facial expression data, and the user's voice and facial expression data are encrypted before being sent to a specific endpoint on the server.

[0177] Step 3: Receiving and analyzing data (server)

[0178] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data are input to the emotion determination means. The emotion determination means analyzes the data using a machine learning model to determine the emotions of the baby and the user. For example, the server may analyze the specific pattern of the baby's crying and the characteristics of the crying face to determine the emotion as "sad," and simultaneously analyze the user's facial expression and tone of voice to determine that the user is "feeling stressed."

[0179] Step 4: Proposing Actions (Server)

[0180] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model uses the baby's and user's emotions, as well as past databases and known knowledge, to generate specific responses. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will suggest "hold the baby and talk to him / her gently" and generate an action such as "the user should take a deep breath and relax."

[0181] Step 5: Proposal notification (device)

[0182] The device receives the suggestions sent from the server and notifies the user. Possible notification methods include a pop-up message or voice guidance. The user checks the notification and performs the suggested action. For example, the app displays the message, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0183] Step 6: Execute Action (User)

[0184] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user will take action such as holding the baby and talking to it gently. If a relaxation method is suggested for the user, the user will take action such as deep breathing or stretching. For example, by taking a deep breath as instructed by the device notification and then holding the baby, the stress of childcare can be reduced.

[0185] In this way, this system, which determines the emotions of the baby and the user from their vocalizations, facial expressions, and physical movements and suggests appropriate responses, helps to reduce anxiety in child-rearing and facilitates communication between parents and children. Furthermore, by taking the user's emotions into consideration, it also contributes to reducing the stress of child-rearing.

[0186] Example 2

[0187] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0188] Conventional childcare support systems have limited methods for responding to specific baby behaviors, making it difficult to accurately assess the emotions of both the baby and the parent and to provide specific responses based on those emotions. This has limited the effectiveness of reducing stress and anxiety in childcare.

[0189] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means including an input device that reads the baby's vocalizations, facial expressions, and body movements, data sending means that encrypts and sends the read data, receiving means that receives and decrypts the sent data, emotion determining means that analyzes the decrypted data and determines the baby's emotion and the user's emotion, a generation engine including suggestion generating means that generates appropriate actions based on the determined emotions, and notification means that notifies the user of the generated suggestions. This makes it possible to quickly and accurately determine the emotions of the baby and the user and to suggest specific and feasible ways of responding based on the determined emotions.

[0190] An "input device" is a device that includes sensors such as a camera or microphone to read a baby's vocalizations, facial expressions, and body movements.

[0191] The "data transmission means" is a means having a function of encrypting the read data and transmitting it safely to the server.

[0192] The "receiving means" is a means for receiving and decrypting the transmitted encrypted data.

[0193] The "emotion determination means" is a means for analyzing received audio data and video data using a machine learning model to determine the emotions of the baby and the user.

[0194] The "suggestion generation means" is a means for using a generative AI model to suggest appropriate actions to the baby and the user based on the determined emotions.

[0195] The "notification means" is a means for notifying the user of the generated proposal, and includes a pop-up message, a voice guide, and the like.

[0196] This invention is a childcare support system that analyzes a baby's vocalizations, facial expressions, and body movements and suggests appropriate actions based on the results. It also has a function that analyzes the user's vocalizations, facial expressions, and body movements to determine the user's emotions and suggest additional actions for the baby. This system is mainly composed of three entities: a terminal (including a smartphone), a server, and a user.

[0197] The device uses a camera and microphone to capture the baby's and user's vocalizations, facial expressions, and body movements. Specifically, the device captures the baby's smiling and crying faces and the user's facial expressions with the camera, and records the baby's crying and cooing sounds and the user's vocalizations with the microphone. The collected data is temporarily stored in the device.

[0198] The collected audio and video data is encrypted and sent to the server using a secure protocol (e.g., HTTPS). For example, the baby's crying voice data and facial expression data, and the user's voice and facial expression data are encrypted and then sent to a specific endpoint on the server.

[0199] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data is input into the emotion determination function, which uses a machine learning model to analyze the data and determine the emotions of the baby and the user. For example, the server analyzes the specific patterns of a baby's crying and the characteristics of its crying face to determine the emotion as "sad." It also analyzes the user's vocalizations and facial expressions to determine that the user is "feeling stressed."

[0200] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model generates specific responses by using the baby's and user's emotions, as well as past databases and known knowledge. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will generate a suggestion to "hold the baby and talk to him gently" to calm the baby, and suggest actions to the user such as "take a deep breath and relax."

[0201] The device receives the suggestions sent from the server and notifies the user. Possible notification methods include a pop-up message or voice guidance. Specifically, the device app displays a message saying, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0202] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user can hold the baby and talk to him / her gently. If the notification suggests relaxation techniques for the user, the user can take actions such as deep breathing or stretching.

[0203] This system can reduce anxiety in child-rearing and facilitate communication between parents and children. It also contributes to reducing stress in child-rearing by taking into consideration the user's emotions.

[0204] For example, you might input the following prompt into a generative AI model:

[0205] "Analyze the baby's crying pattern and determine its emotion. Then, analyze the user's facial expressions and vocalizations and determine the user's emotion as well. Suggest appropriate actions if the baby's emotion is sad and the user's emotion is stressed."

[0206] Using this example, the generative AI model can derive specific countermeasures and make effective suggestions to the user.

[0207] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0208] Step 1:

[0209] The device reads the baby's and user's vocalizations, facial expressions, and body movements. Specifically, the device's built-in camera captures the baby's smiling and crying faces and the user's facial expressions in real time, and the microphone records the baby's crying and cooing, as well as the user's vocalizations. This input data (audio and video data) is temporarily stored in the device.

[0210] Step 2:

[0211] The terminal encrypts the collected audio and video data, specifically by encoding it using a data encryption algorithm and preparing it for transmission. This encrypted data is the output.

[0212] Step 3:

[0213] The device sends the encrypted data to the server using a secure protocol (e.g., HTTPS). Specifically, the encrypted audio and video data is sent over the Internet to a specific endpoint on the server.

[0214] Step 4:

[0215] The server receives the encrypted data sent from the terminal and decrypts it. Specifically, it decodes the data using a decryption algorithm running on the server. This decrypted audio and video data becomes the input data for the next process.

[0216] Step 5:

[0217] The server inputs the decoded audio and video data into the emotion determination means. The machine learning model in the server analyzes this data and determines the emotions of the baby and the user. Specifically, it extracts features from the audio data and recognizes patterns in the video data, and outputs the respective emotions (e.g., "sad" or "stressed").

[0218] Step 6:

[0219] The server uses a generative AI model based on the emotion determination results to suggest appropriate actions. For example, based on the emotion determination results that the baby is "sad" and the user is "stressed," the server generates suggestions such as holding the baby and talking to them gently, or encouraging the user to take a deep breath. These generated suggestions are the output.

[0220] Step 7:

[0221] The server then transmits the generated proposal to the terminal. Specifically, the proposal content is formatted as text data or voice data, and then encrypted again before being transmitted to the terminal.

[0222] Step 8:

[0223] The device receives the encrypted proposal data sent from the server and decrypts it. The decrypted proposal content is notified to the user. Specific notification methods include a pop-up message or voice guidance. For example, a message might be displayed saying, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0224] Step 9:

[0225] The user checks the notification from the device and performs the suggested action. For example, they may hold the baby and talk to it gently based on the notification content. Also, if a relaxation method is suggested for the user, they may perform actions such as deep breathing or stretching.

[0226] Through the above steps, a system is realized that determines the emotions of the baby and the user and suggests specific ways of responding based on the determined emotions.

[0227] (Application example 2)

[0228] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0229] Conventional childcare support systems only provide recommendations regarding the baby's condition and do not consider the user's (parent's) emotions or condition, which limits their ability to reduce childcare stress. Furthermore, physical stores provide insufficient support for parents and respond to baby's condition, making it difficult to provide an appropriate childcare environment.

[0230] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion determination means that analyzes received data and determines the emotions of the baby and the user, a suggestion generation means that generates appropriate actions based on the determined emotions, a notification means that notifies the user of the generated suggestions, and an action support means that supports the user in taking actions corresponding to the generated suggestions. This makes it possible to comprehensively analyze the status of the baby and the user, provide appropriate childcare support even in physical stores, and reduce childcare stress.

[0231] 1. "Sensor means" refers to equipment used to read a baby's vocalizations, facial expressions, and body movements.

[0232] 2. "Means of communication" refers to the functions and technologies used to transmit the captured data.

[0233] 3. "Receiving means" means the functions and technologies for receiving and processing transmitted data.

[0234] 4. "Emotion determination means" refers to the functions and technologies for analyzing received data and determining the emotions of the baby and the user.

[0235] 5. "Proposal generation means" refers to the functions and technologies for generating appropriate actions based on the determined emotions.

[0236] 6. "Notification means" refers to the functions and technologies used to notify users of generated suggestions.

[0237] 7. "Action support means" refers to functions and technologies that support users in taking actions corresponding to the generated suggestions.

[0238] 8. A "machine learning model" is an algorithm or framework for analyzing data to identify patterns and make predictions.

[0239] 9. A “generative AI model” is an artificial intelligence model that generates recommendations using a database or known knowledge to suggest appropriate actions.

[0240] This invention is a system that analyzes a baby's emotions from its vocalizations, facial expressions, and body movements, and suggests appropriate responses based on the analysis. Furthermore, this system has the function of analyzing the user's vocalizations, facial expressions, and body movements to determine the user's emotions, and based on this, suggesting additional actions for the baby.

[0241] Hardware and software used

[0242] This system primarily utilizes the following hardware and software:

[0243] Hardware

[0244] Smartphone: Collects data using a camera and microphone.

[0245] Server: Receives and analyzes data.

[0246] software

[0247] OpenCV: Capture images from the camera.

[0248] Keras (TensorFlow backend): Performs sentiment analysis using machine learning models.

[0249] HTTP client library: Communicates with the server (e.g., requests library).

[0250] Data collection

[0251] The device (smartphone) uses a camera and microphone to capture the baby's and user's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it.

[0252] Data transmission and reception

[0253] The collected audio and video data is encrypted and sent to a server using a secure protocol (e.g., HTTPS). The server receives the encrypted data sent from the device and decrypts it.

[0254] Data analysis

[0255] The server inputs the received audio and video data into a machine learning model to determine the emotions of the baby and the user. The machine learning model analyzes the baby's crying and facial expressions, as well as the user's vocalizations and facial expressions, to identify emotions.

[0256] Suggested Actions

[0257] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. This generative AI model generates specific responses by using the baby's and user's emotions, as well as past databases and existing knowledge. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will generate a suggestion to "hold the baby and talk to him gently" and suggest actions to the user such as "take a deep breath and relax."

[0258] Proposal notification and support

[0259] The device receives the proposed action from the server and notifies the user. The notification method is a pop-up message or voice guidance. The user checks the notification and performs the proposed action. The action support means also supports the proposed action.

[0260] Specific examples

[0261] For example, if a baby starts crying in a physical store selling childcare products, the app will analyze the image and sound of the crying baby and notify the user with a message such as "Hold the baby and talk to him / her gently." It will also suggest actions such as "Take a deep breath and relax." This will provide appropriate childcare support even in physical stores, reducing parenting stress for parents.

[0262] Prompt Sentence Examples

[0263] "Judge the baby's emotions and suggest appropriate parenting actions. Refer to and analyze the baby's facial expressions and cries to inform parents of specific ways to respond."

[0264] This system determines the emotions of the baby and the user from their vocalizations, facial expressions, and physical movements, and by appropriately supporting the user's behavior, it helps to reduce anxiety and stress in child-rearing and facilitate smooth communication between parents and children.

[0265] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0266] Step 1: Data collection

[0267] The device uses a camera to capture the facial expressions of the baby and the user, and a microphone to collect vocalization data. The image data captured by the camera and the audio data collected by the microphone are input. These data are collected in real time and temporarily stored in the device. The output of this step is image data and audio data.

[0268] Step 2: Send data

[0269] The device encrypts the collected image and audio data and sends it to the server using a secure protocol (HTTPS). The encrypted data is input and sent to a specific endpoint on the server. This reduces the risk of the data being stolen or tampered with during transmission. The output is a confirmation that the data has been received by the server.

[0270] Step 3: Receiving and Decrypting Data

[0271] The server receives the encrypted data sent from the device and decrypts it. Encrypted image and audio data are the input, and decrypted data is the output. Decrypting the data converts it into a format that can be used for analysis.

[0272] Step 4: Data analysis

[0273] The server inputs the decoded image data and audio data into a machine learning model to determine the emotions of the baby and the user. The input is the decoded data, which is analyzed using the machine learning model. The emotion determination result is output through data analysis. Specifically, the model analyzes the baby's crying face and crying sound, as well as the user's facial expressions and voice characteristics, to determine emotions such as "sadness" or "stress."

[0274] Step 5: Action Creation

[0275] The server uses a generative AI model based on the emotion determination result to suggest an appropriate action. The input is the emotion determination result, and a specific response method is generated using a past database and existing knowledge. For example, if a baby is determined to be "sad," the generative AI model will make a suggestion such as "hold the baby and talk to him gently." The output is the generated action proposal.

[0276] Step 6: Proposal Notification

[0277] The device receives the suggestions sent from the server and notifies the user. The generated action suggestions are input, and the device displays them as pop-up messages or voice guidance. The output is a notification to the user. Specifically, messages such as "Please hold the baby and speak to him gently" and "The user should take a deep breath and relax" are displayed.

[0278] Step 7: Take action and support

[0279] The user checks the notification from the device and performs the suggested action. The notified action suggestion becomes the input, and the user actually takes the action. The action support means supports the user's action, for example, it can provide specific guides and reminders. The output is the user's execution and the support results.

[0280] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0281] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0282] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0283] [Second embodiment]

[0284] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0285] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0286] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0287] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0288] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0289] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0290] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0291] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0292] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0293] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0294] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0295] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0296] The present invention is a system that analyzes a baby's emotions from their vocalizations, facial expressions, and body movements, and suggests appropriate responses to parents. This system is mainly composed of three components: a terminal (including a smartphone), a server, and a user. Specific embodiments of the present invention will be described below.

[0297] Device data collection:

[0298] The device uses a camera and microphone to read the baby's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it within the device. For example, the device captures the baby's smiling and crying faces with the camera, and records crying and cooing (gentle voices) with the microphone.

[0299] Sending data from device to server:

[0300] The collected data is encrypted and then sent to the server using a secure protocol (e.g., HTTPS). This ensures the safety of the data. As a specific example, the baby's crying voice data and facial expression data are encrypted and then sent to a specific endpoint on the server.

[0301] Data analysis by server:

[0302] The server decodes the received data and performs processing to determine the baby's emotions. The emotion determination means analyzes the data using a machine learning model. For example, the server analyzes the specific patterns of the baby's crying and the characteristics of the crying face to determine that the baby is "sad."

[0303] Server suggested actions:

[0304] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model generates specific responses by referring to the baby's emotions and past data. For example, if the baby is determined to be "sad," the server will suggest "hold the baby and talk to him / her gently."

[0305] Device suggestion notification:

[0306] The server sends the generated suggestions to the device, which then notifies the user. Possible notification methods include a pop-up message or voice guidance. The user checks the notification and performs the suggested action. For example, an app might display a notification saying, "The baby feels sad. Hold the baby and talk to him gently."

[0307] This system allows users (parents and caregivers) to quickly and accurately grasp a baby's emotions and respond appropriately. This facilitates communication between parents and children and helps reduce the burden of childcare. The system we propose will contribute to the healthy growth of babies and reduce the stress of childcare for parents.

[0308] The processing flow will be explained below.

[0309] Step 1: Collect data (device)

[0310] The device activates a microphone and camera to capture the baby's vocalizations, facial expressions, and body movements. The microphone records the baby's vocalizations (crying and cooing), and the camera captures the baby's facial expressions and body movements (such as limb movements). The collected audio and video data is stored in temporary storage within the device.

[0311] Step 2: Sending data (terminal)

[0312] The collected audio and video data is encrypted and sent to a server using a secure protocol (e.g., HTTPS), which reduces the risk of the data being stolen or tampered with during transmission.

[0313] Step 3: Receiving and analyzing data (server)

[0314] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data is input to the emotion determination means. The emotion determination means analyzes the data using a machine learning model and determines the baby's emotion. For example, it detects specific crying patterns and facial features and determines the emotion as "sad."

[0315] Step 4: Proposing Actions (Server)

[0316] The server uses a generative AI model to suggest appropriate actions based on the emotion data determined by the emotion determination means. The generative AI model uses past databases and existing knowledge to suggest how the user should respond to the baby. For example, if the emotion is determined to be "sad," a specific action such as "hold the baby and talk to him / her gently" is generated.

[0317] Step 5: Proposal notification (device)

[0318] The device receives the suggestions sent from the server and notifies the user. The notification is done via a pop-up message or voice guidance, helping the user to respond quickly. For example, a message such as "The baby is feeling sad. Please hold the baby and talk to him / her gently" may be displayed.

[0319] Step 6: Execute Action (User)

[0320] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user can take action by holding the baby and talking to it gently. This series of actions allows the user to understand the baby's emotions and respond appropriately.

[0321] In this way, this system, which determines a baby's emotions from their vocalizations, facial expressions, and body movements and suggests appropriate responses, helps to reduce anxiety in child-rearing and facilitates smooth communication between parents and children.

[0322] Example 1

[0323] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0324] In today's childcare environment, it is important for parents and caregivers to quickly and accurately grasp a baby's emotions and needs and respond appropriately, but this can be difficult, especially for first-time parents. The present invention aims to solve this problem by providing a system that analyzes a baby's emotions and suggests appropriate responses.

[0325] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0326] In this invention, the server includes a sensor means for reading the baby's vocalizations, facial expressions, and body movements, a storage means for temporarily storing the read data, an encryption means for encrypting the stored data, a communication means for transmitting the encrypted data, a receiving means for receiving and decrypting the transmitted data, an emotion determination means for analyzing the decrypted data and determining the baby's emotion, a suggestion generation means using a generative AI model for generating appropriate actions based on the determined emotion, and a notification means for notifying the user of the generated suggestion. This enables parents and caregivers to quickly and accurately grasp the baby's emotion and take appropriate action.

[0327] "Sensor means" refers to hardware that reads the baby's vocalizations, facial expressions, and body movements, as well as software that controls its operation.

[0328] The "storage means" is a memory or storage device for temporarily storing data read by the sensor means.

[0329] "Encryption means" refers to software and its execution environment for encrypting data using an encryption algorithm to protect the stored data.

[0330] A "communication means" is software that implements a network interface and communication protocol for transmitting encrypted data.

[0331] The "receiving means" refers to hardware and software for receiving and decoding data transmitted by the transmitting means.

[0332] The "emotion determination means" is a machine learning model and its execution environment for analyzing the data decoded by the receiving means and determining the baby's emotions.

[0333] The "proposal generation means" refers to software and its execution environment for generating appropriate actions using a generative AI model based on the determined emotions.

[0334] The "notification means" is a part of the application for notifying the user of the generated proposal, and hardware that realizes the notification function.

[0335] This system analyzes a baby's emotions based on their vocalizations, facial expressions, and body movements, and suggests appropriate responses to parents. This system is primarily composed of three components: a terminal, a server, and a user.

[0336] Data collection by terminal

[0337] The device uses a camera and microphone as sensors to read the baby's vocalizations, facial expressions, and body movements. These sensors capture the baby's smiling or crying face with the camera, and record the baby's cries and cooing (gentle voices) with the microphone. For example, the camera takes a picture of the baby's face every second, while the microphone simultaneously collects audio data.

[0338] Data Retention and Encryption

[0339] The device temporarily stores the captured data in its internal memory, which ensures data integrity in real time. The stored data is encrypted using the AES encryption algorithm to ensure security.

[0340] Sending data from the device to the server

[0341] The encrypted data is sent to the server via a communication means. Specifically, the encrypted data is sent to the server via a POST request using HTTPS, a secure communication protocol.

[0342] Data analysis by server

[0343] The server receives the transmitted data and decodes it using a decoding method. The decoded data contains the baby's voice and video, which are then analyzed using a machine learning model. To determine emotions, models such as CNN (convolutional neural network) and RNN (recurrent neural network) are used to extract and analyze voice frequency patterns and facial expression features. For example, if a baby's crying has a specific pattern that indicates "sadness," the server will determine that as the emotion.

[0344] Server-based action suggestions

[0345] The server uses a generative AI model to suggest appropriate actions based on the emotional information determined from the analysis. Examples of generative AI models that can be used include GPT-3. Prompts are sent to the model, which suggests specific actions that parents or caregivers can take. For example, by sending the prompt "What should you do when your baby is sad?", the server generates the specific response of "hold the baby and talk to him / her gently."

[0346] Proposal notification on device

[0347] The generated suggestion is sent back to the terminal and notified to the user via a notification means. Possible notification methods include a pop-up message or voice guidance. For example, a notification message such as "The baby is feeling sad. Please hold the baby and talk to him / her gently" may be displayed.

[0348] Specific examples

[0349] When a baby starts crying, the device's camera captures the crying face and the microphone records the sound. This data is encrypted and sent to a server, where a machine learning model analyzes the emotion as "sad." The generative AI model then generates a suggestion to "hold the baby and talk to him / her gently" and notifies the device.

[0350] Example prompts for generative AI models

[0351] "Analyze the crying data and crying face image data to determine the baby's emotions and suggest appropriate ways to respond."

[0352] This system allows users to understand their baby's emotions in real time and respond appropriately, which is expected to improve communication between parents and children and reduce the burden of childcare.

[0353] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0354] Step 1:

[0355] The device initializes the camera and microphone at system startup, specifically by setting the camera resolution and frame rate and the microphone sampling rate. The input to this process is the camera and microphone hardware, and the output is the ready-to-use sensor device.

[0356] Step 2:

[0357] The device reads and captures the baby's vocalizations, facial expressions, and body movements at regular intervals. The input is real-time video and audio, and the output is the image frame and audio data at that time. Specifically, the device captures an image frame from the camera every second and simultaneously records audio data from the microphone.

[0358] Step 3:

[0359] The device temporarily stores the captured data in its internal memory. The input is the captured image and audio data, and the output is the temporarily stored data. Specifically, this data is organized in chronological order and stored in a buffer.

[0360] Step 4:

[0361] The terminal encrypts the stored data. The input is the temporarily stored data, and the output is the encrypted data. Specifically, the data is encrypted using the AES encryption algorithm.

[0362] Step 5:

[0363] The device sends encrypted data to the server. The input is the encrypted data, and the output is a notification to the server that the data has been sent. Specifically, the device composes an HTTPS POST request and sends the data to the server's endpoint.

[0364] Step 6:

[0365] The server receives the transmitted data and decrypts it. The input is the encrypted data, and the output is the original decrypted data. Specifically, the server decrypts the data using the AES encryption key used on the terminal.

[0366] Step 7:

[0367] The server analyzes the decoded data and determines the baby's emotions. The input is the decoded voice data and image data, and the output is the emotion determination result. Specifically, it uses a machine learning model (e.g., CNN or RNN) to analyze the frequency patterns of the voice and facial features.

[0368] Step 8:

[0369] The server uses a generative AI model to suggest appropriate actions based on the determined emotion information. The input is the emotion determination result, and the output is a specific action suggestion. As a specific operation, the server sends a prompt sentence to the generative AI model (e.g., GPT-3) saying, "What should you do when a baby is sad?"

[0370] Step 9:

[0371] The server sends the generated action proposal to the terminal. The input is the specific action proposal, and the output is a notification to the terminal that the action proposal has been sent.

[0372] Step 10:

[0373] The device notifies the user of the received action suggestion. The input is the action suggestion sent from the server, and the output is the notification displayed to the user. Specific actions include using a pop-up message or voice guidance to display a notification such as "The baby is feeling sad. Please hold the baby and talk to him gently."

[0374] Through these steps, the system can analyze the baby's emotions and suggest appropriate responses to parents or caregivers in real time.

[0375] (Application example 1)

[0376] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0377] Conventional baby emotion analysis systems only address the specific situations of babies, making it difficult to apply them to other purposes. Furthermore, when dealing with customers in brick-and-mortar stores, it is difficult to quickly and accurately grasp their emotions and provide appropriate service. To improve this situation, a system is needed that can analyze the emotions of babies and customers and propose appropriate responses.

[0378] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0379] In this invention, the server includes a sensor means for reading the vocalizations, facial expressions, and body movements of the baby or customer, a communication means for transmitting the read data, a receiving means for receiving and processing the transmitted data, an emotion determination means for analyzing the received data and determining the emotion of the baby or customer, a proposal generation means for generating an appropriate action based on the determined emotion, and a notification means for notifying the user of the generated proposal, thereby enabling an analysis of the emotion of the customer or baby and a proposal of a countermeasure.

[0380] "Sensor means" refers to a device for reading the vocalizations, facial expressions, and body movements of the baby or customer.

[0381] "Communication means" refers to the device or technology used to transmit the scanned data to another device or server.

[0382] A "receiving means" is a device or module for receiving and processing transmitted data.

[0383] The "emotion determination means" is a device or algorithm for analyzing the received data and determining the emotion of the baby or customer.

[0384] The "suggestion generator" is a device or algorithm for generating appropriate actions based on the determined emotion.

[0385] The "notification means" is a device or interface for notifying the user of the generated proposal.

[0386] A "machine learning model" is an algorithm or mathematical model that analyzes data and finds patterns.

[0387] A "generative AI model" is an artificial intelligence model that generates appropriate output based on specific input.

[0388] This invention is a system that analyzes the emotions of a baby or customer based on their vocalizations, facial expressions, and body movements, and proposes appropriate responses. This system is mainly composed of three entities: a terminal, a server, and a user.

[0389] Device data collection:

[0390] The device (such as a smartphone or a robot in a brick-and-mortar store) uses a camera and microphone to read the baby's or customer's vocalizations, facial expressions, and body movements. The data collected by these sensors is temporarily stored on the device. Specific examples include a camera capturing the customer's facial expressions and a microphone recording the tone of voice.

[0391] Sending data from device to server:

[0392] The collected data is encrypted and sent to the server using a secure communication protocol (e.g., HTTPS). This ensures the security of the data. For example, customer voice and facial expression data are encrypted and sent to the server endpoint.

[0393] Data analysis by server:

[0394] The server decodes the received data and analyzes it using an emotion determination method, which uses a machine learning model. For example, the server analyzes specific patterns in the customer's voice and facial features to determine that the customer is "tired."

[0395] Server suggested actions:

[0396] Based on the analysis results, a generative AI model is used to suggest appropriate actions. The generative AI model compares the results with past data to generate specific responses. For example, if the user is determined to be "tired," the server will suggest "please offer me a drink."

[0397] Device suggestion notification:

[0398] The server sends the generated suggestions to the device, which then notifies the user. Possible notification methods include a pop-up message or voice guidance. For example, the app might display a notification saying, "You seem tired. Let's offer you a drink."

[0399] This allows users (store staff and parents) to quickly and accurately grasp the emotions of customers and babies and respond appropriately, which contributes to improving customer satisfaction and facilitating communication between parents and children, and improving the overall service experience.

[0400] Example prompt sentence:

[0401] "If you determine from a customer's facial expression and voice that they are tired, please suggest what kind of hospitality they should provide."

[0402] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0403] Step 1: Data collection

[0404] The device (a smartphone or a robot in a brick-and-mortar store) uses a camera and microphone to read the baby's or customer's vocalizations, facial expressions, and body movements in real time. Specifically, the camera captures facial expressions and the microphone records voice. The input for this step is the baby's or customer's real-time voice and facial expressions, and the output is these audio and video data.

[0405] Step 2: Send data

[0406] The device encrypts the collected data and sends it securely to the server using the HTTPS protocol, ensuring the safety of the data. Specifically, the device encodes and encrypts the audio and video data and sends it to a specific endpoint on the server. The input to this step is the collected audio and video data, and the output is the encrypted data.

[0407] Step 3: Receiving and Decrypting Data

[0408] The server receives and decrypts the data sent from the terminal. Specifically, it decodes the received data and converts it back to the original audio and video data. The input of this step is the encrypted data, and the output is the decrypted audio and video data.

[0409] Step 4: Sentiment Analysis

[0410] The server uses emotion determination means to analyze the received data and determine the baby's or customer's emotion. Using a machine learning model, it analyzes specific patterns of voice tone and facial expressions to determine specific emotions (e.g., "tired," "sad," etc.). The input of this step is the decoded audio and video data, and the output is the emotion determination result.

[0411] Step 5: Action proposal generation

[0412] The server uses a generative AI model to suggest an appropriate action based on the determined emotion. For example, if the customer is determined to be "tired," the server generates the suggestion "Please offer me a drink." Specifically, the emotion determination result is input into the AI ​​model, and an appropriate response prompt is generated. The input for this step is the emotion determination result, and the output is the response prompt.

[0413] Step 6: Proposal Notification

[0414] The server sends the generated suggestion to the terminal, which then notifies the user (parent or store staff). Notification methods include a pop-up message on the terminal or voice guidance. Specifically, the suggestion is sent from the server to the terminal, and the terminal displays the suggestion on the screen or provides voice guidance. The input to this step is a prompt for a solution, and the output is the notification to the user.

[0415] Through the above steps, the system is able to quickly and accurately analyze the emotions of the baby or customer and propose appropriate countermeasures.

[0416] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0417] The present invention is a system that analyzes a baby's emotions from its vocalizations, facial expressions, and body movements, and suggests appropriate responses based on the results.The present invention also includes a system that analyzes a user's vocalizations, facial expressions, and body movements to determine the user's emotions, and suggests additional actions for the baby based on the determined emotions.The present invention is mainly composed of three entities: a terminal (including a smartphone), a server, and a user.Specific embodiments of the present invention will be described below.

[0418] Device data collection:

[0419] The device uses a camera and microphone to capture the baby's and the user's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it within the device. For example, the device captures the baby's smiling and crying faces and the user's facial expressions with the camera, and records the baby's crying and cooing and the user's vocalizations with the microphone.

[0420] Sending data from device to server:

[0421] The collected audio and video data is encrypted and sent to the server using a secure protocol (e.g., HTTPS). Because the data is encrypted before being sent, the risk of the data being stolen or tampered with during transmission is reduced. As a specific example, the baby's crying voice data and facial expression data, and the user's voice and facial expression data are encrypted before being sent to a specific endpoint on the server.

[0422] Data analysis by server:

[0423] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data is input to the emotion determination means. The emotion determination means analyzes the data using a machine learning model to determine the emotions of the baby and the user. For example, the server analyzes the specific pattern of the baby's crying and the characteristics of the crying face to determine the emotion as "sad." It also analyzes the user's vocalizations and facial expressions to determine that the user is "feeling stressed."

[0424] Server suggested actions:

[0425] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model uses the baby's and user's emotions, as well as past databases and known knowledge, to generate specific responses. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will generate a suggestion to "hold the baby and talk to him gently" to calm the baby, and also suggest actions to the user such as "take a deep breath and relax."

[0426] Device suggestion notification:

[0427] The device receives the suggestions sent from the server and notifies the user. Possible notification methods include a pop-up message or voice guidance. The user checks the notification and performs the suggested action. For example, the app displays the message, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0428] User performs the action:

[0429] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user will take action such as holding the baby and talking to it gently. Also, if the notification suggests ways for the user to relax, the user will take action such as deep breathing or stretching.

[0430] In this way, this system, which determines the emotions of the baby and the user from their vocalizations, facial expressions, and physical movements and suggests appropriate responses, helps to reduce anxiety in child-rearing and facilitates communication between parents and children. Furthermore, by taking the user's emotions into consideration, it also contributes to reducing the stress of child-rearing.

[0431] The processing flow will be explained below.

[0432] Step 1: Collect data (device)

[0433] The device activates a microphone and camera to capture the baby's and the user's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it within the device. For example, the device captures the baby's smiling and crying face with the camera and records the baby's crying and cooing with the microphone. For the user, the device captures facial expressions with the camera and records the tone of voice with the microphone.

[0434] Step 2: Sending data (terminal)

[0435] The collected audio and video data is encrypted and sent to the server using a secure protocol (e.g., HTTPS). Because the data is encrypted before being sent, the risk of the data being stolen or tampered with during transmission is reduced. Specifically, the baby's crying data, facial expression data, and the user's voice and facial expression data are encrypted before being sent to a specific endpoint on the server.

[0436] Step 3: Receiving and analyzing data (server)

[0437] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data are input to the emotion determination means. The emotion determination means analyzes the data using a machine learning model to determine the emotions of the baby and the user. For example, the server may analyze the specific pattern of the baby's crying and the characteristics of the crying face to determine the emotion as "sad," and simultaneously analyze the user's facial expression and tone of voice to determine that the user is "feeling stressed."

[0438] Step 4: Proposing Actions (Server)

[0439] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model uses the baby's and user's emotions, as well as past databases and known knowledge, to generate specific responses. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will suggest "hold the baby and talk to him / her gently" and generate an action such as "the user should take a deep breath and relax."

[0440] Step 5: Proposal notification (device)

[0441] The device receives the suggestions sent from the server and notifies the user. Possible notification methods include a pop-up message or voice guidance. The user checks the notification and performs the suggested action. For example, the app displays the message, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0442] Step 6: Execute Action (User)

[0443] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user will take action such as holding the baby and talking to it gently. If a relaxation method is suggested for the user, the user will take action such as deep breathing or stretching. For example, by taking a deep breath as instructed by the device notification and then holding the baby, the stress of childcare can be reduced.

[0444] In this way, this system, which determines the emotions of the baby and the user from their vocalizations, facial expressions, and physical movements and suggests appropriate responses, helps to reduce anxiety in child-rearing and facilitates communication between parents and children. Furthermore, by taking the user's emotions into consideration, it also contributes to reducing the stress of child-rearing.

[0445] Example 2

[0446] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0447] Conventional childcare support systems have limited methods for responding to specific baby behaviors, making it difficult to accurately assess the emotions of both the baby and the parent and to provide specific responses based on those emotions. This has limited the effectiveness of reducing stress and anxiety in childcare.

[0448] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means including an input device that reads the baby's vocalizations, facial expressions, and body movements, data sending means that encrypts and sends the read data, receiving means that receives and decrypts the sent data, emotion determining means that analyzes the decrypted data and determines the baby's emotion and the user's emotion, a generation engine including suggestion generating means that generates appropriate actions based on the determined emotions, and notification means that notifies the user of the generated suggestions. This makes it possible to quickly and accurately determine the emotions of the baby and the user and to suggest specific and feasible ways of responding based on the determined emotions.

[0449] An "input device" is a device that includes sensors such as a camera or microphone to read a baby's vocalizations, facial expressions, and body movements.

[0450] The "data transmission means" is a means having a function of encrypting the read data and transmitting it safely to the server.

[0451] The "receiving means" is a means for receiving and decrypting the transmitted encrypted data.

[0452] The "emotion determination means" is a means for analyzing received audio data and video data using a machine learning model to determine the emotions of the baby and the user.

[0453] The "suggestion generation means" is a means for using a generative AI model to suggest appropriate actions to the baby and the user based on the determined emotions.

[0454] The "notification means" is a means for notifying the user of the generated proposal, and includes a pop-up message, a voice guide, and the like.

[0455] This invention is a childcare support system that analyzes a baby's vocalizations, facial expressions, and body movements and suggests appropriate actions based on the results. It also has a function that analyzes the user's vocalizations, facial expressions, and body movements to determine the user's emotions and suggest additional actions for the baby. This system is mainly composed of three entities: a terminal (including a smartphone), a server, and a user.

[0456] The device uses a camera and microphone to capture the baby's and user's vocalizations, facial expressions, and body movements. Specifically, the device captures the baby's smiling and crying faces and the user's facial expressions with the camera, and records the baby's crying and cooing sounds and the user's vocalizations with the microphone. The collected data is temporarily stored in the device.

[0457] The collected audio and video data is encrypted and sent to the server using a secure protocol (e.g., HTTPS). For example, the baby's crying voice data and facial expression data, and the user's voice and facial expression data are encrypted and then sent to a specific endpoint on the server.

[0458] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data is input into the emotion determination function, which uses a machine learning model to analyze the data and determine the emotions of the baby and the user. For example, the server analyzes the specific patterns of a baby's crying and the characteristics of its crying face to determine the emotion as "sad." It also analyzes the user's vocalizations and facial expressions to determine that the user is "feeling stressed."

[0459] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model generates specific responses by using the baby's and user's emotions, as well as past databases and known knowledge. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will generate a suggestion to "hold the baby and talk to him gently" to calm the baby, and suggest actions to the user such as "take a deep breath and relax."

[0460] The device receives the suggestions sent from the server and notifies the user. Possible notification methods include a pop-up message or voice guidance. Specifically, the device app displays a message saying, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0461] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user can hold the baby and talk to him / her gently. If the notification suggests relaxation techniques for the user, the user can take actions such as deep breathing or stretching.

[0462] This system can reduce anxiety in child-rearing and facilitate communication between parents and children. It also contributes to reducing stress in child-rearing by taking into consideration the user's emotions.

[0463] For example, you might input the following prompt into a generative AI model:

[0464] "Analyze the baby's crying pattern and determine its emotion. Then, analyze the user's facial expressions and vocalizations and determine the user's emotion as well. Suggest appropriate actions if the baby's emotion is sad and the user's emotion is stressed."

[0465] Using this example, the generative AI model can derive specific countermeasures and make effective suggestions to the user.

[0466] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0467] Step 1:

[0468] The device reads the baby's and user's vocalizations, facial expressions, and body movements. Specifically, the device's built-in camera captures the baby's smiling and crying faces and the user's facial expressions in real time, and the microphone records the baby's crying and cooing, as well as the user's vocalizations. This input data (audio and video data) is temporarily stored in the device.

[0469] Step 2:

[0470] The terminal encrypts the collected audio and video data, specifically by encoding it using a data encryption algorithm and preparing it for transmission. This encrypted data is the output.

[0471] Step 3:

[0472] The device sends the encrypted data to the server using a secure protocol (e.g., HTTPS). Specifically, the encrypted audio and video data is sent over the Internet to a specific endpoint on the server.

[0473] Step 4:

[0474] The server receives the encrypted data sent from the terminal and decrypts it. Specifically, it decodes the data using a decryption algorithm running on the server. This decrypted audio and video data becomes the input data for the next process.

[0475] Step 5:

[0476] The server inputs the decoded audio and video data into the emotion determination means. The machine learning model in the server analyzes this data and determines the emotions of the baby and the user. Specifically, it extracts features from the audio data and recognizes patterns in the video data, and outputs the respective emotions (e.g., "sad" or "stressed").

[0477] Step 6:

[0478] The server uses a generative AI model based on the emotion determination results to suggest appropriate actions. For example, based on the emotion determination results that the baby is "sad" and the user is "stressed," the server generates suggestions such as holding the baby and talking to them gently, or encouraging the user to take a deep breath. These generated suggestions are the output.

[0479] Step 7:

[0480] The server then transmits the generated proposal to the terminal. Specifically, the proposal content is formatted as text data or voice data, and then encrypted again before being transmitted to the terminal.

[0481] Step 8:

[0482] The device receives the encrypted proposal data sent from the server and decrypts it. The decrypted proposal content is notified to the user. Specific notification methods include a pop-up message or voice guidance. For example, a message might be displayed saying, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0483] Step 9:

[0484] The user checks the notification from the device and performs the suggested action. For example, they may hold the baby and talk to it gently based on the notification content. Also, if a relaxation method is suggested for the user, they may perform actions such as deep breathing or stretching.

[0485] Through the above steps, a system is realized that determines the emotions of the baby and the user and suggests specific ways of responding based on the determined emotions.

[0486] (Application example 2)

[0487] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0488] Conventional childcare support systems only provide recommendations regarding the baby's condition and do not consider the user's (parent's) emotions or condition, which limits their ability to reduce childcare stress. Furthermore, physical stores provide insufficient support for parents and respond to baby's condition, making it difficult to provide an appropriate childcare environment.

[0489] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion determination means that analyzes received data and determines the emotions of the baby and the user, a suggestion generation means that generates appropriate actions based on the determined emotions, a notification means that notifies the user of the generated suggestions, and an action support means that supports the user in taking actions corresponding to the generated suggestions. This makes it possible to comprehensively analyze the status of the baby and the user, provide appropriate childcare support even in physical stores, and reduce childcare stress.

[0490] 1. "Sensor means" refers to equipment used to read a baby's vocalizations, facial expressions, and body movements.

[0491] 2. "Means of communication" refers to the functions and technologies used to transmit the captured data.

[0492] 3. "Receiving means" means the functions and technologies for receiving and processing transmitted data.

[0493] 4. "Emotion determination means" refers to the functions and technologies for analyzing received data and determining the emotions of the baby and the user.

[0494] 5. "Proposal generation means" refers to the functions and technologies for generating appropriate actions based on the determined emotions.

[0495] 6. "Notification means" refers to the functions and technologies used to notify users of generated suggestions.

[0496] 7. "Action support means" refers to functions and technologies that support users in taking actions corresponding to the generated suggestions.

[0497] 8. A "machine learning model" is an algorithm or framework for analyzing data to identify patterns and make predictions.

[0498] 9. A “generative AI model” is an artificial intelligence model that generates recommendations using a database or known knowledge to suggest appropriate actions.

[0499] This invention is a system that analyzes a baby's emotions from its vocalizations, facial expressions, and body movements, and suggests appropriate responses based on the analysis. Furthermore, this system has the function of analyzing the user's vocalizations, facial expressions, and body movements to determine the user's emotions, and based on this, suggesting additional actions for the baby.

[0500] Hardware and software used

[0501] This system primarily utilizes the following hardware and software:

[0502] Hardware

[0503] Smartphone: Collects data using a camera and microphone.

[0504] Server: Receives and analyzes data.

[0505] software

[0506] OpenCV: Capture images from the camera.

[0507] Keras (TensorFlow backend): Performs sentiment analysis using machine learning models.

[0508] HTTP client library: Communicates with the server (e.g., requests library).

[0509] Data collection

[0510] The device (smartphone) uses a camera and microphone to capture the baby's and user's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it.

[0511] Data transmission and reception

[0512] The collected audio and video data is encrypted and sent to a server using a secure protocol (e.g., HTTPS). The server receives the encrypted data sent from the device and decrypts it.

[0513] Data analysis

[0514] The server inputs the received audio and video data into a machine learning model to determine the emotions of the baby and the user. The machine learning model analyzes the baby's crying and facial expressions, as well as the user's vocalizations and facial expressions, to identify emotions.

[0515] Suggested Actions

[0516] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. This generative AI model generates specific responses by using the baby's and user's emotions, as well as past databases and existing knowledge. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will generate a suggestion to "hold the baby and talk to him gently" and suggest actions to the user such as "take a deep breath and relax."

[0517] Proposal notification and support

[0518] The device receives the proposed action from the server and notifies the user. The notification method is a pop-up message or voice guidance. The user checks the notification and performs the proposed action. The action support means also supports the proposed action.

[0519] Specific examples

[0520] For example, if a baby starts crying in a physical store selling childcare products, the app will analyze the image and sound of the crying baby and notify the user with a message such as "Hold the baby and talk to him / her gently." It will also suggest actions such as "Take a deep breath and relax." This will provide appropriate childcare support even in physical stores, reducing parenting stress for parents.

[0521] Prompt Sentence Examples

[0522] "Judge the baby's emotions and suggest appropriate parenting actions. Refer to and analyze the baby's facial expressions and cries to inform parents of specific ways to respond."

[0523] This system determines the emotions of the baby and the user from their vocalizations, facial expressions, and physical movements, and by appropriately supporting the user's behavior, it helps to reduce anxiety and stress in child-rearing and facilitate smooth communication between parents and children.

[0524] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0525] Step 1: Data collection

[0526] The device uses a camera to capture the facial expressions of the baby and the user, and a microphone to collect vocalization data. The image data captured by the camera and the audio data collected by the microphone are input. These data are collected in real time and temporarily stored in the device. The output of this step is image data and audio data.

[0527] Step 2: Send data

[0528] The device encrypts the collected image and audio data and sends it to the server using a secure protocol (HTTPS). The encrypted data is input and sent to a specific endpoint on the server. This reduces the risk of the data being stolen or tampered with during transmission. The output is a confirmation that the data has been received by the server.

[0529] Step 3: Receiving and Decrypting Data

[0530] The server receives the encrypted data sent from the device and decrypts it. Encrypted image and audio data are the input, and decrypted data is the output. Decrypting the data converts it into a format that can be used for analysis.

[0531] Step 4: Data analysis

[0532] The server inputs the decoded image data and audio data into a machine learning model to determine the emotions of the baby and the user. The input is the decoded data, which is analyzed using the machine learning model. The emotion determination result is output through data analysis. Specifically, the model analyzes the baby's crying face and crying sound, as well as the user's facial expressions and voice characteristics, to determine emotions such as "sadness" or "stress."

[0533] Step 5: Action Creation

[0534] The server uses a generative AI model based on the emotion determination result to suggest an appropriate action. The input is the emotion determination result, and a specific response method is generated using a past database and existing knowledge. For example, if a baby is determined to be "sad," the generative AI model will make a suggestion such as "hold the baby and talk to him gently." The output is the generated action proposal.

[0535] Step 6: Proposal Notification

[0536] The device receives the suggestions sent from the server and notifies the user. The generated action suggestions are input, and the device displays them as pop-up messages or voice guidance. The output is a notification to the user. Specifically, messages such as "Please hold the baby and speak to him gently" and "The user should take a deep breath and relax" are displayed.

[0537] Step 7: Take action and support

[0538] The user checks the notification from the device and performs the suggested action. The notified action suggestion becomes the input, and the user actually takes the action. The action support means supports the user's action, for example, it can provide specific guides and reminders. The output is the user's execution and the support results.

[0539] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0540] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0541] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0542] [Third embodiment]

[0543] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0544] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0545] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0546] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0547] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0548] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0549] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0550] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0551] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0552] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0553] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0554] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0555] The present invention is a system that analyzes a baby's emotions from their vocalizations, facial expressions, and body movements, and suggests appropriate responses to parents. This system is mainly composed of three components: a terminal (including a smartphone), a server, and a user. Specific embodiments of the present invention will be described below.

[0556] Device data collection:

[0557] The device uses a camera and microphone to read the baby's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it within the device. For example, the device captures the baby's smiling and crying faces with the camera, and records crying and cooing (gentle voices) with the microphone.

[0558] Sending data from device to server:

[0559] The collected data is encrypted and then sent to the server using a secure protocol (e.g., HTTPS). This ensures the safety of the data. As a specific example, the baby's crying voice data and facial expression data are encrypted and then sent to a specific endpoint on the server.

[0560] Data analysis by server:

[0561] The server decodes the received data and performs processing to determine the baby's emotions. The emotion determination means analyzes the data using a machine learning model. For example, the server analyzes the specific patterns of the baby's crying and the characteristics of the crying face to determine that the baby is "sad."

[0562] Server suggested actions:

[0563] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model generates specific responses by referring to the baby's emotions and past data. For example, if the baby is determined to be "sad," the server will suggest "hold the baby and talk to him / her gently."

[0564] Device suggestion notification:

[0565] The server sends the generated suggestions to the device, which then notifies the user. Possible notification methods include a pop-up message or voice guidance. The user checks the notification and performs the suggested action. For example, an app might display a notification saying, "The baby feels sad. Hold the baby and talk to him gently."

[0566] This system allows users (parents and caregivers) to quickly and accurately grasp a baby's emotions and respond appropriately. This facilitates communication between parents and children and helps reduce the burden of childcare. The system we propose will contribute to the healthy growth of babies and reduce the stress of childcare for parents.

[0567] The processing flow will be explained below.

[0568] Step 1: Collect data (device)

[0569] The device activates a microphone and camera to capture the baby's vocalizations, facial expressions, and body movements. The microphone records the baby's vocalizations (crying and cooing), and the camera captures the baby's facial expressions and body movements (such as limb movements). The collected audio and video data is stored in temporary storage within the device.

[0570] Step 2: Sending data (terminal)

[0571] The collected audio and video data is encrypted and sent to a server using a secure protocol (e.g., HTTPS), which reduces the risk of the data being stolen or tampered with during transmission.

[0572] Step 3: Receiving and analyzing data (server)

[0573] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data is input to the emotion determination means. The emotion determination means analyzes the data using a machine learning model and determines the baby's emotion. For example, it detects specific crying patterns and facial features and determines the emotion as "sad."

[0574] Step 4: Proposing Actions (Server)

[0575] The server uses a generative AI model to suggest appropriate actions based on the emotion data determined by the emotion determination means. The generative AI model uses past databases and existing knowledge to suggest how the user should respond to the baby. For example, if the emotion is determined to be "sad," a specific action such as "hold the baby and talk to him / her gently" is generated.

[0576] Step 5: Proposal notification (device)

[0577] The device receives the suggestions sent from the server and notifies the user. The notification is done via a pop-up message or voice guidance, helping the user to respond quickly. For example, a message such as "The baby is feeling sad. Please hold the baby and talk to him / her gently" may be displayed.

[0578] Step 6: Execute Action (User)

[0579] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user can take action by holding the baby and talking to it gently. This series of actions allows the user to understand the baby's emotions and respond appropriately.

[0580] In this way, this system, which determines a baby's emotions from their vocalizations, facial expressions, and body movements and suggests appropriate responses, helps to reduce anxiety in child-rearing and facilitates smooth communication between parents and children.

[0581] Example 1

[0582] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0583] In today's childcare environment, it is important for parents and caregivers to quickly and accurately grasp a baby's emotions and needs and respond appropriately, but this can be difficult, especially for first-time parents. The present invention aims to solve this problem by providing a system that analyzes a baby's emotions and suggests appropriate responses.

[0584] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0585] In this invention, the server includes a sensor means for reading the baby's vocalizations, facial expressions, and body movements, a storage means for temporarily storing the read data, an encryption means for encrypting the stored data, a communication means for transmitting the encrypted data, a receiving means for receiving and decrypting the transmitted data, an emotion determination means for analyzing the decrypted data and determining the baby's emotion, a suggestion generation means using a generative AI model for generating appropriate actions based on the determined emotion, and a notification means for notifying the user of the generated suggestion. This enables parents and caregivers to quickly and accurately grasp the baby's emotion and take appropriate action.

[0586] "Sensor means" refers to hardware that reads the baby's vocalizations, facial expressions, and body movements, as well as software that controls its operation.

[0587] The "storage means" is a memory or storage device for temporarily storing data read by the sensor means.

[0588] "Encryption means" refers to software and its execution environment for encrypting data using an encryption algorithm to protect the stored data.

[0589] A "communication means" is software that implements a network interface and communication protocol for transmitting encrypted data.

[0590] The "receiving means" refers to hardware and software for receiving and decoding data transmitted by the transmitting means.

[0591] The "emotion determination means" is a machine learning model and its execution environment for analyzing the data decoded by the receiving means and determining the baby's emotions.

[0592] The "proposal generation means" refers to software and its execution environment for generating appropriate actions using a generative AI model based on the determined emotions.

[0593] The "notification means" is a part of the application for notifying the user of the generated proposal, and hardware that realizes the notification function.

[0594] This system analyzes a baby's emotions based on their vocalizations, facial expressions, and body movements, and suggests appropriate responses to parents. This system is primarily composed of three components: a terminal, a server, and a user.

[0595] Data collection by terminal

[0596] The device uses a camera and microphone as sensors to read the baby's vocalizations, facial expressions, and body movements. These sensors capture the baby's smiling or crying face with the camera, and record the baby's cries and cooing (gentle voices) with the microphone. For example, the camera takes a picture of the baby's face every second, while the microphone simultaneously collects audio data.

[0597] Data Retention and Encryption

[0598] The device temporarily stores the captured data in its internal memory, which ensures data integrity in real time. The stored data is encrypted using the AES encryption algorithm to ensure security.

[0599] Sending data from the device to the server

[0600] The encrypted data is sent to the server via a communication means. Specifically, the encrypted data is sent to the server via a POST request using HTTPS, a secure communication protocol.

[0601] Data analysis by server

[0602] The server receives the transmitted data and decodes it using a decoding method. The decoded data contains the baby's voice and video, which are then analyzed using a machine learning model. To determine emotions, models such as CNN (convolutional neural network) and RNN (recurrent neural network) are used to extract and analyze voice frequency patterns and facial expression features. For example, if a baby's crying has a specific pattern that indicates "sadness," the server will determine that as the emotion.

[0603] Server-based action suggestions

[0604] The server uses a generative AI model to suggest appropriate actions based on the emotional information determined from the analysis. Examples of generative AI models that can be used include GPT-3. Prompts are sent to the model, which suggests specific actions that parents or caregivers can take. For example, by sending the prompt "What should you do when your baby is sad?", the server generates the specific response of "hold the baby and talk to him / her gently."

[0605] Proposal notification on device

[0606] The generated suggestion is sent back to the terminal and notified to the user via a notification means. Possible notification methods include a pop-up message or voice guidance. For example, a notification message such as "The baby is feeling sad. Please hold the baby and talk to him / her gently" may be displayed.

[0607] Specific examples

[0608] When a baby starts crying, the device's camera captures the crying face and the microphone records the sound. This data is encrypted and sent to a server, where a machine learning model analyzes the emotion as "sad." The generative AI model then generates a suggestion to "hold the baby and talk to him / her gently" and notifies the device.

[0609] Example prompts for generative AI models

[0610] "Analyze the crying data and crying face image data to determine the baby's emotions and suggest appropriate ways to respond."

[0611] This system allows users to understand their baby's emotions in real time and respond appropriately, which is expected to improve communication between parents and children and reduce the burden of childcare.

[0612] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0613] Step 1:

[0614] The device initializes the camera and microphone at system startup, specifically by setting the camera resolution and frame rate and the microphone sampling rate. The input to this process is the camera and microphone hardware, and the output is the ready-to-use sensor device.

[0615] Step 2:

[0616] The device reads and captures the baby's vocalizations, facial expressions, and body movements at regular intervals. The input is real-time video and audio, and the output is the image frame and audio data at that time. Specifically, the device captures an image frame from the camera every second and simultaneously records audio data from the microphone.

[0617] Step 3:

[0618] The device temporarily stores the captured data in its internal memory. The input is the captured image and audio data, and the output is the temporarily stored data. Specifically, this data is organized in chronological order and stored in a buffer.

[0619] Step 4:

[0620] The terminal encrypts the stored data. The input is the temporarily stored data, and the output is the encrypted data. Specifically, the data is encrypted using the AES encryption algorithm.

[0621] Step 5:

[0622] The device sends encrypted data to the server. The input is the encrypted data, and the output is a notification to the server that the data has been sent. Specifically, the device composes an HTTPS POST request and sends the data to the server's endpoint.

[0623] Step 6:

[0624] The server receives the transmitted data and decrypts it. The input is the encrypted data, and the output is the original decrypted data. Specifically, the server decrypts the data using the AES encryption key used on the terminal.

[0625] Step 7:

[0626] The server analyzes the decoded data and determines the baby's emotions. The input is the decoded voice data and image data, and the output is the emotion determination result. Specifically, it uses a machine learning model (e.g., CNN or RNN) to analyze the frequency patterns of the voice and facial features.

[0627] Step 8:

[0628] The server uses a generative AI model to suggest appropriate actions based on the determined emotion information. The input is the emotion determination result, and the output is a specific action suggestion. As a specific operation, the server sends a prompt sentence to the generative AI model (e.g., GPT-3) saying, "What should you do when a baby is sad?"

[0629] Step 9:

[0630] The server sends the generated action proposal to the terminal. The input is the specific action proposal, and the output is a notification to the terminal that the action proposal has been sent.

[0631] Step 10:

[0632] The device notifies the user of the received action suggestion. The input is the action suggestion sent from the server, and the output is the notification displayed to the user. Specific actions include using a pop-up message or voice guidance to display a notification such as "The baby is feeling sad. Please hold the baby and talk to him gently."

[0633] Through these steps, the system can analyze the baby's emotions and suggest appropriate responses to parents or caregivers in real time.

[0634] (Application example 1)

[0635] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0636] Conventional baby emotion analysis systems only address the specific situations of babies, making it difficult to apply them to other purposes. Furthermore, when dealing with customers in brick-and-mortar stores, it is difficult to quickly and accurately grasp their emotions and provide appropriate service. To improve this situation, a system is needed that can analyze the emotions of babies and customers and propose appropriate responses.

[0637] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0638] In this invention, the server includes a sensor means for reading the vocalizations, facial expressions, and body movements of the baby or customer, a communication means for transmitting the read data, a receiving means for receiving and processing the transmitted data, an emotion determination means for analyzing the received data and determining the emotion of the baby or customer, a proposal generation means for generating an appropriate action based on the determined emotion, and a notification means for notifying the user of the generated proposal, thereby enabling an analysis of the emotion of the customer or baby and a proposal of a countermeasure.

[0639] "Sensor means" refers to a device for reading the vocalizations, facial expressions, and body movements of the baby or customer.

[0640] "Communication means" refers to the device or technology used to transmit the scanned data to another device or server.

[0641] A "receiving means" is a device or module for receiving and processing transmitted data.

[0642] The "emotion determination means" is a device or algorithm for analyzing the received data and determining the emotion of the baby or customer.

[0643] The "suggestion generator" is a device or algorithm for generating appropriate actions based on the determined emotion.

[0644] The "notification means" is a device or interface for notifying the user of the generated proposal.

[0645] A "machine learning model" is an algorithm or mathematical model that analyzes data and finds patterns.

[0646] A "generative AI model" is an artificial intelligence model that generates appropriate output based on specific input.

[0647] This invention is a system that analyzes the emotions of a baby or customer based on their vocalizations, facial expressions, and body movements, and proposes appropriate responses. This system is mainly composed of three entities: a terminal, a server, and a user.

[0648] Device data collection:

[0649] The device (such as a smartphone or a robot in a brick-and-mortar store) uses a camera and microphone to read the baby's or customer's vocalizations, facial expressions, and body movements. The data collected by these sensors is temporarily stored on the device. Specific examples include a camera capturing the customer's facial expressions and a microphone recording the tone of voice.

[0650] Sending data from device to server:

[0651] The collected data is encrypted and sent to the server using a secure communication protocol (e.g., HTTPS). This ensures the security of the data. For example, customer voice and facial expression data are encrypted and sent to the server endpoint.

[0652] Data analysis by server:

[0653] The server decodes the received data and analyzes it using an emotion determination method, which uses a machine learning model. For example, the server analyzes specific patterns in the customer's voice and facial features to determine that the customer is "tired."

[0654] Server suggested actions:

[0655] Based on the analysis results, a generative AI model is used to suggest appropriate actions. The generative AI model compares the results with past data to generate specific responses. For example, if the user is determined to be "tired," the server will suggest "please offer me a drink."

[0656] Device suggestion notification:

[0657] The server sends the generated suggestions to the device, which then notifies the user. Possible notification methods include a pop-up message or voice guidance. For example, the app might display a notification saying, "You seem tired. Let's offer you a drink."

[0658] This allows users (store staff and parents) to quickly and accurately grasp the emotions of customers and babies and respond appropriately, which contributes to improving customer satisfaction and facilitating communication between parents and children, and improving the overall service experience.

[0659] Example prompt sentence:

[0660] "If you determine from a customer's facial expression and voice that they are tired, please suggest what kind of hospitality they should provide."

[0661] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0662] Step 1: Data collection

[0663] The device (a smartphone or a robot in a brick-and-mortar store) uses a camera and microphone to read the baby's or customer's vocalizations, facial expressions, and body movements in real time. Specifically, the camera captures facial expressions and the microphone records voice. The input for this step is the baby's or customer's real-time voice and facial expressions, and the output is these audio and video data.

[0664] Step 2: Send data

[0665] The device encrypts the collected data and sends it securely to the server using the HTTPS protocol, ensuring the safety of the data. Specifically, the device encodes and encrypts the audio and video data and sends it to a specific endpoint on the server. The input to this step is the collected audio and video data, and the output is the encrypted data.

[0666] Step 3: Receiving and Decrypting Data

[0667] The server receives and decrypts the data sent from the terminal. Specifically, it decodes the received data and converts it back to the original audio and video data. The input of this step is the encrypted data, and the output is the decrypted audio and video data.

[0668] Step 4: Sentiment Analysis

[0669] The server uses emotion determination means to analyze the received data and determine the baby's or customer's emotion. Using a machine learning model, it analyzes specific patterns of voice tone and facial expressions to determine specific emotions (e.g., "tired," "sad," etc.). The input of this step is the decoded audio and video data, and the output is the emotion determination result.

[0670] Step 5: Action proposal generation

[0671] The server uses a generative AI model to suggest an appropriate action based on the determined emotion. For example, if the customer is determined to be "tired," the server generates the suggestion "Please offer me a drink." Specifically, the emotion determination result is input into the AI ​​model, and an appropriate response prompt is generated. The input for this step is the emotion determination result, and the output is the response prompt.

[0672] Step 6: Proposal Notification

[0673] The server sends the generated suggestion to the terminal, which then notifies the user (parent or store staff). Notification methods include a pop-up message on the terminal or voice guidance. Specifically, the suggestion is sent from the server to the terminal, and the terminal displays the suggestion on the screen or provides voice guidance. The input to this step is a prompt for a solution, and the output is the notification to the user.

[0674] Through the above steps, the system is able to quickly and accurately analyze the emotions of the baby or customer and propose appropriate countermeasures.

[0675] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0676] The present invention is a system that analyzes a baby's emotions from its vocalizations, facial expressions, and body movements, and suggests appropriate responses based on the results.The present invention also includes a system that analyzes a user's vocalizations, facial expressions, and body movements to determine the user's emotions, and suggests additional actions for the baby based on the determined emotions.The present invention is mainly composed of three entities: a terminal (including a smartphone), a server, and a user.Specific embodiments of the present invention will be described below.

[0677] Device data collection:

[0678] The device uses a camera and microphone to capture the baby's and the user's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it within the device. For example, the device captures the baby's smiling and crying faces and the user's facial expressions with the camera, and records the baby's crying and cooing and the user's vocalizations with the microphone.

[0679] Sending data from device to server:

[0680] The collected audio and video data is encrypted and sent to the server using a secure protocol (e.g., HTTPS). Because the data is encrypted before being sent, the risk of the data being stolen or tampered with during transmission is reduced. As a specific example, the baby's crying voice data and facial expression data, and the user's voice and facial expression data are encrypted before being sent to a specific endpoint on the server.

[0681] Data analysis by server:

[0682] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data is input to the emotion determination means. The emotion determination means analyzes the data using a machine learning model to determine the emotions of the baby and the user. For example, the server analyzes the specific pattern of the baby's crying and the characteristics of the crying face to determine the emotion as "sad." It also analyzes the user's vocalizations and facial expressions to determine that the user is "feeling stressed."

[0683] Server suggested actions:

[0684] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model uses the baby's and user's emotions, as well as past databases and known knowledge, to generate specific responses. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will generate a suggestion to "hold the baby and talk to him gently" to calm the baby, and also suggest actions to the user such as "take a deep breath and relax."

[0685] Device suggestion notification:

[0686] The device receives the suggestions sent from the server and notifies the user. Possible notification methods include a pop-up message or voice guidance. The user checks the notification and performs the suggested action. For example, the app displays the message, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0687] User performs the action:

[0688] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user will take action such as holding the baby and talking to it gently. Also, if the notification suggests ways for the user to relax, the user will take action such as deep breathing or stretching.

[0689] In this way, this system, which determines the emotions of the baby and the user from their vocalizations, facial expressions, and physical movements and suggests appropriate responses, helps to reduce anxiety in child-rearing and facilitates communication between parents and children. Furthermore, by taking the user's emotions into consideration, it also contributes to reducing the stress of child-rearing.

[0690] The processing flow will be explained below.

[0691] Step 1: Collect data (device)

[0692] The device activates a microphone and camera to capture the baby's and the user's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it within the device. For example, the device captures the baby's smiling and crying face with the camera and records the baby's crying and cooing with the microphone. For the user, the device captures facial expressions with the camera and records the tone of voice with the microphone.

[0693] Step 2: Sending data (terminal)

[0694] The collected audio and video data is encrypted and sent to the server using a secure protocol (e.g., HTTPS). Because the data is encrypted before being sent, the risk of the data being stolen or tampered with during transmission is reduced. Specifically, the baby's crying data, facial expression data, and the user's voice and facial expression data are encrypted before being sent to a specific endpoint on the server.

[0695] Step 3: Receiving and analyzing data (server)

[0696] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data are input to the emotion determination means. The emotion determination means analyzes the data using a machine learning model to determine the emotions of the baby and the user. For example, the server may analyze the specific pattern of the baby's crying and the characteristics of the crying face to determine the emotion as "sad," and simultaneously analyze the user's facial expression and tone of voice to determine that the user is "feeling stressed."

[0697] Step 4: Proposing Actions (Server)

[0698] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model uses the baby's and user's emotions, as well as past databases and known knowledge, to generate specific responses. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will suggest "hold the baby and talk to him / her gently" and generate an action such as "the user should take a deep breath and relax."

[0699] Step 5: Proposal notification (device)

[0700] The device receives the suggestions sent from the server and notifies the user. Possible notification methods include a pop-up message or voice guidance. The user checks the notification and performs the suggested action. For example, the app displays the message, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0701] Step 6: Execute Action (User)

[0702] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user will take action such as holding the baby and talking to it gently. If a relaxation method is suggested for the user, the user will take action such as deep breathing or stretching. For example, by taking a deep breath as instructed by the device notification and then holding the baby, the stress of childcare can be reduced.

[0703] In this way, this system, which determines the emotions of the baby and the user from their vocalizations, facial expressions, and physical movements and suggests appropriate responses, helps to reduce anxiety in child-rearing and facilitates communication between parents and children. Furthermore, by taking the user's emotions into consideration, it also contributes to reducing the stress of child-rearing.

[0704] Example 2

[0705] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0706] Conventional childcare support systems have limited methods for responding to specific baby behaviors, making it difficult to accurately assess the emotions of both the baby and the parent and to provide specific responses based on those emotions. This has limited the effectiveness of reducing stress and anxiety in childcare.

[0707] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means including an input device that reads the baby's vocalizations, facial expressions, and body movements, data sending means that encrypts and sends the read data, receiving means that receives and decrypts the sent data, emotion determining means that analyzes the decrypted data and determines the baby's emotion and the user's emotion, a generation engine including suggestion generating means that generates appropriate actions based on the determined emotions, and notification means that notifies the user of the generated suggestions. This makes it possible to quickly and accurately determine the emotions of the baby and the user and to suggest specific and feasible ways of responding based on the determined emotions.

[0708] An "input device" is a device that includes sensors such as a camera or microphone to read a baby's vocalizations, facial expressions, and body movements.

[0709] The "data transmission means" is a means having a function of encrypting the read data and transmitting it safely to the server.

[0710] The "receiving means" is a means for receiving and decrypting the transmitted encrypted data.

[0711] The "emotion determination means" is a means for analyzing received audio data and video data using a machine learning model to determine the emotions of the baby and the user.

[0712] The "suggestion generation means" is a means for using a generative AI model to suggest appropriate actions to the baby and the user based on the determined emotions.

[0713] The "notification means" is a means for notifying the user of the generated proposal, and includes a pop-up message, a voice guide, and the like.

[0714] This invention is a childcare support system that analyzes a baby's vocalizations, facial expressions, and body movements and suggests appropriate actions based on the results. It also has a function that analyzes the user's vocalizations, facial expressions, and body movements to determine the user's emotions and suggest additional actions for the baby. This system is mainly composed of three entities: a terminal (including a smartphone), a server, and a user.

[0715] The device uses a camera and microphone to capture the baby's and user's vocalizations, facial expressions, and body movements. Specifically, the device captures the baby's smiling and crying faces and the user's facial expressions with the camera, and records the baby's crying and cooing sounds and the user's vocalizations with the microphone. The collected data is temporarily stored in the device.

[0716] The collected audio and video data is encrypted and sent to the server using a secure protocol (e.g., HTTPS). For example, the baby's crying voice data and facial expression data, and the user's voice and facial expression data are encrypted and then sent to a specific endpoint on the server.

[0717] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data is input into the emotion determination function, which uses a machine learning model to analyze the data and determine the emotions of the baby and the user. For example, the server analyzes the specific patterns of a baby's crying and the characteristics of its crying face to determine the emotion as "sad." It also analyzes the user's vocalizations and facial expressions to determine that the user is "feeling stressed."

[0718] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model generates specific responses by using the baby's and user's emotions, as well as past databases and known knowledge. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will generate a suggestion to "hold the baby and talk to him gently" to calm the baby, and suggest actions to the user such as "take a deep breath and relax."

[0719] The device receives the suggestions sent from the server and notifies the user. Possible notification methods include a pop-up message or voice guidance. Specifically, the device app displays a message saying, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0720] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user can hold the baby and talk to him / her gently. If the notification suggests relaxation techniques for the user, the user can take actions such as deep breathing or stretching.

[0721] This system can reduce anxiety in child-rearing and facilitate communication between parents and children. It also contributes to reducing stress in child-rearing by taking into consideration the user's emotions.

[0722] For example, you might input the following prompt into a generative AI model:

[0723] "Analyze the baby's crying pattern and determine its emotion. Then, analyze the user's facial expressions and vocalizations and determine the user's emotion as well. Suggest appropriate actions if the baby's emotion is sad and the user's emotion is stressed."

[0724] Using this example, the generative AI model can derive specific countermeasures and make effective suggestions to the user.

[0725] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0726] Step 1:

[0727] The device reads the baby's and user's vocalizations, facial expressions, and body movements. Specifically, the device's built-in camera captures the baby's smiling and crying faces and the user's facial expressions in real time, and the microphone records the baby's crying and cooing, as well as the user's vocalizations. This input data (audio and video data) is temporarily stored in the device.

[0728] Step 2:

[0729] The terminal encrypts the collected audio and video data, specifically by encoding it using a data encryption algorithm and preparing it for transmission. This encrypted data is the output.

[0730] Step 3:

[0731] The device sends the encrypted data to the server using a secure protocol (e.g., HTTPS). Specifically, the encrypted audio and video data is sent over the Internet to a specific endpoint on the server.

[0732] Step 4:

[0733] The server receives the encrypted data sent from the terminal and decrypts it. Specifically, it decodes the data using a decryption algorithm running on the server. This decrypted audio and video data becomes the input data for the next process.

[0734] Step 5:

[0735] The server inputs the decoded audio and video data into the emotion determination means. The machine learning model in the server analyzes this data and determines the emotions of the baby and the user. Specifically, it extracts features from the audio data and recognizes patterns in the video data, and outputs the respective emotions (e.g., "sad" or "stressed").

[0736] Step 6:

[0737] The server uses a generative AI model based on the emotion determination results to suggest appropriate actions. For example, based on the emotion determination results that the baby is "sad" and the user is "stressed," the server generates suggestions such as holding the baby and talking to them gently, or encouraging the user to take a deep breath. These generated suggestions are the output.

[0738] Step 7:

[0739] The server then transmits the generated proposal to the terminal. Specifically, the proposal content is formatted as text data or voice data, and then encrypted again before being transmitted to the terminal.

[0740] Step 8:

[0741] The device receives the encrypted proposal data sent from the server and decrypts it. The decrypted proposal content is notified to the user. Specific notification methods include a pop-up message or voice guidance. For example, a message might be displayed saying, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0742] Step 9:

[0743] The user checks the notification from the device and performs the suggested action. For example, they may hold the baby and talk to it gently based on the notification content. Also, if a relaxation method is suggested for the user, they may perform actions such as deep breathing or stretching.

[0744] Through the above steps, a system is realized that determines the emotions of the baby and the user and suggests specific ways of responding based on the determined emotions.

[0745] (Application example 2)

[0746] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0747] Conventional childcare support systems only provide recommendations regarding the baby's condition and do not consider the user's (parent's) emotions or condition, which limits their ability to reduce childcare stress. Furthermore, physical stores provide insufficient support for parents and respond to baby's condition, making it difficult to provide an appropriate childcare environment.

[0748] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion determination means that analyzes received data and determines the emotions of the baby and the user, a suggestion generation means that generates appropriate actions based on the determined emotions, a notification means that notifies the user of the generated suggestions, and an action support means that supports the user in taking actions corresponding to the generated suggestions. This makes it possible to comprehensively analyze the status of the baby and the user, provide appropriate childcare support even in physical stores, and reduce childcare stress.

[0749] 1. "Sensor means" refers to equipment used to read a baby's vocalizations, facial expressions, and body movements.

[0750] 2. "Means of communication" refers to the functions and technologies used to transmit the captured data.

[0751] 3. "Receiving means" means the functions and technologies for receiving and processing transmitted data.

[0752] 4. "Emotion determination means" refers to the functions and technologies for analyzing received data and determining the emotions of the baby and the user.

[0753] 5. "Proposal generation means" refers to the functions and technologies for generating appropriate actions based on the determined emotions.

[0754] 6. "Notification means" refers to the functions and technologies used to notify users of generated suggestions.

[0755] 7. "Action support means" refers to functions and technologies that support users in taking actions corresponding to the generated suggestions.

[0756] 8. A "machine learning model" is an algorithm or framework for analyzing data to identify patterns and make predictions.

[0757] 9. A “generative AI model” is an artificial intelligence model that generates recommendations using a database or known knowledge to suggest appropriate actions.

[0758] This invention is a system that analyzes a baby's emotions from its vocalizations, facial expressions, and body movements, and suggests appropriate responses based on the analysis. Furthermore, this system has the function of analyzing the user's vocalizations, facial expressions, and body movements to determine the user's emotions, and based on this, suggesting additional actions for the baby.

[0759] Hardware and software used

[0760] This system primarily utilizes the following hardware and software:

[0761] Hardware

[0762] Smartphone: Collects data using a camera and microphone.

[0763] Server: Receives and analyzes data.

[0764] software

[0765] OpenCV: Capture images from the camera.

[0766] Keras (TensorFlow backend): Performs sentiment analysis using machine learning models.

[0767] HTTP client library: Communicates with the server (e.g., requests library).

[0768] Data collection

[0769] The device (smartphone) uses a camera and microphone to capture the baby's and user's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it.

[0770] Data transmission and reception

[0771] The collected audio and video data is encrypted and sent to a server using a secure protocol (e.g., HTTPS). The server receives the encrypted data sent from the device and decrypts it.

[0772] Data analysis

[0773] The server inputs the received audio and video data into a machine learning model to determine the emotions of the baby and the user. The machine learning model analyzes the baby's crying and facial expressions, as well as the user's vocalizations and facial expressions, to identify emotions.

[0774] Suggested Actions

[0775] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. This generative AI model generates specific responses by using the baby's and user's emotions, as well as past databases and existing knowledge. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will generate a suggestion to "hold the baby and talk to him gently" and suggest actions to the user such as "take a deep breath and relax."

[0776] Proposal notification and support

[0777] The device receives the proposed action from the server and notifies the user. The notification method is a pop-up message or voice guidance. The user checks the notification and performs the proposed action. The action support means also supports the proposed action.

[0778] Specific examples

[0779] For example, if a baby starts crying in a physical store selling childcare products, the app will analyze the image and sound of the crying baby and notify the user with a message such as "Hold the baby and talk to him / her gently." It will also suggest actions such as "Take a deep breath and relax." This will provide appropriate childcare support even in physical stores, reducing parenting stress for parents.

[0780] Prompt Sentence Examples

[0781] "Judge the baby's emotions and suggest appropriate parenting actions. Refer to and analyze the baby's facial expressions and cries to inform parents of specific ways to respond."

[0782] This system determines the emotions of the baby and the user from their vocalizations, facial expressions, and physical movements, and by appropriately supporting the user's behavior, it helps to reduce anxiety and stress in child-rearing and facilitate smooth communication between parents and children.

[0783] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0784] Step 1: Data collection

[0785] The device uses a camera to capture the facial expressions of the baby and the user, and a microphone to collect vocalization data. The image data captured by the camera and the audio data collected by the microphone are input. These data are collected in real time and temporarily stored in the device. The output of this step is image data and audio data.

[0786] Step 2: Send data

[0787] The device encrypts the collected image and audio data and sends it to the server using a secure protocol (HTTPS). The encrypted data is input and sent to a specific endpoint on the server. This reduces the risk of the data being stolen or tampered with during transmission. The output is a confirmation that the data has been received by the server.

[0788] Step 3: Receiving and Decrypting Data

[0789] The server receives the encrypted data sent from the device and decrypts it. Encrypted image and audio data are the input, and decrypted data is the output. Decrypting the data converts it into a format that can be used for analysis.

[0790] Step 4: Data analysis

[0791] The server inputs the decoded image data and audio data into a machine learning model to determine the emotions of the baby and the user. The input is the decoded data, which is analyzed using the machine learning model. The emotion determination result is output through data analysis. Specifically, the model analyzes the baby's crying face and crying sound, as well as the user's facial expressions and voice characteristics, to determine emotions such as "sadness" or "stress."

[0792] Step 5: Action Creation

[0793] The server uses a generative AI model based on the emotion determination result to suggest an appropriate action. The input is the emotion determination result, and a specific response method is generated using a past database and existing knowledge. For example, if a baby is determined to be "sad," the generative AI model will make a suggestion such as "hold the baby and talk to him gently." The output is the generated action proposal.

[0794] Step 6: Proposal Notification

[0795] The device receives the suggestions sent from the server and notifies the user. The generated action suggestions are input, and the device displays them as pop-up messages or voice guidance. The output is a notification to the user. Specifically, messages such as "Please hold the baby and speak to him gently" and "The user should take a deep breath and relax" are displayed.

[0796] Step 7: Take action and support

[0797] The user checks the notification from the device and performs the suggested action. The notified action suggestion becomes the input, and the user actually takes the action. The action support means supports the user's action, for example, it can provide specific guides and reminders. The output is the user's execution and the support results.

[0798] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0799] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0800] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0801] [Fourth embodiment]

[0802] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0803] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0804] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0805] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0806] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0807] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0808] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0809] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0810] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0811] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0812] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0813] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0814] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0815] The present invention is a system that analyzes a baby's emotions from their vocalizations, facial expressions, and body movements, and suggests appropriate responses to parents. This system is mainly composed of three components: a terminal (including a smartphone), a server, and a user. Specific embodiments of the present invention will be described below.

[0816] Device data collection:

[0817] The device uses a camera and microphone to read the baby's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it within the device. For example, the device captures the baby's smiling and crying faces with the camera, and records crying and cooing (gentle voices) with the microphone.

[0818] Sending data from device to server:

[0819] The collected data is encrypted and then sent to the server using a secure protocol (e.g., HTTPS). This ensures the safety of the data. As a specific example, the baby's crying voice data and facial expression data are encrypted and then sent to a specific endpoint on the server.

[0820] Data analysis by server:

[0821] The server decodes the received data and performs processing to determine the baby's emotions. The emotion determination means analyzes the data using a machine learning model. For example, the server analyzes the specific patterns of the baby's crying and the characteristics of the crying face to determine that the baby is "sad."

[0822] Server suggested actions:

[0823] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model generates specific responses by referring to the baby's emotions and past data. For example, if the baby is determined to be "sad," the server will suggest "hold the baby and talk to him / her gently."

[0824] Device suggestion notification:

[0825] The server sends the generated suggestions to the device, which then notifies the user. Possible notification methods include a pop-up message or voice guidance. The user checks the notification and performs the suggested action. For example, an app might display a notification saying, "The baby feels sad. Hold the baby and talk to him gently."

[0826] This system allows users (parents and caregivers) to quickly and accurately grasp a baby's emotions and respond appropriately. This facilitates communication between parents and children and helps reduce the burden of childcare. The system we propose will contribute to the healthy growth of babies and reduce the stress of childcare for parents.

[0827] The processing flow will be explained below.

[0828] Step 1: Collect data (device)

[0829] The device activates a microphone and camera to capture the baby's vocalizations, facial expressions, and body movements. The microphone records the baby's vocalizations (crying and cooing), and the camera captures the baby's facial expressions and body movements (such as limb movements). The collected audio and video data is stored in temporary storage within the device.

[0830] Step 2: Sending data (terminal)

[0831] The collected audio and video data is encrypted and sent to a server using a secure protocol (e.g., HTTPS), which reduces the risk of the data being stolen or tampered with during transmission.

[0832] Step 3: Receiving and analyzing data (server)

[0833] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data is input to the emotion determination means. The emotion determination means analyzes the data using a machine learning model and determines the baby's emotion. For example, it detects specific crying patterns and facial features and determines the emotion as "sad."

[0834] Step 4: Proposing Actions (Server)

[0835] The server uses a generative AI model to suggest appropriate actions based on the emotion data determined by the emotion determination means. The generative AI model uses past databases and existing knowledge to suggest how the user should respond to the baby. For example, if the emotion is determined to be "sad," a specific action such as "hold the baby and talk to him / her gently" is generated.

[0836] Step 5: Proposal notification (device)

[0837] The device receives the suggestions sent from the server and notifies the user. The notification is done via a pop-up message or voice guidance, helping the user to respond quickly. For example, a message such as "The baby is feeling sad. Please hold the baby and talk to him / her gently" may be displayed.

[0838] Step 6: Execute Action (User)

[0839] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user can take action by holding the baby and talking to it gently. This series of actions allows the user to understand the baby's emotions and respond appropriately.

[0840] In this way, this system, which determines a baby's emotions from their vocalizations, facial expressions, and body movements and suggests appropriate responses, helps to reduce anxiety in child-rearing and facilitates smooth communication between parents and children.

[0841] Example 1

[0842] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0843] In today's childcare environment, it is important for parents and caregivers to quickly and accurately grasp a baby's emotions and needs and respond appropriately, but this can be difficult, especially for first-time parents. The present invention aims to solve this problem by providing a system that analyzes a baby's emotions and suggests appropriate responses.

[0844] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0845] In this invention, the server includes a sensor means for reading the baby's vocalizations, facial expressions, and body movements, a storage means for temporarily storing the read data, an encryption means for encrypting the stored data, a communication means for transmitting the encrypted data, a receiving means for receiving and decrypting the transmitted data, an emotion determination means for analyzing the decrypted data and determining the baby's emotion, a suggestion generation means using a generative AI model for generating appropriate actions based on the determined emotion, and a notification means for notifying the user of the generated suggestion. This enables parents and caregivers to quickly and accurately grasp the baby's emotion and take appropriate action.

[0846] "Sensor means" refers to hardware that reads the baby's vocalizations, facial expressions, and body movements, as well as software that controls its operation.

[0847] The "storage means" is a memory or storage device for temporarily storing data read by the sensor means.

[0848] "Encryption means" refers to software and its execution environment for encrypting data using an encryption algorithm to protect the stored data.

[0849] A "communication means" is software that implements a network interface and communication protocol for transmitting encrypted data.

[0850] The "receiving means" refers to hardware and software for receiving and decoding data transmitted by the transmitting means.

[0851] The "emotion determination means" is a machine learning model and its execution environment for analyzing the data decoded by the receiving means and determining the baby's emotions.

[0852] The "proposal generation means" refers to software and its execution environment for generating appropriate actions using a generative AI model based on the determined emotions.

[0853] The "notification means" is a part of the application for notifying the user of the generated proposal, and hardware that realizes the notification function.

[0854] This system analyzes a baby's emotions based on their vocalizations, facial expressions, and body movements, and suggests appropriate responses to parents. This system is primarily composed of three components: a terminal, a server, and a user.

[0855] Data collection by terminal

[0856] The device uses a camera and microphone as sensors to read the baby's vocalizations, facial expressions, and body movements. These sensors capture the baby's smiling or crying face with the camera, and record the baby's cries and cooing (gentle voices) with the microphone. For example, the camera takes a picture of the baby's face every second, while the microphone simultaneously collects audio data.

[0857] Data Retention and Encryption

[0858] The device temporarily stores the captured data in its internal memory, which ensures data integrity in real time. The stored data is encrypted using the AES encryption algorithm to ensure security.

[0859] Sending data from the device to the server

[0860] The encrypted data is sent to the server via a communication means. Specifically, the encrypted data is sent to the server via a POST request using HTTPS, a secure communication protocol.

[0861] Data analysis by server

[0862] The server receives the transmitted data and decodes it using a decoding method. The decoded data contains the baby's voice and video, which are then analyzed using a machine learning model. To determine emotions, models such as CNN (convolutional neural network) and RNN (recurrent neural network) are used to extract and analyze voice frequency patterns and facial expression features. For example, if a baby's crying has a specific pattern that indicates "sadness," the server will determine that as the emotion.

[0863] Server-based action suggestions

[0864] The server uses a generative AI model to suggest appropriate actions based on the emotional information determined from the analysis. Examples of generative AI models that can be used include GPT-3. Prompts are sent to the model, which suggests specific actions that parents or caregivers can take. For example, by sending the prompt "What should you do when your baby is sad?", the server generates the specific response of "hold the baby and talk to him / her gently."

[0865] Proposal notification on device

[0866] The generated suggestion is sent back to the terminal and notified to the user via a notification means. Possible notification methods include a pop-up message or voice guidance. For example, a notification message such as "The baby is feeling sad. Please hold the baby and talk to him / her gently" may be displayed.

[0867] Specific examples

[0868] When a baby starts crying, the device's camera captures the crying face and the microphone records the sound. This data is encrypted and sent to a server, where a machine learning model analyzes the emotion as "sad." The generative AI model then generates a suggestion to "hold the baby and talk to him / her gently" and notifies the device.

[0869] Example prompts for generative AI models

[0870] "Analyze the crying data and crying face image data to determine the baby's emotions and suggest appropriate ways to respond."

[0871] This system allows users to understand their baby's emotions in real time and respond appropriately, which is expected to improve communication between parents and children and reduce the burden of childcare.

[0872] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0873] Step 1:

[0874] The device initializes the camera and microphone at system startup, specifically by setting the camera resolution and frame rate and the microphone sampling rate. The input to this process is the camera and microphone hardware, and the output is the ready-to-use sensor device.

[0875] Step 2:

[0876] The device reads and captures the baby's vocalizations, facial expressions, and body movements at regular intervals. The input is real-time video and audio, and the output is the image frame and audio data at that time. Specifically, the device captures an image frame from the camera every second and simultaneously records audio data from the microphone.

[0877] Step 3:

[0878] The device temporarily stores the captured data in its internal memory. The input is the captured image and audio data, and the output is the temporarily stored data. Specifically, this data is organized in chronological order and stored in a buffer.

[0879] Step 4:

[0880] The terminal encrypts the stored data. The input is the temporarily stored data, and the output is the encrypted data. Specifically, the data is encrypted using the AES encryption algorithm.

[0881] Step 5:

[0882] The device sends encrypted data to the server. The input is the encrypted data, and the output is a notification to the server that the data has been sent. Specifically, the device composes an HTTPS POST request and sends the data to the server's endpoint.

[0883] Step 6:

[0884] The server receives the transmitted data and decrypts it. The input is the encrypted data, and the output is the original decrypted data. Specifically, the server decrypts the data using the AES encryption key used on the terminal.

[0885] Step 7:

[0886] The server analyzes the decoded data and determines the baby's emotions. The input is the decoded voice data and image data, and the output is the emotion determination result. Specifically, it uses a machine learning model (e.g., CNN or RNN) to analyze the frequency patterns of the voice and facial features.

[0887] Step 8:

[0888] The server uses a generative AI model to suggest appropriate actions based on the determined emotion information. The input is the emotion determination result, and the output is a specific action suggestion. As a specific operation, the server sends a prompt sentence to the generative AI model (e.g., GPT-3) saying, "What should you do when a baby is sad?"

[0889] Step 9:

[0890] The server sends the generated action proposal to the terminal. The input is the specific action proposal, and the output is a notification to the terminal that the action proposal has been sent.

[0891] Step 10:

[0892] The device notifies the user of the received action suggestion. The input is the action suggestion sent from the server, and the output is the notification displayed to the user. Specific actions include using a pop-up message or voice guidance to display a notification such as "The baby is feeling sad. Please hold the baby and talk to him gently."

[0893] Through these steps, the system can analyze the baby's emotions and suggest appropriate responses to parents or caregivers in real time.

[0894] (Application example 1)

[0895] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0896] Conventional baby emotion analysis systems only address the specific situations of babies, making it difficult to apply them to other purposes. Furthermore, when dealing with customers in brick-and-mortar stores, it is difficult to quickly and accurately grasp their emotions and provide appropriate service. To improve this situation, a system is needed that can analyze the emotions of babies and customers and propose appropriate responses.

[0897] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0898] In this invention, the server includes a sensor means for reading the vocalizations, facial expressions, and body movements of the baby or customer, a communication means for transmitting the read data, a receiving means for receiving and processing the transmitted data, an emotion determination means for analyzing the received data and determining the emotion of the baby or customer, a proposal generation means for generating an appropriate action based on the determined emotion, and a notification means for notifying the user of the generated proposal, thereby enabling an analysis of the emotion of the customer or baby and a proposal of a countermeasure.

[0899] "Sensor means" refers to a device for reading the vocalizations, facial expressions, and body movements of the baby or customer.

[0900] "Communication means" refers to the device or technology used to transmit the scanned data to another device or server.

[0901] A "receiving means" is a device or module for receiving and processing transmitted data.

[0902] The "emotion determination means" is a device or algorithm for analyzing the received data and determining the emotion of the baby or customer.

[0903] The "suggestion generator" is a device or algorithm for generating appropriate actions based on the determined emotion.

[0904] The "notification means" is a device or interface for notifying the user of the generated proposal.

[0905] A "machine learning model" is an algorithm or mathematical model that analyzes data and finds patterns.

[0906] A "generative AI model" is an artificial intelligence model that generates appropriate output based on specific input.

[0907] This invention is a system that analyzes the emotions of a baby or customer based on their vocalizations, facial expressions, and body movements, and proposes appropriate responses. This system is mainly composed of three entities: a terminal, a server, and a user.

[0908] Device data collection:

[0909] The device (such as a smartphone or a robot in a brick-and-mortar store) uses a camera and microphone to read the baby's or customer's vocalizations, facial expressions, and body movements. The data collected by these sensors is temporarily stored on the device. Specific examples include a camera capturing the customer's facial expressions and a microphone recording the tone of voice.

[0910] Sending data from device to server:

[0911] The collected data is encrypted and sent to the server using a secure communication protocol (e.g., HTTPS). This ensures the security of the data. For example, customer voice and facial expression data are encrypted and sent to the server endpoint.

[0912] Data analysis by server:

[0913] The server decodes the received data and analyzes it using an emotion determination method, which uses a machine learning model. For example, the server analyzes specific patterns in the customer's voice and facial features to determine that the customer is "tired."

[0914] Server suggested actions:

[0915] Based on the analysis results, a generative AI model is used to suggest appropriate actions. The generative AI model compares the results with past data to generate specific responses. For example, if the user is determined to be "tired," the server will suggest "please offer me a drink."

[0916] Device suggestion notification:

[0917] The server sends the generated suggestions to the device, which then notifies the user. Possible notification methods include a pop-up message or voice guidance. For example, the app might display a notification saying, "You seem tired. Let's offer you a drink."

[0918] This allows users (store staff and parents) to quickly and accurately grasp the emotions of customers and babies and respond appropriately, which contributes to improving customer satisfaction and facilitating communication between parents and children, and improving the overall service experience.

[0919] Example prompt sentence:

[0920] "If you determine from a customer's facial expression and voice that they are tired, please suggest what kind of hospitality they should provide."

[0921] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0922] Step 1: Data collection

[0923] The device (a smartphone or a robot in a brick-and-mortar store) uses a camera and microphone to read the baby's or customer's vocalizations, facial expressions, and body movements in real time. Specifically, the camera captures facial expressions and the microphone records voice. The input for this step is the baby's or customer's real-time voice and facial expressions, and the output is these audio and video data.

[0924] Step 2: Send data

[0925] The device encrypts the collected data and sends it securely to the server using the HTTPS protocol, ensuring the safety of the data. Specifically, the device encodes and encrypts the audio and video data and sends it to a specific endpoint on the server. The input to this step is the collected audio and video data, and the output is the encrypted data.

[0926] Step 3: Receiving and Decrypting Data

[0927] The server receives and decrypts the data sent from the terminal. Specifically, it decodes the received data and converts it back to the original audio and video data. The input of this step is the encrypted data, and the output is the decrypted audio and video data.

[0928] Step 4: Sentiment Analysis

[0929] The server uses emotion determination means to analyze the received data and determine the baby's or customer's emotion. Using a machine learning model, it analyzes specific patterns of voice tone and facial expressions to determine specific emotions (e.g., "tired," "sad," etc.). The input of this step is the decoded audio and video data, and the output is the emotion determination result.

[0930] Step 5: Action proposal generation

[0931] The server uses a generative AI model to suggest an appropriate action based on the determined emotion. For example, if the customer is determined to be "tired," the server generates the suggestion "Please offer me a drink." Specifically, the emotion determination result is input into the AI ​​model, and an appropriate response prompt is generated. The input for this step is the emotion determination result, and the output is the response prompt.

[0932] Step 6: Proposal Notification

[0933] The server sends the generated suggestion to the terminal, which then notifies the user (parent or store staff). Notification methods include a pop-up message on the terminal or voice guidance. Specifically, the suggestion is sent from the server to the terminal, and the terminal displays the suggestion on the screen or provides voice guidance. The input to this step is a prompt for a solution, and the output is the notification to the user.

[0934] Through the above steps, the system is able to quickly and accurately analyze the emotions of the baby or customer and propose appropriate countermeasures.

[0935] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0936] The present invention is a system that analyzes a baby's emotions from its vocalizations, facial expressions, and body movements, and suggests appropriate responses based on the results.The present invention also includes a system that analyzes a user's vocalizations, facial expressions, and body movements to determine the user's emotions, and suggests additional actions for the baby based on the determined emotions.The present invention is mainly composed of three entities: a terminal (including a smartphone), a server, and a user.Specific embodiments of the present invention will be described below.

[0937] Device data collection:

[0938] The device uses a camera and microphone to capture the baby's and the user's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it within the device. For example, the device captures the baby's smiling and crying faces and the user's facial expressions with the camera, and records the baby's crying and cooing and the user's vocalizations with the microphone.

[0939] Sending data from device to server:

[0940] The collected audio and video data is encrypted and sent to the server using a secure protocol (e.g., HTTPS). Because the data is encrypted before being sent, the risk of the data being stolen or tampered with during transmission is reduced. As a specific example, the baby's crying voice data and facial expression data, and the user's voice and facial expression data are encrypted before being sent to a specific endpoint on the server.

[0941] Data analysis by server:

[0942] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data is input to the emotion determination means. The emotion determination means analyzes the data using a machine learning model to determine the emotions of the baby and the user. For example, the server analyzes the specific pattern of the baby's crying and the characteristics of the crying face to determine the emotion as "sad." It also analyzes the user's vocalizations and facial expressions to determine that the user is "feeling stressed."

[0943] Server suggested actions:

[0944] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model uses the baby's and user's emotions, as well as past databases and known knowledge, to generate specific responses. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will generate a suggestion to "hold the baby and talk to him gently" to calm the baby, and also suggest actions to the user such as "take a deep breath and relax."

[0945] Device suggestion notification:

[0946] The device receives the suggestions sent from the server and notifies the user. Possible notification methods include a pop-up message or voice guidance. The user checks the notification and performs the suggested action. For example, the app displays the message, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0947] User performs the action:

[0948] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user will take action such as holding the baby and talking to it gently. Also, if the notification suggests ways for the user to relax, the user will take action such as deep breathing or stretching.

[0949] In this way, this system, which determines the emotions of the baby and the user from their vocalizations, facial expressions, and physical movements and suggests appropriate responses, helps to reduce anxiety in child-rearing and facilitates communication between parents and children. Furthermore, by taking the user's emotions into consideration, it also contributes to reducing the stress of child-rearing.

[0950] The processing flow will be explained below.

[0951] Step 1: Collect data (device)

[0952] The device activates a microphone and camera to capture the baby's and the user's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it within the device. For example, the device captures the baby's smiling and crying face with the camera and records the baby's crying and cooing with the microphone. For the user, the device captures facial expressions with the camera and records the tone of voice with the microphone.

[0953] Step 2: Sending data (terminal)

[0954] The collected audio and video data is encrypted and sent to the server using a secure protocol (e.g., HTTPS). Because the data is encrypted before being sent, the risk of the data being stolen or tampered with during transmission is reduced. Specifically, the baby's crying data, facial expression data, and the user's voice and facial expression data are encrypted before being sent to a specific endpoint on the server.

[0955] Step 3: Receiving and analyzing data (server)

[0956] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data are input to the emotion determination means. The emotion determination means analyzes the data using a machine learning model to determine the emotions of the baby and the user. For example, the server may analyze the specific pattern of the baby's crying and the characteristics of the crying face to determine the emotion as "sad," and simultaneously analyze the user's facial expression and tone of voice to determine that the user is "feeling stressed."

[0957] Step 4: Proposing Actions (Server)

[0958] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model uses the baby's and user's emotions, as well as past databases and known knowledge, to generate specific responses. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will suggest "hold the baby and talk to him / her gently" and generate an action such as "the user should take a deep breath and relax."

[0959] Step 5: Proposal notification (device)

[0960] The device receives the suggestions sent from the server and notifies the user. Possible notification methods include a pop-up message or voice guidance. The user checks the notification and performs the suggested action. For example, the app displays the message, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0961] Step 6: Execute Action (User)

[0962] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user will take action such as holding the baby and talking to it gently. If a relaxation method is suggested for the user, the user will take action such as deep breathing or stretching. For example, by taking a deep breath as instructed by the device notification and then holding the baby, the stress of childcare can be reduced.

[0963] In this way, this system, which determines the emotions of the baby and the user from their vocalizations, facial expressions, and physical movements and suggests appropriate responses, helps to reduce anxiety in child-rearing and facilitates communication between parents and children. Furthermore, by taking the user's emotions into consideration, it also contributes to reducing the stress of child-rearing.

[0964] Example 2

[0965] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0966] Conventional childcare support systems have limited methods for responding to specific baby behaviors, making it difficult to accurately assess the emotions of both the baby and the parent and to provide specific responses based on those emotions. This has limited the effectiveness of reducing stress and anxiety in childcare.

[0967] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means including an input device that reads the baby's vocalizations, facial expressions, and body movements, data sending means that encrypts and sends the read data, receiving means that receives and decrypts the sent data, emotion determining means that analyzes the decrypted data and determines the baby's emotion and the user's emotion, a generation engine including suggestion generating means that generates appropriate actions based on the determined emotions, and notification means that notifies the user of the generated suggestions. This makes it possible to quickly and accurately determine the emotions of the baby and the user and to suggest specific and feasible ways of responding based on the determined emotions.

[0968] An "input device" is a device that includes sensors such as a camera or microphone to read a baby's vocalizations, facial expressions, and body movements.

[0969] The "data transmission means" is a means having a function of encrypting the read data and transmitting it safely to the server.

[0970] The "receiving means" is a means for receiving and decrypting the transmitted encrypted data.

[0971] The "emotion determination means" is a means for analyzing received audio data and video data using a machine learning model to determine the emotions of the baby and the user.

[0972] The "suggestion generation means" is a means for using a generative AI model to suggest appropriate actions to the baby and the user based on the determined emotions.

[0973] The "notification means" is a means for notifying the user of the generated proposal, and includes a pop-up message, a voice guide, and the like.

[0974] This invention is a childcare support system that analyzes a baby's vocalizations, facial expressions, and body movements and suggests appropriate actions based on the results. It also has a function that analyzes the user's vocalizations, facial expressions, and body movements to determine the user's emotions and suggest additional actions for the baby. This system is mainly composed of three entities: a terminal (including a smartphone), a server, and a user.

[0975] The device uses a camera and microphone to capture the baby's and user's vocalizations, facial expressions, and body movements. Specifically, the device captures the baby's smiling and crying faces and the user's facial expressions with the camera, and records the baby's crying and cooing sounds and the user's vocalizations with the microphone. The collected data is temporarily stored in the device.

[0976] The collected audio and video data is encrypted and sent to the server using a secure protocol (e.g., HTTPS). For example, the baby's crying voice data and facial expression data, and the user's voice and facial expression data are encrypted and then sent to a specific endpoint on the server.

[0977] The server receives the encrypted data sent from the device and decrypts it. The decrypted audio and video data is input into the emotion determination function, which uses a machine learning model to analyze the data and determine the emotions of the baby and the user. For example, the server analyzes the specific patterns of a baby's crying and the characteristics of its crying face to determine the emotion as "sad." It also analyzes the user's vocalizations and facial expressions to determine that the user is "feeling stressed."

[0978] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. The generative AI model generates specific responses by using the baby's and user's emotions, as well as past databases and known knowledge. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will generate a suggestion to "hold the baby and talk to him gently" to calm the baby, and suggest actions to the user such as "take a deep breath and relax."

[0979] The device receives the suggestions sent from the server and notifies the user. Possible notification methods include a pop-up message or voice guidance. Specifically, the device app displays a message saying, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[0980] The user checks the notification from the device and performs the suggested action. For example, if the notification says "The baby is sad," the user can hold the baby and talk to him / her gently. If the notification suggests relaxation techniques for the user, the user can take actions such as deep breathing or stretching.

[0981] This system can reduce anxiety in child-rearing and facilitate communication between parents and children. It also contributes to reducing stress in child-rearing by taking into consideration the user's emotions.

[0982] For example, you might input the following prompt into a generative AI model:

[0983] "Analyze the baby's crying pattern and determine its emotion. Then, analyze the user's facial expressions and vocalizations and determine the user's emotion as well. Suggest appropriate actions if the baby's emotion is sad and the user's emotion is stressed."

[0984] Using this example, the generative AI model can derive specific countermeasures and make effective suggestions to the user.

[0985] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0986] Step 1:

[0987] The device reads the baby's and user's vocalizations, facial expressions, and body movements. Specifically, the device's built-in camera captures the baby's smiling and crying faces and the user's facial expressions in real time, and the microphone records the baby's crying and cooing, as well as the user's vocalizations. This input data (audio and video data) is temporarily stored in the device.

[0988] Step 2:

[0989] The terminal encrypts the collected audio and video data, specifically by encoding it using a data encryption algorithm and preparing it for transmission. This encrypted data is the output.

[0990] Step 3:

[0991] The device sends the encrypted data to the server using a secure protocol (e.g., HTTPS). Specifically, the encrypted audio and video data is sent over the Internet to a specific endpoint on the server.

[0992] Step 4:

[0993] The server receives the encrypted data sent from the terminal and decrypts it. Specifically, it decodes the data using a decryption algorithm running on the server. This decrypted audio and video data becomes the input data for the next process.

[0994] Step 5:

[0995] The server inputs the decoded audio and video data into the emotion determination means. The machine learning model in the server analyzes this data and determines the emotions of the baby and the user. Specifically, it extracts features from the audio data and recognizes patterns in the video data, and outputs the respective emotions (e.g., "sad" or "stressed").

[0996] Step 6:

[0997] The server uses a generative AI model based on the emotion determination results to suggest appropriate actions. For example, based on the emotion determination results that the baby is "sad" and the user is "stressed," the server generates suggestions such as holding the baby and talking to them gently, or encouraging the user to take a deep breath. These generated suggestions are the output.

[0998] Step 7:

[0999] The server then transmits the generated proposal to the terminal. Specifically, the proposal content is formatted as text data or voice data, and then encrypted again before being transmitted to the terminal.

[1000] Step 8:

[1001] The device receives the encrypted proposal data sent from the server and decrypts it. The decrypted proposal content is notified to the user. Specific notification methods include a pop-up message or voice guidance. For example, a message might be displayed saying, "The baby feels sad. Please hold the baby and talk to him gently. The user should take a deep breath and relax."

[1002] Step 9:

[1003] The user checks the notification from the device and performs the suggested action. For example, they may hold the baby and talk to it gently based on the notification content. Also, if a relaxation method is suggested for the user, they may perform actions such as deep breathing or stretching.

[1004] Through the above steps, a system is realized that determines the emotions of the baby and the user and suggests specific ways of responding based on the determined emotions.

[1005] (Application example 2)

[1006] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1007] Conventional childcare support systems only provide recommendations regarding the baby's condition and do not consider the user's (parent's) emotions or condition, which limits their ability to reduce childcare stress. Furthermore, physical stores provide insufficient support for parents and respond to baby's condition, making it difficult to provide an appropriate childcare environment.

[1008] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion determination means that analyzes received data and determines the emotions of the baby and the user, a suggestion generation means that generates appropriate actions based on the determined emotions, a notification means that notifies the user of the generated suggestions, and an action support means that supports the user in taking actions corresponding to the generated suggestions. This makes it possible to comprehensively analyze the status of the baby and the user, provide appropriate childcare support even in physical stores, and reduce childcare stress.

[1009] 1. "Sensor means" refers to equipment used to read a baby's vocalizations, facial expressions, and body movements.

[1010] 2. "Means of communication" refers to the functions and technologies used to transmit the captured data.

[1011] 3. "Receiving means" means the functions and technologies for receiving and processing transmitted data.

[1012] 4. "Emotion determination means" refers to the functions and technologies for analyzing received data and determining the emotions of the baby and the user.

[1013] 5. "Proposal generation means" refers to the functions and technologies for generating appropriate actions based on the determined emotions.

[1014] 6. "Notification means" refers to the functions and technologies used to notify users of generated suggestions.

[1015] 7. "Action support means" refers to functions and technologies that support users in taking actions corresponding to the generated suggestions.

[1016] 8. A "machine learning model" is an algorithm or framework for analyzing data to identify patterns and make predictions.

[1017] 9. A “generative AI model” is an artificial intelligence model that generates recommendations using a database or known knowledge to suggest appropriate actions.

[1018] This invention is a system that analyzes a baby's emotions from its vocalizations, facial expressions, and body movements, and suggests appropriate responses based on the analysis. Furthermore, this system has the function of analyzing the user's vocalizations, facial expressions, and body movements to determine the user's emotions, and based on this, suggesting additional actions for the baby.

[1019] Hardware and software used

[1020] This system primarily utilizes the following hardware and software:

[1021] Hardware

[1022] Smartphone: Collects data using a camera and microphone.

[1023] Server: Receives and analyzes data.

[1024] software

[1025] OpenCV: Capture images from the camera.

[1026] Keras (TensorFlow backend): Performs sentiment analysis using machine learning models.

[1027] HTTP client library: Communicates with the server (e.g., requests library).

[1028] Data collection

[1029] The device (smartphone) uses a camera and microphone to capture the baby's and user's vocalizations, facial expressions, and body movements. The device collects this data in real time and temporarily stores it.

[1030] Data transmission and reception

[1031] The collected audio and video data is encrypted and sent to a server using a secure protocol (e.g., HTTPS). The server receives the encrypted data sent from the device and decrypts it.

[1032] Data analysis

[1033] The server inputs the received audio and video data into a machine learning model to determine the emotions of the baby and the user. The machine learning model analyzes the baby's crying and facial expressions, as well as the user's vocalizations and facial expressions, to identify emotions.

[1034] Suggested Actions

[1035] Based on the analysis results, the server uses a generative AI model to suggest appropriate actions. This generative AI model generates specific responses by using the baby's and user's emotions, as well as past databases and existing knowledge. For example, if the baby is determined to be "sad" and the user is determined to be "stressed," the server will generate a suggestion to "hold the baby and talk to him gently" and suggest actions to the user such as "take a deep breath and relax."

[1036] Proposal notification and support

[1037] The device receives the proposed action from the server and notifies the user. The notification method is a pop-up message or voice guidance. The user checks the notification and performs the proposed action. The action support means also supports the proposed action.

[1038] Specific examples

[1039] For example, if a baby starts crying in a physical store selling childcare products, the app will analyze the image and sound of the crying baby and notify the user with a message such as "Hold the baby and talk to him / her gently." It will also suggest actions such as "Take a deep breath and relax." This will provide appropriate childcare support even in physical stores, reducing parenting stress for parents.

[1040] Prompt Sentence Examples

[1041] "Judge the baby's emotions and suggest appropriate parenting actions. Refer to and analyze the baby's facial expressions and cries to inform parents of specific ways to respond."

[1042] This system determines the emotions of the baby and the user from their vocalizations, facial expressions, and physical movements, and by appropriately supporting the user's behavior, it helps to reduce anxiety and stress in child-rearing and facilitate smooth communication between parents and children.

[1043] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1044] Step 1: Data collection

[1045] The device uses a camera to capture the facial expressions of the baby and the user, and a microphone to collect vocalization data. The image data captured by the camera and the audio data collected by the microphone are input. These data are collected in real time and temporarily stored in the device. The output of this step is image data and audio data.

[1046] Step 2: Send data

[1047] The device encrypts the collected image and audio data and sends it to the server using a secure protocol (HTTPS). The encrypted data is input and sent to a specific endpoint on the server. This reduces the risk of the data being stolen or tampered with during transmission. The output is a confirmation that the data has been received by the server.

[1048] Step 3: Receiving and Decrypting Data

[1049] The server receives the encrypted data sent from the device and decrypts it. Encrypted image and audio data are the input, and decrypted data is the output. Decrypting the data converts it into a format that can be used for analysis.

[1050] Step 4: Data analysis

[1051] The server inputs the decoded image data and audio data into a machine learning model to determine the emotions of the baby and the user. The input is the decoded data, which is analyzed using the machine learning model. The emotion determination result is output through data analysis. Specifically, the model analyzes the baby's crying face and crying sound, as well as the user's facial expressions and voice characteristics, to determine emotions such as "sadness" or "stress."

[1052] Step 5: Action Creation

[1053] The server uses a generative AI model based on the emotion determination result to suggest an appropriate action. The input is the emotion determination result, and a specific response method is generated using a past database and existing knowledge. For example, if a baby is determined to be "sad," the generative AI model will make a suggestion such as "hold the baby and talk to him gently." The output is the generated action proposal.

[1054] Step 6: Proposal Notification

[1055] The device receives the suggestions sent from the server and notifies the user. The generated action suggestions are input, and the device displays them as pop-up messages or voice guidance. The output is a notification to the user. Specifically, messages such as "Please hold the baby and speak to him gently" and "The user should take a deep breath and relax" are displayed.

[1056] Step 7: Take action and support

[1057] The user checks the notification from the device and performs the suggested action. The notified action suggestion becomes the input, and the user actually takes the action. The action support means supports the user's action, for example, it can provide specific guides and reminders. The output is the user's execution and the support results.

[1058] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1059] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1060] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1061] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1062] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1063] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1064] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1065] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1066] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1067] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1068] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1069] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1070] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1071] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1072] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1073] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1074] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1075] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1076] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1077] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1078] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1079] The following is further disclosed regarding the above embodiment.

[1080] (Claim 1)

[1081] A sensor means to read the baby's vocalizations, facial expressions, and body movements,

[1082] a communication means for transmitting the read data;

[1083] receiving means for receiving and processing the transmitted data;

[1084] emotion determination means for analyzing the received data and determining the emotion of the baby;

[1085] a suggestion generation means for generating an appropriate action based on the determined emotion;

[1086] a notification means for notifying the user of the generated proposal;

[1087] A system including:

[1088] (Claim 2)

[1089] 2. The system according to claim 1, wherein the emotion determination means is a means for analyzing vocalizations, facial expressions, and body movements using a machine learning model.

[1090] (Claim 3)

[1091] 2. The system according to claim 1, wherein the suggestion generating means converts the generated suggestion into a specific action that can be carried out by the user.

[1092] "Example 1"

[1093] (Claim 1)

[1094] A sensor means to read the baby's vocalizations, facial expressions, and body movements,

[1095] a storage means for temporarily storing the read data;

[1096] encryption means for encrypting the stored data;

[1097] a communication means for transmitting encrypted data;

[1098] receiving means for receiving and decoding the transmitted data;

[1099] emotion determination means for analyzing the decoded data and determining the emotion of the baby;

[1100] A proposal generation means using a generative AI model that generates appropriate actions based on the determined emotions;

[1101] a notification means for notifying the user of the generated proposal;

[1102] A system including:

[1103] (Claim 2)

[1104] 2. The system according to claim 1, wherein the emotion determination means is a means for analyzing vocalizations, facial expressions, and body movements using a machine learning model.

[1105] (Claim 3)

[1106] The system according to claim 1, characterized in that the proposal generation means is a means for converting the proposals generated using a generative AI model into specific actions that can be taken by the user.

[1107] "Application Example 1"

[1108] (Claim 1)

[1109] A sensor means for reading the vocalizations, facial expressions, and body movements of the baby or customer;

[1110] a communication means for transmitting the read data;

[1111] receiving means for receiving and processing the transmitted data;

[1112] emotion determination means for analyzing the received data and determining the emotion of the baby or the customer;

[1113] a suggestion generation means for generating an appropriate action based on the determined emotion;

[1114] a notification means for notifying the user of the generated proposal;

[1115] A system including:

[1116] (Claim 2)

[1117] 2. The system according to claim 1, wherein the emotion determination means is a means for analyzing vocalizations, facial expressions, and body movements using a machine learning model.

[1118] (Claim 3)

[1119] 2. The system according to claim 1, wherein the suggestion generating means converts the generated suggestion into a specific action that can be carried out by the user.

[1120] "Example 2: Combining Emotion Engines"

[1121] (Claim 1)

[1122] a means including an input device for reading the baby's vocalizations, facial expressions, and body movements;

[1123] a data transmission means for encrypting and transmitting the read data;

[1124] receiving means for receiving and decoding the transmitted data;

[1125] emotion determination means for analyzing the decoded data and determining the emotions of the baby and the user;

[1126] a generation engine including a suggestion generation means for generating an appropriate action based on the determined emotion;

[1127] a notification means for notifying the user of the generated proposal;

[1128] A system including:

[1129] (Claim 2)

[1130] 2. The system according to claim 1, wherein the emotion determination means is a means for analyzing vocalizations, facial expressions, and body movements using a machine learning model.

[1131] (Claim 3)

[1132] The system according to claim 1, characterized in that the proposal generation means is a means for converting proposals generated using a generative AI model into specific actions that can be carried out by the user.

[1133] "Application example 2 when combining emotion engines"

[1134] (Claim 1)

[1135] A sensor means to read the baby's vocalizations, facial expressions, and body movements,

[1136] a communication means for transmitting the read data;

[1137] receiving means for receiving and processing the transmitted data;

[1138] emotion determination means for analyzing the received data and determining the emotions of the baby and the user;

[1139] a suggestion generation means for generating an appropriate action based on the determined emotion;

[1140] a notification means for notifying the user of the generated proposal;

[1141] an action support means for supporting the user in taking an action corresponding to the generated suggestion;

[1142] A system including:

[1143] (Claim 2)

[1144] 2. The system according to claim 1, wherein the emotion determination means is a means for analyzing vocalizations, facial expressions, and body movements using a machine learning model.

[1145] (Claim 3)

[1146] The system of claim 1, wherein the suggestion generation means converts the generated suggestions into specific actions that the user can take, and further uses a generative AI model that generates suggestions using a database of the baby's and the user's past and known knowledge. [Explanation of symbols]

[1147] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A sensor means to read the baby's vocalizations, facial expressions, and body movements, a communication means for transmitting the read data; receiving means for receiving and processing the transmitted data; emotion determination means for analyzing the received data and determining the emotion of the baby; a suggestion generation means for generating an appropriate action based on the determined emotion; a notification means for notifying the user of the generated proposal; A system including:

2. 2. The system according to claim 1, wherein the emotion determination means is a means for analyzing vocalizations, facial expressions, and body movements using a machine learning model.

3. 2. The system according to claim 1, wherein the suggestion generating means converts the generated suggestion into a specific action that can be carried out by the user.

Images