System

Abstract

A system is provided.SOLUTION: A system comprising: means for analyzing video information obtained from a mobile communication device; means for analyzing audio information obtained from the mobile communication device; means for integrating the video information and the audio information and using a learning algorithm to infer a state of the infant; means for predicting a future potential state based on the inferred state of the infant; and means for providing a caregiver with a response to the predicted future state.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In Japan, where the birthrate is declining, many families are giving up on having children due to economic reasons or the difficulty of balancing work, housework, and child-rearing. Therefore, there is a need for a system that can reduce the burden on parents and make child-rearing more efficient and effective. There is also a need for a method to effectively provide information about a baby's daily patterns and preferences when relatives are temporarily in charge of childcare. [Means for solving the problem]

[0005] The present invention provides a system including: means for analyzing video information acquired from a mobile communication device; means for analyzing audio information acquired from the mobile communication device; means for integrating the video information and the audio information and using a learning algorithm to estimate an infant's condition; means for predicting a potential future condition based on the estimated infant's condition; and means for providing parents with countermeasures for the predicted future condition. The system further includes means for accumulating data on the estimated infant's condition and countermeasures and improving the accuracy of the prediction through comparative analysis. The system also includes means for providing this data to relatives and generating a manual for optimal care. This reduces the burden on parents of childcare, facilitates temporary childcare by relatives, and improves the childcare environment.

[0006] A "mobile communication device" is a device such as a smartphone or tablet that has communication capabilities and can be carried and used by a user.

[0007] "Video information" refers to video data or still image data obtained from a visual input device such as a camera.

[0008] "Audio information" is audio data obtained from a voice input device such as a microphone.

[0009] A "learning algorithm" is a mathematical model or program designed to make predictions or classifications based on past data.

[0010] "Infant state" refers to the infant's current emotions and physical needs (e.g., hunger, sleepiness, desire to play, etc.).

[0011] "Future potential states" are predictions of a child's future emotional and physical needs based on current data and past patterns.

[0012] "Countermeasures" refer to specific measures or advice that instruct appropriate behavior or ways of dealing with the child's predicted condition.

[0013] "Parent" refers to a person (e.g., father or mother) who is responsible for raising a young child.

[0014] "Relatives" refers to family members other than the infant's parents (e.g., grandparents, uncles, aunts, etc.).

[0015] A "data log" is a collection of data that includes records of acquired video information, audio information, and analysis results and countermeasures based on the information.

[0016] "Comparative analysis" is an analytical technique that aims to uncover specific patterns and relationships by comparing multiple data sets.

[0017] A "manual" is a document or guide that contains procedures or instructions for performing a particular operation or action. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention relates to the "AI Mimamori" system, which aims to reduce the burden on parents raising children and make child-rearing more effective and efficient amid the declining birthrate. This system analyzes video and audio information obtained from mobile communication devices (e.g., smartphones), predicts the condition of infants, and provides parents with countermeasures based on that information. The detailed configuration and operation for realizing this system are described below.

[0040] System Overview

[0041] The device uses a camera and microphone to capture real-time video and audio information about the baby. For example, a smartphone's camera and microphone can be used to capture the baby's facial expressions, crying, and surrounding environmental sounds. This captured data is then sent to a server via the Internet.

[0042] The server analyzes this data using AI models. Deep learning models such as convolutional neural networks (CNNs) are used to analyze visual data, and speech recognition models are used to analyze auditory data. The results of each analysis (e.g., whether the baby is hungry, sleepy, or wants to play) are integrated by the server to estimate the baby's overall state.

[0043] The estimated infant's condition is compared and analyzed with previously collected data logs to predict potential future conditions. Based on this prediction, appropriate countermeasures (such as notifying the next feeding time or providing a sleep-friendly environment) are notified to the device. Furthermore, by linking with smart home devices, the system can automatically adjust the room temperature, play music, etc.

[0044] Program processing

[0045] The device activates the camera and microphone to capture the baby's video and audio information, and captures the data in real time. This data is pre-processed (e.g., image resizing and audio data format conversion) and then sent to a server via the Internet.

[0046] The server first analyzes visual information using a convolutional neural network (CNN). It receives image data as input, extracts features, and infers the infant's current emotions and physical needs. Similarly, audio information is fed into a speech recognition model, which analyzes the infant's state from crying patterns and other audio signals.

[0047] The results of both analyses are integrated on a server and recorded as a comprehensive infant status. This information is stored in a data log and compared with past data to predict future behavior (e.g., whether the infant will be hungry in two hours).

[0048] The prediction results and countermeasures are sent to the device via the Internet and notified to the parent. For example, if the baby seems sleepy, a notification such as "The baby seems sleepy. The room temperature is being adjusted appropriately" will be displayed on the smartphone.

[0049] In addition, the server can also connect to smart home devices to automatically adjust the temperature in the baby's room and play music to soothe them to sleep, helping to ensure that the baby is raised in a comfortable and safe environment.

[0050] Specific examples

[0051] The user activates the "AI Mimamori" system. The device (such as a smartphone) then captures video and audio of the baby. The server analyzes this data and infers that the baby is hungry. Based on this, the server predicts that the baby will become hungry again in approximately two hours. The server then issues instructions to smart home devices to set the room temperature to 22 degrees. The smartphone also receives a notification that the baby is likely to become hungry in two hours. This allows the parent to prepare the next meal at the appropriate time.

[0052] This series of actions reduces the burden of childcare on parents and provides a comfortable environment for babies. The "AI Mimamori" system of this invention offers a new means of supporting childcare and is expected to contribute to measures to combat the declining birthrate.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user launches the "AI Mimamori" system.

[0056] Users start the system by tapping a dedicated application on their smartphone.

[0057] Step 2:

[0058] The device activates the camera and acquires video information.

[0059] The device uses a built-in camera to capture images of the baby in real time and obtain frame data.

[0060] Step 3:

[0061] The device activates the microphone and captures audio information.

[0062] The device uses a built-in microphone to record the baby's crying and environmental sounds in real time, capturing audio data.

[0063] Step 4:

[0064] The video and audio information acquired by the terminal is preprocessed.

[0065] The video information is resized, noise removed, and otherwise processed to convert it into a format suitable for analysis.

[0066] The audio information is processed by adjusting the sampling rate and removing noise, and then converted into a format suitable for analysis.

[0067] Step 5:

[0068] The terminal transmits the preprocessed data to a server via the Internet.

[0069] The device converts the acquired video and audio data into an appropriate format and sends it to the server using encrypted communication such as SSL.

[0070] Step 6:

[0071] The server uses an AI model to analyze the video information.

[0072] The server inputs the preprocessed video data into a convolutional neural network (CNN), which infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements.

[0073] Step 7:

[0074] The server uses an AI model to analyze the audio information.

[0075] The server inputs the preprocessed audio data into a speech recognition model and assesses the baby's condition based on crying patterns and surrounding sounds.

[0076] Step 8:

[0077] The server integrates the video analysis results and audio analysis results.

[0078] The server integrates the results of the analysis of visual and auditory information to estimate the overall state of the child.

[0079] Step 9:

[0080] The server logs the inferred infant state.

[0081] The server stores the guesses and their timestamps in a data log.

[0082] Step 10:

[0083] The server predicts future conditions based on the accumulated data logs.

[0084] The server analyzes past data logs and uses learning algorithms to predict the baby's future condition (for example, the next feeding time or when the baby will feel sleepy).

[0085] Step 11:

[0086] The server notifies the parent to provide appropriate action.

[0087] Based on the results of the estimation and prediction, the server notifies the parent's smartphone with advice on appropriate measures (e.g., preparing for breastfeeding or adjusting the room temperature).

[0088] Step 12:

[0089] The device receives the notification and displays it to the user.

[0090] The device receives notifications from the server and displays them to the user as push notifications or alerts. For example, "Your baby will be hungry in 2 hours."

[0091] Step 13:

[0092] The server controls the smart home devices.

[0093] The server controls smart home devices (e.g., smart thermostats and speakers) to maintain a comfortable environment for the baby, adjusting the room temperature and playing music to soothe the baby to sleep.

[0094] Step 14:

[0095] The user acts on the notification.

[0096] Users can take necessary measures according to the notifications on their smartphones, such as preparing for breastfeeding or preparing the environment for putting the baby to sleep.

[0097] Example 1

[0098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] There is a need for systems that can reduce the burden faced by parents raising children and enable them to raise their children more effectively and efficiently. Especially with the declining birthrate, there is an urgent need to monitor a baby's condition in real time and provide appropriate measures. There is also a need to reduce the burden on parents by integrating smart home devices and automating environmental adjustments.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0101] In this invention, the server includes means for analyzing video information acquired from the mobile communication device, means for analyzing audio information acquired from the mobile communication device, means for integrating the video information and the audio information and using a machine learning algorithm to predict the infant's condition, means for predicting the infant's potential future condition based on the predicted infant's condition, means for providing the guardian with countermeasures for the predicted future condition, means for storing the analysis results and countermeasures in a database and performing comparative analysis based on past data, means for integrating the extracted analysis results and countermeasures in the server and notifying the mobile communication device via the Internet, and means for automatically adjusting the physical environment using the mobile communication device and smart home devices. This allows parents raising their baby to understand the baby's condition in real time and obtain appropriate countermeasures. Furthermore, environmental adjustments are automatically performed in cooperation with the smart home devices, effectively reducing the parental burden on parents.

[0102] A "mobile communication device" refers to a device that is mobile and has communication capabilities, and specifically includes smartphones and tablets.

[0103] "Video information" refers to image data or video data acquired using a camera.

[0104] "Audio information" refers to audio data or acoustic data acquired using a microphone.

[0105] "Machine learning algorithm" is a general term for algorithms that learn patterns from data and make predictions and classifications for future data.

[0106] "Infant states" refers to the baby's various emotions and physical needs (e.g., hunger, sleepiness, discomfort, etc.).

[0107] "Inferred infant state" refers to the infant's emotional and needs state inferred by a machine learning algorithm based on the analysis of video and audio information.

[0108] "Future potential state" refers to the infant's future state predicted based on the infant's estimated current state and past data logs.

[0109] "Countermeasures" refer to specific responses and guidelines for action that parents should take based on the estimated condition of the child and their predicted future condition.

[0110] "Guardian" refers to a person who is responsible for caring for a young child, typically a parent or relative.

[0111] A "database" refers to a system that stores data in an organized form and allows it to be searched and updated efficiently.

[0112] "Comparative analysis" refers to an analytical technique that compares different data sets to reveal relationships and patterns between them.

[0113] A "server" refers to a computer system that provides services and data to clients over a network.

[0114] "Internet" refers to a global information and communications network that interconnects computer networks around the world.

[0115] "Smart home devices" refer to appliances and devices in the home that can be controlled via the Internet, including thermostats and smart speakers.

[0116] "Automatically adjusting the physical environment" refers to automatically adjusting environmental settings such as room temperature, lighting, and music through smart home devices.

[0117] The present invention is a system for reducing the burden on parents raising children and for raising them more effectively and efficiently. This system analyzes video and audio information acquired from a mobile communication device (e.g., a smartphone), predicts the baby's condition, and provides parents with countermeasures based on the predictions. The specific configuration and operation of the system are described in detail below.

[0118] Hardware and software configuration

[0119] Terminal

[0120] The terminal is a mobile communication device equipped with a camera and a microphone, and is typically a smartphone. The terminal has the following functions:

[0121] 1. Camera: Used to capture the baby's visual information (facial expressions and movements).

[0122] 2. Microphone: Used to record the baby's crying and surrounding environmental sounds.

[0123] 3. Data preprocessing: Converting the captured video and audio data into a format suitable for analysis.

[0124] 4. Data transmission: The preprocessed data is transmitted to the server via the Internet.

[0125] server

[0126] The server has the following analysis and inference functions:

[0127] 1. Video analysis: Analyze the baby's facial expressions using a convolutional neural network (CNN) and extract features.

[0128] 2. Speech analysis: Analyze the baby's crying patterns using a speech recognition model.

[0129] 3. Data integration: The results of the analysis of visual and audio information are integrated to estimate the overall state of the child.

[0130] 4. Prediction function: Predicts future potential states based on past data logs.

[0131] 5. Notification function: Prediction results and countermeasures are notified to the device.

[0132] Smart Home Devices

[0133] The server connects to smart home devices via the Internet to achieve the following functions:

[0134] 1. Environmental adjustment: Adjust the temperature and lighting in your baby's room appropriately.

[0135] 2. Play music: Play music that helps your baby relax.

[0136] Specific examples

[0137] As a concrete example, the following scenario can be considered.

[0138] The user launches the "childcare support system." The device (e.g., a smartphone) then captures video and audio of the baby. The baby's facial expression and audio data are sent to a server via the Internet. The server first inputs the video data into a CNN model, extracts features, and analyzes the baby's emotions and needs. Next, it inputs the audio data into a speech recognition model, and analyzes the crying patterns to infer the baby's condition.

[0139] For example, if the server determines that the baby is crying because it seems sleepy, it will send a notification to the device saying, "The baby seems sleepy. The room temperature is being adjusted appropriately." At the same time, the server will send instructions to connected smart home devices to set the room temperature to 22 degrees and play relaxing music. This allows the baby to sleep in a comfortable environment, allowing parents to respond at the appropriate time.

[0140] Prompt Sentence Examples

[0141] Below are some example prompts that can be used as input to a generative AI model.

[0142] "My baby is crying. Please help me guess what his current emotions and physical needs are and suggest appropriate responses."

[0143] Using this prompt, the AI ​​model will perform specific analysis and make suggestions. In this way, the system of the present invention aims to support childcare and reduce the burden on parents.

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1:

[0146] The device activates the camera and microphone to capture the baby's video and audio information in real time. Specifically, the smartphone camera captures the baby's face, and the microphone records the baby's crying and surrounding environmental sounds. The input is the baby's actual video and audio, and the output is the raw data captured by the camera and microphone.

[0147] Step 2:

[0148] The device preprocesses the captured video and audio data. For example, video data is resized and audio data is converted to a specific format. The input to this process is the raw data captured in step 1, and the output is preprocessed data suitable for analysis.

[0149] Step 3:

[0150] Send the preprocessed data to a server via the internet. The input is the preprocessed data generated in step 2, and the output is the data sent to the server.

[0151] Step 4:

[0152] A convolutional neural network (CNN) is used to analyze the video data received by the server. Specifically, the server inputs the video data into a CNN model, identifies the baby's facial expressions, and extracts features. The input for this analysis is the video data received in step 3, and the output is the baby's facial expressions and their features.

[0153] Step 5:

[0154] The server uses a speech recognition model to analyze the received voice data. Specifically, the server inputs the voice data into the model, analyzes the crying pattern, and extracts its features. The input for this analysis is the voice data received in step 3, and the output is the crying pattern and its features.

[0155] Step 6:

[0156] The server integrates the results of the video and audio analysis to infer the baby's overall state. The server uses the video and audio features as examples to input into a statistical model to infer the baby's emotions and needs. The input for this inference is the features extracted in steps 4 and 5, and the output is the baby's current state (e.g., hunger, sleepiness, discomfort, etc.).

[0157] Step 7:

[0158] Based on the estimated state of the infant, the server refers to past data logs to predict the future state. The server uses statistical methods and machine learning models to predict the future state (e.g., the infant will be hungry in two hours). The inputs to this prediction are the current state estimated in step 6 and past data logs, and the output is the predicted future state.

[0159] Step 8:

[0160] The server notifies the device of the prediction results and countermeasures via the Internet. The server generates a specific countermeasure (e.g., notification of the next breastfeeding time) and sends a notification to the smartphone. The input of this notification is the prediction result generated in step 7, and the output is the notification content sent to the smartphone.

[0161] Step 9:

[0162] The server interacts with the smart home devices and issues necessary instructions, such as adjusting the room temperature or playing relaxing music. Based on the prediction results, the server instructs the smart home devices to set appropriate settings. The input of this instruction is the result predicted in step 7, and the output is an operation instruction to the smart home devices.

[0163] (Application example 1)

[0164] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0165] It is necessary to reduce the burden on parents raising children and to make child-rearing more effective and efficient, but current systems lack the data analysis necessary to accurately predict the condition of young children, the prediction of potential future conditions, and the provision of appropriate countermeasures. Furthermore, the lack of an environment in which information can be shared and parenting can be coordinated not only with parents but also with other stakeholders and family members is slowing the reduction of the burden of child-rearing. Furthermore, there is currently a lack of coordination when it comes to automatically adjusting the environment using home automation devices.

[0166] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0167] In this invention, the server includes: a means for analyzing video information of the baby; a means for analyzing audio information; a means for integrating the video information and audio information and using a learning algorithm to estimate the infant's condition; a means for predicting a potential future condition based on the estimated infant's condition; a means for providing countermeasures for the predicted future condition; a means for sending an alert to a caregiver based on the estimated infant's condition; a means for controlling home automation devices based on the alert; a means for generating the content of the countermeasures using a generative AI; and a means for creating prompts to support the caregiver's actions based on the generated countermeasures. This reduces the burden on parents raising children and enables appropriate responses through efficient collaboration within the household and among related parties. Furthermore, by linking with home automation devices, it is possible to automatically optimize the childcare environment.

[0168] A "mobile communication device" is a device with communication capabilities that can be used by a user while on the move, such as a mobile phone or smartphone.

[0169] "Video information" refers to video and still image data captured using a camera.

[0170] "Audio information" refers to sound data acquired using a microphone.

[0171] A "learning algorithm" is a method for analyzing data to find patterns and relationships and make predictions or classifications.

[0172] "Infant states" are situations that indicate the emotional and physical needs of babies and young children.

[0173] "Future potential states" are future states of infants predicted based on current and past data.

[0174] "Countermeasures" refer to specific guidelines and measures provided depending on the child's condition.

[0175] An "alert" is a warning message intended to notify you of urgent or important information.

[0176] A "caregiver" is a parent or other guardian who cares for a baby or young child.

[0177] "Home automation devices" refer to systems and equipment that automatically control various devices in the home.

[0178] "Generative AI" refers to technology that uses artificial intelligence to generate new information and data.

[0179] A "prompt" is a sentence that guides the user in instructing them on what to do.

[0180] This invention is a home support system based on the "AI Mimamori" system that monitors the condition of infants in real time and provides appropriate countermeasures. This system uses a mobile communication device (e.g., a smartphone) to acquire and analyze video and audio information of the baby, inferring the infant's condition and providing the user with countermeasures based on that condition.

[0181] Hardware and Software Configuration

[0182] Device (e.g. smartphone)

[0183] The device uses a camera and microphone to capture real-time video and audio information of the baby, which is then sent to a server via the Internet.

[0184] server

[0185] The server analyzes the received video information using deep learning models such as convolutional neural networks (CNNs) to infer the infant's condition from their facial expressions and movements. Similarly, it analyzes audio information using speech recognition models to understand the infant's condition from crying patterns and other audio signals. Based on these results, the infant's overall condition is inferred.

[0186] The server then uses the inference results to predict potential future states. Based on this prediction, an appropriate response is generated using a generative AI model and provided to the user as a prompt, such as "The baby will be hungry in two hours."

[0187] Home Automation Devices

[0188] The server also connects with home automation devices (e.g., smart home systems) to automate environmental settings such as adjusting the temperature and playing music in the baby's room, ensuring a comfortable and safe environment for the baby.

[0189] Examples and prompts

[0190] For example, the "AI Mimamori" system may detect a baby's crying and send an alert to the user's smartphone saying, "Your baby is crying excessively. Please check on him / her immediately." At the same time, the server will instruct the smart home system to turn on the lights in the baby's room and provide the camera feed to the parents remotely.

[0191] Prompt Sentence Examples

[0192] "What is the best way to respond if my baby is crying?"

[0193] "Infer your baby's emotional state from their facial expressions."

[0194] "How do I connect it to my smart home devices to keep my baby safe?"

[0195] This allows parents to keep track of their baby's condition and respond quickly, while automated home devices can adjust the environment to provide a comfortable and safe environment for the baby.

[0196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0197] Step 1:

[0198] Data acquisition (device)

[0199] The device (smartphone) activates the camera and microphone to capture the baby's video and audio information in real time. At this time, the camera captures continuous still images, and the microphone captures continuous audio samples.

[0200] Input: Camera images, audio data

[0201] Output: Pre-processed video data, audio data

[0202] Step 2:

[0203] Data preprocessing (terminal)

[0204] The device resizes the captured video data and converts the audio data into an appropriate format for preprocessing, which removes noise for data analysis. Specifically, the image size is resized to 224x224 pixels, and the audio data is extracted to a certain time range.

[0205] Input: Camera images, audio data

[0206] Output: Pre-processed video data, audio data

[0207] Step 3:

[0208] Data transmission (terminal)

[0209] The device transmits the preprocessed video and audio data to a server via the Internet, using a secure communication protocol such as HTTPS.

[0210] Input: Preprocessed video data, audio data

[0211] Output: Send data to the server

[0212] Step 4:

[0213] Video data analysis (server)

[0214] The video data received by the server is analyzed using a convolutional neural network (CNN). Specifically, the video data is input into a CNN model to extract features such as the baby's facial expressions and movements. This analysis allows the system to predict the baby's state, such as whether it is crying or about to fall asleep.

[0215] Input: Preprocessed video data

[0216] Output: Image features, inferred state

[0217] Step 5:

[0218] Voice data analysis (server)

[0219] The server analyzes the received voice data using a voice recognition model, which inputs the voice data into the model and uses crying patterns and other audio signals to understand the baby's condition.

[0220] Input: Preprocessed audio data

[0221] Output: Audio features, inferred state

[0222] Step 6:

[0223] State estimation integration (server)

[0224] The server combines the results of the video and audio analysis to comprehensively estimate the baby's condition. Specifically, it uses a machine learning model to combine the features of both data to determine the final condition.

[0225] Input: Image features, audio features

[0226] Output: Overall guess state

[0227] Step 7:

[0228] Future state prediction (server)

[0229] The server predicts potential future states based on the current state, specifically by comparing it with past data logs to predict when the next person will be hungry or when they will fall asleep.

[0230] Input: Overall guess state, historical data log

[0231] Output: Future potential states

[0232] Step 8:

[0233] Generate countermeasures (server)

[0234] The server generates appropriate childcare responses based on the predicted future state, which are created using a generative AI model and provide a concrete action plan for the caregiver.

[0235] Input: Future potential state

[0236] Output: Generated workarounds

[0237] Step 9:

[0238] Prompt statement generation (server)

[0239] The server creates a prompt sentence to support the caregiver's actions based on the generated countermeasure. Specifically, it creates a sentence that succinctly expresses the generated countermeasure and notifies the caregiver.

[0240] Input: Generated countermeasures

[0241] Output: prompt statement

[0242] Step 10:

[0243] Notification and control of home automation devices (server)

[0244] The server then sends the generated prompts to the caregiver's device and simultaneously controls home automation devices, such as sending notifications to a smartphone and issuing commands to the smart home system to adjust the room temperature or play music.

[0245] Input: prompt statement, home automation device control instructions

[0246] Output: Notification to caregivers, activation of home automation devices

[0247] The above is the specific flow of the system program processing.

[0248] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0249] This invention combines an emotion engine with the "AI Mimamori" system, which aims to reduce the burden on parents raising children and make child-rearing more effective and efficient amid a declining birthrate. This system analyzes video and audio information obtained from mobile communication devices (e.g., smartphones), predicts the state of a child, and provides parents with countermeasures based on that information. It also recognizes the user's emotions and provides more appropriate countermeasures based on those.

[0250] System Overview

[0251] The device uses a camera and microphone to capture video and audio information of the baby, as well as that of the user (parent) in real time. For example, the camera and microphone of a smartphone can be used to capture the facial expressions and voices of the baby and parent. This captured data is then sent to a server via the Internet.

[0252] The server analyzes this data using AI models. Deep learning models such as convolutional neural networks (CNNs) are used to analyze the baby's visual data, and speech recognition models are used to analyze the auditory data. Similarly, an emotion recognition engine is used to analyze the parent's emotions. The results of each analysis (e.g., the baby is hungry, the parent is stressed, etc.) are integrated by the server to infer the overall state of the baby and parent.

[0253] The estimated state of the infant and parent is compared and analyzed with previously collected data logs to predict potential future states. Based on this prediction, appropriate countermeasures (such as notifications of the next feeding time or suggestions to reduce parental stress) are notified to the device. Furthermore, by linking with smart home devices, the system can automatically adjust the room temperature, play music, etc.

[0254] Program processing

[0255] The device activates the camera and microphone to capture video and audio information of the baby and parent, and captures data in real time. This data is pre-processed (e.g., image resizing and audio data format conversion) and sent to a server via the Internet.

[0256] The server first analyzes the baby's video information using a convolutional neural network (CNN). It receives image data as input, extracts features, and infers the baby's state, such as hunger, sleepiness, or desire to play, from its facial expressions and movements. Similarly, the baby's audio information is input into a speech recognition model, which evaluates the baby's state based on crying patterns and other audio signals.

[0257] The server then inputs the parent's video and audio information into an emotion recognition engine, which analyzes the parent's facial expressions and tone of voice to identify emotions such as stress, joy, and fatigue.

[0258] The results of both analyses are integrated on a server and recorded as the overall state of the infant and parent. This information is stored in a data log and compared with past data to predict future states (for example, the baby may become hungry in two hours, which may cause stress for the parent).

[0259] The prediction results and countermeasures are sent to the device via the Internet and notified to the parent. For example, if the baby seems sleepy and the parent is tired, a notification will appear on the smartphone saying, "The baby seems sleepy. The room temperature is being adjusted. We recommend that the parent also take a break."

[0260] In addition, the server can also connect to smart home devices to automatically adjust the room temperature and play relaxing music to maintain a comfortable environment for both baby and parent, ensuring a safe and comfortable environment for both baby and parent.

[0261] Specific examples

[0262] The user activates the "AI Mimamori" system. The device (e.g., a smartphone) then captures video and audio of the baby and parent. The server analyzes this data and infers whether the baby is hungry and whether the parent is feeling stressed. Based on this, the server predicts that the baby will become hungry again in approximately two hours and suggests that the parent take a break to relax. The server then instructs smart home devices to set the room temperature to 22 degrees and play relaxing music. The smartphone also receives a notification saying, "The baby is likely to become hungry in two hours. Please relax, parent." This allows parents to prepare the next meal at the appropriate time while also maintaining their own health.

[0263] This series of actions reduces the burden of childcare on parents and provides a comfortable environment for both the baby and the parent. The "AI Mimamori" system of this invention provides a new means of supporting childcare and is expected to contribute to measures to combat the declining birthrate.

[0264] The processing flow will be explained below.

[0265] Step 1:

[0266] The user launches the "AI Mimamori" system.

[0267] Users start the system by tapping a dedicated application on their smartphone.

[0268] Step 2:

[0269] The device activates the camera and acquires video information.

[0270] The device uses a built-in camera to capture images of the baby in real time and obtain frame data.

[0271] Step 3:

[0272] The device activates the microphone and captures audio information.

[0273] The device uses a built-in microphone to record the baby's crying and environmental sounds in real time, capturing audio data.

[0274] Step 4:

[0275] The terminal uses a camera to acquire video information of the user.

[0276] The device captures the user's facial expressions using a built-in camera and acquires the data.

[0277] Step 5:

[0278] The terminal acquires the user's voice information using a microphone.

[0279] The device uses a built-in microphone to record the user's tone of voice and speaking style.

[0280] Step 6:

[0281] The video and audio information acquired by the terminal is preprocessed.

[0282] The video information is resized, noise removed, and otherwise processed to convert it into a format suitable for analysis.

[0283] The audio information is processed by adjusting the sampling rate and removing noise, and then converted into a format suitable for analysis.

[0284] Step 7:

[0285] The terminal transmits the preprocessed data to a server via the Internet.

[0286] The device converts the acquired video and audio data into an appropriate format and sends it to the server using encrypted communication such as SSL.

[0287] Step 8:

[0288] The server uses an AI model to analyze the video information.

[0289] The server inputs the preprocessed video data into a convolutional neural network (CNN), which infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements.

[0290] Step 9:

[0291] The server uses an AI model to analyze the audio information.

[0292] The server inputs the preprocessed audio data into a speech recognition model to assess the baby's condition from crying patterns and other audio signals.

[0293] Step 10:

[0294] The server uses an emotion recognition engine to analyze the user's video and audio information.

[0295] The server analyzes the preprocessed video data of the user and infers emotions such as stress, joy, and fatigue from facial expressions.

[0296] Similarly, the system analyzes the user's voice data and assesses emotions based on tone of voice and speaking style.

[0297] Step 11:

[0298] The server integrates the results of video analysis, audio analysis, and emotion analysis.

[0299] The server compiles the results of the analysis of visual, auditory and emotional information into unified data and estimates the overall state of the infant and parent.

[0300] Step 12:

[0301] The server logs the inferred infant and parent states.

[0302] The server stores the guesses and their timestamps in a data log.

[0303] Step 13:

[0304] The server predicts future conditions based on the accumulated data logs.

[0305] The server analyzes past data logs and uses learning algorithms to predict future conditions of the infant and parent (e.g., the next feeding time or the parent's stress level).

[0306] Step 14:

[0307] The server notifies the parent to provide appropriate action.

[0308] Based on the results of the estimation and prediction, the server notifies the parent's smartphone with advice on appropriate measures (e.g., preparing for breastfeeding, suggestions for reducing parental stress).

[0309] Step 15:

[0310] The device receives the notification and displays it to the user.

[0311] The device receives notifications from the server and displays them to the user as push notifications or alerts. For example, "Your baby will be hungry in 2 hours. Please relax."

[0312] Step 16:

[0313] The server controls the smart home devices.

[0314] The server controls smart home devices (e.g., smart thermostats and speakers) to adjust the room temperature and play relaxing music to maintain a comfortable environment for the baby and parents.

[0315] Step 17:

[0316] The user acts on the notification.

[0317] Users can take necessary measures based on notifications from their smartphones, such as preparing to breastfeed, preparing the environment for putting the baby to sleep, or taking a break to relieve their own stress.

[0318] Example 2

[0319] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0320] As the birthrate continues to decline, there is a need to reduce the burden on parents raising children and provide more effective and efficient methods for raising them. In particular, there is a need for a system that can accurately grasp the baby's condition and quickly provide appropriate countermeasures. There is also a need for a function that takes into account the parents' own emotional state and provides comprehensive support. However, existing systems are limited to estimating the baby's condition, and there are no systems that perform integrated analysis including the parent's emotional state or automatically adjust environmental conditions. Therefore, the challenge is to provide a system that provides comprehensive support for both, thereby significantly reducing the burden on parents raising children and providing a more comfortable childcare environment.

[0321] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0322] In this invention, the server includes means for analyzing video information acquired from a mobile communication device, means for analyzing audio information acquired from the mobile communication device, means for integrating the video information and the audio information and using a learning algorithm to predict the infant's condition, means for integrating and analyzing the estimated infant's condition and the guardian's emotional state to evaluate the overall condition, means for predicting a future potential condition based on the integrated analysis, means for providing the guardian with countermeasures for the predicted future condition, and means for automatically adjusting environmental conditions based on the countermeasures and cooperating with a device to provide a comfortable environment for the guardian and infant. This enables a comprehensive analysis of the baby's condition and the parent's emotional state, and makes it possible to reduce the burden of child-rearing on parents by suggesting appropriate countermeasures and automatically adjusting the environment.

[0323] "Mobile communications device" refers to a mobile device such as a mobile phone or smartphone, which has the ability to send and receive data using communications technology.

[0324] "Video information" is data containing visual content captured by a camera, video device, or the like.

[0325] "Audio information" is information that includes sound data collected by an audio capture device such as a microphone.

[0326] A "learning algorithm" is a set of computational methods and procedures for analyzing data and finding patterns and regularities within it, and plays an important role in machine learning.

[0327] "Infant state" refers to the physical and emotional state of a young child, such as a baby, including hunger, sleepiness, pleasure and pain, etc.

[0328] "Parental emotional state" refers to the psychological emotional state of the parent or guardian, including stress, joy, fatigue, etc.

[0329] "Integrated analysis" is an analytical method that combines data obtained from multiple different sources to make a single comprehensive evaluation.

[0330] "Future potential state" refers to the likely future state of the child and caregiver, predicted based on current data.

[0331] A "countermeasure" is a proposed action or method based on an estimated state or prediction, with the aim of solving a problem or improving a situation.

[0332] "Automatic adjustment of environmental conditions" is an operation that automatically adjusts environmental elements such as room temperature, lighting, and music to provide a comfortable living environment for infants and their guardians.

[0333] "Working with devices" refers to a system exchanging information and instructions with other devices or equipment to perform functions in collaboration.

[0334] MODE FOR CARRYING OUT THE INVENTION

[0335] The present invention relates to the "AI Mimamori" system, which aims to reduce the burden on parents raising children and help them do so more effectively and efficiently. This system analyzes video and audio information obtained from mobile communication devices (such as smartphones), predicts the state of the child, and provides parents with countermeasures based on that information. It is also possible to recognize the emotions of the user (parent) and provide more appropriate countermeasures.

[0336] Hardware and software used

[0337] Devices: Smartphone, camera, microphone

[0338] Server: High-performance computer

[0339] software:

[0340] Video analysis: Convolutional neural networks (CNN)

[0341] Speech analysis: Speech recognition model

[0342] Sentiment Analysis: Emotion Recognition Engine

[0343] Data transmission: Internet Protocol

[0344] System Operation

[0345] The user activates the "AI Mimamori" system using a smartphone. The device uses a camera and microphone to capture video and audio information of the baby and parent in real time. For example, the smartphone's camera captures the baby's face, and the microphone records its crying. At the same time, the parent's facial expressions and voice are also captured. This captured data is preprocessed and then sent to a server via the Internet.

[0346] The server first analyzes the baby's video information using a convolutional neural network (CNN). It receives image data as input, extracts features, and infers the baby's state, such as hunger, sleepiness, or desire to play, from its facial expressions and movements. Similarly, the baby's audio information is input into a speech recognition model, which evaluates the baby's state based on crying patterns and other audio signals.

[0347] The server then inputs the parent's video and audio information into an emotion recognition engine, which analyzes the parent's facial expressions and tone of voice to identify emotions such as stress, joy, and fatigue.

[0348] The results of both analyses are integrated on a server and recorded as the overall state of the infant and parent. This information is stored in a data log and compared with past data to predict future states (for example, the baby may become hungry in two hours, which may cause stress for the parent).

[0349] The prediction results and countermeasures are sent to the device via the Internet and notified to the parent. For example, if the baby seems sleepy and the parent is tired, a notification will appear on the smartphone saying, "The baby seems sleepy. The room temperature is being adjusted. We recommend that the parent also take a break."

[0350] In addition, the server can also connect to smart home devices to automatically adjust the room temperature and play relaxing music to maintain a comfortable environment for both baby and parent, ensuring a safe and comfortable environment for both baby and parent.

[0351] Specific examples

[0352] The user activates the "AI Mimamori" system. The device (e.g., a smartphone) then captures video and audio of the baby and parent. The server analyzes this data and infers whether the baby is hungry and whether the parent is feeling stressed. Based on this, the server predicts that the baby will become hungry again in approximately two hours and suggests that the parent take a break to relax. The server then instructs smart home devices to set the room temperature to 22 degrees and play relaxing music. The smartphone also receives a notification saying, "The baby is likely to become hungry in two hours. Please relax, parent." This allows parents to prepare the next meal at the appropriate time while also maintaining their own health.

[0353] Prompt Sentence Examples

[0354] Contracting System:

[0355] 1. Use a camera and microphone to capture video and audio of the baby and parent.

[0356] 2. Video and audio data is sent to a server via the Internet.

[0357] 3. The server analyzes the data, proposes appropriate countermeasures, and notifies the smartphone.

[0358] Situation:

[0359] Imagine when a baby is crying and the parents are tired, and generate the best response.

[0360] Expected output:

[0361] "It's likely your baby is hungry and may be hungry again in about two hours. I set the room temperature to 72 degrees and played relaxing music. I encourage parents to take breaks as well."

[0362] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0363] Step 1:

[0364] The device uses a camera and microphone to capture real-time video and audio information of the baby and parent. The camera captures the baby's face, and the microphone records the baby's crying. At the same time, the device also captures the parent's facial expressions and voice. This data is temporarily stored on the device.

[0365] Input: Video and audio information from the camera and microphone

[0366] Output: A dataset of captured video and audio information

[0367] Step 2:

[0368] The device preprocesses the captured video and audio data, specifically resizing the video data and converting the audio data into a digital format. This preprocessing ensures uniform data quality for efficient analysis.

[0369] Input: A dataset of captured video and audio information.

[0370] Output: Pre-processed video and audio information

[0371] Step 3:

[0372] The device then transmits the preprocessed data to a server over the internet, using encryption protocols to prevent data leakage or tampering.

[0373] Input: Preprocessed video and audio information

[0374] Output: Data sent to the server

[0375] Step 4:

[0376] The server inputs the transmitted video data of the baby into a convolutional neural network (CNN). The CNN extracts features from the image data and infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements. For example, it analyzes video of a baby with its mouth open and determines that the baby is hungry.

[0377] Input: Baby video data sent to the server

[0378] Output: Analysis results showing the baby's condition

[0379] Step 5:

[0380] The server inputs the baby's voice data into a speech recognition model, which evaluates the baby's state from crying patterns and other audio signals. For example, if the crying pattern matches "sleepy," that information is output.

[0381] Input: Baby's voice data sent to the server

[0382] Output: Voice analysis results showing the baby's condition

[0383] Step 6:

[0384] The server inputs the parent's video and audio information into an emotion recognition engine. The emotion recognition engine analyzes the parent's facial expressions and tone of voice to determine emotions such as stress, joy, and fatigue. For example, it analyzes a video of the parent frowning and determines that the parent is feeling "stressed."

[0385] Input: Parent's video and audio information sent to the server

[0386] Output: Analysis results showing the parent's emotional state

[0387] Step 7:

[0388] The server combines the results of the baby's and parent's analysis to assess their overall condition. This information is stored in a data log and compared with past data. This allows for predictions of future conditions (e.g., the baby will be hungry in two hours).

[0389] Input: Analysis results showing the baby's condition, analysis results showing the parent's emotional state

[0390] Output: Comprehensive condition assessment and future condition prediction

[0391] Step 8:

[0392] The server generates appropriate countermeasures based on the prediction results and notifies the device, such as the next feeding time or suggestions for parental relaxation.

[0393] Input: Comprehensive condition assessment and future condition prediction

[0394] Output: Notification of action to be taken

[0395] Step 9:

[0396] The server works in conjunction with smart home devices to automatically adjust the room temperature and play relaxing music to maintain a comfortable environment for both baby and parent.

[0397] Input: Comprehensive condition assessment and future condition prediction

[0398] Output: Instructions to smart home devices

[0399] (Application example 2)

[0400] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0401] In order to reduce the burden on parents raising children and to raise them more effectively and efficiently amid a declining birthrate, it is necessary to consider not only the condition of the infant but also the emotional state of the parent. However, conventional systems lack the functionality to analyze the parent's emotional state, making it difficult to provide appropriate countermeasures. In addition, there is a lack of integration with smart home devices to maintain a comfortable environment for infants and parents, which means the burden on parents is not reduced.

[0402] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video information acquired from a mobile communication device, means for analyzing audio information acquired from the mobile communication device, means for integrating the video information and the audio information and using a learning algorithm to estimate the infant's state, means for predicting a future potential state based on the estimated infant's state, means for analyzing the parent's emotions, means for providing countermeasures based on the estimated infant's state and the parent's emotions, means for adjusting the environment in cooperation with a smart home device, and means for notifying the parent of the countermeasures. This provides appropriate childcare support that also takes into account the parent's emotional state, making it possible to reduce the burden of childcare and provide a comfortable environment for both the baby and the parent.

[0403] A "mobile communication device" is a portable information communication terminal equipped with a camera and a microphone, and has the function of acquiring video information and audio information and transmitting and receiving data via a communication network.

[0404] "Video information" refers to visual data captured by a camera on a mobile communication device and presented in the form of images or video.

[0405] "Audio information" refers to auditory data captured by a microphone on a mobile communication device and expressed in voice or acoustic form.

[0406] A "learning algorithm" refers to a computational method or model that learns patterns and relationships based on previously collected data and makes inferences and predictions about new data.

[0407] "Infant condition" refers to the baby's physical or psychological state, such as hunger, sleepiness, or a desire to play.

[0408] "Future potential states" refer to the infant's future state predicted based on current data and analysis results, such as hunger or sleepiness after a certain time.

[0409] "Parental emotions" refers to the emotional state that the parent is currently experiencing, such as specific emotions such as stress, joy, or fatigue.

[0410] "Countermeasures" refer to specific actions or coping methods suggested based on the estimated infant's condition and the parent's emotions, such as notification of the next feeding time or advice on stress reduction.

[0411] "Smart home devices" refer to home electronic devices that are connected to a network and have remote control or automation capabilities, such as smart thermostats and smart speakers.

[0412] "Environmental adjustment" refers to operations such as changing temperature and sound settings to maintain a comfortable indoor environment.

[0413] "Notification" refers to a function of informing a user of information via a mobile communication device, and includes notification by screen display or voice.

[0414] The "childcare support system" of the present invention is a system that uses a mobile communication device (e.g., a smartphone) to reduce the burden on parents raising children and allow them to raise children more effectively and efficiently. This system analyzes not only the condition of the infant but also the emotional state of the parent, and provides the user with appropriate countermeasures. The following describes in detail an embodiment of the present invention.

[0415] 1. System Configuration

[0416] The system consists of three main components: terminals, servers, and smart home devices.

[0417] Terminal

[0418] The device is a smartphone equipped with a camera and microphone, which captures real-time video and audio information of the baby and parent. The captured data is then sent to a server via the Internet.

[0419] server

[0420] The server has the following analytical methods and capabilities:

[0421] Video information analysis method: Using a convolutional neural network (CNN), the video data obtained from the camera is analyzed to extract the baby's facial expressions and movements.

[0422] Audio information analysis means: Using a speech recognition model, the audio data captured from the microphone is analyzed to evaluate crying and other audio signals.

[0423] Emotion recognition: An emotion recognition engine is used to analyze emotions from the parent's facial expressions and tone of voice.

[0424] The analyzed results are integrated to estimate the status of the infant and parent. Based on this, potential future conditions are predicted and appropriate countermeasures are generated. The appropriate countermeasures are notified to the terminal via the Internet, and instructions for cooperation are sent to smart home devices.

[0425] Smart Home Devices

[0426] Smart home devices automatically adjust the environment, such as networked thermostats and music players, to keep children and parents comfortable by adjusting the temperature or playing relaxing music.

[0427] 2. System Operation

[0428] Data Acquisition and Transmission

[0429] When a user launches the smartphone app, the camera and microphone begin capturing data in real time, which is then sent to a server via the Internet for detailed analysis.

[0430] Analysis and Notification

[0431] The server analyzes video and audio data to infer the baby's condition. For example, it can infer hunger from the baby's facial expression and sleepiness from the crying pattern. It can also analyze the parent's emotions from their facial expression and tone of voice to infer stress or fatigue. Based on these analysis results, it predicts future potential conditions and generates countermeasures.

[0432] environmental adjustment

[0433] By connecting with smart home devices, it can automatically adjust the room temperature and play relaxing music, reducing the burden on parents while providing a comfortable environment for both baby and parent.

[0434] 3. Examples of concrete examples and prompts

[0435] Specific examples

[0436] After the user activates the system, the smartphone's camera and microphone capture video and audio of the baby and parent. The server analyzes the data and infers that the baby is hungry and the parent is stressed. The following notification is displayed on the smartphone as a response generated by the server:

[0437] "Your baby is hungry. The next feeding is in about an hour. Set the room temperature to 72 degrees and play some relaxing music."

[0438] Prompt Sentence Examples

[0439] Analyze your baby's condition:

[0440] Capture the baby's facial expressions and movements with a camera and analyze them using a CNN model.

[0441] Use a voice recognition model to analyze crying patterns and assess states such as hunger and sleepiness.

[0442] Analyzing parental emotions:

[0443] Capture the parent's facial expressions and voice and analyze them with an emotion recognition engine.

[0444] Integrate baby and parent status and generate notifications:

[0445] If the baby is hungry, notify them of appropriate action (e.g., notify them of the next feeding time).

[0446] If a parent is tired, suggest a break and adjust the temperature with a smart home device.

[0447] As described above, the childcare support system of the present invention is capable of providing appropriate childcare support that takes into consideration the emotional state of the parents, thereby reducing the burden on the parents and improving the comfort of both the baby and the parents.

[0448] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0449] Step 1:

[0450] The user launches the smartphone app.

[0451] Input: Smartphone camera and microphone

[0452] Output: Real-time video and audio information

[0453] How it works: The user taps to launch the smartphone app, which automatically turns on the camera and microphone, capturing video and audio of the baby and parent.

[0454] Step 2:

[0455] The video and audio information acquired by the terminal is transmitted to a server via the Internet.

[0456] Input: Real-time video and audio information

[0457] Output: Data sent to the server

[0458] How it works: The device transfers the captured video and audio data to the server via a communications network. The data is encrypted and securely transmitted.

[0459] Step 3:

[0460] The server receives the video data and analyzes it using a convolutional neural network (CNN).

[0461] Input: Video information obtained from the camera

[0462] Output: Baby's facial expressions and behavior patterns

[0463] How it works: The server preprocesses the received video data (e.g., resizes and normalizes it) and then inputs it into a CNN model. The CNN model extracts features and infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements.

[0464] Step 4:

[0465] The server receives the voice data and analyzes it using a voice recognition model.

[0466] Input: Audio information obtained from the microphone

[0467] Output: Baby's crying and voice patterns

[0468] Specific operation: The server preprocesses the received voice data (e.g., noise reduction, normalization) and then inputs it into a voice recognition model. The voice recognition model analyzes the characteristics of the crying voice and evaluates the baby's state (e.g., hunger, sleepiness, demands).

[0469] Step 5:

[0470] The server inputs the parent's video and audio data into an emotion recognition engine for analysis.

[0471] Input: Parent video and audio information obtained from camera and microphone

[0472] Output: Parent's emotional state (e.g., stress, joy, fatigue)

[0473] Specific operation: The server inputs the parent's video data into an emotion recognition engine and analyzes their facial expressions. It also uses audio data to analyze their tone of voice and the content of their speech. This allows it to infer the parent's emotional state.

[0474] Step 6:

[0475] The server integrates the baby and parent condition data it has acquired and predicts potential future conditions.

[0476] Input: Baby's state data, Parent's emotional state data

[0477] Output: Predicted future state (e.g. next feeding time, parental stress level)

[0478] How it works: The server performs statistical analysis and comparison with past data to predict the future state of the baby and parents, for example, calculating the likelihood that the baby will be hungry after a certain time, or the likelihood that the parents' stress levels will increase.

[0479] Step 7:

[0480] The server generates appropriate countermeasures based on the prediction results and notifies the device.

[0481] Input: Predicted future state data

[0482] Output: Notifications sent to smartphones (e.g., notifications about breastfeeding times, suggestions for adjusting the environment)

[0483] Specific operation: Based on the prediction results, the server generates appropriate responses for the baby and the parent. For example, it predicts that the baby will become hungry again in about two hours and suggests that the parent take a break to relax. This information is then sent to the smartphone.

[0484] Step 8:

[0485] The server works in conjunction with smart home devices to adjust the environment.

[0486] Input: Generated countermeasure data, control information of smart home devices

[0487] Output: Adjusted comfort environment (e.g., adjusting room temperature, playing relaxing music)

[0488] How it works: The server sends instructions to smart home devices, which then automatically change the room temperature or play relaxing music, optimizing the childcare environment.

[0489] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0490] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0491] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0492] [Second embodiment]

[0493] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0494] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0495] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0496] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0497] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0498] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0499] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0500] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0501] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0502] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0503] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0504] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0505] The present invention relates to the "AI Mimamori" system, which aims to reduce the burden on parents raising children and make child-rearing more effective and efficient amid the declining birthrate. This system analyzes video and audio information obtained from mobile communication devices (e.g., smartphones), predicts the condition of infants, and provides parents with countermeasures based on that information. The detailed configuration and operation for realizing this system are described below.

[0506] System Overview

[0507] The device uses a camera and microphone to capture real-time video and audio information about the baby. For example, a smartphone's camera and microphone can be used to capture the baby's facial expressions, crying, and surrounding environmental sounds. This captured data is then sent to a server via the Internet.

[0508] The server analyzes this data using AI models. Deep learning models such as convolutional neural networks (CNNs) are used to analyze visual data, and speech recognition models are used to analyze auditory data. The results of each analysis (e.g., whether the baby is hungry, sleepy, or wants to play) are integrated by the server to estimate the baby's overall state.

[0509] The estimated infant's condition is compared and analyzed with previously collected data logs to predict potential future conditions. Based on this prediction, appropriate countermeasures (such as notifying the next feeding time or providing a sleep-friendly environment) are notified to the device. Furthermore, by linking with smart home devices, the system can automatically adjust the room temperature, play music, etc.

[0510] Program processing

[0511] The device activates the camera and microphone to capture the baby's video and audio information, and captures the data in real time. This data is pre-processed (e.g., image resizing and audio data format conversion) and then sent to a server via the Internet.

[0512] The server first analyzes visual information using a convolutional neural network (CNN). It receives image data as input, extracts features, and infers the infant's current emotions and physical needs. Similarly, audio information is fed into a speech recognition model, which analyzes the infant's state from crying patterns and other audio signals.

[0513] The results of both analyses are integrated on a server and recorded as a comprehensive infant status. This information is stored in a data log and compared with past data to predict future behavior (e.g., whether the infant will be hungry in two hours).

[0514] The prediction results and countermeasures are sent to the device via the Internet and notified to the parent. For example, if the baby seems sleepy, a notification such as "The baby seems sleepy. The room temperature is being adjusted appropriately" will be displayed on the smartphone.

[0515] In addition, the server can also connect to smart home devices to automatically adjust the temperature in the baby's room and play music to soothe them to sleep, helping to ensure that the baby is raised in a comfortable and safe environment.

[0516] Specific examples

[0517] The user activates the "AI Mimamori" system. The device (such as a smartphone) then captures video and audio of the baby. The server analyzes this data and infers that the baby is hungry. Based on this, the server predicts that the baby will become hungry again in approximately two hours. The server then issues instructions to smart home devices to set the room temperature to 22 degrees. The smartphone also receives a notification that the baby is likely to become hungry in two hours. This allows the parent to prepare the next meal at the appropriate time.

[0518] This series of actions reduces the burden of childcare on parents and provides a comfortable environment for babies. The "AI Mimamori" system of this invention offers a new means of supporting childcare and is expected to contribute to measures to combat the declining birthrate.

[0519] The processing flow will be explained below.

[0520] Step 1:

[0521] The user launches the "AI Mimamori" system.

[0522] Users start the system by tapping a dedicated application on their smartphone.

[0523] Step 2:

[0524] The device activates the camera and acquires video information.

[0525] The device uses a built-in camera to capture images of the baby in real time and obtain frame data.

[0526] Step 3:

[0527] The device activates the microphone and captures audio information.

[0528] The device uses a built-in microphone to record the baby's crying and environmental sounds in real time, capturing audio data.

[0529] Step 4:

[0530] The video and audio information acquired by the terminal is preprocessed.

[0531] The video information is resized, noise removed, and otherwise processed to convert it into a format suitable for analysis.

[0532] The audio information is processed by adjusting the sampling rate and removing noise, and then converted into a format suitable for analysis.

[0533] Step 5:

[0534] The terminal transmits the preprocessed data to a server via the Internet.

[0535] The device converts the acquired video and audio data into an appropriate format and sends it to the server using encrypted communication such as SSL.

[0536] Step 6:

[0537] The server uses an AI model to analyze the video information.

[0538] The server inputs the preprocessed video data into a convolutional neural network (CNN), which infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements.

[0539] Step 7:

[0540] The server uses an AI model to analyze the audio information.

[0541] The server inputs the preprocessed audio data into a speech recognition model and assesses the baby's condition based on crying patterns and surrounding sounds.

[0542] Step 8:

[0543] The server integrates the video analysis results and audio analysis results.

[0544] The server integrates the results of the analysis of visual and auditory information to estimate the overall state of the child.

[0545] Step 9:

[0546] The server logs the inferred infant state.

[0547] The server stores the guesses and their timestamps in a data log.

[0548] Step 10:

[0549] The server predicts future conditions based on the accumulated data logs.

[0550] The server analyzes past data logs and uses learning algorithms to predict the baby's future condition (for example, the next feeding time or when the baby will feel sleepy).

[0551] Step 11:

[0552] The server notifies the parent to provide appropriate action.

[0553] Based on the results of the estimation and prediction, the server notifies the parent's smartphone with advice on appropriate measures (e.g., preparing for breastfeeding or adjusting the room temperature).

[0554] Step 12:

[0555] The device receives the notification and displays it to the user.

[0556] The device receives notifications from the server and displays them to the user as push notifications or alerts. For example, "Your baby will be hungry in 2 hours."

[0557] Step 13:

[0558] The server controls the smart home devices.

[0559] The server controls smart home devices (e.g., smart thermostats and speakers) to maintain a comfortable environment for the baby, adjusting the room temperature and playing music to soothe the baby to sleep.

[0560] Step 14:

[0561] The user acts on the notification.

[0562] Users can take necessary measures according to the notifications on their smartphones, such as preparing for breastfeeding or preparing the environment for putting the baby to sleep.

[0563] Example 1

[0564] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0565] There is a need for systems that can reduce the burden faced by parents raising children and enable them to raise their children more effectively and efficiently. Especially with the declining birthrate, there is an urgent need to monitor a baby's condition in real time and provide appropriate measures. There is also a need to reduce the burden on parents by integrating smart home devices and automating environmental adjustments.

[0566] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0567] In this invention, the server includes means for analyzing video information acquired from the mobile communication device, means for analyzing audio information acquired from the mobile communication device, means for integrating the video information and the audio information and using a machine learning algorithm to predict the infant's condition, means for predicting the infant's potential future condition based on the predicted infant's condition, means for providing the guardian with countermeasures for the predicted future condition, means for storing the analysis results and countermeasures in a database and performing comparative analysis based on past data, means for integrating the extracted analysis results and countermeasures in the server and notifying the mobile communication device via the Internet, and means for automatically adjusting the physical environment using the mobile communication device and smart home devices. This allows parents raising their baby to understand the baby's condition in real time and obtain appropriate countermeasures. Furthermore, environmental adjustments are automatically performed in cooperation with the smart home devices, effectively reducing the parental burden on parents.

[0568] A "mobile communication device" refers to a device that is mobile and has communication capabilities, and specifically includes smartphones and tablets.

[0569] "Video information" refers to image data or video data acquired using a camera.

[0570] "Audio information" refers to audio data or acoustic data acquired using a microphone.

[0571] "Machine learning algorithm" is a general term for algorithms that learn patterns from data and make predictions and classifications for future data.

[0572] "Infant states" refers to the baby's various emotions and physical needs (e.g., hunger, sleepiness, discomfort, etc.).

[0573] "Inferred infant state" refers to the infant's emotional and needs state inferred by a machine learning algorithm based on the analysis of video and audio information.

[0574] "Future potential state" refers to the infant's future state predicted based on the infant's estimated current state and past data logs.

[0575] "Countermeasures" refer to specific responses and guidelines for action that parents should take based on the estimated condition of the child and their predicted future condition.

[0576] "Guardian" refers to a person who is responsible for caring for a young child, typically a parent or relative.

[0577] A "database" refers to a system that stores data in an organized form and allows it to be searched and updated efficiently.

[0578] "Comparative analysis" refers to an analytical technique that compares different data sets to reveal relationships and patterns between them.

[0579] A "server" refers to a computer system that provides services and data to clients over a network.

[0580] "Internet" refers to a global information and communications network that interconnects computer networks around the world.

[0581] "Smart home devices" refer to appliances and devices in the home that can be controlled via the Internet, including thermostats and smart speakers.

[0582] "Automatically adjusting the physical environment" refers to automatically adjusting environmental settings such as room temperature, lighting, and music through smart home devices.

[0583] The present invention is a system for reducing the burden on parents raising children and for raising them more effectively and efficiently. This system analyzes video and audio information acquired from a mobile communication device (e.g., a smartphone), predicts the baby's condition, and provides parents with countermeasures based on the predictions. The specific configuration and operation of the system are described in detail below.

[0584] Hardware and software configuration

[0585] Terminal

[0586] The terminal is a mobile communication device equipped with a camera and a microphone, and is typically a smartphone. The terminal has the following functions:

[0587] 1. Camera: Used to capture the baby's visual information (facial expressions and movements).

[0588] 2. Microphone: Used to record the baby's crying and surrounding environmental sounds.

[0589] 3. Data preprocessing: Converting the captured video and audio data into a format suitable for analysis.

[0590] 4. Data transmission: The preprocessed data is transmitted to the server via the Internet.

[0591] server

[0592] The server has the following analysis and inference functions:

[0593] 1. Video analysis: Analyze the baby's facial expressions using a convolutional neural network (CNN) and extract features.

[0594] 2. Speech analysis: Analyze the baby's crying patterns using a speech recognition model.

[0595] 3. Data integration: The results of the analysis of visual and audio information are integrated to estimate the overall state of the child.

[0596] 4. Prediction function: Predicts future potential states based on past data logs.

[0597] 5. Notification function: Prediction results and countermeasures are notified to the device.

[0598] Smart Home Devices

[0599] The server connects to smart home devices via the Internet to achieve the following functions:

[0600] 1. Environmental adjustment: Adjust the temperature and lighting in your baby's room appropriately.

[0601] 2. Play music: Play music that helps your baby relax.

[0602] Specific examples

[0603] As a concrete example, the following scenario can be considered.

[0604] The user launches the "childcare support system." The device (e.g., a smartphone) then captures video and audio of the baby. The baby's facial expression and audio data are sent to a server via the Internet. The server first inputs the video data into a CNN model, extracts features, and analyzes the baby's emotions and needs. Next, it inputs the audio data into a speech recognition model, and analyzes the crying patterns to infer the baby's condition.

[0605] For example, if the server determines that the baby is crying because it seems sleepy, it will send a notification to the device saying, "The baby seems sleepy. The room temperature is being adjusted appropriately." At the same time, the server will send instructions to connected smart home devices to set the room temperature to 22 degrees and play relaxing music. This allows the baby to sleep in a comfortable environment, allowing parents to respond at the appropriate time.

[0606] Prompt Sentence Examples

[0607] Below are some example prompts that can be used as input to a generative AI model.

[0608] "My baby is crying. Please help me guess what his current emotions and physical needs are and suggest appropriate responses."

[0609] Using this prompt, the AI ​​model will perform specific analysis and make suggestions. In this way, the system of the present invention aims to support childcare and reduce the burden on parents.

[0610] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0611] Step 1:

[0612] The device activates the camera and microphone to capture the baby's video and audio information in real time. Specifically, the smartphone camera captures the baby's face, and the microphone records the baby's crying and surrounding environmental sounds. The input is the baby's actual video and audio, and the output is the raw data captured by the camera and microphone.

[0613] Step 2:

[0614] The device preprocesses the captured video and audio data. For example, video data is resized and audio data is converted to a specific format. The input to this process is the raw data captured in step 1, and the output is preprocessed data suitable for analysis.

[0615] Step 3:

[0616] Send the preprocessed data to a server via the internet. The input is the preprocessed data generated in step 2, and the output is the data sent to the server.

[0617] Step 4:

[0618] A convolutional neural network (CNN) is used to analyze the video data received by the server. Specifically, the server inputs the video data into a CNN model, identifies the baby's facial expressions, and extracts features. The input for this analysis is the video data received in step 3, and the output is the baby's facial expressions and their features.

[0619] Step 5:

[0620] The server uses a speech recognition model to analyze the received voice data. Specifically, the server inputs the voice data into the model, analyzes the crying pattern, and extracts its features. The input for this analysis is the voice data received in step 3, and the output is the crying pattern and its features.

[0621] Step 6:

[0622] The server integrates the results of the video and audio analysis to infer the baby's overall state. The server uses the video and audio features as examples to input into a statistical model to infer the baby's emotions and needs. The input for this inference is the features extracted in steps 4 and 5, and the output is the baby's current state (e.g., hunger, sleepiness, discomfort, etc.).

[0623] Step 7:

[0624] Based on the estimated state of the infant, the server refers to past data logs to predict the future state. The server uses statistical methods and machine learning models to predict the future state (e.g., the infant will be hungry in two hours). The inputs to this prediction are the current state estimated in step 6 and past data logs, and the output is the predicted future state.

[0625] Step 8:

[0626] The server notifies the device of the prediction results and countermeasures via the Internet. The server generates a specific countermeasure (e.g., notification of the next breastfeeding time) and sends a notification to the smartphone. The input of this notification is the prediction result generated in step 7, and the output is the notification content sent to the smartphone.

[0627] Step 9:

[0628] The server interacts with the smart home devices and issues necessary instructions, such as adjusting the room temperature or playing relaxing music. Based on the prediction results, the server instructs the smart home devices to set appropriate settings. The input of this instruction is the result predicted in step 7, and the output is an operation instruction to the smart home devices.

[0629] (Application example 1)

[0630] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0631] It is necessary to reduce the burden on parents raising children and to make child-rearing more effective and efficient, but current systems lack the data analysis necessary to accurately predict the condition of young children, the prediction of potential future conditions, and the provision of appropriate countermeasures. Furthermore, the lack of an environment in which information can be shared and parenting can be coordinated not only with parents but also with other stakeholders and family members is slowing the reduction of the burden of child-rearing. Furthermore, there is currently a lack of coordination when it comes to automatically adjusting the environment using home automation devices.

[0632] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0633] In this invention, the server includes: a means for analyzing video information of the baby; a means for analyzing audio information; a means for integrating the video information and audio information and using a learning algorithm to estimate the infant's condition; a means for predicting a potential future condition based on the estimated infant's condition; a means for providing countermeasures for the predicted future condition; a means for sending an alert to a caregiver based on the estimated infant's condition; a means for controlling home automation devices based on the alert; a means for generating the content of the countermeasures using a generative AI; and a means for creating prompts to support the caregiver's actions based on the generated countermeasures. This reduces the burden on parents raising children and enables appropriate responses through efficient collaboration within the household and among related parties. Furthermore, by linking with home automation devices, it is possible to automatically optimize the childcare environment.

[0634] A "mobile communication device" is a device with communication capabilities that can be used by a user while on the move, such as a mobile phone or smartphone.

[0635] "Video information" refers to video and still image data captured using a camera.

[0636] "Audio information" refers to sound data acquired using a microphone.

[0637] A "learning algorithm" is a method for analyzing data to find patterns and relationships and make predictions or classifications.

[0638] "Infant states" are situations that indicate the emotional and physical needs of babies and young children.

[0639] "Future potential states" are future states of infants predicted based on current and past data.

[0640] "Countermeasures" refer to specific guidelines and measures provided depending on the child's condition.

[0641] An "alert" is a warning message intended to notify you of urgent or important information.

[0642] A "caregiver" is a parent or other guardian who cares for a baby or young child.

[0643] "Home automation devices" refer to systems and equipment that automatically control various devices in the home.

[0644] "Generative AI" refers to technology that uses artificial intelligence to generate new information and data.

[0645] A "prompt" is a sentence that guides the user in instructing them on what to do.

[0646] This invention is a home support system based on the "AI Mimamori" system that monitors the condition of infants in real time and provides appropriate countermeasures. This system uses a mobile communication device (e.g., a smartphone) to acquire and analyze video and audio information of the baby, inferring the infant's condition and providing the user with countermeasures based on that condition.

[0647] Hardware and Software Configuration

[0648] Device (e.g. smartphone)

[0649] The device uses a camera and microphone to capture real-time video and audio information of the baby, which is then sent to a server via the Internet.

[0650] server

[0651] The server analyzes the received video information using deep learning models such as convolutional neural networks (CNNs) to infer the infant's condition from their facial expressions and movements. Similarly, it analyzes audio information using speech recognition models to understand the infant's condition from crying patterns and other audio signals. Based on these results, the infant's overall condition is inferred.

[0652] The server then uses the inference results to predict potential future states. Based on this prediction, an appropriate response is generated using a generative AI model and provided to the user as a prompt, such as "The baby will be hungry in two hours."

[0653] Home Automation Devices

[0654] The server also connects with home automation devices (e.g., smart home systems) to automate environmental settings such as adjusting the temperature and playing music in the baby's room, ensuring a comfortable and safe environment for the baby.

[0655] Examples and prompts

[0656] For example, the "AI Mimamori" system may detect a baby's crying and send an alert to the user's smartphone saying, "Your baby is crying excessively. Please check on him / her immediately." At the same time, the server will instruct the smart home system to turn on the lights in the baby's room and provide the camera feed to the parents remotely.

[0657] Prompt Sentence Examples

[0658] "What is the best way to respond if my baby is crying?"

[0659] "Infer your baby's emotional state from their facial expressions."

[0660] "How do I connect it to my smart home devices to keep my baby safe?"

[0661] This allows parents to keep track of their baby's condition and respond quickly, while automated home devices can adjust the environment to provide a comfortable and safe environment for the baby.

[0662] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0663] Step 1:

[0664] Data acquisition (device)

[0665] The device (smartphone) activates the camera and microphone to capture the baby's video and audio information in real time. At this time, the camera captures continuous still images, and the microphone captures continuous audio samples.

[0666] Input: Camera images, audio data

[0667] Output: Pre-processed video data, audio data

[0668] Step 2:

[0669] Data preprocessing (terminal)

[0670] The device resizes the captured video data and converts the audio data into an appropriate format for preprocessing, which removes noise for data analysis. Specifically, the image size is resized to 224x224 pixels, and the audio data is extracted to a certain time range.

[0671] Input: Camera images, audio data

[0672] Output: Pre-processed video data, audio data

[0673] Step 3:

[0674] Data transmission (terminal)

[0675] The device transmits the preprocessed video and audio data to a server via the Internet, using a secure communication protocol such as HTTPS.

[0676] Input: Preprocessed video data, audio data

[0677] Output: Send data to the server

[0678] Step 4:

[0679] Video data analysis (server)

[0680] The video data received by the server is analyzed using a convolutional neural network (CNN). Specifically, the video data is input into a CNN model to extract features such as the baby's facial expressions and movements. This analysis allows the system to predict the baby's state, such as whether it is crying or about to fall asleep.

[0681] Input: Preprocessed video data

[0682] Output: Image features, inferred state

[0683] Step 5:

[0684] Voice data analysis (server)

[0685] The server analyzes the received voice data using a voice recognition model, which inputs the voice data into the model and uses crying patterns and other audio signals to understand the baby's condition.

[0686] Input: Preprocessed audio data

[0687] Output: Audio features, inferred state

[0688] Step 6:

[0689] State estimation integration (server)

[0690] The server combines the results of the video and audio analysis to comprehensively estimate the baby's condition. Specifically, it uses a machine learning model to combine the features of both data to determine the final condition.

[0691] Input: Image features, audio features

[0692] Output: Overall guess state

[0693] Step 7:

[0694] Future state prediction (server)

[0695] The server predicts potential future states based on the current state, specifically by comparing it with past data logs to predict when the next person will be hungry or when they will fall asleep.

[0696] Input: Overall guess state, historical data log

[0697] Output: Future potential states

[0698] Step 8:

[0699] Generate countermeasures (server)

[0700] The server generates appropriate childcare responses based on the predicted future state, which are created using a generative AI model and provide a concrete action plan for the caregiver.

[0701] Input: Future potential state

[0702] Output: Generated workarounds

[0703] Step 9:

[0704] Prompt statement generation (server)

[0705] The server creates a prompt sentence to support the caregiver's actions based on the generated countermeasure. Specifically, it creates a sentence that succinctly expresses the generated countermeasure and notifies the caregiver.

[0706] Input: Generated countermeasures

[0707] Output: prompt statement

[0708] Step 10:

[0709] Notification and control of home automation devices (server)

[0710] The server then sends the generated prompts to the caregiver's device and simultaneously controls home automation devices, such as sending notifications to a smartphone and issuing commands to the smart home system to adjust the room temperature or play music.

[0711] Input: prompt statement, home automation device control instructions

[0712] Output: Notification to caregivers, activation of home automation devices

[0713] The above is the specific flow of the system program processing.

[0714] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0715] This invention combines an emotion engine with the "AI Mimamori" system, which aims to reduce the burden on parents raising children and make child-rearing more effective and efficient amid a declining birthrate. This system analyzes video and audio information obtained from mobile communication devices (e.g., smartphones), predicts the state of a child, and provides parents with countermeasures based on that information. It also recognizes the user's emotions and provides more appropriate countermeasures based on those.

[0716] System Overview

[0717] The device uses a camera and microphone to capture video and audio information of the baby, as well as that of the user (parent) in real time. For example, the camera and microphone of a smartphone can be used to capture the facial expressions and voices of the baby and parent. This captured data is then sent to a server via the Internet.

[0718] The server analyzes this data using AI models. Deep learning models such as convolutional neural networks (CNNs) are used to analyze the baby's visual data, and speech recognition models are used to analyze the auditory data. Similarly, an emotion recognition engine is used to analyze the parent's emotions. The results of each analysis (e.g., the baby is hungry, the parent is stressed, etc.) are integrated by the server to infer the overall state of the baby and parent.

[0719] The estimated state of the infant and parent is compared and analyzed with previously collected data logs to predict potential future states. Based on this prediction, appropriate countermeasures (such as notifications of the next feeding time or suggestions to reduce parental stress) are notified to the device. Furthermore, by linking with smart home devices, the system can automatically adjust the room temperature, play music, etc.

[0720] Program processing

[0721] The device activates the camera and microphone to capture video and audio information of the baby and parent, and captures data in real time. This data is pre-processed (e.g., image resizing and audio data format conversion) and sent to a server via the Internet.

[0722] The server first analyzes the baby's video information using a convolutional neural network (CNN). It receives image data as input, extracts features, and infers the baby's state, such as hunger, sleepiness, or desire to play, from its facial expressions and movements. Similarly, the baby's audio information is input into a speech recognition model, which evaluates the baby's state based on crying patterns and other audio signals.

[0723] The server then inputs the parent's video and audio information into an emotion recognition engine, which analyzes the parent's facial expressions and tone of voice to identify emotions such as stress, joy, and fatigue.

[0724] The results of both analyses are integrated on a server and recorded as the overall state of the infant and parent. This information is stored in a data log and compared with past data to predict future states (for example, the baby may become hungry in two hours, which may cause stress for the parent).

[0725] The prediction results and countermeasures are sent to the device via the Internet and notified to the parent. For example, if the baby seems sleepy and the parent is tired, a notification will appear on the smartphone saying, "The baby seems sleepy. The room temperature is being adjusted. We recommend that the parent also take a break."

[0726] In addition, the server can also connect to smart home devices to automatically adjust the room temperature and play relaxing music to maintain a comfortable environment for both baby and parent, ensuring a safe and comfortable environment for both baby and parent.

[0727] Specific examples

[0728] The user activates the "AI Mimamori" system. The device (e.g., a smartphone) then captures video and audio of the baby and parent. The server analyzes this data and infers whether the baby is hungry and whether the parent is feeling stressed. Based on this, the server predicts that the baby will become hungry again in approximately two hours and suggests that the parent take a break to relax. The server then instructs smart home devices to set the room temperature to 22 degrees and play relaxing music. The smartphone also receives a notification saying, "The baby is likely to become hungry in two hours. Please relax, parent." This allows parents to prepare the next meal at the appropriate time while also maintaining their own health.

[0729] This series of actions reduces the burden of childcare on parents and provides a comfortable environment for both the baby and the parent. The "AI Mimamori" system of this invention provides a new means of supporting childcare and is expected to contribute to measures to combat the declining birthrate.

[0730] The processing flow will be explained below.

[0731] Step 1:

[0732] The user launches the "AI Mimamori" system.

[0733] Users start the system by tapping a dedicated application on their smartphone.

[0734] Step 2:

[0735] The device activates the camera and acquires video information.

[0736] The device uses a built-in camera to capture images of the baby in real time and obtain frame data.

[0737] Step 3:

[0738] The device activates the microphone and captures audio information.

[0739] The device uses a built-in microphone to record the baby's crying and environmental sounds in real time, capturing audio data.

[0740] Step 4:

[0741] The terminal uses a camera to acquire video information of the user.

[0742] The device captures the user's facial expressions using a built-in camera and acquires the data.

[0743] Step 5:

[0744] The terminal acquires the user's voice information using a microphone.

[0745] The device uses a built-in microphone to record the user's tone of voice and speaking style.

[0746] Step 6:

[0747] The video and audio information acquired by the terminal is preprocessed.

[0748] The video information is resized, noise removed, and otherwise processed to convert it into a format suitable for analysis.

[0749] The audio information is processed by adjusting the sampling rate and removing noise, and then converted into a format suitable for analysis.

[0750] Step 7:

[0751] The terminal transmits the preprocessed data to a server via the Internet.

[0752] The device converts the acquired video and audio data into an appropriate format and sends it to the server using encrypted communication such as SSL.

[0753] Step 8:

[0754] The server uses an AI model to analyze the video information.

[0755] The server inputs the preprocessed video data into a convolutional neural network (CNN), which infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements.

[0756] Step 9:

[0757] The server uses an AI model to analyze the audio information.

[0758] The server inputs the preprocessed audio data into a speech recognition model to assess the baby's condition from crying patterns and other audio signals.

[0759] Step 10:

[0760] The server uses an emotion recognition engine to analyze the user's video and audio information.

[0761] The server analyzes the preprocessed video data of the user and infers emotions such as stress, joy, and fatigue from facial expressions.

[0762] Similarly, the system analyzes the user's voice data and assesses emotions based on tone of voice and speaking style.

[0763] Step 11:

[0764] The server integrates the results of video analysis, audio analysis, and emotion analysis.

[0765] The server compiles the results of the analysis of visual, auditory and emotional information into unified data and estimates the overall state of the infant and parent.

[0766] Step 12:

[0767] The server logs the inferred infant and parent states.

[0768] The server stores the guesses and their timestamps in a data log.

[0769] Step 13:

[0770] The server predicts future conditions based on the accumulated data logs.

[0771] The server analyzes past data logs and uses learning algorithms to predict future conditions of the infant and parent (e.g., the next feeding time or the parent's stress level).

[0772] Step 14:

[0773] The server notifies the parent to provide appropriate action.

[0774] Based on the results of the estimation and prediction, the server notifies the parent's smartphone with advice on appropriate measures (e.g., preparing for breastfeeding, suggestions for reducing parental stress).

[0775] Step 15:

[0776] The device receives the notification and displays it to the user.

[0777] The device receives notifications from the server and displays them to the user as push notifications or alerts. For example, "Your baby will be hungry in 2 hours. Please relax."

[0778] Step 16:

[0779] The server controls the smart home devices.

[0780] The server controls smart home devices (e.g., smart thermostats and speakers) to adjust the room temperature and play relaxing music to maintain a comfortable environment for the baby and parents.

[0781] Step 17:

[0782] The user acts on the notification.

[0783] Users can take necessary measures based on notifications from their smartphones, such as preparing to breastfeed, preparing the environment for putting the baby to sleep, or taking a break to relieve their own stress.

[0784] Example 2

[0785] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0786] As the birthrate continues to decline, there is a need to reduce the burden on parents raising children and provide more effective and efficient methods for raising them. In particular, there is a need for a system that can accurately grasp the baby's condition and quickly provide appropriate countermeasures. There is also a need for a function that takes into account the parents' own emotional state and provides comprehensive support. However, existing systems are limited to estimating the baby's condition, and there are no systems that perform integrated analysis including the parent's emotional state or automatically adjust environmental conditions. Therefore, the challenge is to provide a system that provides comprehensive support for both, thereby significantly reducing the burden on parents raising children and providing a more comfortable childcare environment.

[0787] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0788] In this invention, the server includes means for analyzing video information acquired from a mobile communication device, means for analyzing audio information acquired from the mobile communication device, means for integrating the video information and the audio information and using a learning algorithm to predict the infant's condition, means for integrating and analyzing the estimated infant's condition and the guardian's emotional state to evaluate the overall condition, means for predicting a future potential condition based on the integrated analysis, means for providing the guardian with countermeasures for the predicted future condition, and means for automatically adjusting environmental conditions based on the countermeasures and cooperating with a device to provide a comfortable environment for the guardian and infant. This enables a comprehensive analysis of the baby's condition and the parent's emotional state, and makes it possible to reduce the burden of child-rearing on parents by suggesting appropriate countermeasures and automatically adjusting the environment.

[0789] "Mobile communications device" refers to a mobile device such as a mobile phone or smartphone, which has the ability to send and receive data using communications technology.

[0790] "Video information" is data containing visual content captured by a camera, video device, or the like.

[0791] "Audio information" is information that includes sound data collected by an audio capture device such as a microphone.

[0792] A "learning algorithm" is a set of computational methods and procedures for analyzing data and finding patterns and regularities within it, and plays an important role in machine learning.

[0793] "Infant state" refers to the physical and emotional state of a young child, such as a baby, including hunger, sleepiness, pleasure and pain, etc.

[0794] "Parental emotional state" refers to the psychological emotional state of the parent or guardian, including stress, joy, fatigue, etc.

[0795] "Integrated analysis" is an analytical method that combines data obtained from multiple different sources to make a single comprehensive evaluation.

[0796] "Future potential state" refers to the likely future state of the child and caregiver, predicted based on current data.

[0797] A "countermeasure" is a proposed action or method based on an estimated state or prediction, with the aim of solving a problem or improving a situation.

[0798] "Automatic adjustment of environmental conditions" is an operation that automatically adjusts environmental elements such as room temperature, lighting, and music to provide a comfortable living environment for infants and their guardians.

[0799] "Working with devices" refers to a system exchanging information and instructions with other devices or equipment to perform functions in collaboration.

[0800] MODE FOR CARRYING OUT THE INVENTION

[0801] The present invention relates to the "AI Mimamori" system, which aims to reduce the burden on parents raising children and help them do so more effectively and efficiently. This system analyzes video and audio information obtained from mobile communication devices (such as smartphones), predicts the state of the child, and provides parents with countermeasures based on that information. It is also possible to recognize the emotions of the user (parent) and provide more appropriate countermeasures.

[0802] Hardware and software used

[0803] Devices: Smartphone, camera, microphone

[0804] Server: High-performance computer

[0805] software:

[0806] Video analysis: Convolutional neural networks (CNN)

[0807] Speech analysis: Speech recognition model

[0808] Sentiment Analysis: Emotion Recognition Engine

[0809] Data transmission: Internet Protocol

[0810] System Operation

[0811] The user activates the "AI Mimamori" system using a smartphone. The device uses a camera and microphone to capture video and audio information of the baby and parent in real time. For example, the smartphone's camera captures the baby's face, and the microphone records its crying. At the same time, the parent's facial expressions and voice are also captured. This captured data is preprocessed and then sent to a server via the Internet.

[0812] The server first analyzes the baby's video information using a convolutional neural network (CNN). It receives image data as input, extracts features, and infers the baby's state, such as hunger, sleepiness, or desire to play, from its facial expressions and movements. Similarly, the baby's audio information is input into a speech recognition model, which evaluates the baby's state based on crying patterns and other audio signals.

[0813] The server then inputs the parent's video and audio information into an emotion recognition engine, which analyzes the parent's facial expressions and tone of voice to identify emotions such as stress, joy, and fatigue.

[0814] The results of both analyses are integrated on a server and recorded as the overall state of the infant and parent. This information is stored in a data log and compared with past data to predict future states (for example, the baby may become hungry in two hours, which may cause stress for the parent).

[0815] The prediction results and countermeasures are sent to the device via the Internet and notified to the parent. For example, if the baby seems sleepy and the parent is tired, a notification will appear on the smartphone saying, "The baby seems sleepy. The room temperature is being adjusted. We recommend that the parent also take a break."

[0816] In addition, the server can also connect to smart home devices to automatically adjust the room temperature and play relaxing music to maintain a comfortable environment for both baby and parent, ensuring a safe and comfortable environment for both baby and parent.

[0817] Specific examples

[0818] The user activates the "AI Mimamori" system. The device (e.g., a smartphone) then captures video and audio of the baby and parent. The server analyzes this data and infers whether the baby is hungry and whether the parent is feeling stressed. Based on this, the server predicts that the baby will become hungry again in approximately two hours and suggests that the parent take a break to relax. The server then instructs smart home devices to set the room temperature to 22 degrees and play relaxing music. The smartphone also receives a notification saying, "The baby is likely to become hungry in two hours. Please relax, parent." This allows parents to prepare the next meal at the appropriate time while also maintaining their own health.

[0819] Prompt Sentence Examples

[0820] Contracting System:

[0821] 1. Use a camera and microphone to capture video and audio of the baby and parent.

[0822] 2. Video and audio data is sent to a server via the Internet.

[0823] 3. The server analyzes the data, proposes appropriate countermeasures, and notifies the smartphone.

[0824] Situation:

[0825] Imagine when a baby is crying and the parents are tired, and generate the best response.

[0826] Expected output:

[0827] "It's likely your baby is hungry and may be hungry again in about two hours. I set the room temperature to 72 degrees and played relaxing music. I encourage parents to take breaks as well."

[0828] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0829] Step 1:

[0830] The device uses a camera and microphone to capture real-time video and audio information of the baby and parent. The camera captures the baby's face, and the microphone records the baby's crying. At the same time, the device also captures the parent's facial expressions and voice. This data is temporarily stored on the device.

[0831] Input: Video and audio information from the camera and microphone

[0832] Output: A dataset of captured video and audio information

[0833] Step 2:

[0834] The device preprocesses the captured video and audio data, specifically resizing the video data and converting the audio data into a digital format. This preprocessing ensures uniform data quality for efficient analysis.

[0835] Input: A dataset of captured video and audio information.

[0836] Output: Pre-processed video and audio information

[0837] Step 3:

[0838] The device then transmits the preprocessed data to a server over the internet, using encryption protocols to prevent data leakage or tampering.

[0839] Input: Preprocessed video and audio information

[0840] Output: Data sent to the server

[0841] Step 4:

[0842] The server inputs the transmitted video data of the baby into a convolutional neural network (CNN). The CNN extracts features from the image data and infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements. For example, it analyzes video of a baby with its mouth open and determines that the baby is hungry.

[0843] Input: Baby video data sent to the server

[0844] Output: Analysis results showing the baby's condition

[0845] Step 5:

[0846] The server inputs the baby's voice data into a speech recognition model, which evaluates the baby's state from crying patterns and other audio signals. For example, if the crying pattern matches "sleepy," that information is output.

[0847] Input: Baby's voice data sent to the server

[0848] Output: Voice analysis results showing the baby's condition

[0849] Step 6:

[0850] The server inputs the parent's video and audio information into an emotion recognition engine. The emotion recognition engine analyzes the parent's facial expressions and tone of voice to determine emotions such as stress, joy, and fatigue. For example, it analyzes a video of the parent frowning and determines that the parent is feeling "stressed."

[0851] Input: Parent's video and audio information sent to the server

[0852] Output: Analysis results showing the parent's emotional state

[0853] Step 7:

[0854] The server combines the results of the baby's and parent's analysis to assess their overall condition. This information is stored in a data log and compared with past data. This allows for predictions of future conditions (e.g., the baby will be hungry in two hours).

[0855] Input: Analysis results showing the baby's condition, analysis results showing the parent's emotional state

[0856] Output: Comprehensive condition assessment and future condition prediction

[0857] Step 8:

[0858] The server generates appropriate countermeasures based on the prediction results and notifies the device, such as the next feeding time or suggestions for parental relaxation.

[0859] Input: Comprehensive condition assessment and future condition prediction

[0860] Output: Notification of action to be taken

[0861] Step 9:

[0862] The server works in conjunction with smart home devices to automatically adjust the room temperature and play relaxing music to maintain a comfortable environment for both baby and parent.

[0863] Input: Comprehensive condition assessment and future condition prediction

[0864] Output: Instructions to smart home devices

[0865] (Application example 2)

[0866] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0867] In order to reduce the burden on parents raising children and to raise them more effectively and efficiently amid a declining birthrate, it is necessary to consider not only the condition of the infant but also the emotional state of the parent. However, conventional systems lack the functionality to analyze the parent's emotional state, making it difficult to provide appropriate countermeasures. In addition, there is a lack of integration with smart home devices to maintain a comfortable environment for infants and parents, which means the burden on parents is not reduced.

[0868] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video information acquired from a mobile communication device, means for analyzing audio information acquired from the mobile communication device, means for integrating the video information and the audio information and using a learning algorithm to estimate the infant's state, means for predicting a future potential state based on the estimated infant's state, means for analyzing the parent's emotions, means for providing countermeasures based on the estimated infant's state and the parent's emotions, means for adjusting the environment in cooperation with a smart home device, and means for notifying the parent of the countermeasures. This provides appropriate childcare support that also takes into account the parent's emotional state, making it possible to reduce the burden of childcare and provide a comfortable environment for both the baby and the parent.

[0869] A "mobile communication device" is a portable information communication terminal equipped with a camera and a microphone, and has the function of acquiring video information and audio information and transmitting and receiving data via a communication network.

[0870] "Video information" refers to visual data captured by a camera on a mobile communication device and presented in the form of images or video.

[0871] "Audio information" refers to auditory data captured by a microphone on a mobile communication device and expressed in voice or acoustic form.

[0872] A "learning algorithm" refers to a computational method or model that learns patterns and relationships based on previously collected data and makes inferences and predictions about new data.

[0873] "Infant condition" refers to the baby's physical or psychological state, such as hunger, sleepiness, or a desire to play.

[0874] "Future potential states" refer to the infant's future state predicted based on current data and analysis results, such as hunger or sleepiness after a certain time.

[0875] "Parental emotions" refers to the emotional state that the parent is currently experiencing, such as specific emotions such as stress, joy, or fatigue.

[0876] "Countermeasures" refer to specific actions or coping methods suggested based on the estimated infant's condition and the parent's emotions, such as notification of the next feeding time or advice on stress reduction.

[0877] "Smart home devices" refer to home electronic devices that are connected to a network and have remote control or automation capabilities, such as smart thermostats and smart speakers.

[0878] "Environmental adjustment" refers to operations such as changing temperature and sound settings to maintain a comfortable indoor environment.

[0879] "Notification" refers to a function of informing a user of information via a mobile communication device, and includes notification by screen display or voice.

[0880] The "childcare support system" of the present invention is a system that uses a mobile communication device (e.g., a smartphone) to reduce the burden on parents raising children and allow them to raise children more effectively and efficiently. This system analyzes not only the condition of the infant but also the emotional state of the parent, and provides the user with appropriate countermeasures. The following describes in detail an embodiment of the present invention.

[0881] 1. System Configuration

[0882] The system consists of three main components: terminals, servers, and smart home devices.

[0883] Terminal

[0884] The device is a smartphone equipped with a camera and microphone, which captures real-time video and audio information of the baby and parent. The captured data is then sent to a server via the Internet.

[0885] server

[0886] The server has the following analytical methods and capabilities:

[0887] Video information analysis method: Using a convolutional neural network (CNN), the video data obtained from the camera is analyzed to extract the baby's facial expressions and movements.

[0888] Audio information analysis means: Using a speech recognition model, the audio data captured from the microphone is analyzed to evaluate crying and other audio signals.

[0889] Emotion recognition: An emotion recognition engine is used to analyze emotions from the parent's facial expressions and tone of voice.

[0890] The analyzed results are integrated to estimate the status of the infant and parent. Based on this, potential future conditions are predicted and appropriate countermeasures are generated. The appropriate countermeasures are notified to the terminal via the Internet, and instructions for cooperation are sent to smart home devices.

[0891] Smart Home Devices

[0892] Smart home devices automatically adjust the environment, such as networked thermostats and music players, to keep children and parents comfortable by adjusting the temperature or playing relaxing music.

[0893] 2. System Operation

[0894] Data Acquisition and Transmission

[0895] When a user launches the smartphone app, the camera and microphone begin capturing data in real time, which is then sent to a server via the Internet for detailed analysis.

[0896] Analysis and Notification

[0897] The server analyzes video and audio data to infer the baby's condition. For example, it can infer hunger from the baby's facial expression and sleepiness from the crying pattern. It can also analyze the parent's emotions from their facial expression and tone of voice to infer stress or fatigue. Based on these analysis results, it predicts future potential conditions and generates countermeasures.

[0898] environmental adjustment

[0899] By connecting with smart home devices, it can automatically adjust the room temperature and play relaxing music, reducing the burden on parents while providing a comfortable environment for both baby and parent.

[0900] 3. Examples of concrete examples and prompts

[0901] Specific examples

[0902] After the user activates the system, the smartphone's camera and microphone capture video and audio of the baby and parent. The server analyzes the data and infers that the baby is hungry and the parent is stressed. The following notification is displayed on the smartphone as a response generated by the server:

[0903] "Your baby is hungry. The next feeding is in about an hour. Set the room temperature to 72 degrees and play some relaxing music."

[0904] Prompt Sentence Examples

[0905] Analyze your baby's condition:

[0906] Capture the baby's facial expressions and movements with a camera and analyze them using a CNN model.

[0907] Use a voice recognition model to analyze crying patterns and assess states such as hunger and sleepiness.

[0908] Analyzing parental emotions:

[0909] Capture the parent's facial expressions and voice and analyze them with an emotion recognition engine.

[0910] Integrate baby and parent status and generate notifications:

[0911] If the baby is hungry, notify them of appropriate action (e.g., notify them of the next feeding time).

[0912] If a parent is tired, suggest a break and adjust the temperature with a smart home device.

[0913] As described above, the childcare support system of the present invention is capable of providing appropriate childcare support that takes into consideration the emotional state of the parents, thereby reducing the burden on the parents and improving the comfort of both the baby and the parents.

[0914] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0915] Step 1:

[0916] The user launches the smartphone app.

[0917] Input: Smartphone camera and microphone

[0918] Output: Real-time video and audio information

[0919] How it works: The user taps to launch the smartphone app, which automatically turns on the camera and microphone, capturing video and audio of the baby and parent.

[0920] Step 2:

[0921] The video and audio information acquired by the terminal is transmitted to a server via the Internet.

[0922] Input: Real-time video and audio information

[0923] Output: Data sent to the server

[0924] How it works: The device transfers the captured video and audio data to the server via a communications network. The data is encrypted and securely transmitted.

[0925] Step 3:

[0926] The server receives the video data and analyzes it using a convolutional neural network (CNN).

[0927] Input: Video information obtained from the camera

[0928] Output: Baby's facial expressions and behavior patterns

[0929] How it works: The server preprocesses the received video data (e.g., resizes and normalizes it) and then inputs it into a CNN model. The CNN model extracts features and infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements.

[0930] Step 4:

[0931] The server receives the voice data and analyzes it using a voice recognition model.

[0932] Input: Audio information obtained from the microphone

[0933] Output: Baby's crying and voice patterns

[0934] Specific operation: The server preprocesses the received voice data (e.g., noise reduction, normalization) and then inputs it into a voice recognition model. The voice recognition model analyzes the characteristics of the crying voice and evaluates the baby's state (e.g., hunger, sleepiness, demands).

[0935] Step 5:

[0936] The server inputs the parent's video and audio data into an emotion recognition engine for analysis.

[0937] Input: Parent video and audio information obtained from camera and microphone

[0938] Output: Parent's emotional state (e.g., stress, joy, fatigue)

[0939] Specific operation: The server inputs the parent's video data into an emotion recognition engine and analyzes their facial expressions. It also uses audio data to analyze their tone of voice and the content of their speech. This allows it to infer the parent's emotional state.

[0940] Step 6:

[0941] The server integrates the baby and parent condition data it has acquired and predicts potential future conditions.

[0942] Input: Baby's state data, Parent's emotional state data

[0943] Output: Predicted future state (e.g. next feeding time, parental stress level)

[0944] How it works: The server performs statistical analysis and comparison with past data to predict the future state of the baby and parents, for example, calculating the likelihood that the baby will be hungry after a certain time, or the likelihood that the parents' stress levels will increase.

[0945] Step 7:

[0946] The server generates appropriate countermeasures based on the prediction results and notifies the device.

[0947] Input: Predicted future state data

[0948] Output: Notifications sent to smartphones (e.g., notifications about breastfeeding times, suggestions for adjusting the environment)

[0949] Specific operation: Based on the prediction results, the server generates appropriate responses for the baby and the parent. For example, it predicts that the baby will become hungry again in about two hours and suggests that the parent take a break to relax. This information is then sent to the smartphone.

[0950] Step 8:

[0951] The server works in conjunction with smart home devices to adjust the environment.

[0952] Input: Generated countermeasure data, control information of smart home devices

[0953] Output: Adjusted comfort environment (e.g., adjusting room temperature, playing relaxing music)

[0954] How it works: The server sends instructions to smart home devices, which then automatically change the room temperature or play relaxing music, optimizing the childcare environment.

[0955] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0956] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0957] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0958] [Third embodiment]

[0959] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0960] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0961] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0962] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0963] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0964] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0965] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0966] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0967] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0968] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0969] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0970] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0971] The present invention relates to the "AI Mimamori" system, which aims to reduce the burden on parents raising children and make child-rearing more effective and efficient amid the declining birthrate. This system analyzes video and audio information obtained from mobile communication devices (e.g., smartphones), predicts the condition of infants, and provides parents with countermeasures based on that information. The detailed configuration and operation for realizing this system are described below.

[0972] System Overview

[0973] The device uses a camera and microphone to capture real-time video and audio information about the baby. For example, a smartphone's camera and microphone can be used to capture the baby's facial expressions, crying, and surrounding environmental sounds. This captured data is then sent to a server via the Internet.

[0974] The server analyzes this data using AI models. Deep learning models such as convolutional neural networks (CNNs) are used to analyze visual data, and speech recognition models are used to analyze auditory data. The results of each analysis (e.g., whether the baby is hungry, sleepy, or wants to play) are integrated by the server to estimate the baby's overall state.

[0975] The estimated infant's condition is compared and analyzed with previously collected data logs to predict potential future conditions. Based on this prediction, appropriate countermeasures (such as notifying the next feeding time or providing a sleep-friendly environment) are notified to the device. Furthermore, by linking with smart home devices, the system can automatically adjust the room temperature, play music, etc.

[0976] Program processing

[0977] The device activates the camera and microphone to capture the baby's video and audio information, and captures the data in real time. This data is pre-processed (e.g., image resizing and audio data format conversion) and then sent to a server via the Internet.

[0978] The server first analyzes visual information using a convolutional neural network (CNN). It receives image data as input, extracts features, and infers the infant's current emotions and physical needs. Similarly, audio information is fed into a speech recognition model, which analyzes the infant's state from crying patterns and other audio signals.

[0979] The results of both analyses are integrated on a server and recorded as a comprehensive infant status. This information is stored in a data log and compared with past data to predict future behavior (e.g., whether the infant will be hungry in two hours).

[0980] The prediction results and countermeasures are sent to the device via the Internet and notified to the parent. For example, if the baby seems sleepy, a notification such as "The baby seems sleepy. The room temperature is being adjusted appropriately" will be displayed on the smartphone.

[0981] In addition, the server can also connect to smart home devices to automatically adjust the temperature in the baby's room and play music to soothe them to sleep, helping to ensure that the baby is raised in a comfortable and safe environment.

[0982] Specific examples

[0983] The user activates the "AI Mimamori" system. The device (such as a smartphone) then captures video and audio of the baby. The server analyzes this data and infers that the baby is hungry. Based on this, the server predicts that the baby will become hungry again in approximately two hours. The server then issues instructions to smart home devices to set the room temperature to 22 degrees. The smartphone also receives a notification that the baby is likely to become hungry in two hours. This allows the parent to prepare the next meal at the appropriate time.

[0984] This series of actions reduces the burden of childcare on parents and provides a comfortable environment for babies. The "AI Mimamori" system of this invention offers a new means of supporting childcare and is expected to contribute to measures to combat the declining birthrate.

[0985] The processing flow will be explained below.

[0986] Step 1:

[0987] The user launches the "AI Mimamori" system.

[0988] Users start the system by tapping a dedicated application on their smartphone.

[0989] Step 2:

[0990] The device activates the camera and acquires video information.

[0991] The device uses a built-in camera to capture images of the baby in real time and obtain frame data.

[0992] Step 3:

[0993] The device activates the microphone and captures audio information.

[0994] The device uses a built-in microphone to record the baby's crying and environmental sounds in real time, capturing audio data.

[0995] Step 4:

[0996] The video and audio information acquired by the terminal is preprocessed.

[0997] The video information is resized, noise removed, and otherwise processed to convert it into a format suitable for analysis.

[0998] The audio information is processed by adjusting the sampling rate and removing noise, and then converted into a format suitable for analysis.

[0999] Step 5:

[1000] The terminal transmits the preprocessed data to a server via the Internet.

[1001] The device converts the acquired video and audio data into an appropriate format and sends it to the server using encrypted communication such as SSL.

[1002] Step 6:

[1003] The server uses an AI model to analyze the video information.

[1004] The server inputs the preprocessed video data into a convolutional neural network (CNN), which infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements.

[1005] Step 7:

[1006] The server uses an AI model to analyze the audio information.

[1007] The server inputs the preprocessed audio data into a speech recognition model and assesses the baby's condition based on crying patterns and surrounding sounds.

[1008] Step 8:

[1009] The server integrates the video analysis results and audio analysis results.

[1010] The server integrates the results of the analysis of visual and auditory information to estimate the overall state of the child.

[1011] Step 9:

[1012] The server logs the inferred infant state.

[1013] The server stores the guesses and their timestamps in a data log.

[1014] Step 10:

[1015] The server predicts future conditions based on the accumulated data logs.

[1016] The server analyzes past data logs and uses learning algorithms to predict the baby's future condition (for example, the next feeding time or when the baby will feel sleepy).

[1017] Step 11:

[1018] The server notifies the parent to provide appropriate action.

[1019] Based on the results of the estimation and prediction, the server notifies the parent's smartphone with advice on appropriate measures (e.g., preparing for breastfeeding or adjusting the room temperature).

[1020] Step 12:

[1021] The device receives the notification and displays it to the user.

[1022] The device receives notifications from the server and displays them to the user as push notifications or alerts. For example, "Your baby will be hungry in 2 hours."

[1023] Step 13:

[1024] The server controls the smart home devices.

[1025] The server controls smart home devices (e.g., smart thermostats and speakers) to maintain a comfortable environment for the baby, adjusting the room temperature and playing music to soothe the baby to sleep.

[1026] Step 14:

[1027] The user acts on the notification.

[1028] Users can take necessary measures according to the notifications on their smartphones, such as preparing for breastfeeding or preparing the environment for putting the baby to sleep.

[1029] Example 1

[1030] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1031] There is a need for systems that can reduce the burden faced by parents raising children and enable them to raise their children more effectively and efficiently. Especially with the declining birthrate, there is an urgent need to monitor a baby's condition in real time and provide appropriate measures. There is also a need to reduce the burden on parents by integrating smart home devices and automating environmental adjustments.

[1032] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1033] In this invention, the server includes means for analyzing video information acquired from the mobile communication device, means for analyzing audio information acquired from the mobile communication device, means for integrating the video information and the audio information and using a machine learning algorithm to predict the infant's condition, means for predicting the infant's potential future condition based on the predicted infant's condition, means for providing the guardian with countermeasures for the predicted future condition, means for storing the analysis results and countermeasures in a database and performing comparative analysis based on past data, means for integrating the extracted analysis results and countermeasures in the server and notifying the mobile communication device via the Internet, and means for automatically adjusting the physical environment using the mobile communication device and smart home devices. This allows parents raising their baby to understand the baby's condition in real time and obtain appropriate countermeasures. Furthermore, environmental adjustments are automatically performed in cooperation with the smart home devices, effectively reducing the parental burden on parents.

[1034] A "mobile communication device" refers to a device that is mobile and has communication capabilities, and specifically includes smartphones and tablets.

[1035] "Video information" refers to image data or video data acquired using a camera.

[1036] "Audio information" refers to audio data or acoustic data acquired using a microphone.

[1037] "Machine learning algorithm" is a general term for algorithms that learn patterns from data and make predictions and classifications for future data.

[1038] "Infant states" refers to the baby's various emotions and physical needs (e.g., hunger, sleepiness, discomfort, etc.).

[1039] "Inferred infant state" refers to the infant's emotional and needs state inferred by a machine learning algorithm based on the analysis of video and audio information.

[1040] "Future potential state" refers to the infant's future state predicted based on the infant's estimated current state and past data logs.

[1041] "Countermeasures" refer to specific responses and guidelines for action that parents should take based on the estimated condition of the child and their predicted future condition.

[1042] "Guardian" refers to a person who is responsible for caring for a young child, typically a parent or relative.

[1043] A "database" refers to a system that stores data in an organized form and allows it to be searched and updated efficiently.

[1044] "Comparative analysis" refers to an analytical technique that compares different data sets to reveal relationships and patterns between them.

[1045] A "server" refers to a computer system that provides services and data to clients over a network.

[1046] "Internet" refers to a global information and communications network that interconnects computer networks around the world.

[1047] "Smart home devices" refer to appliances and devices in the home that can be controlled via the Internet, including thermostats and smart speakers.

[1048] "Automatically adjusting the physical environment" refers to automatically adjusting environmental settings such as room temperature, lighting, and music through smart home devices.

[1049] The present invention is a system for reducing the burden on parents raising children and for raising them more effectively and efficiently. This system analyzes video and audio information acquired from a mobile communication device (e.g., a smartphone), predicts the baby's condition, and provides parents with countermeasures based on the predictions. The specific configuration and operation of the system are described in detail below.

[1050] Hardware and software configuration

[1051] Terminal

[1052] The terminal is a mobile communication device equipped with a camera and a microphone, and is typically a smartphone. The terminal has the following functions:

[1053] 1. Camera: Used to capture the baby's visual information (facial expressions and movements).

[1054] 2. Microphone: Used to record the baby's crying and surrounding environmental sounds.

[1055] 3. Data preprocessing: Converting the captured video and audio data into a format suitable for analysis.

[1056] 4. Data transmission: The preprocessed data is transmitted to the server via the Internet.

[1057] server

[1058] The server has the following analysis and inference functions:

[1059] 1. Video analysis: Analyze the baby's facial expressions using a convolutional neural network (CNN) and extract features.

[1060] 2. Speech analysis: Analyze the baby's crying patterns using a speech recognition model.

[1061] 3. Data integration: The results of the analysis of visual and audio information are integrated to estimate the overall state of the child.

[1062] 4. Prediction function: Predicts future potential states based on past data logs.

[1063] 5. Notification function: Prediction results and countermeasures are notified to the device.

[1064] Smart Home Devices

[1065] The server connects to smart home devices via the Internet to achieve the following functions:

[1066] 1. Environmental adjustment: Adjust the temperature and lighting in your baby's room appropriately.

[1067] 2. Play music: Play music that helps your baby relax.

[1068] Specific examples

[1069] As a concrete example, the following scenario can be considered.

[1070] The user launches the "childcare support system." The device (e.g., a smartphone) then captures video and audio of the baby. The baby's facial expression and audio data are sent to a server via the Internet. The server first inputs the video data into a CNN model, extracts features, and analyzes the baby's emotions and needs. Next, it inputs the audio data into a speech recognition model, and analyzes the crying patterns to infer the baby's condition.

[1071] For example, if the server determines that the baby is crying because it seems sleepy, it will send a notification to the device saying, "The baby seems sleepy. The room temperature is being adjusted appropriately." At the same time, the server will send instructions to connected smart home devices to set the room temperature to 22 degrees and play relaxing music. This allows the baby to sleep in a comfortable environment, allowing parents to respond at the appropriate time.

[1072] Prompt Sentence Examples

[1073] Below are some example prompts that can be used as input to a generative AI model.

[1074] "My baby is crying. Please help me guess what his current emotions and physical needs are and suggest appropriate responses."

[1075] Using this prompt, the AI ​​model will perform specific analysis and make suggestions. In this way, the system of the present invention aims to support childcare and reduce the burden on parents.

[1076] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1077] Step 1:

[1078] The device activates the camera and microphone to capture the baby's video and audio information in real time. Specifically, the smartphone camera captures the baby's face, and the microphone records the baby's crying and surrounding environmental sounds. The input is the baby's actual video and audio, and the output is the raw data captured by the camera and microphone.

[1079] Step 2:

[1080] The device preprocesses the captured video and audio data. For example, video data is resized and audio data is converted to a specific format. The input to this process is the raw data captured in step 1, and the output is preprocessed data suitable for analysis.

[1081] Step 3:

[1082] Send the preprocessed data to a server via the internet. The input is the preprocessed data generated in step 2, and the output is the data sent to the server.

[1083] Step 4:

[1084] A convolutional neural network (CNN) is used to analyze the video data received by the server. Specifically, the server inputs the video data into a CNN model, identifies the baby's facial expressions, and extracts features. The input for this analysis is the video data received in step 3, and the output is the baby's facial expressions and their features.

[1085] Step 5:

[1086] The server uses a speech recognition model to analyze the received voice data. Specifically, the server inputs the voice data into the model, analyzes the crying pattern, and extracts its features. The input for this analysis is the voice data received in step 3, and the output is the crying pattern and its features.

[1087] Step 6:

[1088] The server integrates the results of the video and audio analysis to infer the baby's overall state. The server uses the video and audio features as examples to input into a statistical model to infer the baby's emotions and needs. The input for this inference is the features extracted in steps 4 and 5, and the output is the baby's current state (e.g., hunger, sleepiness, discomfort, etc.).

[1089] Step 7:

[1090] Based on the estimated state of the infant, the server refers to past data logs to predict the future state. The server uses statistical methods and machine learning models to predict the future state (e.g., the infant will be hungry in two hours). The inputs to this prediction are the current state estimated in step 6 and past data logs, and the output is the predicted future state.

[1091] Step 8:

[1092] The server notifies the device of the prediction results and countermeasures via the Internet. The server generates a specific countermeasure (e.g., notification of the next breastfeeding time) and sends a notification to the smartphone. The input of this notification is the prediction result generated in step 7, and the output is the notification content sent to the smartphone.

[1093] Step 9:

[1094] The server interacts with the smart home devices and issues necessary instructions, such as adjusting the room temperature or playing relaxing music. Based on the prediction results, the server instructs the smart home devices to set appropriate settings. The input of this instruction is the result predicted in step 7, and the output is an operation instruction to the smart home devices.

[1095] (Application example 1)

[1096] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1097] It is necessary to reduce the burden on parents raising children and to make child-rearing more effective and efficient, but current systems lack the data analysis necessary to accurately predict the condition of young children, the prediction of potential future conditions, and the provision of appropriate countermeasures. Furthermore, the lack of an environment in which information can be shared and parenting can be coordinated not only with parents but also with other stakeholders and family members is slowing the reduction of the burden of child-rearing. Furthermore, there is currently a lack of coordination when it comes to automatically adjusting the environment using home automation devices.

[1098] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1099] In this invention, the server includes: a means for analyzing video information of the baby; a means for analyzing audio information; a means for integrating the video information and audio information and using a learning algorithm to estimate the infant's condition; a means for predicting a potential future condition based on the estimated infant's condition; a means for providing countermeasures for the predicted future condition; a means for sending an alert to a caregiver based on the estimated infant's condition; a means for controlling home automation devices based on the alert; a means for generating the content of the countermeasures using a generative AI; and a means for creating prompts to support the caregiver's actions based on the generated countermeasures. This reduces the burden on parents raising children and enables appropriate responses through efficient collaboration within the household and among related parties. Furthermore, by linking with home automation devices, it is possible to automatically optimize the childcare environment.

[1100] A "mobile communication device" is a device with communication capabilities that can be used by a user while on the move, such as a mobile phone or smartphone.

[1101] "Video information" refers to video and still image data captured using a camera.

[1102] "Audio information" refers to sound data acquired using a microphone.

[1103] A "learning algorithm" is a method for analyzing data to find patterns and relationships and make predictions or classifications.

[1104] "Infant states" are situations that indicate the emotional and physical needs of babies and young children.

[1105] "Future potential states" are future states of infants predicted based on current and past data.

[1106] "Countermeasures" refer to specific guidelines and measures provided depending on the child's condition.

[1107] An "alert" is a warning message intended to notify you of urgent or important information.

[1108] A "caregiver" is a parent or other guardian who cares for a baby or young child.

[1109] "Home automation devices" refer to systems and equipment that automatically control various devices in the home.

[1110] "Generative AI" refers to technology that uses artificial intelligence to generate new information and data.

[1111] A "prompt" is a sentence that guides the user in instructing them on what to do.

[1112] This invention is a home support system based on the "AI Mimamori" system that monitors the condition of infants in real time and provides appropriate countermeasures. This system uses a mobile communication device (e.g., a smartphone) to acquire and analyze video and audio information of the baby, inferring the infant's condition and providing the user with countermeasures based on that condition.

[1113] Hardware and Software Configuration

[1114] Device (e.g. smartphone)

[1115] The device uses a camera and microphone to capture real-time video and audio information of the baby, which is then sent to a server via the Internet.

[1116] server

[1117] The server analyzes the received video information using deep learning models such as convolutional neural networks (CNNs) to infer the infant's condition from their facial expressions and movements. Similarly, it analyzes audio information using speech recognition models to understand the infant's condition from crying patterns and other audio signals. Based on these results, the infant's overall condition is inferred.

[1118] The server then uses the inference results to predict potential future states. Based on this prediction, an appropriate response is generated using a generative AI model and provided to the user as a prompt, such as "The baby will be hungry in two hours."

[1119] Home Automation Devices

[1120] The server also connects with home automation devices (e.g., smart home systems) to automate environmental settings such as adjusting the temperature and playing music in the baby's room, ensuring a comfortable and safe environment for the baby.

[1121] Examples and prompts

[1122] For example, the "AI Mimamori" system may detect a baby's crying and send an alert to the user's smartphone saying, "Your baby is crying excessively. Please check on him / her immediately." At the same time, the server will instruct the smart home system to turn on the lights in the baby's room and provide the camera feed to the parents remotely.

[1123] Prompt Sentence Examples

[1124] "What is the best way to respond if my baby is crying?"

[1125] "Infer your baby's emotional state from their facial expressions."

[1126] "How do I connect it to my smart home devices to keep my baby safe?"

[1127] This allows parents to keep track of their baby's condition and respond quickly, while automated home devices can adjust the environment to provide a comfortable and safe environment for the baby.

[1128] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1129] Step 1:

[1130] Data acquisition (device)

[1131] The device (smartphone) activates the camera and microphone to capture the baby's video and audio information in real time. At this time, the camera captures continuous still images, and the microphone captures continuous audio samples.

[1132] Input: Camera images, audio data

[1133] Output: Pre-processed video data, audio data

[1134] Step 2:

[1135] Data preprocessing (terminal)

[1136] The device resizes the captured video data and converts the audio data into an appropriate format for preprocessing, which removes noise for data analysis. Specifically, the image size is resized to 224x224 pixels, and the audio data is extracted to a certain time range.

[1137] Input: Camera images, audio data

[1138] Output: Pre-processed video data, audio data

[1139] Step 3:

[1140] Data transmission (terminal)

[1141] The device transmits the preprocessed video and audio data to a server via the Internet, using a secure communication protocol such as HTTPS.

[1142] Input: Preprocessed video data, audio data

[1143] Output: Send data to the server

[1144] Step 4:

[1145] Video data analysis (server)

[1146] The video data received by the server is analyzed using a convolutional neural network (CNN). Specifically, the video data is input into a CNN model to extract features such as the baby's facial expressions and movements. This analysis allows the system to predict the baby's state, such as whether it is crying or about to fall asleep.

[1147] Input: Preprocessed video data

[1148] Output: Image features, inferred state

[1149] Step 5:

[1150] Voice data analysis (server)

[1151] The server analyzes the received voice data using a voice recognition model, which inputs the voice data into the model and uses crying patterns and other audio signals to understand the baby's condition.

[1152] Input: Preprocessed audio data

[1153] Output: Audio features, inferred state

[1154] Step 6:

[1155] State estimation integration (server)

[1156] The server combines the results of the video and audio analysis to comprehensively estimate the baby's condition. Specifically, it uses a machine learning model to combine the features of both data to determine the final condition.

[1157] Input: Image features, audio features

[1158] Output: Overall guess state

[1159] Step 7:

[1160] Future state prediction (server)

[1161] The server predicts potential future states based on the current state, specifically by comparing it with past data logs to predict when the next person will be hungry or when they will fall asleep.

[1162] Input: Overall guess state, historical data log

[1163] Output: Future potential states

[1164] Step 8:

[1165] Generate countermeasures (server)

[1166] The server generates appropriate childcare responses based on the predicted future state, which are created using a generative AI model and provide a concrete action plan for the caregiver.

[1167] Input: Future potential state

[1168] Output: Generated workarounds

[1169] Step 9:

[1170] Prompt statement generation (server)

[1171] The server creates a prompt sentence to support the caregiver's actions based on the generated countermeasure. Specifically, it creates a sentence that succinctly expresses the generated countermeasure and notifies the caregiver.

[1172] Input: Generated countermeasures

[1173] Output: prompt statement

[1174] Step 10:

[1175] Notification and control of home automation devices (server)

[1176] The server then sends the generated prompts to the caregiver's device and simultaneously controls home automation devices, such as sending notifications to a smartphone and issuing commands to the smart home system to adjust the room temperature or play music.

[1177] Input: prompt statement, home automation device control instructions

[1178] Output: Notification to caregivers, activation of home automation devices

[1179] The above is the specific flow of the system program processing.

[1180] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1181] This invention combines an emotion engine with the "AI Mimamori" system, which aims to reduce the burden on parents raising children and make child-rearing more effective and efficient amid a declining birthrate. This system analyzes video and audio information obtained from mobile communication devices (e.g., smartphones), predicts the state of a child, and provides parents with countermeasures based on that information. It also recognizes the user's emotions and provides more appropriate countermeasures based on those.

[1182] System Overview

[1183] The device uses a camera and microphone to capture video and audio information of the baby, as well as that of the user (parent) in real time. For example, the camera and microphone of a smartphone can be used to capture the facial expressions and voices of the baby and parent. This captured data is then sent to a server via the Internet.

[1184] The server analyzes this data using AI models. Deep learning models such as convolutional neural networks (CNNs) are used to analyze the baby's visual data, and speech recognition models are used to analyze the auditory data. Similarly, an emotion recognition engine is used to analyze the parent's emotions. The results of each analysis (e.g., the baby is hungry, the parent is stressed, etc.) are integrated by the server to infer the overall state of the baby and parent.

[1185] The estimated state of the infant and parent is compared and analyzed with previously collected data logs to predict potential future states. Based on this prediction, appropriate countermeasures (such as notifications of the next feeding time or suggestions to reduce parental stress) are notified to the device. Furthermore, by linking with smart home devices, the system can automatically adjust the room temperature, play music, etc.

[1186] Program processing

[1187] The device activates the camera and microphone to capture video and audio information of the baby and parent, and captures data in real time. This data is pre-processed (e.g., image resizing and audio data format conversion) and sent to a server via the Internet.

[1188] The server first analyzes the baby's video information using a convolutional neural network (CNN). It receives image data as input, extracts features, and infers the baby's state, such as hunger, sleepiness, or desire to play, from its facial expressions and movements. Similarly, the baby's audio information is input into a speech recognition model, which evaluates the baby's state based on crying patterns and other audio signals.

[1189] The server then inputs the parent's video and audio information into an emotion recognition engine, which analyzes the parent's facial expressions and tone of voice to identify emotions such as stress, joy, and fatigue.

[1190] The results of both analyses are integrated on a server and recorded as the overall state of the infant and parent. This information is stored in a data log and compared with past data to predict future states (for example, the baby may become hungry in two hours, which may cause stress for the parent).

[1191] The prediction results and countermeasures are sent to the device via the Internet and notified to the parent. For example, if the baby seems sleepy and the parent is tired, a notification will appear on the smartphone saying, "The baby seems sleepy. The room temperature is being adjusted. We recommend that the parent also take a break."

[1192] In addition, the server can also connect to smart home devices to automatically adjust the room temperature and play relaxing music to maintain a comfortable environment for both baby and parent, ensuring a safe and comfortable environment for both baby and parent.

[1193] Specific examples

[1194] The user activates the "AI Mimamori" system. The device (e.g., a smartphone) then captures video and audio of the baby and parent. The server analyzes this data and infers whether the baby is hungry and whether the parent is feeling stressed. Based on this, the server predicts that the baby will become hungry again in approximately two hours and suggests that the parent take a break to relax. The server then instructs smart home devices to set the room temperature to 22 degrees and play relaxing music. The smartphone also receives a notification saying, "The baby is likely to become hungry in two hours. Please relax, parent." This allows parents to prepare the next meal at the appropriate time while also maintaining their own health.

[1195] This series of actions reduces the burden of childcare on parents and provides a comfortable environment for both the baby and the parent. The "AI Mimamori" system of this invention provides a new means of supporting childcare and is expected to contribute to measures to combat the declining birthrate.

[1196] The processing flow will be explained below.

[1197] Step 1:

[1198] The user launches the "AI Mimamori" system.

[1199] Users start the system by tapping a dedicated application on their smartphone.

[1200] Step 2:

[1201] The device activates the camera and acquires video information.

[1202] The device uses a built-in camera to capture images of the baby in real time and obtain frame data.

[1203] Step 3:

[1204] The device activates the microphone and captures audio information.

[1205] The device uses a built-in microphone to record the baby's crying and environmental sounds in real time, capturing audio data.

[1206] Step 4:

[1207] The terminal uses a camera to acquire video information of the user.

[1208] The device captures the user's facial expressions using a built-in camera and acquires the data.

[1209] Step 5:

[1210] The terminal acquires the user's voice information using a microphone.

[1211] The device uses a built-in microphone to record the user's tone of voice and speaking style.

[1212] Step 6:

[1213] The video and audio information acquired by the terminal is preprocessed.

[1214] The video information is resized, noise removed, and otherwise processed to convert it into a format suitable for analysis.

[1215] The audio information is processed by adjusting the sampling rate and removing noise, and then converted into a format suitable for analysis.

[1216] Step 7:

[1217] The terminal transmits the preprocessed data to a server via the Internet.

[1218] The device converts the acquired video and audio data into an appropriate format and sends it to the server using encrypted communication such as SSL.

[1219] Step 8:

[1220] The server uses an AI model to analyze the video information.

[1221] The server inputs the preprocessed video data into a convolutional neural network (CNN), which infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements.

[1222] Step 9:

[1223] The server uses an AI model to analyze the audio information.

[1224] The server inputs the preprocessed audio data into a speech recognition model to assess the baby's condition from crying patterns and other audio signals.

[1225] Step 10:

[1226] The server uses an emotion recognition engine to analyze the user's video and audio information.

[1227] The server analyzes the preprocessed video data of the user and infers emotions such as stress, joy, and fatigue from facial expressions.

[1228] Similarly, the system analyzes the user's voice data and assesses emotions based on tone of voice and speaking style.

[1229] Step 11:

[1230] The server integrates the results of video analysis, audio analysis, and emotion analysis.

[1231] The server compiles the results of the analysis of visual, auditory and emotional information into unified data and estimates the overall state of the infant and parent.

[1232] Step 12:

[1233] The server logs the inferred infant and parent states.

[1234] The server stores the guesses and their timestamps in a data log.

[1235] Step 13:

[1236] The server predicts future conditions based on the accumulated data logs.

[1237] The server analyzes past data logs and uses learning algorithms to predict future conditions of the infant and parent (e.g., the next feeding time or the parent's stress level).

[1238] Step 14:

[1239] The server notifies the parent to provide appropriate action.

[1240] Based on the results of the estimation and prediction, the server notifies the parent's smartphone with advice on appropriate measures (e.g., preparing for breastfeeding, suggestions for reducing parental stress).

[1241] Step 15:

[1242] The device receives the notification and displays it to the user.

[1243] The device receives notifications from the server and displays them to the user as push notifications or alerts. For example, "Your baby will be hungry in 2 hours. Please relax."

[1244] Step 16:

[1245] The server controls the smart home devices.

[1246] The server controls smart home devices (e.g., smart thermostats and speakers) to adjust the room temperature and play relaxing music to maintain a comfortable environment for the baby and parents.

[1247] Step 17:

[1248] The user acts on the notification.

[1249] Users can take necessary measures based on notifications from their smartphones, such as preparing to breastfeed, preparing the environment for putting the baby to sleep, or taking a break to relieve their own stress.

[1250] Example 2

[1251] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1252] As the birthrate continues to decline, there is a need to reduce the burden on parents raising children and provide more effective and efficient methods for raising them. In particular, there is a need for a system that can accurately grasp the baby's condition and quickly provide appropriate countermeasures. There is also a need for a function that takes into account the parents' own emotional state and provides comprehensive support. However, existing systems are limited to estimating the baby's condition, and there are no systems that perform integrated analysis including the parent's emotional state or automatically adjust environmental conditions. Therefore, the challenge is to provide a system that provides comprehensive support for both, thereby significantly reducing the burden on parents raising children and providing a more comfortable childcare environment.

[1253] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1254] In this invention, the server includes means for analyzing video information acquired from a mobile communication device, means for analyzing audio information acquired from the mobile communication device, means for integrating the video information and the audio information and using a learning algorithm to predict the infant's condition, means for integrating and analyzing the estimated infant's condition and the guardian's emotional state to evaluate the overall condition, means for predicting a future potential condition based on the integrated analysis, means for providing the guardian with countermeasures for the predicted future condition, and means for automatically adjusting environmental conditions based on the countermeasures and cooperating with a device to provide a comfortable environment for the guardian and infant. This enables a comprehensive analysis of the baby's condition and the parent's emotional state, and makes it possible to reduce the burden of child-rearing on parents by suggesting appropriate countermeasures and automatically adjusting the environment.

[1255] "Mobile communications device" refers to a mobile device such as a mobile phone or smartphone, which has the ability to send and receive data using communications technology.

[1256] "Video information" is data containing visual content captured by a camera, video device, or the like.

[1257] "Audio information" is information that includes sound data collected by an audio capture device such as a microphone.

[1258] A "learning algorithm" is a set of computational methods and procedures for analyzing data and finding patterns and regularities within it, and plays an important role in machine learning.

[1259] "Infant state" refers to the physical and emotional state of a young child, such as a baby, including hunger, sleepiness, pleasure and pain, etc.

[1260] "Parental emotional state" refers to the psychological emotional state of the parent or guardian, including stress, joy, fatigue, etc.

[1261] "Integrated analysis" is an analytical method that combines data obtained from multiple different sources to make a single comprehensive evaluation.

[1262] "Future potential state" refers to the likely future state of the child and caregiver, predicted based on current data.

[1263] A "countermeasure" is a proposed action or method based on an estimated state or prediction, with the aim of solving a problem or improving a situation.

[1264] "Automatic adjustment of environmental conditions" is an operation that automatically adjusts environmental elements such as room temperature, lighting, and music to provide a comfortable living environment for infants and their guardians.

[1265] "Working with devices" refers to a system exchanging information and instructions with other devices or equipment to perform functions in collaboration.

[1266] MODE FOR CARRYING OUT THE INVENTION

[1267] The present invention relates to the "AI Mimamori" system, which aims to reduce the burden on parents raising children and help them do so more effectively and efficiently. This system analyzes video and audio information obtained from mobile communication devices (such as smartphones), predicts the state of the child, and provides parents with countermeasures based on that information. It is also possible to recognize the emotions of the user (parent) and provide more appropriate countermeasures.

[1268] Hardware and software used

[1269] Devices: Smartphone, camera, microphone

[1270] Server: High-performance computer

[1271] software:

[1272] Video analysis: Convolutional neural networks (CNN)

[1273] Speech analysis: Speech recognition model

[1274] Sentiment Analysis: Emotion Recognition Engine

[1275] Data transmission: Internet Protocol

[1276] System Operation

[1277] The user activates the "AI Mimamori" system using a smartphone. The device uses a camera and microphone to capture video and audio information of the baby and parent in real time. For example, the smartphone's camera captures the baby's face, and the microphone records its crying. At the same time, the parent's facial expressions and voice are also captured. This captured data is preprocessed and then sent to a server via the Internet.

[1278] The server first analyzes the baby's video information using a convolutional neural network (CNN). It receives image data as input, extracts features, and infers the baby's state, such as hunger, sleepiness, or desire to play, from its facial expressions and movements. Similarly, the baby's audio information is input into a speech recognition model, which evaluates the baby's state based on crying patterns and other audio signals.

[1279] The server then inputs the parent's video and audio information into an emotion recognition engine, which analyzes the parent's facial expressions and tone of voice to identify emotions such as stress, joy, and fatigue.

[1280] The results of both analyses are integrated on a server and recorded as the overall state of the infant and parent. This information is stored in a data log and compared with past data to predict future states (for example, the baby may become hungry in two hours, which may cause stress for the parent).

[1281] The prediction results and countermeasures are sent to the device via the Internet and notified to the parent. For example, if the baby seems sleepy and the parent is tired, a notification will appear on the smartphone saying, "The baby seems sleepy. The room temperature is being adjusted. We recommend that the parent also take a break."

[1282] In addition, the server can also connect to smart home devices to automatically adjust the room temperature and play relaxing music to maintain a comfortable environment for both baby and parent, ensuring a safe and comfortable environment for both baby and parent.

[1283] Specific examples

[1284] The user activates the "AI Mimamori" system. The device (e.g., a smartphone) then captures video and audio of the baby and parent. The server analyzes this data and infers whether the baby is hungry and whether the parent is feeling stressed. Based on this, the server predicts that the baby will become hungry again in approximately two hours and suggests that the parent take a break to relax. The server then instructs smart home devices to set the room temperature to 22 degrees and play relaxing music. The smartphone also receives a notification saying, "The baby is likely to become hungry in two hours. Please relax, parent." This allows parents to prepare the next meal at the appropriate time while also maintaining their own health.

[1285] Prompt Sentence Examples

[1286] Contracting System:

[1287] 1. Use a camera and microphone to capture video and audio of the baby and parent.

[1288] 2. Video and audio data is sent to a server via the Internet.

[1289] 3. The server analyzes the data, proposes appropriate countermeasures, and notifies the smartphone.

[1290] Situation:

[1291] Imagine when a baby is crying and the parents are tired, and generate the best response.

[1292] Expected output:

[1293] "It's likely your baby is hungry and may be hungry again in about two hours. I set the room temperature to 72 degrees and played relaxing music. I encourage parents to take breaks as well."

[1294] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1295] Step 1:

[1296] The device uses a camera and microphone to capture real-time video and audio information of the baby and parent. The camera captures the baby's face, and the microphone records the baby's crying. At the same time, the device also captures the parent's facial expressions and voice. This data is temporarily stored on the device.

[1297] Input: Video and audio information from the camera and microphone

[1298] Output: A dataset of captured video and audio information

[1299] Step 2:

[1300] The device preprocesses the captured video and audio data, specifically resizing the video data and converting the audio data into a digital format. This preprocessing ensures uniform data quality for efficient analysis.

[1301] Input: A dataset of captured video and audio information.

[1302] Output: Pre-processed video and audio information

[1303] Step 3:

[1304] The device then transmits the preprocessed data to a server over the internet, using encryption protocols to prevent data leakage or tampering.

[1305] Input: Preprocessed video and audio information

[1306] Output: Data sent to the server

[1307] Step 4:

[1308] The server inputs the transmitted video data of the baby into a convolutional neural network (CNN). The CNN extracts features from the image data and infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements. For example, it analyzes video of a baby with its mouth open and determines that the baby is hungry.

[1309] Input: Baby video data sent to the server

[1310] Output: Analysis results showing the baby's condition

[1311] Step 5:

[1312] The server inputs the baby's voice data into a speech recognition model, which evaluates the baby's state from crying patterns and other audio signals. For example, if the crying pattern matches "sleepy," that information is output.

[1313] Input: Baby's voice data sent to the server

[1314] Output: Voice analysis results showing the baby's condition

[1315] Step 6:

[1316] The server inputs the parent's video and audio information into an emotion recognition engine. The emotion recognition engine analyzes the parent's facial expressions and tone of voice to determine emotions such as stress, joy, and fatigue. For example, it analyzes a video of the parent frowning and determines that the parent is feeling "stressed."

[1317] Input: Parent's video and audio information sent to the server

[1318] Output: Analysis results showing the parent's emotional state

[1319] Step 7:

[1320] The server combines the results of the baby's and parent's analysis to assess their overall condition. This information is stored in a data log and compared with past data. This allows for predictions of future conditions (e.g., the baby will be hungry in two hours).

[1321] Input: Analysis results showing the baby's condition, analysis results showing the parent's emotional state

[1322] Output: Comprehensive condition assessment and future condition prediction

[1323] Step 8:

[1324] The server generates appropriate countermeasures based on the prediction results and notifies the device, such as the next feeding time or suggestions for parental relaxation.

[1325] Input: Comprehensive condition assessment and future condition prediction

[1326] Output: Notification of action to be taken

[1327] Step 9:

[1328] The server works in conjunction with smart home devices to automatically adjust the room temperature and play relaxing music to maintain a comfortable environment for both baby and parent.

[1329] Input: Comprehensive condition assessment and future condition prediction

[1330] Output: Instructions to smart home devices

[1331] (Application example 2)

[1332] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1333] In order to reduce the burden on parents raising children and to raise them more effectively and efficiently amid a declining birthrate, it is necessary to consider not only the condition of the infant but also the emotional state of the parent. However, conventional systems lack the functionality to analyze the parent's emotional state, making it difficult to provide appropriate countermeasures. In addition, there is a lack of integration with smart home devices to maintain a comfortable environment for infants and parents, which means the burden on parents is not reduced.

[1334] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video information acquired from a mobile communication device, means for analyzing audio information acquired from the mobile communication device, means for integrating the video information and the audio information and using a learning algorithm to estimate the infant's state, means for predicting a future potential state based on the estimated infant's state, means for analyzing the parent's emotions, means for providing countermeasures based on the estimated infant's state and the parent's emotions, means for adjusting the environment in cooperation with a smart home device, and means for notifying the parent of the countermeasures. This provides appropriate childcare support that also takes into account the parent's emotional state, making it possible to reduce the burden of childcare and provide a comfortable environment for both the baby and the parent.

[1335] A "mobile communication device" is a portable information communication terminal equipped with a camera and a microphone, and has the function of acquiring video information and audio information and transmitting and receiving data via a communication network.

[1336] "Video information" refers to visual data captured by a camera on a mobile communication device and presented in the form of images or video.

[1337] "Audio information" refers to auditory data captured by a microphone on a mobile communication device and expressed in voice or acoustic form.

[1338] A "learning algorithm" refers to a computational method or model that learns patterns and relationships based on previously collected data and makes inferences and predictions about new data.

[1339] "Infant condition" refers to the baby's physical or psychological state, such as hunger, sleepiness, or a desire to play.

[1340] "Future potential states" refer to the infant's future state predicted based on current data and analysis results, such as hunger or sleepiness after a certain time.

[1341] "Parental emotions" refers to the emotional state that the parent is currently experiencing, such as specific emotions such as stress, joy, or fatigue.

[1342] "Countermeasures" refer to specific actions or coping methods suggested based on the estimated infant's condition and the parent's emotions, such as notification of the next feeding time or advice on stress reduction.

[1343] "Smart home devices" refer to home electronic devices that are connected to a network and have remote control or automation capabilities, such as smart thermostats and smart speakers.

[1344] "Environmental adjustment" refers to operations such as changing temperature and sound settings to maintain a comfortable indoor environment.

[1345] "Notification" refers to a function of informing a user of information via a mobile communication device, and includes notification by screen display or voice.

[1346] The "childcare support system" of the present invention is a system that uses a mobile communication device (e.g., a smartphone) to reduce the burden on parents raising children and allow them to raise children more effectively and efficiently. This system analyzes not only the condition of the infant but also the emotional state of the parent, and provides the user with appropriate countermeasures. The following describes in detail an embodiment of the present invention.

[1347] 1. System Configuration

[1348] The system consists of three main components: terminals, servers, and smart home devices.

[1349] Terminal

[1350] The device is a smartphone equipped with a camera and microphone, which captures real-time video and audio information of the baby and parent. The captured data is then sent to a server via the Internet.

[1351] server

[1352] The server has the following analytical methods and capabilities:

[1353] Video information analysis method: Using a convolutional neural network (CNN), the video data obtained from the camera is analyzed to extract the baby's facial expressions and movements.

[1354] Audio information analysis means: Using a speech recognition model, the audio data captured from the microphone is analyzed to evaluate crying and other audio signals.

[1355] Emotion recognition: An emotion recognition engine is used to analyze emotions from the parent's facial expressions and tone of voice.

[1356] The analyzed results are integrated to estimate the status of the infant and parent. Based on this, potential future conditions are predicted and appropriate countermeasures are generated. The appropriate countermeasures are notified to the terminal via the Internet, and instructions for cooperation are sent to smart home devices.

[1357] Smart Home Devices

[1358] Smart home devices automatically adjust the environment, such as networked thermostats and music players, to keep children and parents comfortable by adjusting the temperature or playing relaxing music.

[1359] 2. System Operation

[1360] Data Acquisition and Transmission

[1361] When a user launches the smartphone app, the camera and microphone begin capturing data in real time, which is then sent to a server via the Internet for detailed analysis.

[1362] Analysis and Notification

[1363] The server analyzes video and audio data to infer the baby's condition. For example, it can infer hunger from the baby's facial expression and sleepiness from the crying pattern. It can also analyze the parent's emotions from their facial expression and tone of voice to infer stress or fatigue. Based on these analysis results, it predicts future potential conditions and generates countermeasures.

[1364] environmental adjustment

[1365] By connecting with smart home devices, it can automatically adjust the room temperature and play relaxing music, reducing the burden on parents while providing a comfortable environment for both baby and parent.

[1366] 3. Examples of concrete examples and prompts

[1367] Specific examples

[1368] After the user activates the system, the smartphone's camera and microphone capture video and audio of the baby and parent. The server analyzes the data and infers that the baby is hungry and the parent is stressed. The following notification is displayed on the smartphone as a response generated by the server:

[1369] "Your baby is hungry. The next feeding is in about an hour. Set the room temperature to 72 degrees and play some relaxing music."

[1370] Prompt Sentence Examples

[1371] Analyze your baby's condition:

[1372] Capture the baby's facial expressions and movements with a camera and analyze them using a CNN model.

[1373] Use a voice recognition model to analyze crying patterns and assess states such as hunger and sleepiness.

[1374] Analyzing parental emotions:

[1375] Capture the parent's facial expressions and voice and analyze them with an emotion recognition engine.

[1376] Integrate baby and parent status and generate notifications:

[1377] If the baby is hungry, notify them of appropriate action (e.g., notify them of the next feeding time).

[1378] If a parent is tired, suggest a break and adjust the temperature with a smart home device.

[1379] As described above, the childcare support system of the present invention is capable of providing appropriate childcare support that takes into consideration the emotional state of the parents, thereby reducing the burden on the parents and improving the comfort of both the baby and the parents.

[1380] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1381] Step 1:

[1382] The user launches the smartphone app.

[1383] Input: Smartphone camera and microphone

[1384] Output: Real-time video and audio information

[1385] How it works: The user taps to launch the smartphone app, which automatically turns on the camera and microphone, capturing video and audio of the baby and parent.

[1386] Step 2:

[1387] The video and audio information acquired by the terminal is transmitted to a server via the Internet.

[1388] Input: Real-time video and audio information

[1389] Output: Data sent to the server

[1390] How it works: The device transfers the captured video and audio data to the server via a communications network. The data is encrypted and securely transmitted.

[1391] Step 3:

[1392] The server receives the video data and analyzes it using a convolutional neural network (CNN).

[1393] Input: Video information obtained from the camera

[1394] Output: Baby's facial expressions and behavior patterns

[1395] How it works: The server preprocesses the received video data (e.g., resizes and normalizes it) and then inputs it into a CNN model. The CNN model extracts features and infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements.

[1396] Step 4:

[1397] The server receives the voice data and analyzes it using a voice recognition model.

[1398] Input: Audio information obtained from the microphone

[1399] Output: Baby's crying and voice patterns

[1400] Specific operation: The server preprocesses the received voice data (e.g., noise reduction, normalization) and then inputs it into a voice recognition model. The voice recognition model analyzes the characteristics of the crying voice and evaluates the baby's state (e.g., hunger, sleepiness, demands).

[1401] Step 5:

[1402] The server inputs the parent's video and audio data into an emotion recognition engine for analysis.

[1403] Input: Parent video and audio information obtained from camera and microphone

[1404] Output: Parent's emotional state (e.g., stress, joy, fatigue)

[1405] Specific operation: The server inputs the parent's video data into an emotion recognition engine and analyzes their facial expressions. It also uses audio data to analyze their tone of voice and the content of their speech. This allows it to infer the parent's emotional state.

[1406] Step 6:

[1407] The server integrates the baby and parent condition data it has acquired and predicts potential future conditions.

[1408] Input: Baby's state data, Parent's emotional state data

[1409] Output: Predicted future state (e.g. next feeding time, parental stress level)

[1410] How it works: The server performs statistical analysis and comparison with past data to predict the future state of the baby and parents, for example, calculating the likelihood that the baby will be hungry after a certain time, or the likelihood that the parents' stress levels will increase.

[1411] Step 7:

[1412] The server generates appropriate countermeasures based on the prediction results and notifies the device.

[1413] Input: Predicted future state data

[1414] Output: Notifications sent to smartphones (e.g., notifications about breastfeeding times, suggestions for adjusting the environment)

[1415] Specific operation: Based on the prediction results, the server generates appropriate responses for the baby and the parent. For example, it predicts that the baby will become hungry again in about two hours and suggests that the parent take a break to relax. This information is then sent to the smartphone.

[1416] Step 8:

[1417] The server works in conjunction with smart home devices to adjust the environment.

[1418] Input: Generated countermeasure data, control information of smart home devices

[1419] Output: Adjusted comfort environment (e.g., adjusting room temperature, playing relaxing music)

[1420] How it works: The server sends instructions to smart home devices, which then automatically change the room temperature or play relaxing music, optimizing the childcare environment.

[1421] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1422] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1423] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1424] [Fourth embodiment]

[1425] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1426] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1427] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1428] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1429] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1430] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1431] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1432] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1433] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1434] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1435] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1436] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1437] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1438] The present invention relates to the "AI Mimamori" system, which aims to reduce the burden on parents raising children and make child-rearing more effective and efficient amid the declining birthrate. This system analyzes video and audio information obtained from mobile communication devices (e.g., smartphones), predicts the condition of infants, and provides parents with countermeasures based on that information. The detailed configuration and operation for realizing this system are described below.

[1439] System Overview

[1440] The device uses a camera and microphone to capture real-time video and audio information about the baby. For example, a smartphone's camera and microphone can be used to capture the baby's facial expressions, crying, and surrounding environmental sounds. This captured data is then sent to a server via the Internet.

[1441] The server analyzes this data using AI models. Deep learning models such as convolutional neural networks (CNNs) are used to analyze visual data, and speech recognition models are used to analyze auditory data. The results of each analysis (e.g., whether the baby is hungry, sleepy, or wants to play) are integrated by the server to estimate the baby's overall state.

[1442] The estimated infant's condition is compared and analyzed with previously collected data logs to predict potential future conditions. Based on this prediction, appropriate countermeasures (such as notifying the next feeding time or providing a sleep-friendly environment) are notified to the device. Furthermore, by linking with smart home devices, the system can automatically adjust the room temperature, play music, etc.

[1443] Program processing

[1444] The device activates the camera and microphone to capture the baby's video and audio information, and captures the data in real time. This data is pre-processed (e.g., image resizing and audio data format conversion) and then sent to a server via the Internet.

[1445] The server first analyzes visual information using a convolutional neural network (CNN). It receives image data as input, extracts features, and infers the infant's current emotions and physical needs. Similarly, audio information is fed into a speech recognition model, which analyzes the infant's state from crying patterns and other audio signals.

[1446] The results of both analyses are integrated on a server and recorded as a comprehensive infant status. This information is stored in a data log and compared with past data to predict future behavior (e.g., whether the infant will be hungry in two hours).

[1447] The prediction results and countermeasures are sent to the device via the Internet and notified to the parent. For example, if the baby seems sleepy, a notification such as "The baby seems sleepy. The room temperature is being adjusted appropriately" will be displayed on the smartphone.

[1448] In addition, the server can also connect to smart home devices to automatically adjust the temperature in the baby's room and play music to soothe them to sleep, helping to ensure that the baby is raised in a comfortable and safe environment.

[1449] Specific examples

[1450] The user activates the "AI Mimamori" system. The device (such as a smartphone) then captures video and audio of the baby. The server analyzes this data and infers that the baby is hungry. Based on this, the server predicts that the baby will become hungry again in approximately two hours. The server then issues instructions to smart home devices to set the room temperature to 22 degrees. The smartphone also receives a notification that the baby is likely to become hungry in two hours. This allows the parent to prepare the next meal at the appropriate time.

[1451] This series of actions reduces the burden of childcare on parents and provides a comfortable environment for babies. The "AI Mimamori" system of this invention offers a new means of supporting childcare and is expected to contribute to measures to combat the declining birthrate.

[1452] The processing flow will be explained below.

[1453] Step 1:

[1454] The user launches the "AI Mimamori" system.

[1455] Users start the system by tapping a dedicated application on their smartphone.

[1456] Step 2:

[1457] The device activates the camera and acquires video information.

[1458] The device uses a built-in camera to capture images of the baby in real time and obtain frame data.

[1459] Step 3:

[1460] The device activates the microphone and captures audio information.

[1461] The device uses a built-in microphone to record the baby's crying and environmental sounds in real time, capturing audio data.

[1462] Step 4:

[1463] The video and audio information acquired by the terminal is preprocessed.

[1464] The video information is resized, noise removed, and otherwise processed to convert it into a format suitable for analysis.

[1465] The audio information is processed by adjusting the sampling rate and removing noise, and then converted into a format suitable for analysis.

[1466] Step 5:

[1467] The terminal transmits the preprocessed data to a server via the Internet.

[1468] The device converts the acquired video and audio data into an appropriate format and sends it to the server using encrypted communication such as SSL.

[1469] Step 6:

[1470] The server uses an AI model to analyze the video information.

[1471] The server inputs the preprocessed video data into a convolutional neural network (CNN), which infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements.

[1472] Step 7:

[1473] The server uses an AI model to analyze the audio information.

[1474] The server inputs the preprocessed audio data into a speech recognition model and assesses the baby's condition based on crying patterns and surrounding sounds.

[1475] Step 8:

[1476] The server integrates the video analysis results and audio analysis results.

[1477] The server integrates the results of the analysis of visual and auditory information to estimate the overall state of the child.

[1478] Step 9:

[1479] The server logs the inferred infant state.

[1480] The server stores the guesses and their timestamps in a data log.

[1481] Step 10:

[1482] The server predicts future conditions based on the accumulated data logs.

[1483] The server analyzes past data logs and uses learning algorithms to predict the baby's future condition (for example, the next feeding time or when the baby will feel sleepy).

[1484] Step 11:

[1485] The server notifies the parent to provide appropriate action.

[1486] Based on the results of the estimation and prediction, the server notifies the parent's smartphone with advice on appropriate measures (e.g., preparing for breastfeeding or adjusting the room temperature).

[1487] Step 12:

[1488] The device receives the notification and displays it to the user.

[1489] The device receives notifications from the server and displays them to the user as push notifications or alerts. For example, "Your baby will be hungry in 2 hours."

[1490] Step 13:

[1491] The server controls the smart home devices.

[1492] The server controls smart home devices (e.g., smart thermostats and speakers) to maintain a comfortable environment for the baby, adjusting the room temperature and playing music to soothe the baby to sleep.

[1493] Step 14:

[1494] The user acts on the notification.

[1495] Users can take necessary measures according to the notifications on their smartphones, such as preparing for breastfeeding or preparing the environment for putting the baby to sleep.

[1496] Example 1

[1497] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1498] There is a need for systems that can reduce the burden faced by parents raising children and enable them to raise their children more effectively and efficiently. Especially with the declining birthrate, there is an urgent need to monitor a baby's condition in real time and provide appropriate measures. There is also a need to reduce the burden on parents by integrating smart home devices and automating environmental adjustments.

[1499] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1500] In this invention, the server includes means for analyzing video information acquired from the mobile communication device, means for analyzing audio information acquired from the mobile communication device, means for integrating the video information and the audio information and using a machine learning algorithm to predict the infant's condition, means for predicting the infant's potential future condition based on the predicted infant's condition, means for providing the guardian with countermeasures for the predicted future condition, means for storing the analysis results and countermeasures in a database and performing comparative analysis based on past data, means for integrating the extracted analysis results and countermeasures in the server and notifying the mobile communication device via the Internet, and means for automatically adjusting the physical environment using the mobile communication device and smart home devices. This allows parents raising their baby to understand the baby's condition in real time and obtain appropriate countermeasures. Furthermore, environmental adjustments are automatically performed in cooperation with the smart home devices, effectively reducing the parental burden on parents.

[1501] A "mobile communication device" refers to a device that is mobile and has communication capabilities, and specifically includes smartphones and tablets.

[1502] "Video information" refers to image data or video data acquired using a camera.

[1503] "Audio information" refers to audio data or acoustic data acquired using a microphone.

[1504] "Machine learning algorithm" is a general term for algorithms that learn patterns from data and make predictions and classifications for future data.

[1505] "Infant states" refers to the baby's various emotions and physical needs (e.g., hunger, sleepiness, discomfort, etc.).

[1506] "Inferred infant state" refers to the infant's emotional and needs state inferred by a machine learning algorithm based on the analysis of video and audio information.

[1507] "Future potential state" refers to the infant's future state predicted based on the infant's estimated current state and past data logs.

[1508] "Countermeasures" refer to specific responses and guidelines for action that parents should take based on the estimated condition of the child and their predicted future condition.

[1509] "Guardian" refers to a person who is responsible for caring for a young child, typically a parent or relative.

[1510] A "database" refers to a system that stores data in an organized form and allows it to be searched and updated efficiently.

[1511] "Comparative analysis" refers to an analytical technique that compares different data sets to reveal relationships and patterns between them.

[1512] A "server" refers to a computer system that provides services and data to clients over a network.

[1513] "Internet" refers to a global information and communications network that interconnects computer networks around the world.

[1514] "Smart home devices" refer to appliances and devices in the home that can be controlled via the Internet, including thermostats and smart speakers.

[1515] "Automatically adjusting the physical environment" refers to automatically adjusting environmental settings such as room temperature, lighting, and music through smart home devices.

[1516] The present invention is a system for reducing the burden on parents raising children and for raising them more effectively and efficiently. This system analyzes video and audio information acquired from a mobile communication device (e.g., a smartphone), predicts the baby's condition, and provides parents with countermeasures based on the predictions. The specific configuration and operation of the system are described in detail below.

[1517] Hardware and software configuration

[1518] Terminal

[1519] The terminal is a mobile communication device equipped with a camera and a microphone, and is typically a smartphone. The terminal has the following functions:

[1520] 1. Camera: Used to capture the baby's visual information (facial expressions and movements).

[1521] 2. Microphone: Used to record the baby's crying and surrounding environmental sounds.

[1522] 3. Data preprocessing: Converting the captured video and audio data into a format suitable for analysis.

[1523] 4. Data transmission: The preprocessed data is transmitted to the server via the Internet.

[1524] server

[1525] The server has the following analysis and inference functions:

[1526] 1. Video analysis: Analyze the baby's facial expressions using a convolutional neural network (CNN) and extract features.

[1527] 2. Speech analysis: Analyze the baby's crying patterns using a speech recognition model.

[1528] 3. Data integration: The results of the analysis of visual and audio information are integrated to estimate the overall state of the child.

[1529] 4. Prediction function: Predicts future potential states based on past data logs.

[1530] 5. Notification function: Prediction results and countermeasures are notified to the device.

[1531] Smart Home Devices

[1532] The server connects to smart home devices via the Internet to achieve the following functions:

[1533] 1. Environmental adjustment: Adjust the temperature and lighting in your baby's room appropriately.

[1534] 2. Play music: Play music that helps your baby relax.

[1535] Specific examples

[1536] As a concrete example, the following scenario can be considered.

[1537] The user launches the "childcare support system." The device (e.g., a smartphone) then captures video and audio of the baby. The baby's facial expression and audio data are sent to a server via the Internet. The server first inputs the video data into a CNN model, extracts features, and analyzes the baby's emotions and needs. Next, it inputs the audio data into a speech recognition model, and analyzes the crying patterns to infer the baby's condition.

[1538] For example, if the server determines that the baby is crying because it seems sleepy, it will send a notification to the device saying, "The baby seems sleepy. The room temperature is being adjusted appropriately." At the same time, the server will send instructions to connected smart home devices to set the room temperature to 22 degrees and play relaxing music. This allows the baby to sleep in a comfortable environment, allowing parents to respond at the appropriate time.

[1539] Prompt Sentence Examples

[1540] Below are some example prompts that can be used as input to a generative AI model.

[1541] "My baby is crying. Please help me guess what his current emotions and physical needs are and suggest appropriate responses."

[1542] Using this prompt, the AI ​​model will perform specific analysis and make suggestions. In this way, the system of the present invention aims to support childcare and reduce the burden on parents.

[1543] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1544] Step 1:

[1545] The device activates the camera and microphone to capture the baby's video and audio information in real time. Specifically, the smartphone camera captures the baby's face, and the microphone records the baby's crying and surrounding environmental sounds. The input is the baby's actual video and audio, and the output is the raw data captured by the camera and microphone.

[1546] Step 2:

[1547] The device preprocesses the captured video and audio data. For example, video data is resized and audio data is converted to a specific format. The input to this process is the raw data captured in step 1, and the output is preprocessed data suitable for analysis.

[1548] Step 3:

[1549] Send the preprocessed data to a server via the internet. The input is the preprocessed data generated in step 2, and the output is the data sent to the server.

[1550] Step 4:

[1551] A convolutional neural network (CNN) is used to analyze the video data received by the server. Specifically, the server inputs the video data into a CNN model, identifies the baby's facial expressions, and extracts features. The input for this analysis is the video data received in step 3, and the output is the baby's facial expressions and their features.

[1552] Step 5:

[1553] The server uses a speech recognition model to analyze the received voice data. Specifically, the server inputs the voice data into the model, analyzes the crying pattern, and extracts its features. The input for this analysis is the voice data received in step 3, and the output is the crying pattern and its features.

[1554] Step 6:

[1555] The server integrates the results of the video and audio analysis to infer the baby's overall state. The server uses the video and audio features as examples to input into a statistical model to infer the baby's emotions and needs. The input for this inference is the features extracted in steps 4 and 5, and the output is the baby's current state (e.g., hunger, sleepiness, discomfort, etc.).

[1556] Step 7:

[1557] Based on the estimated state of the infant, the server refers to past data logs to predict the future state. The server uses statistical methods and machine learning models to predict the future state (e.g., the infant will be hungry in two hours). The inputs to this prediction are the current state estimated in step 6 and past data logs, and the output is the predicted future state.

[1558] Step 8:

[1559] The server notifies the device of the prediction results and countermeasures via the Internet. The server generates a specific countermeasure (e.g., notification of the next breastfeeding time) and sends a notification to the smartphone. The input of this notification is the prediction result generated in step 7, and the output is the notification content sent to the smartphone.

[1560] Step 9:

[1561] The server interacts with the smart home devices and issues necessary instructions, such as adjusting the room temperature or playing relaxing music. Based on the prediction results, the server instructs the smart home devices to set appropriate settings. The input of this instruction is the result predicted in step 7, and the output is an operation instruction to the smart home devices.

[1562] (Application example 1)

[1563] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1564] It is necessary to reduce the burden on parents raising children and to make child-rearing more effective and efficient, but current systems lack the data analysis necessary to accurately predict the condition of young children, the prediction of potential future conditions, and the provision of appropriate countermeasures. Furthermore, the lack of an environment in which information can be shared and parenting can be coordinated not only with parents but also with other stakeholders and family members is slowing the reduction of the burden of child-rearing. Furthermore, there is currently a lack of coordination when it comes to automatically adjusting the environment using home automation devices.

[1565] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1566] In this invention, the server includes: a means for analyzing video information of the baby; a means for analyzing audio information; a means for integrating the video information and audio information and using a learning algorithm to estimate the infant's condition; a means for predicting a potential future condition based on the estimated infant's condition; a means for providing countermeasures for the predicted future condition; a means for sending an alert to a caregiver based on the estimated infant's condition; a means for controlling home automation devices based on the alert; a means for generating the content of the countermeasures using a generative AI; and a means for creating prompts to support the caregiver's actions based on the generated countermeasures. This reduces the burden on parents raising children and enables appropriate responses through efficient collaboration within the household and among related parties. Furthermore, by linking with home automation devices, it is possible to automatically optimize the childcare environment.

[1567] A "mobile communication device" is a device with communication capabilities that can be used by a user while on the move, such as a mobile phone or smartphone.

[1568] "Video information" refers to video and still image data captured using a camera.

[1569] "Audio information" refers to sound data acquired using a microphone.

[1570] A "learning algorithm" is a method for analyzing data to find patterns and relationships and make predictions or classifications.

[1571] "Infant states" are situations that indicate the emotional and physical needs of babies and young children.

[1572] "Future potential states" are future states of infants predicted based on current and past data.

[1573] "Countermeasures" refer to specific guidelines and measures provided depending on the child's condition.

[1574] An "alert" is a warning message intended to notify you of urgent or important information.

[1575] A "caregiver" is a parent or other guardian who cares for a baby or young child.

[1576] "Home automation devices" refer to systems and equipment that automatically control various devices in the home.

[1577] "Generative AI" refers to technology that uses artificial intelligence to generate new information and data.

[1578] A "prompt" is a sentence that guides the user in instructing them on what to do.

[1579] This invention is a home support system based on the "AI Mimamori" system that monitors the condition of infants in real time and provides appropriate countermeasures. This system uses a mobile communication device (e.g., a smartphone) to acquire and analyze video and audio information of the baby, inferring the infant's condition and providing the user with countermeasures based on that condition.

[1580] Hardware and Software Configuration

[1581] Device (e.g. smartphone)

[1582] The device uses a camera and microphone to capture real-time video and audio information of the baby, which is then sent to a server via the Internet.

[1583] server

[1584] The server analyzes the received video information using deep learning models such as convolutional neural networks (CNNs) to infer the infant's condition from their facial expressions and movements. Similarly, it analyzes audio information using speech recognition models to understand the infant's condition from crying patterns and other audio signals. Based on these results, the infant's overall condition is inferred.

[1585] The server then uses the inference results to predict potential future states. Based on this prediction, an appropriate response is generated using a generative AI model and provided to the user as a prompt, such as "The baby will be hungry in two hours."

[1586] Home Automation Devices

[1587] The server also connects with home automation devices (e.g., smart home systems) to automate environmental settings such as adjusting the temperature and playing music in the baby's room, ensuring a comfortable and safe environment for the baby.

[1588] Examples and prompts

[1589] For example, the "AI Mimamori" system may detect a baby's crying and send an alert to the user's smartphone saying, "Your baby is crying excessively. Please check on him / her immediately." At the same time, the server will instruct the smart home system to turn on the lights in the baby's room and provide the camera feed to the parents remotely.

[1590] Prompt Sentence Examples

[1591] "What is the best way to respond if my baby is crying?"

[1592] "Infer your baby's emotional state from their facial expressions."

[1593] "How do I connect it to my smart home devices to keep my baby safe?"

[1594] This allows parents to keep track of their baby's condition and respond quickly, while automated home devices can adjust the environment to provide a comfortable and safe environment for the baby.

[1595] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1596] Step 1:

[1597] Data acquisition (device)

[1598] The device (smartphone) activates the camera and microphone to capture the baby's video and audio information in real time. At this time, the camera captures continuous still images, and the microphone captures continuous audio samples.

[1599] Input: Camera images, audio data

[1600] Output: Pre-processed video data, audio data

[1601] Step 2:

[1602] Data preprocessing (terminal)

[1603] The device resizes the captured video data and converts the audio data into an appropriate format for preprocessing, which removes noise for data analysis. Specifically, the image size is resized to 224x224 pixels, and the audio data is extracted to a certain time range.

[1604] Input: Camera images, audio data

[1605] Output: Pre-processed video data, audio data

[1606] Step 3:

[1607] Data transmission (terminal)

[1608] The device transmits the preprocessed video and audio data to a server via the Internet, using a secure communication protocol such as HTTPS.

[1609] Input: Preprocessed video data, audio data

[1610] Output: Send data to the server

[1611] Step 4:

[1612] Video data analysis (server)

[1613] The video data received by the server is analyzed using a convolutional neural network (CNN). Specifically, the video data is input into a CNN model to extract features such as the baby's facial expressions and movements. This analysis allows the system to predict the baby's state, such as whether it is crying or about to fall asleep.

[1614] Input: Preprocessed video data

[1615] Output: Image features, inferred state

[1616] Step 5:

[1617] Voice data analysis (server)

[1618] The server analyzes the received voice data using a voice recognition model, which inputs the voice data into the model and uses crying patterns and other audio signals to understand the baby's condition.

[1619] Input: Preprocessed audio data

[1620] Output: Audio features, inferred state

[1621] Step 6:

[1622] State estimation integration (server)

[1623] The server combines the results of the video and audio analysis to comprehensively estimate the baby's condition. Specifically, it uses a machine learning model to combine the features of both data to determine the final condition.

[1624] Input: Image features, audio features

[1625] Output: Overall guess state

[1626] Step 7:

[1627] Future state prediction (server)

[1628] The server predicts potential future states based on the current state, specifically by comparing it with past data logs to predict when the next person will be hungry or when they will fall asleep.

[1629] Input: Overall guess state, historical data log

[1630] Output: Future potential states

[1631] Step 8:

[1632] Generate countermeasures (server)

[1633] The server generates appropriate childcare responses based on the predicted future state, which are created using a generative AI model and provide a concrete action plan for the caregiver.

[1634] Input: Future potential state

[1635] Output: Generated workarounds

[1636] Step 9:

[1637] Prompt statement generation (server)

[1638] The server creates a prompt sentence to support the caregiver's actions based on the generated countermeasure. Specifically, it creates a sentence that succinctly expresses the generated countermeasure and notifies the caregiver.

[1639] Input: Generated countermeasures

[1640] Output: prompt statement

[1641] Step 10:

[1642] Notification and control of home automation devices (server)

[1643] The server then sends the generated prompts to the caregiver's device and simultaneously controls home automation devices, such as sending notifications to a smartphone and issuing commands to the smart home system to adjust the room temperature or play music.

[1644] Input: prompt statement, home automation device control instructions

[1645] Output: Notification to caregivers, activation of home automation devices

[1646] The above is the specific flow of the system program processing.

[1647] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1648] This invention combines an emotion engine with the "AI Mimamori" system, which aims to reduce the burden on parents raising children and make child-rearing more effective and efficient amid a declining birthrate. This system analyzes video and audio information obtained from mobile communication devices (e.g., smartphones), predicts the state of a child, and provides parents with countermeasures based on that information. It also recognizes the user's emotions and provides more appropriate countermeasures based on those.

[1649] System Overview

[1650] The device uses a camera and microphone to capture video and audio information of the baby, as well as that of the user (parent) in real time. For example, the camera and microphone of a smartphone can be used to capture the facial expressions and voices of the baby and parent. This captured data is then sent to a server via the Internet.

[1651] The server analyzes this data using AI models. Deep learning models such as convolutional neural networks (CNNs) are used to analyze the baby's visual data, and speech recognition models are used to analyze the auditory data. Similarly, an emotion recognition engine is used to analyze the parent's emotions. The results of each analysis (e.g., the baby is hungry, the parent is stressed, etc.) are integrated by the server to infer the overall state of the baby and parent.

[1652] The estimated state of the infant and parent is compared and analyzed with previously collected data logs to predict potential future states. Based on this prediction, appropriate countermeasures (such as notifications of the next feeding time or suggestions to reduce parental stress) are notified to the device. Furthermore, by linking with smart home devices, the system can automatically adjust the room temperature, play music, etc.

[1653] Program processing

[1654] The device activates the camera and microphone to capture video and audio information of the baby and parent, and captures data in real time. This data is pre-processed (e.g., image resizing and audio data format conversion) and sent to a server via the Internet.

[1655] The server first analyzes the baby's video information using a convolutional neural network (CNN). It receives image data as input, extracts features, and infers the baby's state, such as hunger, sleepiness, or desire to play, from its facial expressions and movements. Similarly, the baby's audio information is input into a speech recognition model, which evaluates the baby's state based on crying patterns and other audio signals.

[1656] The server then inputs the parent's video and audio information into an emotion recognition engine, which analyzes the parent's facial expressions and tone of voice to identify emotions such as stress, joy, and fatigue.

[1657] The results of both analyses are integrated on a server and recorded as the overall state of the infant and parent. This information is stored in a data log and compared with past data to predict future states (for example, the baby may become hungry in two hours, which may cause stress for the parent).

[1658] The prediction results and countermeasures are sent to the device via the Internet and notified to the parent. For example, if the baby seems sleepy and the parent is tired, a notification will appear on the smartphone saying, "The baby seems sleepy. The room temperature is being adjusted. We recommend that the parent also take a break."

[1659] In addition, the server can also connect to smart home devices to automatically adjust the room temperature and play relaxing music to maintain a comfortable environment for both baby and parent, ensuring a safe and comfortable environment for both baby and parent.

[1660] Specific examples

[1661] The user activates the "AI Mimamori" system. The device (e.g., a smartphone) then captures video and audio of the baby and parent. The server analyzes this data and infers whether the baby is hungry and whether the parent is feeling stressed. Based on this, the server predicts that the baby will become hungry again in approximately two hours and suggests that the parent take a break to relax. The server then instructs smart home devices to set the room temperature to 22 degrees and play relaxing music. The smartphone also receives a notification saying, "The baby is likely to become hungry in two hours. Please relax, parent." This allows parents to prepare the next meal at the appropriate time while also maintaining their own health.

[1662] This series of actions reduces the burden of childcare on parents and provides a comfortable environment for both the baby and the parent. The "AI Mimamori" system of this invention provides a new means of supporting childcare and is expected to contribute to measures to combat the declining birthrate.

[1663] The processing flow will be explained below.

[1664] Step 1:

[1665] The user launches the "AI Mimamori" system.

[1666] Users start the system by tapping a dedicated application on their smartphone.

[1667] Step 2:

[1668] The device activates the camera and acquires video information.

[1669] The device uses a built-in camera to capture images of the baby in real time and obtain frame data.

[1670] Step 3:

[1671] The device activates the microphone and captures audio information.

[1672] The device uses a built-in microphone to record the baby's crying and environmental sounds in real time, capturing audio data.

[1673] Step 4:

[1674] The terminal uses a camera to acquire video information of the user.

[1675] The device captures the user's facial expressions using a built-in camera and acquires the data.

[1676] Step 5:

[1677] The terminal acquires the user's voice information using a microphone.

[1678] The device uses a built-in microphone to record the user's tone of voice and speaking style.

[1679] Step 6:

[1680] The video and audio information acquired by the terminal is preprocessed.

[1681] The video information is resized, noise removed, and otherwise processed to convert it into a format suitable for analysis.

[1682] The audio information is processed by adjusting the sampling rate and removing noise, and then converted into a format suitable for analysis.

[1683] Step 7:

[1684] The terminal transmits the preprocessed data to a server via the Internet.

[1685] The device converts the acquired video and audio data into an appropriate format and sends it to the server using encrypted communication such as SSL.

[1686] Step 8:

[1687] The server uses an AI model to analyze the video information.

[1688] The server inputs the preprocessed video data into a convolutional neural network (CNN), which infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements.

[1689] Step 9:

[1690] The server uses an AI model to analyze the audio information.

[1691] The server inputs the preprocessed audio data into a speech recognition model to assess the baby's condition from crying patterns and other audio signals.

[1692] Step 10:

[1693] The server uses an emotion recognition engine to analyze the user's video and audio information.

[1694] The server analyzes the preprocessed video data of the user and infers emotions such as stress, joy, and fatigue from facial expressions.

[1695] Similarly, the system analyzes the user's voice data and assesses emotions based on tone of voice and speaking style.

[1696] Step 11:

[1697] The server integrates the results of video analysis, audio analysis, and emotion analysis.

[1698] The server compiles the results of the analysis of visual, auditory and emotional information into unified data and estimates the overall state of the infant and parent.

[1699] Step 12:

[1700] The server logs the inferred infant and parent states.

[1701] The server stores the guesses and their timestamps in a data log.

[1702] Step 13:

[1703] The server predicts future conditions based on the accumulated data logs.

[1704] The server analyzes past data logs and uses learning algorithms to predict future conditions of the infant and parent (e.g., the next feeding time or the parent's stress level).

[1705] Step 14:

[1706] The server notifies the parent to provide appropriate action.

[1707] Based on the results of the estimation and prediction, the server notifies the parent's smartphone with advice on appropriate measures (e.g., preparing for breastfeeding, suggestions for reducing parental stress).

[1708] Step 15:

[1709] The device receives the notification and displays it to the user.

[1710] The device receives notifications from the server and displays them to the user as push notifications or alerts. For example, "Your baby will be hungry in 2 hours. Please relax."

[1711] Step 16:

[1712] The server controls the smart home devices.

[1713] The server controls smart home devices (e.g., smart thermostats and speakers) to adjust the room temperature and play relaxing music to maintain a comfortable environment for the baby and parents.

[1714] Step 17:

[1715] The user acts on the notification.

[1716] Users can take necessary measures based on notifications from their smartphones, such as preparing to breastfeed, preparing the environment for putting the baby to sleep, or taking a break to relieve their own stress.

[1717] Example 2

[1718] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1719] As the birthrate continues to decline, there is a need to reduce the burden on parents raising children and provide more effective and efficient methods for raising them. In particular, there is a need for a system that can accurately grasp the baby's condition and quickly provide appropriate countermeasures. There is also a need for a function that takes into account the parents' own emotional state and provides comprehensive support. However, existing systems are limited to estimating the baby's condition, and there are no systems that perform integrated analysis including the parent's emotional state or automatically adjust environmental conditions. Therefore, the challenge is to provide a system that provides comprehensive support for both, thereby significantly reducing the burden on parents raising children and providing a more comfortable childcare environment.

[1720] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1721] In this invention, the server includes means for analyzing video information acquired from a mobile communication device, means for analyzing audio information acquired from the mobile communication device, means for integrating the video information and the audio information and using a learning algorithm to predict the infant's condition, means for integrating and analyzing the estimated infant's condition and the guardian's emotional state to evaluate the overall condition, means for predicting a future potential condition based on the integrated analysis, means for providing the guardian with countermeasures for the predicted future condition, and means for automatically adjusting environmental conditions based on the countermeasures and cooperating with a device to provide a comfortable environment for the guardian and infant. This enables a comprehensive analysis of the baby's condition and the parent's emotional state, and makes it possible to reduce the burden of child-rearing on parents by suggesting appropriate countermeasures and automatically adjusting the environment.

[1722] "Mobile communications device" refers to a mobile device such as a mobile phone or smartphone, which has the ability to send and receive data using communications technology.

[1723] "Video information" is data containing visual content captured by a camera, video device, or the like.

[1724] "Audio information" is information that includes sound data collected by an audio capture device such as a microphone.

[1725] A "learning algorithm" is a set of computational methods and procedures for analyzing data and finding patterns and regularities within it, and plays an important role in machine learning.

[1726] "Infant state" refers to the physical and emotional state of a young child, such as a baby, including hunger, sleepiness, pleasure and pain, etc.

[1727] "Parental emotional state" refers to the psychological emotional state of the parent or guardian, including stress, joy, fatigue, etc.

[1728] "Integrated analysis" is an analytical method that combines data obtained from multiple different sources to make a single comprehensive evaluation.

[1729] "Future potential state" refers to the likely future state of the child and caregiver, predicted based on current data.

[1730] A "countermeasure" is a proposed action or method based on an estimated state or prediction, with the aim of solving a problem or improving a situation.

[1731] "Automatic adjustment of environmental conditions" is an operation that automatically adjusts environmental elements such as room temperature, lighting, and music to provide a comfortable living environment for infants and their guardians.

[1732] "Working with devices" refers to a system exchanging information and instructions with other devices or equipment to perform functions in collaboration.

[1733] MODE FOR CARRYING OUT THE INVENTION

[1734] The present invention relates to the "AI Mimamori" system, which aims to reduce the burden on parents raising children and help them do so more effectively and efficiently. This system analyzes video and audio information obtained from mobile communication devices (such as smartphones), predicts the state of the child, and provides parents with countermeasures based on that information. It is also possible to recognize the emotions of the user (parent) and provide more appropriate countermeasures.

[1735] Hardware and software used

[1736] Devices: Smartphone, camera, microphone

[1737] Server: High-performance computer

[1738] software:

[1739] Video analysis: Convolutional neural networks (CNN)

[1740] Speech analysis: Speech recognition model

[1741] Sentiment Analysis: Emotion Recognition Engine

[1742] Data transmission: Internet Protocol

[1743] System Operation

[1744] The user activates the "AI Mimamori" system using a smartphone. The device uses a camera and microphone to capture video and audio information of the baby and parent in real time. For example, the smartphone's camera captures the baby's face, and the microphone records its crying. At the same time, the parent's facial expressions and voice are also captured. This captured data is preprocessed and then sent to a server via the Internet.

[1745] The server first analyzes the baby's video information using a convolutional neural network (CNN). It receives image data as input, extracts features, and infers the baby's state, such as hunger, sleepiness, or desire to play, from its facial expressions and movements. Similarly, the baby's audio information is input into a speech recognition model, which evaluates the baby's state based on crying patterns and other audio signals.

[1746] The server then inputs the parent's video and audio information into an emotion recognition engine, which analyzes the parent's facial expressions and tone of voice to identify emotions such as stress, joy, and fatigue.

[1747] The results of both analyses are integrated on a server and recorded as the overall state of the infant and parent. This information is stored in a data log and compared with past data to predict future states (for example, the baby may become hungry in two hours, which may cause stress for the parent).

[1748] The prediction results and countermeasures are sent to the device via the Internet and notified to the parent. For example, if the baby seems sleepy and the parent is tired, a notification will appear on the smartphone saying, "The baby seems sleepy. The room temperature is being adjusted. We recommend that the parent also take a break."

[1749] In addition, the server can also connect to smart home devices to automatically adjust the room temperature and play relaxing music to maintain a comfortable environment for both baby and parent, ensuring a safe and comfortable environment for both baby and parent.

[1750] Specific examples

[1751] The user activates the "AI Mimamori" system. The device (e.g., a smartphone) then captures video and audio of the baby and parent. The server analyzes this data and infers whether the baby is hungry and whether the parent is feeling stressed. Based on this, the server predicts that the baby will become hungry again in approximately two hours and suggests that the parent take a break to relax. The server then instructs smart home devices to set the room temperature to 22 degrees and play relaxing music. The smartphone also receives a notification saying, "The baby is likely to become hungry in two hours. Please relax, parent." This allows parents to prepare the next meal at the appropriate time while also maintaining their own health.

[1752] Prompt Sentence Examples

[1753] Contracting System:

[1754] 1. Use a camera and microphone to capture video and audio of the baby and parent.

[1755] 2. Video and audio data is sent to a server via the Internet.

[1756] 3. The server analyzes the data, proposes appropriate countermeasures, and notifies the smartphone.

[1757] Situation:

[1758] Imagine when a baby is crying and the parents are tired, and generate the best response.

[1759] Expected output:

[1760] "It's likely your baby is hungry and may be hungry again in about two hours. I set the room temperature to 72 degrees and played relaxing music. I encourage parents to take breaks as well."

[1761] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1762] Step 1:

[1763] The device uses a camera and microphone to capture real-time video and audio information of the baby and parent. The camera captures the baby's face, and the microphone records the baby's crying. At the same time, the device also captures the parent's facial expressions and voice. This data is temporarily stored on the device.

[1764] Input: Video and audio information from the camera and microphone

[1765] Output: A dataset of captured video and audio information

[1766] Step 2:

[1767] The device preprocesses the captured video and audio data, specifically resizing the video data and converting the audio data into a digital format. This preprocessing ensures uniform data quality for efficient analysis.

[1768] Input: A dataset of captured video and audio information.

[1769] Output: Pre-processed video and audio information

[1770] Step 3:

[1771] The device then transmits the preprocessed data to a server over the internet, using encryption protocols to prevent data leakage or tampering.

[1772] Input: Preprocessed video and audio information

[1773] Output: Data sent to the server

[1774] Step 4:

[1775] The server inputs the transmitted video data of the baby into a convolutional neural network (CNN). The CNN extracts features from the image data and infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements. For example, it analyzes video of a baby with its mouth open and determines that the baby is hungry.

[1776] Input: Baby video data sent to the server

[1777] Output: Analysis results showing the baby's condition

[1778] Step 5:

[1779] The server inputs the baby's voice data into a speech recognition model, which evaluates the baby's state from crying patterns and other audio signals. For example, if the crying pattern matches "sleepy," that information is output.

[1780] Input: Baby's voice data sent to the server

[1781] Output: Voice analysis results showing the baby's condition

[1782] Step 6:

[1783] The server inputs the parent's video and audio information into an emotion recognition engine. The emotion recognition engine analyzes the parent's facial expressions and tone of voice to determine emotions such as stress, joy, and fatigue. For example, it analyzes a video of the parent frowning and determines that the parent is feeling "stressed."

[1784] Input: Parent's video and audio information sent to the server

[1785] Output: Analysis results showing the parent's emotional state

[1786] Step 7:

[1787] The server combines the results of the baby's and parent's analysis to assess their overall condition. This information is stored in a data log and compared with past data. This allows for predictions of future conditions (e.g., the baby will be hungry in two hours).

[1788] Input: Analysis results showing the baby's condition, analysis results showing the parent's emotional state

[1789] Output: Comprehensive condition assessment and future condition prediction

[1790] Step 8:

[1791] The server generates appropriate countermeasures based on the prediction results and notifies the device, such as the next feeding time or suggestions for parental relaxation.

[1792] Input: Comprehensive condition assessment and future condition prediction

[1793] Output: Notification of action to be taken

[1794] Step 9:

[1795] The server works in conjunction with smart home devices to automatically adjust the room temperature and play relaxing music to maintain a comfortable environment for both baby and parent.

[1796] Input: Comprehensive condition assessment and future condition prediction

[1797] Output: Instructions to smart home devices

[1798] (Application example 2)

[1799] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1800] In order to reduce the burden on parents raising children and to raise them more effectively and efficiently amid a declining birthrate, it is necessary to consider not only the condition of the infant but also the emotional state of the parent. However, conventional systems lack the functionality to analyze the parent's emotional state, making it difficult to provide appropriate countermeasures. In addition, there is a lack of integration with smart home devices to maintain a comfortable environment for infants and parents, which means the burden on parents is not reduced.

[1801] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video information acquired from a mobile communication device, means for analyzing audio information acquired from the mobile communication device, means for integrating the video information and the audio information and using a learning algorithm to estimate the infant's state, means for predicting a future potential state based on the estimated infant's state, means for analyzing the parent's emotions, means for providing countermeasures based on the estimated infant's state and the parent's emotions, means for adjusting the environment in cooperation with a smart home device, and means for notifying the parent of the countermeasures. This provides appropriate childcare support that also takes into account the parent's emotional state, making it possible to reduce the burden of childcare and provide a comfortable environment for both the baby and the parent.

[1802] A "mobile communication device" is a portable information communication terminal equipped with a camera and a microphone, and has the function of acquiring video information and audio information and transmitting and receiving data via a communication network.

[1803] "Video information" refers to visual data captured by a camera on a mobile communication device and presented in the form of images or video.

[1804] "Audio information" refers to auditory data captured by a microphone on a mobile communication device and expressed in voice or acoustic form.

[1805] A "learning algorithm" refers to a computational method or model that learns patterns and relationships based on previously collected data and makes inferences and predictions about new data.

[1806] "Infant condition" refers to the baby's physical or psychological state, such as hunger, sleepiness, or a desire to play.

[1807] "Future potential states" refer to the infant's future state predicted based on current data and analysis results, such as hunger or sleepiness after a certain time.

[1808] "Parental emotions" refers to the emotional state that the parent is currently experiencing, such as specific emotions such as stress, joy, or fatigue.

[1809] "Countermeasures" refer to specific actions or coping methods suggested based on the estimated infant's condition and the parent's emotions, such as notification of the next feeding time or advice on stress reduction.

[1810] "Smart home devices" refer to home electronic devices that are connected to a network and have remote control or automation capabilities, such as smart thermostats and smart speakers.

[1811] "Environmental adjustment" refers to operations such as changing temperature and sound settings to maintain a comfortable indoor environment.

[1812] "Notification" refers to a function of informing a user of information via a mobile communication device, and includes notification by screen display or voice.

[1813] The "childcare support system" of the present invention is a system that uses a mobile communication device (e.g., a smartphone) to reduce the burden on parents raising children and allow them to raise children more effectively and efficiently. This system analyzes not only the condition of the infant but also the emotional state of the parent, and provides the user with appropriate countermeasures. The following describes in detail an embodiment of the present invention.

[1814] 1. System Configuration

[1815] The system consists of three main components: terminals, servers, and smart home devices.

[1816] Terminal

[1817] The device is a smartphone equipped with a camera and microphone, which captures real-time video and audio information of the baby and parent. The captured data is then sent to a server via the Internet.

[1818] server

[1819] The server has the following analytical methods and capabilities:

[1820] Video information analysis method: Using a convolutional neural network (CNN), the video data obtained from the camera is analyzed to extract the baby's facial expressions and movements.

[1821] Audio information analysis means: Using a speech recognition model, the audio data captured from the microphone is analyzed to evaluate crying and other audio signals.

[1822] Emotion recognition: An emotion recognition engine is used to analyze emotions from the parent's facial expressions and tone of voice.

[1823] The analyzed results are integrated to estimate the status of the infant and parent. Based on this, potential future conditions are predicted and appropriate countermeasures are generated. The appropriate countermeasures are notified to the terminal via the Internet, and instructions for cooperation are sent to smart home devices.

[1824] Smart Home Devices

[1825] Smart home devices automatically adjust the environment, such as networked thermostats and music players, to keep children and parents comfortable by adjusting the temperature or playing relaxing music.

[1826] 2. System Operation

[1827] Data Acquisition and Transmission

[1828] When a user launches the smartphone app, the camera and microphone begin capturing data in real time, which is then sent to a server via the Internet for detailed analysis.

[1829] Analysis and Notification

[1830] The server analyzes video and audio data to infer the baby's condition. For example, it can infer hunger from the baby's facial expression and sleepiness from the crying pattern. It can also analyze the parent's emotions from their facial expression and tone of voice to infer stress or fatigue. Based on these analysis results, it predicts future potential conditions and generates countermeasures.

[1831] environmental adjustment

[1832] By connecting with smart home devices, it can automatically adjust the room temperature and play relaxing music, reducing the burden on parents while providing a comfortable environment for both baby and parent.

[1833] 3. Examples of concrete examples and prompts

[1834] Specific examples

[1835] After the user activates the system, the smartphone's camera and microphone capture video and audio of the baby and parent. The server analyzes the data and infers that the baby is hungry and the parent is stressed. The following notification is displayed on the smartphone as a response generated by the server:

[1836] "Your baby is hungry. The next feeding is in about an hour. Set the room temperature to 72 degrees and play some relaxing music."

[1837] Prompt Sentence Examples

[1838] Analyze your baby's condition:

[1839] Capture the baby's facial expressions and movements with a camera and analyze them using a CNN model.

[1840] Use a voice recognition model to analyze crying patterns and assess states such as hunger and sleepiness.

[1841] Analyzing parental emotions:

[1842] Capture the parent's facial expressions and voice and analyze them with an emotion recognition engine.

[1843] Integrate baby and parent status and generate notifications:

[1844] If the baby is hungry, notify them of appropriate action (e.g., notify them of the next feeding time).

[1845] If a parent is tired, suggest a break and adjust the temperature with a smart home device.

[1846] As described above, the childcare support system of the present invention is capable of providing appropriate childcare support that takes into consideration the emotional state of the parents, thereby reducing the burden on the parents and improving the comfort of both the baby and the parents.

[1847] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1848] Step 1:

[1849] The user launches the smartphone app.

[1850] Input: Smartphone camera and microphone

[1851] Output: Real-time video and audio information

[1852] How it works: The user taps to launch the smartphone app, which automatically turns on the camera and microphone, capturing video and audio of the baby and parent.

[1853] Step 2:

[1854] The video and audio information acquired by the terminal is transmitted to a server via the Internet.

[1855] Input: Real-time video and audio information

[1856] Output: Data sent to the server

[1857] How it works: The device transfers the captured video and audio data to the server via a communications network. The data is encrypted and securely transmitted.

[1858] Step 3:

[1859] The server receives the video data and analyzes it using a convolutional neural network (CNN).

[1860] Input: Video information obtained from the camera

[1861] Output: Baby's facial expressions and behavior patterns

[1862] How it works: The server preprocesses the received video data (e.g., resizes and normalizes it) and then inputs it into a CNN model. The CNN model extracts features and infers the baby's state, such as hunger, sleepiness, or desire to play, from the baby's facial expressions and movements.

[1863] Step 4:

[1864] The server receives the voice data and analyzes it using a voice recognition model.

[1865] Input: Audio information obtained from the microphone

[1866] Output: Baby's crying and voice patterns

[1867] Specific operation: The server preprocesses the received voice data (e.g., noise reduction, normalization) and then inputs it into a voice recognition model. The voice recognition model analyzes the characteristics of the crying voice and evaluates the baby's state (e.g., hunger, sleepiness, demands).

[1868] Step 5:

[1869] The server inputs the parent's video and audio data into an emotion recognition engine for analysis.

[1870] Input: Parent video and audio information obtained from camera and microphone

[1871] Output: Parent's emotional state (e.g., stress, joy, fatigue)

[1872] Specific operation: The server inputs the parent's video data into an emotion recognition engine and analyzes their facial expressions. It also uses audio data to analyze their tone of voice and the content of their speech. This allows it to infer the parent's emotional state.

[1873] Step 6:

[1874] The server integrates the baby and parent condition data it has acquired and predicts potential future conditions.

[1875] Input: Baby's state data, Parent's emotional state data

[1876] Output: Predicted future state (e.g. next feeding time, parental stress level)

[1877] How it works: The server performs statistical analysis and comparison with past data to predict the future state of the baby and parents, for example, calculating the likelihood that the baby will be hungry after a certain time, or the likelihood that the parents' stress levels will increase.

[1878] Step 7:

[1879] The server generates appropriate countermeasures based on the prediction results and notifies the device.

[1880] Input: Predicted future state data

[1881] Output: Notifications sent to smartphones (e.g., notifications about breastfeeding times, suggestions for adjusting the environment)

[1882] Specific operation: Based on the prediction results, the server generates appropriate responses for the baby and the parent. For example, it predicts that the baby will become hungry again in about two hours and suggests that the parent take a break to relax. This information is then sent to the smartphone.

[1883] Step 8:

[1884] The server works in conjunction with smart home devices to adjust the environment.

[1885] Input: Generated countermeasure data, control information of smart home devices

[1886] Output: Adjusted comfort environment (e.g., adjusting room temperature, playing relaxing music)

[1887] How it works: The server sends instructions to smart home devices, which then automatically change the room temperature or play relaxing music, optimizing the childcare environment.

[1888] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1889] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1890] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1891] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1892] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1893] The...

Claims

1. means for analyzing video information acquired from a mobile communication device; means for analyzing voice information obtained from the mobile communication device; a means for integrating the video information and the audio information and using a learning algorithm to estimate the state of the infant; a means for predicting a future potential state based on the estimated state of the infant; means for providing a parent with a course of action regarding said predicted future condition; A system including:

2. The system of claim 1 further comprising means for accumulating data on the predicted infant condition and the countermeasures and for improving the accuracy of predictions through comparative analysis.

3. The system according to claim 1 , further comprising means for providing data on the estimated condition of the infant and the countermeasures to relatives, and generating a manual for dealing with the infant in an optimal manner.

Images