system

Abstract

We provide the system. [Solution] A means for capturing video of the baby and sending it to a server, A means for detecting the baby's behavior from the video data using an image analysis algorithm, A means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model, A means for generating childcare advice based on the aforementioned evaluation results, A means for transmitting and displaying the aforementioned childcare advice on a terminal, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Many baby cameras currently on the market are mainly limited to a monitoring function for protecting the safety of babies, so they lack the function of providing parents with valuable information about the growth and behavior of babies. As a result, it is difficult for parents to grasp the growth stage of their babies in real time and obtain appropriate parenting advice. The purpose of the present invention is to solve this problem and provide a system that can more effectively monitor the growth of babies and enable parents to receive sufficient parenting support.

Means for Solving the Problems

[0005] The present invention includes means for capturing video of a baby and transmitting it to a server. It also includes means for detecting the baby's behavior from the video data using an image analysis algorithm. Furthermore, it provides means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model. It also includes means for generating childcare advice based on the evaluation results and transmitting and displaying the childcare advice on a terminal. Thus, the present invention provides a system that allows parents to understand the baby's behavior and developmental status in real time and receive appropriate childcare advice.

[0006] "Means for capturing video" refers to a set of hardware and software for capturing video of a baby in real time and converting it into digital data format.

[0007] "Means of sending to the server" refers to a set of communication protocols and software for transferring captured video data to a server via a network.

[0008] An "image analysis algorithm" is a mathematical and computational method for identifying and classifying specific objects or movements from video data.

[0009] "Means of detecting behavior" refers to a set of software that uses image analysis algorithms to identify and track a baby's movements over time.

[0010] A "machine learning model" is a learning system that uses statistical and computational methods to make predictions and classifications based on collected data.

[0011] "Means for assessing growth stages" refers to a set of software that uses machine learning models to estimate a baby's current growth stage based on behavioral data.

[0012] "Means for generating parenting advice" refers to a set of software that creates appropriate parenting suggestions and feedback for parents based on the results of an assessment of their child's developmental stage.

[0013] "Means of transmitting and displaying on a terminal" refers to a set of communication system and display application for notifying a terminal used by a parent of the generated childcare advice. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention is a system that monitors a baby's growth and provides parents with appropriate childcare advice. The following describes the system's program processing in natural language, along with specific examples.

[0036] 1. Data collection and transmission

[0037] terminal

[0038] A camera installed in the baby's room captures video of the baby in real time. The captured video data is compressed and sent to a server via a secure communication protocol.

[0039] 2. Video Analysis

[0040] server

[0041] The server processes the video data received from the terminal. First, it preprocesses the data by removing noise and preparing the video data for analysis. Next, it uses an image analysis algorithm (e.g., an object detection algorithm) to detect the baby's actions. Specifically, it identifies actions such as the baby rolling over, sitting up, and crawling.

[0042] 3. Evaluation using machine learning

[0043] server

[0044] The detected behavioral data is used to evaluate the baby's developmental stage. This is done using a pre-trained machine learning model. The data obtained in the feature extraction process is used as input to determine the baby's developmental stage (e.g., starting to roll over, starting to sit up, etc.). In addition, future behaviors and potential risk factors are predicted based on the behavioral history.

[0045] 4. Advice generation

[0046] server

[0047] Based on the evaluation results of a machine learning model, it generates advice and feedback on childcare. For example, it creates specific advice such as, "Now that your baby has started rolling over, it's a good idea to start practicing sitting up next." This advice is formatted in a way that is easy for parents to understand.

[0048] 5. Notifications and Display

[0049] Device & User

[0050] The server sends generated advice and feedback to the device (e.g., a smartphone app). The device notifies the user of the received message using a pop-up notification or sound. When the user opens the app, they can view detailed parenting advice.

[0051] Specific example

[0052] Let's take the example of a baby who has just started rolling over in their crib.

[0053] terminal

[0054] The camera captures the baby rolling over and sends this video data to the server.

[0055] server

[0056] The server analyzes the received video data and detects when the baby rolls over. This motion data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0057] Device & User

[0058] A smartphone app receives this parenting advice and notifies the user. When the user opens the app, they can view specific advice and take appropriate action according to their baby's development.

[0059] The above describes the detailed configuration for carrying out the invention. This system allows parents to monitor their baby's growth in real time and receive appropriate childcare advice.

[0060] The following describes the processing flow.

[0061] Step 1: Data Capture (Device)

[0062] The device activates the camera and captures the baby's image in real time. The camera continuously records the baby's movements and prepares to send the acquired video data to the server in real time.

[0063] Step 2: Data transmission (terminal)

[0064] The terminal compresses the captured video data and sends it to the server using a secure communication protocol. Data encryption is performed during this process to prevent interception by third parties.

[0065] Step 3: Data reception (server)

[0066] The server receives video data transmitted from the terminal. The received data is temporarily stored in a buffer and then passed on to the analysis process.

[0067] Step 4: Data preprocessing (server)

[0068] The server removes noise from the received video data and extracts only the necessary parts. This improves the accuracy of data analysis.

[0069] Step 5: Behavior detection (server)

[0070] The server uses image analysis algorithms to detect specific baby behaviors (e.g., rolling over, sitting up, crawling) from video data. The analyzed data is then categorized by type of behavior.

[0071] Step 6: Feature Extraction (Server)

[0072] The server extracts features from the detected behavioral data. For example, the number of times a cat turns over in its sleep or the duration of a specific behavior may be extracted as features.

[0073] Step 7: Evaluation and Prediction (Server)

[0074] The server applies a machine learning model and uses the extracted features to evaluate the baby's developmental stage. It also predicts likely future behaviors and potential risks based on the history of behavioral data.

[0075] Step 8: Advice generation (server)

[0076] The server generates parenting advice for parents based on the evaluation and prediction results. The advice is formatted in a way that is easy for parents to understand.

[0077] Step 9: Send notification (server)

[0078] The server generates childcare advice and sends it to the device. The sent notification arrives on the device, which is a smartphone or tablet.

[0079] Step 10: Notification display (device)

[0080] The device notifies the user of childcare advice it has received. The notification is delivered via a pop-up message or sound.

[0081] Step 11: View Details (User)

[0082] Users can check notifications on their devices and view detailed parenting advice. Through the app, users can also access past advice and prediction results.

[0083] The above outlines the specific processing steps of the program. This series of steps enables a system that monitors the baby's growth in real time and provides parents with appropriate childcare advice.

[0084] (Example 1)

[0085] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0086] Traditional childcare support systems often do not provide real-time monitoring of a baby's growth or offer childcare advice, and therefore do not adequately help parents understand their baby's development. Furthermore, they lacked the means to accurately detect a baby's behavior and generate appropriate advice based on their developmental stage.

[0087] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0088] In this invention, the server includes means for capturing and transmitting video of the baby to the server, means for preprocessing by applying a noise reduction filter, means for detecting the baby's actions from the video data using an object detection algorithm, means for extracting the baby's actions data, means for evaluating the baby's developmental stage based on the detected actions using a machine learning model, means for generating childcare advice based on the evaluation results, and means for transmitting, notifying, and displaying the childcare advice to a terminal. This enables parents to understand their baby's developmental status in real time and receive appropriate childcare advice.

[0089] A "noise reduction filter" is an algorithm used to remove random noise from video data.

[0090] An "object detection algorithm" is an algorithm used to detect specific objects within video data, such as the posture or movements of a baby.

[0091] "Behavioral data" refers to information about a baby's specific movements and postures, including the type of movement detected and a timestamp.

[0092] A "machine learning model" is an algorithm that learns behavioral patterns and action stages based on training data, and then uses that to evaluate and predict new data.

[0093] "Parenting advice" refers to specific suggestions and instructions for parents, generated based on the baby's developmental stage and behavioral data.

[0094] A "server" is a computing device that analyzes video data of babies, detects their behavior, evaluates their developmental stage using machine learning models, and generates parenting advice.

[0095] A "terminal" refers to a device, such as a camera or smartphone, used to capture images of the baby or to receive and display advice sent from a server.

[0096] This invention is a system for monitoring a baby's growth and providing parents with appropriate childcare advice. The system consists of a video capture device, communication means, a server, an analysis algorithm, a machine learning model, and a notification system.

[0097] Hardware and software configuration

[0098] terminal

[0099] The device includes a camera installed in the baby's room. This camera (e.g., an IP camera) captures video of the baby in real time. The video data is encoded in a compressed format such as H.264 and securely transmitted to the server using the SSL / TLS protocol.

[0100] server

[0101] The server receives and processes video data transmitted from the terminal. Specifically, it preprocesses the data by applying a noise reduction filter (e.g., a Gaussian filter), and then analyzes the baby's behavior using object detection algorithms such as OpenCV or YOLO. The baby's behavior data is stored in a database with timestamps.

[0102] The server also uses machine learning models (e.g., TENSORFLOW®, PyTorch) to assess the baby's developmental stage. These models are trained on pre-collected datasets and can automatically evaluate and classify the baby's behavioral patterns. Based on the evaluation results, natural language generation (NLG) algorithms are used to generate specific parenting advice.

[0103] User notification system

[0104] The advice generated by the server is sent to the device (e.g., a smartphone app). The device notifies the user of the received advice via a pop-up notification or sound notification. When the user opens the app, detailed parenting advice is displayed.

[0105] Specific example

[0106] For example, consider a situation where a baby starts rolling over in their crib. A camera captures this movement and sends the video to a server. The server analyzes the video data and detects that the baby is rolling over. A machine learning model evaluates this movement as a developmental stage, specifically "rolling over." The server then generates advice such as, "Your baby has started rolling over. Next, try practicing sitting up," and sends this advice to the device. The device notifies the user of this advice, and when the user opens the app, more detailed advice is displayed.

[0107] Example of a prompt

[0108] An example of a prompt to input into a generative AI model is: "Describe a system that detects behavior from video data of a baby and generates childcare advice. Describe in detail the entire process from preprocessing of input data to notification of advice. Include the names of specific algorithms and software."

[0109] This invention allows parents to monitor their baby's development in real time and receive appropriate childcare advice.

[0110] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0111] Step 1:

[0112] terminal

[0113] A camera in the baby's room captures video. This camera captures the baby's video in real time, frame by frame, and encodes it in a compressed format such as H.264.

[0114] Input: Real-time video data

[0115] Output: Compressed video data

[0116] Step 2:

[0117] terminal

[0118] The video data transmitted from the camera is encrypted using the SSL / TLS protocol and securely sent to the server.

[0119] Input: Compressed video data

[0120] Output: Encrypted video data

[0121] Step 3:

[0122] server

[0123] The server decrypts the encrypted video data received from the terminal. Next, it applies a noise reduction filter (e.g., a Gaussian filter) to preprocess the data. This removes random noise from the video.

[0124] Input: Encrypted video data

[0125] Output: Preprocessed video data

[0126] Step 4:

[0127] server

[0128] Preprocessed video data is divided into frames, and object detection algorithms (e.g., OpenCV, YOLO) are used to detect the baby's actions. For example, actions such as rolling over, sitting up, and crawling are analyzed to identify the baby's posture and movements.

[0129] Input: Preprocessed video data

[0130] Output: Baby behavior data

[0131] Step 5:

[0132] server

[0133] The baby's behavioral data is extracted and stored in a database with timestamps. This data includes the baby's actions and the time they occurred.

[0134] Input: Baby's behavioral data

[0135] Output: Timestamped behavioral data

[0136] Step 6:

[0137] server

[0138] Using machine learning models (e.g., TensorFlow, PyTorch), we evaluate the developmental stage of a baby based on extracted behavioral data. For example, behavioral data is input as features to determine developmental stages such as "the baby has started rolling over." We also predict future behaviors and potential risk factors based on the behavioral history.

[0139] Input: Timestamped behavioral data

[0140] Output: Growth stage evaluation results

[0141] Step 7:

[0142] server

[0143] Based on the results of the developmental stage assessment, a natural language generation (NLG) algorithm is used to generate parenting advice. For example, it can create specific advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0144] Input: Growth stage evaluation results

[0145] Output: Childcare advice

[0146] Step 8:

[0147] server

[0148] The generated parenting advice is sent to the device (e.g., a smartphone app).

[0149] Input: Childcare advice

[0150] Output: Advice sent to the terminal

[0151] Step 9:

[0152] Device & User

[0153] The device notifies the user of received advice through pop-up notifications and sounds. When the user opens the app, detailed parenting advice is displayed.

[0154] Input: Advice sent to the device

[0155] Output: Advice notified to the user

[0156] The above outlines the specific processing steps of this system.

[0157] (Application Example 1)

[0158] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0159] Systems already exist that appropriately monitor a baby's growth and provide parents with accurate childcare advice. However, these systems typically only provide childcare advice and do not recommend or assist with the purchase of childcare-related products. As a result, there is a challenge in that parents have difficulty choosing appropriate products for their baby's development. Furthermore, there is a lack of integration with virtual stores, making it difficult to purchase childcare-related products in a centralized manner.

[0160] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0161] In this invention, the server includes means for capturing images of a baby and transmitting them to a data server; means for detecting the baby's behavior from the image data using an image analysis algorithm; means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model; means for generating childcare advice based on the evaluation results; means for transmitting and displaying the childcare advice on a user terminal; and means for coordinating with a virtual store to recommend childcare-related products. This makes it possible for parents to not only receive appropriate advice according to their baby's growth, but also to easily purchase products corresponding to that growth stage.

[0162] "Means for capturing images of a baby" refers to a device or method for capturing images of a baby's activities in real time and acquiring that image data.

[0163] "Means of sending to a data server" refers to a device or software that transfers captured video data to a data server via a network.

[0164] An "image analysis algorithm" is a set of computational procedures or programs used to analyze acquired video data and recognize specific actions or objects.

[0165] "Means for detecting a baby's behavior" refers to a device or method that uses an image analysis algorithm to identify a baby's specific movements from video data.

[0166] A "machine learning model" is a computational model that learns from large amounts of data and uses that data to make predictions and classifications about unknown data.

[0167] "Means for evaluating the developmental stage of an infant" refers to a device or method that uses a machine learning model to determine the progress of an infant's development based on their current behavioral data.

[0168] A "means for generating childcare advice" refers to a device or software that automatically creates specific advice and suggestions for parents based on the results of an assessment of the baby's developmental stage.

[0169] "Means of transmitting and displaying on a user terminal" refers to a device or software that transfers and displays the generated childcare advice on a device such as a smartphone or tablet used by the parent.

[0170] "Means of recommending childcare-related products in conjunction with virtual stores" refers to a device or method that links information with virtual stores that provide appropriate products online according to the developmental stage of a baby, and recommends products to parents along with childcare advice.

[0171] This invention is a system for monitoring a baby's growth and providing childcare advice based on that growth. The system includes a camera, a data server, an image analysis algorithm, a machine learning model, a notification system, and integration with a virtual store.

[0172] 1. Data collection and transmission

[0173] The server uses a camera to capture the baby's activity in real time. The captured video data is compressed and sent to the data server via a secure communication protocol. This process uses video processing libraries such as OpenCV.

[0174] 2. Video Analysis

[0175] The server preprocesses the received video data to remove noise. Next, it uses image analysis algorithms to identify the baby's actions. In this process, for example, object detection algorithms are used to analyze actions such as the baby rolling over, sitting up, and crawling.

[0176] 3. Evaluation of Growth Stages

[0177] The server uses a machine learning model to assess the baby's developmental stage based on detected behavioral data. This model utilizes a pre-trained dataset to determine the developmental stage based on the baby's current behavior. Furthermore, it analyzes the behavioral history to predict future behavior and potential risk factors.

[0178] 4. Generating parenting advice

[0179] The server generates parenting advice based on the evaluation results of the machine learning model. For example, it creates specific advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0180] 5. Notifications and Display

[0181] The server sends the generated advice to the user's device. The user's device (such as a smartphone or tablet) notifies the user of the received message with a pop-up notification or sound, and displays detailed childcare advice within the app.

[0182] 6. Collaboration with virtual stores

[0183] The server integrates information with a virtual store to recommend appropriate products based on the baby's developmental stage. For example, it might provide product recommendations such as, "Your baby has started rolling over. This cushion would be a good next step."

[0184] Specific example

[0185] Consider a scenario where a baby starts rolling over in their crib. A camera captures this movement and sends the video data to a server. The server analyzes the received video and detects the baby's rolling over. Based on this information, a machine learning model evaluates the developmental stage as "rolling over." Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up," and also makes product recommendations such as, "This cushion would be helpful." This information is then sent to the parent's smartphone.

[0186] Example of input prompt text for a generative AI model:

[0187] "Your baby has started rolling over. Next, try practicing sitting up."

[0188] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0189] Step 1:

[0190] The server uses a camera to capture the baby's activities in real time. The input is video data from the camera, and the output is compressed video data. Specifically, the camera films the baby's movements, acquires the video frame by frame, compresses it in JPEG format, and sends the prepared video data to the data server.

[0191] Step 2:

[0192] The server preprocesses the video data it receives. The input is compressed video data, and the output is video data formatted for analysis. Specifically, preprocessing is performed to remove video noise and adjust the resolution to an appropriate level. Video processing libraries such as OpenCV are used for this process.

[0193] Step 3:

[0194] The server identifies the baby's actions from pre-processed video data using an image analysis algorithm. The input is the pre-processed video data, and the output is the detection result of specific baby actions (e.g., rolling over, sitting up, crawling). Specifically, an object detection algorithm identifies the baby's movements, and the result is extracted as data.

[0195] Step 4:

[0196] The server uses a machine learning model to detect behavioral data and evaluate the baby's developmental stage. The input is the baby's behavioral data, and the output is an evaluation of the baby's developmental stage (e.g., rolling over, sitting up). Specifically, it uses a pre-trained dataset to determine the developmental stage based on the current behavioral data.

[0197] Step 5:

[0198] The server generates parenting advice based on the evaluation results of a machine learning model. The input is the evaluation result of the baby's developmental stage, and the output is specific parenting advice. For example, it might generate advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0199] Step 6:

[0200] The server generates childcare advice and sends it to the user's device. The input is the generated childcare advice, and the output is a notification to the user's device. Specifically, the childcare advice is sent as a push notification or message to the user's smartphone or tablet.

[0201] Step 7:

[0202] The server interacts with a virtual store to recommend childcare-related products. The input is the result of an assessment of the baby's developmental stage, and the output is information about the corresponding product. Specifically, it retrieves products suitable for the baby's growth (e.g., cushions, toys) from the virtual store and generates recommendations for parents.

[0203] Step 8:

[0204] The user's device receives notifications from the server and displays them to the parent. The input is childcare advice and product recommendations sent from the server, and the output is display and notifications within the app. Specifically, when the user opens the app, detailed childcare advice and recommended products are displayed for viewing.

[0205] The above outlines the specific processing steps of the system for implementing the invention. This allows parents to monitor their baby's growth in real time and receive appropriate childcare advice and product recommendations.

[0206] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0207] This invention is a system for monitoring a baby's growth and providing parents with appropriate childcare advice. The system includes means for capturing video of the baby and transmitting it to a server, means for detecting the baby's behavior from the video data using an image analysis algorithm, means for evaluating the baby's growth stage based on the detected behavior using a machine learning model, means for generating childcare advice based on the evaluation results, and means for transmitting and displaying the generated childcare advice on a terminal. It also incorporates an emotion engine that recognizes the user's emotions and has a function to customize childcare advice based on the user's emotional state.

[0208] 1. Data collection and transmission

[0209] terminal

[0210] The device activates its camera and captures video of the baby in real time. The captured video data is compressed and sent to the server via a secure communication protocol.

[0211] 2. Video Analysis

[0212] server

[0213] The server processes the video data received from the terminal. First, it preprocesses the data by removing noise and preparing it for analysis. Next, it uses an image analysis algorithm to detect the baby's actions. For example, it identifies actions such as the baby rolling over or sitting up.

[0214] 3. Evaluation using machine learning

[0215] server

[0216] Based on the detected behavioral data, a machine learning model is used to evaluate the baby's developmental stage. The data obtained in the feature extraction process is input into the model to determine the developmental stage. In addition, future behaviors and potential risk factors are predicted based on the behavioral history.

[0217] 4. Advice generation

[0218] server

[0219] Based on the evaluation and prediction results, specific parenting advice is generated. The generated advice is formatted in a way that is easy for parents to understand.

[0220] 5. Emotion Recognition and Customization

[0221] terminal

[0222] The device is also equipped with a camera and microphone to capture the user's facial expressions and voice tone. This data is sent to an emotion engine to analyze the user's emotional state.

[0223] server

[0224] The emotion engine identifies the user's emotions (e.g., stress, fatigue, exhilaration, etc.) and customizes parenting advice based on the user's emotional state.

[0225] 6. Sending and displaying notifications

[0226] server

[0227] The server sends customized parenting advice to the device.

[0228] Device & User

[0229] The device notifies the user of any advice it receives. The user checks the notification and can view detailed parenting advice within the app.

[0230] Specific example

[0231] Let's take the example of a baby who has just started rolling over in their crib.

[0232] terminal

[0233] The camera captures the baby rolling over and sends this video data to the server.

[0234] server

[0235] The server analyzes the video data and detects when the baby rolls over. The detected data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0236] terminal

[0237] The device captures the user's facial expressions and tone of voice, and an emotion engine analyzes whether the user is experiencing stress.

[0238] server

[0239] Based on this emotional data, the server generates customized advice such as, "Let's have your partner help you practice together."

[0240] Device & User

[0241] A smartphone app receives this parenting advice and notifies the user. When the user opens the app, they can view customized, specific advice.

[0242] The above describes the detailed configuration for carrying out the invention. This system makes it possible not only to monitor the baby's growth in real time, but also to provide appropriate childcare advice tailored to the parents' emotional state.

[0243] The following describes the processing flow.

[0244] Step 1: Data Capture (Device)

[0245] The device activates the camera and captures the baby's image in real time. The camera continuously records the baby's movements and prepares to send the captured video data to the server in real time.

[0246] Step 2: Data transmission (terminal)

[0247] The device compresses the captured video data and sends it to the server using a secure communication protocol. Encryption technology is also used to ensure data security.

[0248] Step 3: Data reception (server)

[0249] The server receives video data transmitted from the terminal. The received data is temporarily stored in a buffer and then passed on to the analysis process.

[0250] Step 4: Data preprocessing (server)

[0251] The server removes noise from the received video data and prepares it for easier analysis. The pre-processed data is then used in the next analysis step.

[0252] Step 5: Behavior detection (server)

[0253] The server uses image analysis algorithms to detect specific baby behaviors (e.g., rolling over, sitting up, crawling) from video data. The detected behavior data is then categorized by type of behavior.

[0254] Step 6: Feature Extraction (Server)

[0255] The server extracts features from the detected behavioral data. For example, the number of times a cat turns over in its sleep or the duration of a specific behavior may be extracted as features.

[0256] Step 7: Evaluation and Prediction (Server)

[0257] The server applies a machine learning model and uses extracted feature data to assess the baby's developmental stage. Based on behavioral history, it also predicts potential next behaviors and risk factors.

[0258] Step 8: Advice generation (server)

[0259] The server generates specific parenting advice based on the evaluation and prediction results. The generated advice is formatted in a way that is easy for parents to understand.

[0260] Step 9: Emotional Data Capture (Device)

[0261] The device uses its camera and microphone to capture the user's facial expressions and voice tone. This data is then prepared to be sent to the emotion engine.

[0262] Step 10: Emotional Data Analysis (Server)

[0263] The server uses an emotion engine to analyze the user's emotional data. From the analyzed data, it identifies the user's emotional state (e.g., stress, exhilaration, fatigue, etc.).

[0264] Step 11: Customizing parenting advice (server)

[0265] The server customizes parenting advice based on the user's emotional state, as identified by the emotion engine. For example, if the user is feeling stressed, advice such as "get your partner to help you" will be generated.

[0266] Step 12: Send notification (server)

[0267] The server sends customized parenting advice to the device. The sent notifications arrive on the device, which is a smartphone or tablet.

[0268] Step 13: Notification display (device)

[0269] The device notifies the user of childcare advice it has received. Notifications are delivered to the user via pop-up messages and sounds.

[0270] Step 14: View Details (User)

[0271] Users can check notifications on their devices and view detailed parenting advice. Through the app, users can also access past advice and prediction results.

[0272] The above outlines the specific processing steps of the program. This sequence of steps enables a system that monitors the baby's growth in real time and provides appropriate childcare advice tailored to the user's emotional state.

[0273] (Example 2)

[0274] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0275] Conventional infant growth monitoring systems can identify an infant's behavior, but they do not take into account the parent's emotional state, meaning the parenting advice provided may not be appropriate to the parent's mental state. Furthermore, they lacked the ability to predict future behavior and potential risks, in addition to evaluating the infant's developmental stage. This invention aims to solve these problems.

[0276] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0277] In this invention, the server includes means for capturing and transmitting images of the baby to the server; means for detecting the baby's behavior from the image data using an image analysis algorithm; means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model; means for generating childcare advice based on the evaluation results; means for transmitting and displaying the childcare advice on a terminal; means for capturing the user's facial expressions and tone of voice and analyzing their emotional state; and means for customizing the childcare advice based on the emotional state. This makes it possible to accurately monitor the baby's development in real time and provide appropriate childcare advice that takes into account the parents' mental state.

[0278] "Means for capturing video" refers to a camera and its control device that captures video of the baby in real time.

[0279] The means for transmitting to a server refers to a communication device that compresses the captured video data and transmits it to a remote server through a secure communication protocol.

[0280] The "image analysis algorithm" refers to a program and method for detecting the behavior of a baby based on the received video data. Specifically, it includes methods such as object detection and pose estimation.

[0281] The "machine learning model" refers to a data model for evaluating the growth stage of a baby based on past data. Specific examples include neural networks.

[0282] The means for generating childcare advice refers to a program that creates appropriate childcare methods and advice for parents based on the evaluation results by the machine learning model.

[0283] The means for transmitting and displaying on a terminal refers to a communication device and a display device for transmitting and displaying childcare advice on a terminal (e.g., smartphone or tablet) used by parents.

[0284] The means for capturing expressions and voice tones refers to a camera, a microphone, and its control device for real-time recording of the expressions and voice tones of parents to grasp the emotional state of parents.

[0285] The means for analyzing the emotional state refers to a program and method for analyzing the emotions of parents from the captured expressions and voice tones. Specific examples include emotion recognition algorithms.

[0286] The means for customizing childcare advice refers to a program that adjusts and modifies childcare advice into a situation-appropriate form based on the analyzed emotional state of parents.

[0287] This invention is a system for monitoring a baby's growth and providing parents with appropriate childcare advice. The system includes means for capturing video of the baby and transmitting it to a server, means for detecting the baby's behavior from the video data using an image analysis algorithm, means for evaluating the baby's growth stage based on the detected behavior using a machine learning model, means for generating childcare advice based on the evaluation results, and means for transmitting and displaying the generated childcare advice on a terminal. It also incorporates an emotion engine that recognizes the user's emotions and has a function to customize childcare advice based on the user's emotional state.

[0288] Data collection and transmission

[0289] terminal

[0290] The device activates its camera and captures video of the baby in real time. For example, if the baby starts rolling over in the crib, it will record that movement. The captured video data is compressed using a standard compression format (e.g., H.264) and sent to the server via a secure communication protocol (e.g., SSL / TLS).

[0291] Video analysis

[0292] server

[0293] The server processes the video data received from the terminal. First, it preprocesses the data using libraries such as OpenCV to remove noise and prepare it for analysis. Next, it detects the baby's actions using image analysis algorithms such as ResNet or YOLO. For example, it detects when the baby rolls over.

[0294] Evaluation using machine learning

[0295] server

[0296] Based on the detected behavioral data, machine learning models such as TensorFlow and PyTorch are used to evaluate the baby's developmental stage. The data obtained in the feature extraction process is input into the model to determine the developmental stage (e.g., the stage of rolling over). In addition, future behaviors and potential risk factors are predicted based on the behavioral history.

[0297] Advice generation

[0298] server

[0299] Based on the evaluation and prediction results, it generates specific parenting advice. For example, it might generate advice such as, "Your baby has started rolling over. Next, try practicing sitting up." The generated advice is formatted in a way that is easy for parents to understand (e.g., text messages or illustrated guides).

[0300] Emotion recognition and customization

[0301] terminal

[0302] The device is equipped with a camera and microphone to capture the user's facial expressions and voice tone. This data is sent to an emotion engine to analyze the user's emotional state. This process utilizes services such as Amazon Rekognition and Microsoft Azure Cognitive Services.

[0303] server

[0304] The emotion engine identifies the user's emotions (e.g., stress, fatigue, exhilaration) and customizes parenting advice based on the user's emotional state. For example, if the user is feeling stressed, advice such as "Ask your partner to help you practice together" might be added.

[0305] Notification sent and displayed

[0306] server

[0307] The server sends customized childcare advice to the terminal.

[0308] Terminal & User

[0309] The terminal notifies the user of the received advice. The user checks the notification and views detailed childcare advice on the smartphone app. For example, when the app is opened, specific instructions such as "The baby has started to turn over. Next, try practicing sitting with a partner" are displayed.

[0310] Specific Example

[0311] Take the situation where the baby starts to turn over in the baby bed as an example.

[0312] Terminal

[0313] The camera captures the baby's turning over and sends this video data to the server.

[0314] Server

[0315] The server analyzes the video data, detects the baby's turning-over motion. The detected data is input into the machine learning model, and it is evaluated that the baby is in the growth stage (the stage of turning over). Next, the server generates childcare advice such as "The baby has started to turn over. Next, try practicing sitting."

[0316] Terminal

[0317] The terminal captures the user's expression and voice tone, and the emotion engine analyzes that the user is feeling stressed.

[0318] Server

[0319] Based on this emotion data, the server customizes and generates advice such as "Let's ask a partner to help with the practice together."

[0320] Device & User

[0321] A smartphone app receives this parenting advice and notifies the user. When the user opens the app, they can view customized, specific advice.

[0322] Example of a prompt

[0323] Please describe how a program works to generate parenting advice for parents who are feeling stressed when their baby starts rolling over.

[0324] This system not only allows for real-time monitoring of the baby's development but also provides appropriate parenting advice tailored to the parents' emotional state.

[0325] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0326] Step 1:

[0327] terminal

[0328] The device activates its camera and captures video of the baby in real time. For example, if the baby starts rolling over in the crib, it will record that movement. The captured video data is compressed in H.264 format. The compressed data is sent to the server as input using a secure communication protocol (SSL / TLS). The output is compressed video data.

[0329] Step 2:

[0330] server

[0331] The server analyzes the video data received from the terminal. First, it preprocesses the data using libraries such as OpenCV to remove noise. Using the preprocessed data as input, it identifies the baby's actions using image analysis algorithms such as ResNet and YOLO. Specifically, it detects when the baby is rolling over. The output is the detected action data.

[0332] Step 3:

[0333] server

[0334] The server inputs the behavioral data obtained in the previous step into a machine learning model. TensorFlow or PyTorch is used to evaluate the baby's developmental stage. Based on the extracted features, the neural network determines the developmental stage (e.g., "rolling over"). It also uses the behavioral history to predict future behavior and potential risk factors. The output is the developmental stage and the predicted result.

[0335] Step 4:

[0336] server

[0337] The server generates parenting advice based on the evaluation results obtained in step 3. For example, it might generate specific advice such as, "Your baby has started rolling over. Next, try practicing sitting up." The generated advice is formatted in a way that is easy for parents to understand (text messages or illustrated guides). The output is the formatted parenting advice.

[0338] Step 5:

[0339] terminal

[0340] The device captures the user's facial expressions and voice tone. Using the camera and microphone, it records the user's facial expressions and voice tone in real time. The captured facial and voice data are sent to the emotion engine as input. The output is the captured emotion data.

[0341] Step 6:

[0342] server

[0343] The server analyzes the user's emotional state based on data received by the emotion engine. It uses Amazon Rekognition or Microsoft Azure Cognitive Services to identify emotions such as stress, fatigue, and exhilaration. The emotion analysis results are used as input to customize parenting advice based on the user's emotions. For example, if the user is feeling stressed, the advice might be changed to "Ask your partner to help you practice together." The output is the customized advice.

[0344] Step 7:

[0345] server

[0346] The server sends customized parenting advice to the terminal. The output is the customized parenting advice.

[0347] Device & User

[0348] The device notifies the user of any advice it receives. The user checks the notification and views detailed parenting advice in the smartphone app. Specifically, when the user opens the app, specific instructions are displayed, such as, "Your baby has started rolling over. Next, try practicing sitting up with your partner." The output is the displayed advice.

[0349] (Application Example 2)

[0350] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0351] In modern society, a lack of support for childcare is a significant problem. First-time parents, in particular, often experience a great deal of anxiety and struggle to find appropriate advice and products. Furthermore, baby supply stores face the challenge of providing advice and product recommendations tailored to the individual developmental stages of each baby.

[0352] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing and transmitting images of a baby to the server, means for detecting the baby's behavior from the image data using an image analysis algorithm, means for evaluating the baby's growth stage based on the detected behavior using a machine learning model, means for generating childcare advice based on the evaluation results, means for transmitting and displaying the childcare advice on a terminal, and means for providing childcare advice to a terminal used by a customer in a baby goods store. This enables the provision of appropriate childcare advice in real time according to the growth stage of each baby, and allows parents to choose products with confidence in a baby goods store.

[0353] "Means for capturing images of a baby and sending them to a server" refers to a device that includes a camera for taking images of a baby and a communication interface for sending the captured video data to a central server via a network.

[0354] An "image analysis algorithm" is a set of computational procedures for identifying and extracting specific patterns or movements from video data, and is a software program used to detect a baby's movements.

[0355] A "machine learning model" is an algorithm that learns from data and uses that knowledge to analyze and evaluate new data. Specifically, it is used to evaluate a baby's developmental stage using behavioral data.

[0356] A "means for generating childcare advice" refers to a software program that automatically generates appropriate childcare guidance and suggestions based on evaluation results and provides them to parents.

[0357] "Means of sending and displaying on a device" refers to a device that has the function of sending the generated childcare advice to a user's smartphone, tablet, or other device and displaying it on its screen.

[0358] "Terminals used by customers in baby product stores" refers to information terminals installed in physical stores that sell baby products, which customers use to receive childcare advice.

[0359] "A baby's movements such as rolling over, sitting up, and crawling" refer to specific physical movements that occur during a baby's developmental stages and are important indicators for evaluating their developmental stage.

[0360] "Behavioral history" refers to a record of a baby's past actions and behaviors, and is data used to predict future behavior and potential risk factors.

[0361] "Potential risk factors" refer to dangerous situations or risks that a baby may face in the future, and serve as foundational data to provide preventative advice.

[0362] This invention is a system that monitors a baby's growth in real time and provides parents with appropriate childcare advice. The system functions through a series of processes including video capture, video analysis, machine learning evaluation, advice generation, and notification display. It can also provide childcare advice to customer terminals in baby product stores.

[0363] Data collection and transmission

[0364] The device is equipped with a camera that captures video of the baby in real time. The captured video data is compressed and sent to the server via a secure communication protocol (e.g., HTTPS).

[0365] Video analysis

[0366] The server processes the video data received from the terminal. As a preprocessing step, it removes noise and prepares the data for analysis. Then, it uses OpenCV or similar software to detect the baby's movements (rolling over, sitting up, crawling, etc.) from the video data.

[0367] Evaluation using machine learning

[0368] The server uses a machine learning model (e.g., built using TensorFlow or Keras) to evaluate the baby's developmental stage based on detected behavioral data. The model is input with data obtained during the feature extraction process to determine the developmental stage. In this process, future behaviors and potential risk factors are also predicted based on the behavioral history.

[0369] Advice generation

[0370] The server generates specific parenting advice based on the evaluation and prediction results. The generated advice is formatted in a way that is easy for parents to understand. For example, it might say, "Your baby has started rolling over. Next, try practicing sitting up."

[0371] Notification sent and displayed

[0372] The server sends the generated parenting advice to the terminal. The terminal notifies the user of the received advice and displays the specific content of the advice. In a baby goods store, the advice is displayed on the terminal used by the customer, allowing parents to choose appropriate products in the store.

[0373] Specific example

[0374] Let's take the example of a baby who has just started rolling over in their crib.

[0375] Device: The camera captures the baby rolling over and sends this video data to the server.

[0376] Server: The server analyzes the video data and detects when the baby rolls over. The detected data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0377] Device: Smartphones and tablets receive this parenting advice and notify the user. When the user opens the app, they can view the specific advice.

[0378] Examples of prompts to input into a generative AI model

[0379] Create a program for a system that captures a baby's behavior with a camera and provides parenting advice based on that behavior. The advice should include specific product suggestions for parents when they visit a baby goods store.

[0380] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0381] Step 1:

[0382] The device activates the camera and captures video of the baby in real time. The captured video data is compressed and sent to the server via a secure communication protocol (e.g., HTTPS). The input here is real-time video data, and the output is compressed video data.

[0383] Step 2:

[0384] The server processes the video data received from the terminal. First, it preprocesses the data by removing noise and preparing it for analysis. Then, it uses OpenCV or similar software to detect the baby's movements (e.g., rolling over, sitting up, crawling) from the video data. The input is compressed video data, and the output is the detected baby movement data.

[0385] Step 3:

[0386] The server uses a machine learning model based on detected behavioral data to evaluate the baby's developmental stage. Data obtained during the feature extraction process is input into the model to determine the developmental stage. Furthermore, during this process, future behaviors and potential risk factors are predicted based on the behavioral history. The input is the detected behavioral data, and the output is the developmental stage and predicted behaviors.

[0387] Step 4:

[0388] The server generates specific parenting advice based on the evaluation and prediction results. The generated advice is formatted in a way that is easy for parents to understand. For example, it might say, "Your baby has started rolling over. Next, try practicing sitting up." The input is the developmental stage and predicted behavior, and the output is the text of the parenting advice.

[0389] Step 5:

[0390] The server sends the generated childcare advice to the terminal. The input here is the text of the childcare advice, and the output is communication data.

[0391] Step 6:

[0392] The device notifies the user of received advice and displays the specific advice on the app screen. The input here is communication data from the server, and the output is the displayed text of the parenting advice. The user checks the notification and then views the detailed parenting advice within the app.

[0393] Step 7:

[0394] In baby product stores, parenting advice is ultimately displayed on a terminal used by the customer. This allows parents to choose appropriate products within the store. The input is the text of the parenting advice, and the output is the displayed product suggestions.

[0395] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0396] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0397] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0398] [Second Embodiment]

[0399] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0400] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0401] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0402] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0403] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0404] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0405] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0406] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0407] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0408] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0409] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0410] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0411] This invention is a system that monitors a baby's growth and provides parents with appropriate childcare advice. The following describes the system's program processing in natural language, along with specific examples.

[0412] 1. Data collection and transmission

[0413] terminal

[0414] A camera installed in the baby's room captures video of the baby in real time. The captured video data is compressed and sent to a server via a secure communication protocol.

[0415] 2. Video Analysis

[0416] server

[0417] The server processes the video data received from the terminal. First, it preprocesses the data by removing noise and preparing the video data for analysis. Next, it uses an image analysis algorithm (e.g., an object detection algorithm) to detect the baby's actions. Specifically, it identifies actions such as the baby rolling over, sitting up, and crawling.

[0418] 3. Evaluation using machine learning

[0419] server

[0420] The detected behavioral data is used to evaluate the baby's developmental stage. This is done using a pre-trained machine learning model. The data obtained in the feature extraction process is used as input to determine the baby's developmental stage (e.g., starting to roll over, starting to sit up, etc.). In addition, future behaviors and potential risk factors are predicted based on the behavioral history.

[0421] 4. Advice generation

[0422] server

[0423] Based on the evaluation results of a machine learning model, it generates advice and feedback on childcare. For example, it creates specific advice such as, "Now that your baby has started rolling over, it's a good idea to start practicing sitting up next." This advice is formatted in a way that is easy for parents to understand.

[0424] 5. Notifications and Display

[0425] Device & User

[0426] The server sends generated advice and feedback to the device (e.g., a smartphone app). The device notifies the user of the received message using a pop-up notification or sound. When the user opens the app, they can view detailed parenting advice.

[0427] Specific example

[0428] Let's take the example of a baby who has just started rolling over in their crib.

[0429] terminal

[0430] The camera captures the baby rolling over and sends this video data to the server.

[0431] server

[0432] The server analyzes the received video data and detects when the baby rolls over. This motion data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0433] Device & User

[0434] A smartphone app receives this parenting advice and notifies the user. When the user opens the app, they can view specific advice and take appropriate action according to their baby's development.

[0435] The above describes the detailed configuration for carrying out the invention. This system allows parents to monitor their baby's growth in real time and receive appropriate childcare advice.

[0436] The following describes the processing flow.

[0437] Step 1: Data Capture (Device)

[0438] The device activates the camera and captures the baby's image in real time. The camera continuously records the baby's movements and prepares to send the acquired video data to the server in real time.

[0439] Step 2: Data transmission (terminal)

[0440] The terminal compresses the captured video data and sends it to the server using a secure communication protocol. Data encryption is performed during this process to prevent interception by third parties.

[0441] Step 3: Data reception (server)

[0442] The server receives video data transmitted from the terminal. The received data is temporarily stored in a buffer and then passed on to the analysis process.

[0443] Step 4: Data preprocessing (server)

[0444] The server removes noise from the received video data and extracts only the necessary parts. This improves the accuracy of data analysis.

[0445] Step 5: Behavior detection (server)

[0446] The server uses image analysis algorithms to detect specific baby behaviors (e.g., rolling over, sitting up, crawling) from video data. The analyzed data is then categorized by type of behavior.

[0447] Step 6: Feature Extraction (Server)

[0448] The server extracts features from the detected behavioral data. For example, the number of times a cat turns over in its sleep or the duration of a specific behavior may be extracted as features.

[0449] Step 7: Evaluation and Prediction (Server)

[0450] The server applies a machine learning model and uses the extracted features to evaluate the baby's developmental stage. It also predicts likely future behaviors and potential risks based on the history of behavioral data.

[0451] Step 8: Advice generation (server)

[0452] The server generates parenting advice for parents based on the evaluation and prediction results. The advice is formatted in a way that is easy for parents to understand.

[0453] Step 9: Send notification (server)

[0454] The server generates childcare advice and sends it to the device. The sent notification arrives on the device, which is a smartphone or tablet.

[0455] Step 10: Notification display (device)

[0456] The device notifies the user of childcare advice it has received. The notification is delivered via a pop-up message or sound.

[0457] Step 11: View Details (User)

[0458] Users can check notifications on their devices and view detailed parenting advice. Through the app, users can also access past advice and prediction results.

[0459] The above outlines the specific processing steps of the program. This series of steps enables a system that monitors the baby's growth in real time and provides parents with appropriate childcare advice.

[0460] (Example 1)

[0461] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0462] Traditional childcare support systems often do not provide real-time monitoring of a baby's growth or offer childcare advice, and therefore do not adequately help parents understand their baby's development. Furthermore, they lacked the means to accurately detect a baby's behavior and generate appropriate advice based on their developmental stage.

[0463] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0464] In this invention, the server includes means for capturing and transmitting video of the baby to the server, means for preprocessing by applying a noise reduction filter, means for detecting the baby's actions from the video data using an object detection algorithm, means for extracting the baby's actions data, means for evaluating the baby's developmental stage based on the detected actions using a machine learning model, means for generating childcare advice based on the evaluation results, and means for transmitting, notifying, and displaying the childcare advice to a terminal. This enables parents to understand their baby's developmental status in real time and receive appropriate childcare advice.

[0465] A "noise reduction filter" is an algorithm used to remove random noise from video data.

[0466] An "object detection algorithm" is an algorithm used to detect specific objects within video data, such as the posture or movements of a baby.

[0467] "Behavioral data" refers to information about a baby's specific movements and postures, including the type of movement detected and a timestamp.

[0468] A "machine learning model" is an algorithm that learns behavioral patterns and action stages based on training data, and then uses that knowledge to evaluate and predict new data.

[0469] "Parenting advice" refers to specific suggestions and instructions for parents, generated based on the baby's developmental stage and behavioral data.

[0470] A "server" is a computing device that analyzes video data of babies, detects their behavior, evaluates their developmental stage using machine learning models, and generates parenting advice.

[0471] A "terminal" refers to a device, such as a camera or smartphone, used to capture images of the baby or to receive and display advice sent from a server.

[0472] This invention is a system for monitoring a baby's growth and providing parents with appropriate childcare advice. The system consists of a video capture device, communication means, a server, an analysis algorithm, a machine learning model, and a notification system.

[0473] Hardware and software configuration

[0474] terminal

[0475] The device includes a camera installed in the baby's room. This camera (e.g., an IP camera) captures video of the baby in real time. The video data is encoded in a compressed format such as H.264 and securely transmitted to the server using the SSL / TLS protocol.

[0476] server

[0477] The server receives and processes video data transmitted from the terminal. Specifically, it preprocesses the data by applying a noise reduction filter (e.g., a Gaussian filter), and then analyzes the baby's behavior using object detection algorithms such as OpenCV or YOLO. The baby's behavior data is stored in a database with timestamps.

[0478] The server also uses machine learning models (e.g., TensorFlow, PyTorch) to assess the baby's developmental stage. These models are trained on pre-collected datasets and can automatically evaluate and classify the baby's behavioral patterns. Based on the evaluation results, natural language generation (NLG) algorithms are used to generate specific parenting advice.

[0479] User notification system

[0480] The advice generated by the server is sent to the device (e.g., a smartphone app). The device notifies the user of the received advice via a pop-up notification or sound notification. When the user opens the app, detailed parenting advice is displayed.

[0481] Specific example

[0482] For example, consider a situation where a baby starts rolling over in their crib. A camera captures this movement and sends the video to a server. The server analyzes the video data and detects that the baby is rolling over. A machine learning model evaluates this movement as a developmental stage, specifically "rolling over." The server then generates advice such as, "Your baby has started rolling over. Next, try practicing sitting up," and sends this advice to the device. The device notifies the user of this advice, and when the user opens the app, more detailed advice is displayed.

[0483] Example of a prompt

[0484] An example of a prompt to input into a generative AI model is: "Describe a system that detects behavior from video data of a baby and generates childcare advice. Describe in detail the entire process from preprocessing of input data to notification of advice. Include the names of specific algorithms and software."

[0485] This invention allows parents to monitor their baby's development in real time and receive appropriate childcare advice.

[0486] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0487] Step 1:

[0488] terminal

[0489] A camera in the baby's room captures video. This camera captures the baby's video in real time, frame by frame, and encodes it in a compressed format such as H.264.

[0490] Input: Real-time video data

[0491] Output: Compressed video data

[0492] Step 2:

[0493] terminal

[0494] The video data transmitted from the camera is encrypted using the SSL / TLS protocol and securely sent to the server.

[0495] Input: Compressed video data

[0496] Output: Encrypted video data

[0497] Step 3:

[0498] server

[0499] The server decrypts the encrypted video data received from the terminal. Next, it applies a noise reduction filter (e.g., a Gaussian filter) to preprocess the data. This removes random noise from the video.

[0500] Input: Encrypted video data

[0501] Output: Preprocessed video data

[0502] Step 4:

[0503] server

[0504] Preprocessed video data is divided into frames, and object detection algorithms (e.g., OpenCV, YOLO) are used to detect the baby's actions. For example, actions such as rolling over, sitting up, and crawling are analyzed to identify the baby's posture and movements.

[0505] Input: Preprocessed video data

[0506] Output: Baby behavior data

[0507] Step 5:

[0508] server

[0509] The baby's behavioral data is extracted and stored in a database with timestamps. This data includes the baby's actions and the time they occurred.

[0510] Input: Baby's behavioral data

[0511] Output: Timestamped behavioral data

[0512] Step 6:

[0513] server

[0514] Using machine learning models (e.g., TensorFlow, PyTorch), we evaluate the developmental stage of a baby based on extracted behavioral data. For example, behavioral data is input as features to determine developmental stages such as "the baby has started rolling over." We also predict future behaviors and potential risk factors based on the behavioral history.

[0515] Input: Timestamped behavioral data

[0516] Output: Growth stage evaluation results

[0517] Step 7:

[0518] server

[0519] Based on the results of the developmental stage assessment, a natural language generation (NLG) algorithm is used to generate parenting advice. For example, it can create specific advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0520] Input: Growth stage evaluation results

[0521] Output: Childcare advice

[0522] Step 8:

[0523] server

[0524] The generated parenting advice is sent to the device (e.g., a smartphone app).

[0525] Input: Childcare advice

[0526] Output: Advice sent to the terminal

[0527] Step 9:

[0528] Device & User

[0529] The device notifies the user of received advice through pop-up notifications and sounds. When the user opens the app, detailed parenting advice is displayed.

[0530] Input: Advice sent to the device

[0531] Output: Advice notified to the user

[0532] The above outlines the specific processing steps of this system.

[0533] (Application Example 1)

[0534] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0535] Systems already exist that appropriately monitor a baby's growth and provide parents with accurate childcare advice. However, these systems typically only provide childcare advice and do not recommend or assist with the purchase of childcare-related products. As a result, there is a challenge in that parents have difficulty choosing appropriate products for their baby's development. Furthermore, there is a lack of integration with virtual stores, making it difficult to purchase childcare-related products in a centralized manner.

[0536] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0537] In this invention, the server includes means for capturing images of a baby and transmitting them to a data server; means for detecting the baby's behavior from the image data using an image analysis algorithm; means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model; means for generating childcare advice based on the evaluation results; means for transmitting and displaying the childcare advice on a user terminal; and means for coordinating with a virtual store to recommend childcare-related products. This makes it possible for parents to not only receive appropriate advice according to their baby's growth, but also to easily purchase products corresponding to that growth stage.

[0538] "Means for capturing images of a baby" refers to a device or method for capturing images of a baby's activities in real time and acquiring that image data.

[0539] "Means of sending to a data server" refers to a device or software that transfers captured video data to a data server via a network.

[0540] An "image analysis algorithm" is a set of computational procedures or programs used to analyze acquired video data and recognize specific actions or objects.

[0541] "Means for detecting a baby's behavior" refers to a device or method that uses an image analysis algorithm to identify a baby's specific movements from video data.

[0542] A "machine learning model" is a computational model that learns from large amounts of data and uses that data to make predictions and classifications about unknown data.

[0543] "Means for evaluating the developmental stage of an infant" refers to a device or method that uses a machine learning model to determine the progress of an infant's development based on their current behavioral data.

[0544] "Means for generating childcare advice" refers to a device or software that automatically creates specific advice and suggestions for parents based on the results of an assessment of the baby's developmental stage.

[0545] "Means of transmitting and displaying on a user terminal" refers to a device or software that transfers and displays the generated childcare advice on a device such as a smartphone or tablet used by the parent.

[0546] "Means of recommending childcare-related products in conjunction with virtual stores" refers to a device or method that links information with virtual stores that provide appropriate products online according to the developmental stage of a baby, and recommends products to parents along with childcare advice.

[0547] This invention is a system for monitoring a baby's growth and providing childcare advice based on that growth. The system includes a camera, a data server, an image analysis algorithm, a machine learning model, a notification system, and integration with a virtual store.

[0548] 1. Data collection and transmission

[0549] The server uses a camera to capture the baby's activity in real time. The captured video data is compressed and sent to the data server via a secure communication protocol. This process uses video processing libraries such as OpenCV.

[0550] 2. Video Analysis

[0551] The server preprocesses the received video data to remove noise. Next, it uses image analysis algorithms to identify the baby's actions. In this process, for example, object detection algorithms are used to analyze actions such as the baby rolling over, sitting up, and crawling.

[0552] 3. Evaluation of Growth Stages

[0553] The server uses a machine learning model to assess the baby's developmental stage based on detected behavioral data. This model utilizes a pre-trained dataset to determine the developmental stage based on the baby's current behavior. Furthermore, it analyzes the behavioral history to predict future behavior and potential risk factors.

[0554] 4. Generating parenting advice

[0555] The server generates parenting advice based on the evaluation results of the machine learning model. For example, it creates specific advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0556] 5. Notifications and Display

[0557] The server sends the generated advice to the user's device. The user's device (such as a smartphone or tablet) notifies the user of the received message with a pop-up notification or sound, and displays detailed childcare advice within the app.

[0558] 6. Collaboration with virtual stores

[0559] The server integrates information with a virtual store to recommend appropriate products based on the baby's developmental stage. For example, it might provide product recommendations such as, "Your baby has started rolling over. This cushion would be a good next step."

[0560] Specific example

[0561] Consider a scenario where a baby starts rolling over in their crib. A camera captures this movement and sends the video data to a server. The server analyzes the received video and detects the baby's rolling over. Based on this information, a machine learning model evaluates the developmental stage as "rolling over." Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up," and also makes product recommendations such as, "This cushion would be helpful." This information is then sent to the parent's smartphone.

[0562] Example of input prompt text for a generative AI model:

[0563] "Your baby has started rolling over. Next, try practicing sitting up."

[0564] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0565] Step 1:

[0566] The server uses a camera to capture the baby's activities in real time. The input is video data from the camera, and the output is compressed video data. Specifically, the camera films the baby's movements, acquires the video frame by frame, compresses it in JPEG format, and sends the prepared video data to the data server.

[0567] Step 2:

[0568] The server preprocesses the video data it receives. The input is compressed video data, and the output is video data formatted for analysis. Specifically, preprocessing is performed to remove video noise and adjust the resolution to an appropriate level. Video processing libraries such as OpenCV are used for this process.

[0569] Step 3:

[0570] The server identifies the baby's actions from pre-processed video data using an image analysis algorithm. The input is the pre-processed video data, and the output is the detection result of specific baby actions (e.g., rolling over, sitting up, crawling). Specifically, an object detection algorithm identifies the baby's movements, and the result is extracted as data.

[0571] Step 4:

[0572] The server uses a machine learning model to detect behavioral data and evaluate the baby's developmental stage. The input is the baby's behavioral data, and the output is an evaluation of the baby's developmental stage (e.g., rolling over, sitting up). Specifically, it uses a pre-trained dataset to determine the developmental stage based on the current behavioral data.

[0573] Step 5:

[0574] The server generates parenting advice based on the evaluation results of a machine learning model. The input is the evaluation result of the baby's developmental stage, and the output is specific parenting advice. For example, it might generate advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0575] Step 6:

[0576] The server generates childcare advice and sends it to the user's device. The input is the generated childcare advice, and the output is a notification to the user's device. Specifically, the childcare advice is sent as a push notification or message to the user's smartphone or tablet.

[0577] Step 7:

[0578] The server interacts with a virtual store to recommend childcare-related products. The input is the result of an assessment of the baby's developmental stage, and the output is information about the corresponding product. Specifically, it retrieves products suitable for the baby's growth (e.g., cushions, toys) from the virtual store and generates recommendations for parents.

[0579] Step 8:

[0580] The user's device receives notifications from the server and displays them to the parent. The input is childcare advice and product recommendations sent from the server, and the output is display and notification within the app. Specifically, when the user opens the app, detailed childcare advice and recommended products are displayed for review.

[0581] The above outlines the specific processing steps of the system for implementing the invention. This allows parents to monitor their baby's growth in real time and receive appropriate childcare advice and product recommendations.

[0582] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0583] This invention is a system for monitoring a baby's growth and providing parents with appropriate childcare advice. The system includes means for capturing video of the baby and transmitting it to a server, means for detecting the baby's behavior from the video data using an image analysis algorithm, means for evaluating the baby's growth stage based on the detected behavior using a machine learning model, means for generating childcare advice based on the evaluation results, and means for transmitting and displaying the generated childcare advice on a terminal. It also incorporates an emotion engine that recognizes the user's emotions and has a function to customize childcare advice based on the user's emotional state.

[0584] 1. Data collection and transmission

[0585] terminal

[0586] The device activates its camera and captures video of the baby in real time. The captured video data is compressed and sent to the server via a secure communication protocol.

[0587] 2. Video Analysis

[0588] server

[0589] The server processes the video data received from the terminal. First, it preprocesses the data by removing noise and preparing it for analysis. Next, it uses an image analysis algorithm to detect the baby's actions. For example, it identifies actions such as the baby rolling over or sitting up.

[0590] 3. Evaluation using machine learning

[0591] server

[0592] Based on the detected behavioral data, a machine learning model is used to evaluate the baby's developmental stage. The data obtained in the feature extraction process is input into the model to determine the developmental stage. In addition, future behaviors and potential risk factors are predicted based on the behavioral history.

[0593] 4. Advice generation

[0594] server

[0595] Based on the evaluation and prediction results, specific parenting advice is generated. The generated advice is formatted in a way that is easy for parents to understand.

[0596] 5. Emotion Recognition and Customization

[0597] terminal

[0598] The device is also equipped with a camera and microphone to capture the user's facial expressions and voice tone. This data is sent to an emotion engine to analyze the user's emotional state.

[0599] server

[0600] The emotion engine identifies the user's emotions (e.g., stress, fatigue, exhilaration, etc.) and customizes parenting advice based on the user's emotional state.

[0601] 6. Sending and displaying notifications

[0602] server

[0603] The server sends customized parenting advice to the device.

[0604] Device & User

[0605] The device notifies the user of any advice it receives. The user checks the notification and can view detailed parenting advice within the app.

[0606] Specific example

[0607] Let's take the example of a baby who has just started rolling over in their crib.

[0608] terminal

[0609] The camera captures the baby rolling over and sends this video data to the server.

[0610] server

[0611] The server analyzes the video data and detects when the baby rolls over. The detected data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0612] terminal

[0613] The device captures the user's facial expressions and tone of voice, and an emotion engine analyzes whether the user is experiencing stress.

[0614] server

[0615] Based on this emotional data, the server generates customized advice such as, "Let's have your partner help you practice together."

[0616] Device & User

[0617] A smartphone app receives this parenting advice and notifies the user. When the user opens the app, they can view customized, specific advice.

[0618] The above describes the detailed configuration for carrying out the invention. This system makes it possible not only to monitor the baby's growth in real time, but also to provide appropriate childcare advice tailored to the parents' emotional state.

[0619] The following describes the processing flow.

[0620] Step 1: Data Capture (Device)

[0621] The device activates the camera and captures the baby's image in real time. The camera continuously records the baby's movements and prepares to send the captured video data to the server in real time.

[0622] Step 2: Data transmission (terminal)

[0623] The device compresses the captured video data and sends it to the server using a secure communication protocol. Encryption technology is also used to ensure data security.

[0624] Step 3: Data reception (server)

[0625] The server receives video data transmitted from the terminal. The received data is temporarily stored in a buffer and then passed on to the analysis process.

[0626] Step 4: Data preprocessing (server)

[0627] The server removes noise from the received video data and prepares it for easier analysis. The pre-processed data is then used in the next analysis step.

[0628] Step 5: Behavior detection (server)

[0629] The server uses image analysis algorithms to detect specific baby behaviors (e.g., rolling over, sitting up, crawling) from video data. The detected behavior data is then categorized by type of behavior.

[0630] Step 6: Feature Extraction (Server)

[0631] The server extracts features from the detected behavioral data. For example, the number of times a cat turns over in its sleep or the duration of a specific behavior may be extracted as features.

[0632] Step 7: Evaluation and Prediction (Server)

[0633] The server applies a machine learning model and uses extracted feature data to assess the baby's developmental stage. Based on behavioral history, it also predicts potential next behaviors and risk factors.

[0634] Step 8: Advice generation (server)

[0635] The server generates specific parenting advice based on the evaluation and prediction results. The generated advice is formatted in a way that is easy for parents to understand.

[0636] Step 9: Emotional Data Capture (Device)

[0637] The device uses its camera and microphone to capture the user's facial expressions and voice tone. This data is then prepared to be sent to the emotion engine.

[0638] Step 10: Emotional Data Analysis (Server)

[0639] The server uses an emotion engine to analyze the user's emotional data. From the analyzed data, it identifies the user's emotional state (e.g., stress, exhilaration, fatigue, etc.).

[0640] Step 11: Customizing parenting advice (server)

[0641] The server customizes parenting advice based on the user's emotional state, as identified by the emotion engine. For example, if the user is feeling stressed, advice such as "get your partner to help you" will be generated.

[0642] Step 12: Send notification (server)

[0643] The server sends customized parenting advice to the device. The sent notifications arrive on the device, such as a smartphone or tablet.

[0644] Step 13: Notification display (device)

[0645] The device notifies the user of childcare advice it has received. Notifications are delivered via pop-up messages and sounds.

[0646] Step 14: View Details (User)

[0647] Users can check notifications on their devices and view detailed parenting advice. Through the app, users can also access past advice and prediction results.

[0648] The above outlines the specific processing steps of the program. This sequence of steps enables a system that monitors the baby's growth in real time and provides appropriate childcare advice tailored to the user's emotional state.

[0649] (Example 2)

[0650] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0651] Conventional infant growth monitoring systems can identify an infant's behavior, but they do not take into account the parent's emotional state, meaning the parenting advice provided may not be appropriate to the parent's mental state. Furthermore, they lacked the ability to predict future behavior and potential risks, in addition to evaluating the infant's developmental stage. This invention aims to solve these problems.

[0652] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0653] In this invention, the server includes means for capturing and transmitting images of the baby to the server; means for detecting the baby's behavior from the image data using an image analysis algorithm; means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model; means for generating childcare advice based on the evaluation results; means for transmitting and displaying the childcare advice on a terminal; means for capturing the user's facial expressions and tone of voice and analyzing their emotional state; and means for customizing the childcare advice based on the emotional state. This makes it possible to accurately monitor the baby's development in real time and provide appropriate childcare advice that takes into account the parents' mental state.

[0654] "Means for capturing video" refers to a camera and its control device that captures video of the baby in real time.

[0655] "Means of sending to the server" refers to a communication device that compresses the captured video data and sends it to a remote server via a secure communication protocol.

[0656] An "image analysis algorithm" refers to a program and method for detecting a baby's behavior based on received video data. Specifically, it includes methods such as object detection and posture estimation.

[0657] A "machine learning model" refers to a data model used to evaluate a baby's developmental stage based on past data. A specific example is a neural network.

[0658] "Means for generating parenting advice" refers to a program that creates appropriate parenting methods and advice for parents based on evaluation results from machine learning models.

[0659] "Means of transmitting and displaying on a device" refers to communication devices and display devices used to transmit and display childcare advice on a device used by a parent (e.g., a smartphone or tablet).

[0660] "Means of capturing facial expressions and voice tone" refers to a camera and microphone, as well as their control device, that record the parent's facial expressions and voice tone in real time in order to understand the parent's emotional state.

[0661] "Means for analyzing emotional states" refers to programs and methods for analyzing a parent's emotions from captured facial expressions and voice tones. Specific examples include emotion recognition algorithms.

[0662] "Methods for customizing parenting advice" refers to a program that adjusts and modifies parenting advice to a situation-appropriate format based on the analyzed emotional state of the parent.

[0663] This invention is a system for monitoring a baby's growth and providing parents with appropriate childcare advice. The system includes means for capturing video of the baby and transmitting it to a server, means for detecting the baby's behavior from the video data using an image analysis algorithm, means for evaluating the baby's growth stage based on the detected behavior using a machine learning model, means for generating childcare advice based on the evaluation results, and means for transmitting and displaying the generated childcare advice on a terminal. It also incorporates an emotion engine that recognizes the user's emotions and has a function to customize childcare advice based on the user's emotional state.

[0664] Data collection and transmission

[0665] terminal

[0666] The device activates its camera and captures video of the baby in real time. For example, if the baby starts rolling over in the crib, it will record that movement. The captured video data is compressed using a standard compression format (e.g., H.264) and sent to the server via a secure communication protocol (e.g., SSL / TLS).

[0667] Video analysis

[0668] server

[0669] The server processes the video data received from the terminal. First, it preprocesses the data using libraries such as OpenCV to remove noise and prepare it for analysis. Next, it detects the baby's actions using image analysis algorithms such as ResNet or YOLO. For example, it detects when the baby rolls over.

[0670] Evaluation using machine learning

[0671] server

[0672] Based on the detected behavioral data, machine learning models such as TensorFlow and PyTorch are used to evaluate the baby's developmental stage. The data obtained in the feature extraction process is input into the model to determine the developmental stage (e.g., the stage of rolling over). In addition, future behaviors and potential risk factors are predicted based on the behavioral history.

[0673] Advice generation

[0674] server

[0675] Based on the evaluation and prediction results, it generates specific parenting advice. For example, it might generate advice such as, "Your baby has started rolling over. Next, try practicing sitting up." The generated advice is formatted in a way that is easy for parents to understand (e.g., text messages or illustrated guides).

[0676] Emotion recognition and customization

[0677] terminal

[0678] The device is also equipped with a camera and microphone to capture the user's facial expressions and voice tone. This data is sent to an emotion engine to analyze the user's emotional state. This process utilizes tools such as Amazon Rekognition and Microsoft Azure's Cognitive Services.

[0679] server

[0680] The emotion engine identifies the user's emotions (e.g., stress, fatigue, exhilaration) and customizes parenting advice based on the user's emotional state. For example, if the user is feeling stressed, advice such as "Ask your partner to help you practice together" might be added.

[0681] Notification sent and displayed

[0682] server

[0683] The server sends customized parenting advice to the device.

[0684] Device & User

[0685] The device notifies the user of any advice it receives. The user checks the notification and views detailed parenting advice in the smartphone app. For example, when the app is opened, specific instructions such as "Your baby has started rolling over. Next, try practicing sitting up with your partner" will be displayed.

[0686] Specific example

[0687] Let's take the example of a baby who has just started rolling over in their crib.

[0688] terminal

[0689] The camera captures the baby rolling over and sends this video data to the server.

[0690] server

[0691] The server analyzes the video data and detects when the baby rolls over. The detected data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0692] terminal

[0693] The device captures the user's facial expressions and tone of voice, and an emotion engine analyzes whether the user is experiencing stress.

[0694] server

[0695] Based on this emotional data, the server generates customized advice such as, "Let's have your partner help you practice together."

[0696] Device & User

[0697] A smartphone app receives this parenting advice and notifies the user. When the user opens the app, they can view customized, specific advice.

[0698] Example of a prompt

[0699] Please describe how a program works to generate parenting advice for when a baby starts rolling over and display it to stressed parents.

[0700] This system not only allows for real-time monitoring of the baby's development but also provides appropriate parenting advice tailored to the parents' emotional state.

[0701] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0702] Step 1:

[0703] terminal

[0704] The device activates its camera and captures video of the baby in real time. For example, if the baby starts rolling over in the crib, it will record that movement. The captured video data is compressed in H.264 format. The compressed data is sent to the server as input using a secure communication protocol (SSL / TLS). The output is compressed video data.

[0705] Step 2:

[0706] server

[0707] The server analyzes the video data received from the terminal. First, it preprocesses the data using libraries such as OpenCV to remove noise. Using the preprocessed data as input, it identifies the baby's actions using image analysis algorithms such as ResNet and YOLO. Specifically, it detects when the baby is rolling over. The output is the detected action data.

[0708] Step 3:

[0709] server

[0710] The server inputs the behavioral data obtained in the previous step into a machine learning model. TensorFlow or PyTorch is used to evaluate the baby's developmental stage. Based on the extracted features, the neural network determines the developmental stage (e.g., "rolling over"). It also uses the behavioral history to predict future behavior and potential risk factors. The output is the developmental stage and the predicted result.

[0711] Step 4:

[0712] server

[0713] The server generates parenting advice based on the evaluation results obtained in step 3. For example, it might generate specific advice such as, "Your baby has started rolling over. Next, try practicing sitting up." The generated advice is formatted in a way that is easy for parents to understand (text messages or illustrated guides). The output is the formatted parenting advice.

[0714] Step 5:

[0715] terminal

[0716] The device captures the user's facial expressions and voice tone. Using the camera and microphone, it records the user's facial expressions and voice tone in real time. The captured facial and voice data are sent to the emotion engine as input. The output is the captured emotion data.

[0717] Step 6:

[0718] server

[0719] The server analyzes the user's emotional state based on data received by the emotion engine. It uses Amazon Rekognition or Microsoft Azure Cognitive Services to identify emotions such as stress, fatigue, and exhilaration. The emotion analysis results are used as input to customize parenting advice based on the user's emotions. For example, if the user is feeling stressed, the advice might be changed to "Ask your partner to help you practice together." The output is the customized advice.

[0720] Step 7:

[0721] server

[0722] The server sends customized parenting advice to the terminal. The output is the customized parenting advice.

[0723] Device & User

[0724] The device notifies the user of any advice it receives. The user checks the notification and views detailed parenting advice in the smartphone app. Specifically, when the user opens the app, specific instructions are displayed, such as, "Your baby has started rolling over. Next, try practicing sitting up with your partner." The output is the displayed advice.

[0725] (Application Example 2)

[0726] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0727] In modern society, a lack of support for childcare is a significant problem. First-time parents, in particular, often experience a great deal of anxiety and struggle to find appropriate advice and products. Furthermore, baby supply stores face the challenge of providing advice and product recommendations tailored to the individual developmental stages of each baby.

[0728] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing and transmitting images of a baby to the server, means for detecting the baby's behavior from the image data using an image analysis algorithm, means for evaluating the baby's growth stage based on the detected behavior using a machine learning model, means for generating childcare advice based on the evaluation results, means for transmitting and displaying the childcare advice on a terminal, and means for providing childcare advice to a terminal used by a customer in a baby goods store. This enables the provision of appropriate childcare advice in real time according to the growth stage of each baby, and allows parents to choose products with confidence in a baby goods store.

[0729] "Means for capturing images of a baby and sending them to a server" refers to a device that includes a camera for taking images of a baby and a communication interface for sending the captured video data to a central server via a network.

[0730] An "image analysis algorithm" is a set of computational procedures for identifying and extracting specific patterns or movements from video data, and is a software program used to detect a baby's movements.

[0731] A "machine learning model" is an algorithm that learns from data and uses that knowledge to analyze and evaluate new data. Specifically, it is used to evaluate a baby's developmental stage using behavioral data.

[0732] A "means for generating childcare advice" refers to a software program that automatically generates appropriate childcare guidance and suggestions based on evaluation results and provides them to parents.

[0733] "Means of sending and displaying on a device" refers to a device that has the function of sending the generated childcare advice to a user's smartphone, tablet, or other device and displaying it on its screen.

[0734] "Terminals used by customers in baby product stores" refers to information terminals installed in physical stores that sell baby products, which customers use to receive childcare advice.

[0735] "A baby's movements such as rolling over, sitting up, and crawling" refer to specific physical movements that occur during a baby's developmental stages and are important indicators for evaluating their developmental stage.

[0736] "Behavioral history" refers to a record of a baby's past actions and behaviors, and is data used to predict future behavior and potential risk factors.

[0737] "Potential risk factors" refer to dangerous situations or risks that a baby may face in the future, and serve as foundational data to provide preventative advice.

[0738] This invention is a system that monitors a baby's growth in real time and provides parents with appropriate childcare advice. The system functions through a series of processes including video capture, video analysis, machine learning evaluation, advice generation, and notification display. It can also provide childcare advice to customer terminals in baby product stores.

[0739] Data collection and transmission

[0740] The device is equipped with a camera that captures video of the baby in real time. The captured video data is compressed and sent to the server via a secure communication protocol (e.g., HTTPS).

[0741] Video analysis

[0742] The server processes the video data received from the terminal. As a preprocessing step, it removes noise and prepares the data for analysis. Then, it uses OpenCV or similar software to detect the baby's movements (rolling over, sitting up, crawling, etc.) from the video data.

[0743] Evaluation using machine learning

[0744] The server uses a machine learning model (e.g., built using TensorFlow or Keras) to evaluate the baby's developmental stage based on detected behavioral data. The model is input with data obtained during the feature extraction process to determine the developmental stage. In this process, future behaviors and potential risk factors are also predicted based on the behavioral history.

[0745] Advice generation

[0746] The server generates specific parenting advice based on the evaluation and prediction results. The generated advice is formatted in a way that is easy for parents to understand. For example, it might say, "Your baby has started rolling over. Next, try practicing sitting up."

[0747] Notification sent and displayed

[0748] The server sends the generated parenting advice to the terminal. The terminal notifies the user of the received advice and displays the specific content of the advice. In a baby goods store, the advice is displayed on the terminal used by the customer, allowing parents to choose appropriate products in the store.

[0749] Specific example

[0750] Let's take the example of a baby who has just started rolling over in their crib.

[0751] Device: The camera captures the baby rolling over and sends this video data to the server.

[0752] Server: The server analyzes the video data and detects when the baby rolls over. The detected data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0753] Device: Smartphones and tablets receive this parenting advice and notify the user. When the user opens the app, they can view the specific advice.

[0754] Examples of prompts to input into a generative AI model

[0755] Please create a program for a system that captures a baby's behavior with a camera and provides parenting advice based on that behavior. The advice should include specific product suggestions for parents when they visit a baby goods store.

[0756] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0757] Step 1:

[0758] The device activates the camera and captures video of the baby in real time. The captured video data is compressed and sent to the server via a secure communication protocol (e.g., HTTPS). The input here is real-time video data, and the output is compressed video data.

[0759] Step 2:

[0760] The server processes the video data received from the terminal. First, it preprocesses the data by removing noise and preparing it for analysis. Then, it uses OpenCV or similar software to detect the baby's movements (e.g., rolling over, sitting up, crawling) from the video data. The input is compressed video data, and the output is the detected baby movement data.

[0761] Step 3:

[0762] The server uses a machine learning model based on detected behavioral data to evaluate the baby's developmental stage. Data obtained during the feature extraction process is input into the model to determine the developmental stage. Furthermore, during this process, future behaviors and potential risk factors are predicted based on the behavioral history. The input is the detected behavioral data, and the output is the developmental stage and predicted behaviors.

[0763] Step 4:

[0764] The server generates specific parenting advice based on the evaluation and prediction results. The generated advice is formatted in a way that is easy for parents to understand. For example, it might say, "Your baby has started rolling over. Next, try practicing sitting up." The input is the developmental stage and predicted behavior, and the output is the text of the parenting advice.

[0765] Step 5:

[0766] The server sends the generated childcare advice to the terminal. The input here is the text of the childcare advice, and the output is communication data.

[0767] Step 6:

[0768] The device notifies the user of received advice and displays the specific advice on the app screen. The input here is communication data from the server, and the output is the displayed text of the parenting advice. The user checks the notification and then views the detailed parenting advice within the app.

[0769] Step 7:

[0770] In baby product stores, parenting advice is ultimately displayed on a terminal used by the customer. This allows parents to choose appropriate products within the store. The input is the text of the parenting advice, and the output is the displayed product suggestions.

[0771] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0772] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0773] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0774] [Third Embodiment]

[0775] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0776] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0777] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0778] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0779] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0780] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0781] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0782] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0783] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0784] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0785] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0786] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0787] This invention is a system that monitors a baby's growth and provides parents with appropriate childcare advice. The following describes the system's program processing in natural language, along with specific examples.

[0788] 1. Data collection and transmission

[0789] terminal

[0790] A camera installed in the baby's room captures video of the baby in real time. The captured video data is compressed and sent to a server via a secure communication protocol.

[0791] 2. Video Analysis

[0792] server

[0793] The server processes the video data received from the terminal. First, it preprocesses the data by removing noise and preparing the video data for analysis. Next, it uses an image analysis algorithm (e.g., an object detection algorithm) to detect the baby's actions. Specifically, it identifies actions such as the baby rolling over, sitting up, and crawling.

[0794] 3. Evaluation using machine learning

[0795] server

[0796] The detected behavioral data is used to evaluate the baby's developmental stage. This is done using a pre-trained machine learning model. The data obtained in the feature extraction process is used as input to determine the baby's developmental stage (e.g., starting to roll over, starting to sit up, etc.). In addition, future behaviors and potential risk factors are predicted based on the behavioral history.

[0797] 4. Advice generation

[0798] server

[0799] Based on the evaluation results of a machine learning model, it generates advice and feedback on childcare. For example, it creates specific advice such as, "Now that your baby has started rolling over, it's a good idea to start practicing sitting up next." This advice is formatted in a way that is easy for parents to understand.

[0800] 5. Notifications and Display

[0801] Device & User

[0802] The server sends generated advice and feedback to the device (e.g., a smartphone app). The device notifies the user of the received message using a pop-up notification or sound. When the user opens the app, they can view detailed parenting advice.

[0803] Specific example

[0804] Let's take the example of a baby who has just started rolling over in their crib.

[0805] terminal

[0806] The camera captures the baby rolling over and sends this video data to the server.

[0807] server

[0808] The server analyzes the received video data and detects when the baby rolls over. This motion data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0809] Device & User

[0810] A smartphone app receives this parenting advice and notifies the user. When the user opens the app, they can view specific advice and take appropriate action according to their baby's development.

[0811] The above describes the detailed configuration for carrying out the invention. This system allows parents to monitor their baby's growth in real time and receive appropriate childcare advice.

[0812] The following describes the processing flow.

[0813] Step 1: Data Capture (Device)

[0814] The device activates the camera and captures the baby's image in real time. The camera continuously records the baby's movements and prepares to send the acquired video data to the server in real time.

[0815] Step 2: Data transmission (terminal)

[0816] The terminal compresses the captured video data and sends it to the server using a secure communication protocol. Data encryption is performed during this process to prevent interception by third parties.

[0817] Step 3: Data reception (server)

[0818] The server receives video data transmitted from the terminal. The received data is temporarily stored in a buffer and then passed on to the analysis process.

[0819] Step 4: Data preprocessing (server)

[0820] The server removes noise from the received video data and extracts only the necessary parts. This improves the accuracy of data analysis.

[0821] Step 5: Behavior detection (server)

[0822] The server uses image analysis algorithms to detect specific baby behaviors (e.g., rolling over, sitting up, crawling) from video data. The analyzed data is then categorized by type of behavior.

[0823] Step 6: Feature Extraction (Server)

[0824] The server extracts features from the detected behavioral data. For example, the number of times a cat turns over in its sleep or the duration of a specific behavior may be extracted as features.

[0825] Step 7: Evaluation and Prediction (Server)

[0826] The server applies a machine learning model and uses the extracted features to evaluate the baby's developmental stage. It also predicts likely future behaviors and potential risks based on the history of behavioral data.

[0827] Step 8: Advice generation (server)

[0828] The server generates parenting advice for parents based on the evaluation and prediction results. The advice is formatted in a way that is easy for parents to understand.

[0829] Step 9: Send notification (server)

[0830] The server generates childcare advice and sends it to the device. The sent notification arrives on the device, which is a smartphone or tablet.

[0831] Step 10: Notification display (device)

[0832] The device notifies the user of childcare advice it has received. The notification is delivered via a pop-up message or sound.

[0833] Step 11: View Details (User)

[0834] Users can check notifications on their devices and view detailed parenting advice. Through the app, users can also access past advice and prediction results.

[0835] The above outlines the specific processing steps of the program. This series of steps enables a system that monitors the baby's growth in real time and provides parents with appropriate childcare advice.

[0836] (Example 1)

[0837] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0838] Traditional childcare support systems often do not provide real-time monitoring of a baby's growth or offer childcare advice, and therefore do not adequately help parents understand their baby's development. Furthermore, they lacked the means to accurately detect a baby's behavior and generate appropriate advice based on their developmental stage.

[0839] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0840] In this invention, the server includes means for capturing and transmitting video of the baby to the server, means for preprocessing by applying a noise reduction filter, means for detecting the baby's actions from the video data using an object detection algorithm, means for extracting the baby's actions data, means for evaluating the baby's developmental stage based on the detected actions using a machine learning model, means for generating childcare advice based on the evaluation results, and means for transmitting, notifying, and displaying the childcare advice to a terminal. This enables parents to understand their baby's developmental status in real time and receive appropriate childcare advice.

[0841] A "noise reduction filter" is an algorithm used to remove random noise from video data.

[0842] An "object detection algorithm" is an algorithm used to detect specific objects within video data, such as the posture or movements of a baby.

[0843] "Behavioral data" refers to information about a baby's specific movements and postures, including the type of movement detected and a timestamp.

[0844] A "machine learning model" is an algorithm that learns behavioral patterns and action stages based on training data, and then uses that knowledge to evaluate and predict new data.

[0845] "Parenting advice" refers to specific suggestions and instructions for parents, generated based on the baby's developmental stage and behavioral data.

[0846] A "server" is a computing device that analyzes video data of babies, detects their behavior, evaluates their developmental stage using machine learning models, and generates parenting advice.

[0847] A "terminal" refers to a device, such as a camera or smartphone, used to capture images of the baby or to receive and display advice sent from a server.

[0848] This invention is a system for monitoring a baby's growth and providing parents with appropriate childcare advice. The system consists of a video capture device, communication means, a server, an analysis algorithm, a machine learning model, and a notification system.

[0849] Hardware and software configuration

[0850] terminal

[0851] The device includes a camera installed in the baby's room. This camera (e.g., an IP camera) captures video of the baby in real time. The video data is encoded in a compressed format such as H.264 and securely transmitted to the server using the SSL / TLS protocol.

[0852] server

[0853] The server receives and processes video data transmitted from the terminal. Specifically, it preprocesses the data by applying a noise reduction filter (e.g., a Gaussian filter), and then analyzes the baby's behavior using object detection algorithms such as OpenCV or YOLO. The baby's behavior data is stored in a database with timestamps.

[0854] The server also uses machine learning models (e.g., TensorFlow, PyTorch) to assess the baby's developmental stage. These models are trained on pre-collected datasets and can automatically evaluate and classify the baby's behavioral patterns. Based on the evaluation results, natural language generation (NLG) algorithms are used to generate specific parenting advice.

[0855] User notification system

[0856] The advice generated by the server is sent to the device (e.g., a smartphone app). The device notifies the user of the received advice via a pop-up notification or sound notification. When the user opens the app, detailed parenting advice is displayed.

[0857] Specific example

[0858] For example, consider a situation where a baby starts rolling over in their crib. A camera captures this movement and sends the video to a server. The server analyzes the video data and detects that the baby is rolling over. A machine learning model evaluates this movement as a developmental stage, specifically "rolling over." The server then generates advice such as, "Your baby has started rolling over. Next, try practicing sitting up," and sends this advice to the device. The device notifies the user of this advice, and when the user opens the app, more detailed advice is displayed.

[0859] Example of a prompt

[0860] An example of a prompt to input into a generative AI model is: "Describe a system that detects behavior from video data of a baby and generates childcare advice. Describe in detail the entire process from preprocessing of input data to notification of advice. Include the names of specific algorithms and software."

[0861] This invention allows parents to monitor their baby's development in real time and receive appropriate childcare advice.

[0862] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0863] Step 1:

[0864] terminal

[0865] A camera in the baby's room captures video. This camera captures the baby's video in real time, frame by frame, and encodes it in a compressed format such as H.264.

[0866] Input: Real-time video data

[0867] Output: Compressed video data

[0868] Step 2:

[0869] terminal

[0870] The video data transmitted from the camera is encrypted using the SSL / TLS protocol and securely sent to the server.

[0871] Input: Compressed video data

[0872] Output: Encrypted video data

[0873] Step 3:

[0874] server

[0875] The server decrypts the encrypted video data received from the terminal. Next, it applies a noise reduction filter (e.g., a Gaussian filter) to preprocess the data. This removes random noise from the video.

[0876] Input: Encrypted video data

[0877] Output: Preprocessed video data

[0878] Step 4:

[0879] server

[0880] Preprocessed video data is divided into frames, and object detection algorithms (e.g., OpenCV, YOLO) are used to detect the baby's actions. For example, actions such as rolling over, sitting up, and crawling are analyzed to identify the baby's posture and movements.

[0881] Input: Preprocessed video data

[0882] Output: Baby behavior data

[0883] Step 5:

[0884] server

[0885] The baby's behavioral data is extracted and stored in a database with timestamps. This data includes the baby's actions and the time they occurred.

[0886] Input: Baby's behavioral data

[0887] Output: Timestamped behavioral data

[0888] Step 6:

[0889] server

[0890] Using machine learning models (e.g., TensorFlow, PyTorch), we evaluate the developmental stage of a baby based on extracted behavioral data. For example, behavioral data is input as features to determine developmental stages such as "the baby has started rolling over." We also predict future behaviors and potential risk factors based on the behavioral history.

[0891] Input: Timestamped behavioral data

[0892] Output: Growth stage evaluation results

[0893] Step 7:

[0894] server

[0895] Based on the results of the developmental stage assessment, a natural language generation (NLG) algorithm is used to generate parenting advice. For example, it can create specific advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0896] Input: Growth stage evaluation results

[0897] Output: Childcare advice

[0898] Step 8:

[0899] server

[0900] The generated parenting advice is sent to the device (e.g., a smartphone app).

[0901] Input: Childcare advice

[0902] Output: Advice sent to the terminal

[0903] Step 9:

[0904] Device & User

[0905] The device notifies the user of received advice through pop-up notifications and sounds. When the user opens the app, detailed parenting advice is displayed.

[0906] Input: Advice sent to the device

[0907] Output: Advice notified to the user

[0908] The above outlines the specific processing steps of this system.

[0909] (Application Example 1)

[0910] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0911] Systems already exist that appropriately monitor a baby's growth and provide parents with accurate childcare advice. However, these systems typically only provide childcare advice and do not recommend or assist with the purchase of childcare-related products. As a result, there is a challenge in that parents have difficulty choosing appropriate products for their baby's development. Furthermore, there is a lack of integration with virtual stores, making it difficult to purchase childcare-related products in a centralized manner.

[0912] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0913] In this invention, the server includes means for capturing images of a baby and transmitting them to a data server; means for detecting the baby's behavior from the image data using an image analysis algorithm; means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model; means for generating childcare advice based on the evaluation results; means for transmitting and displaying the childcare advice on a user terminal; and means for coordinating with a virtual store to recommend childcare-related products. This makes it possible for parents to not only receive appropriate advice according to their baby's growth, but also to easily purchase products corresponding to that growth stage.

[0914] "Means for capturing images of a baby" refers to a device or method for capturing images of a baby's activities in real time and acquiring that image data.

[0915] "Means of sending to a data server" refers to a device or software that transfers captured video data to a data server via a network.

[0916] An "image analysis algorithm" is a set of computational procedures or programs used to analyze acquired video data and recognize specific actions or objects.

[0917] "Means for detecting a baby's behavior" refers to a device or method that uses an image analysis algorithm to identify a baby's specific movements from video data.

[0918] A "machine learning model" is a computational model that learns from large amounts of data and uses that data to make predictions and classifications about unknown data.

[0919] "Means for evaluating the developmental stage of an infant" refers to a device or method that uses a machine learning model to determine the progress of an infant's development based on their current behavioral data.

[0920] "Means for generating childcare advice" refers to a device or software that automatically creates specific advice and suggestions for parents based on the results of an assessment of the baby's developmental stage.

[0921] "Means of transmitting and displaying on a user terminal" refers to a device or software that transfers and displays the generated childcare advice on a device such as a smartphone or tablet used by the parent.

[0922] "Means of recommending childcare-related products in conjunction with virtual stores" refers to a device or method that links information with virtual stores that provide appropriate products online according to the developmental stage of a baby, and recommends products to parents along with childcare advice.

[0923] This invention is a system for monitoring a baby's growth and providing childcare advice based on that growth. The system includes a camera, a data server, an image analysis algorithm, a machine learning model, a notification system, and integration with a virtual store.

[0924] 1. Data collection and transmission

[0925] The server uses a camera to capture the baby's activity in real time. The captured video data is compressed and sent to the data server via a secure communication protocol. This process uses video processing libraries such as OpenCV.

[0926] 2. Video Analysis

[0927] The server preprocesses the received video data to remove noise. Next, it uses image analysis algorithms to identify the baby's actions. In this process, for example, object detection algorithms are used to analyze actions such as the baby rolling over, sitting up, and crawling.

[0928] 3. Evaluation of Growth Stages

[0929] The server uses a machine learning model to assess the baby's developmental stage based on detected behavioral data. This model utilizes a pre-trained dataset to determine the developmental stage based on the baby's current behavior. Furthermore, it analyzes the behavioral history to predict future behavior and potential risk factors.

[0930] 4. Generating parenting advice

[0931] The server generates parenting advice based on the evaluation results of the machine learning model. For example, it creates specific advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0932] 5. Notifications and Display

[0933] The server sends the generated advice to the user's device. The user's device (such as a smartphone or tablet) notifies the user of the received message with a pop-up notification or sound, and displays detailed childcare advice within the app.

[0934] 6. Collaboration with virtual stores

[0935] The server integrates information with a virtual store to recommend appropriate products based on the baby's developmental stage. For example, it might provide product recommendations such as, "Your baby has started rolling over. This cushion would be a good next step."

[0936] Specific example

[0937] Consider a scenario where a baby starts rolling over in their crib. A camera captures this movement and sends the video data to a server. The server analyzes the received video and detects the baby's rolling over. Based on this information, a machine learning model evaluates the developmental stage as "rolling over." Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up," and also makes product recommendations such as, "This cushion would be helpful." This information is then sent to the parent's smartphone.

[0938] Example of input prompt text for a generative AI model:

[0939] "Your baby has started rolling over. Next, try practicing sitting up."

[0940] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0941] Step 1:

[0942] The server uses a camera to capture the baby's activities in real time. The input is video data from the camera, and the output is compressed video data. Specifically, the camera films the baby's movements, acquires the video frame by frame, compresses it in JPEG format, and sends the prepared video data to the data server.

[0943] Step 2:

[0944] The server preprocesses the video data it receives. The input is compressed video data, and the output is video data formatted for analysis. Specifically, preprocessing is performed to remove video noise and adjust the resolution to an appropriate level. Video processing libraries such as OpenCV are used for this process.

[0945] Step 3:

[0946] The server identifies the baby's actions from pre-processed video data using an image analysis algorithm. The input is the pre-processed video data, and the output is the detection result of specific baby actions (e.g., rolling over, sitting up, crawling). Specifically, an object detection algorithm identifies the baby's movements, and the result is extracted as data.

[0947] Step 4:

[0948] The server uses a machine learning model to detect behavioral data and evaluate the baby's developmental stage. The input is the baby's behavioral data, and the output is an evaluation of the baby's developmental stage (e.g., rolling over, sitting up). Specifically, it uses a pre-trained dataset to determine the developmental stage based on the current behavioral data.

[0949] Step 5:

[0950] The server generates parenting advice based on the evaluation results of a machine learning model. The input is the evaluation result of the baby's developmental stage, and the output is specific parenting advice. For example, it might generate advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0951] Step 6:

[0952] The server generates childcare advice and sends it to the user's device. The input is the generated childcare advice, and the output is a notification to the user's device. Specifically, the childcare advice is sent as a push notification or message to the user's smartphone or tablet.

[0953] Step 7:

[0954] The server interacts with a virtual store to recommend childcare-related products. The input is the result of an assessment of the baby's developmental stage, and the output is information about the corresponding product. Specifically, it retrieves products suitable for the baby's growth (e.g., cushions, toys) from the virtual store and generates recommendations for parents.

[0955] Step 8:

[0956] The user's device receives notifications from the server and displays them to the parent. The input is childcare advice and product recommendations sent from the server, and the output is display and notification within the app. Specifically, when the user opens the app, detailed childcare advice and recommended products are displayed for review.

[0957] The above outlines the specific processing steps of the system for implementing the invention. This allows parents to monitor their baby's growth in real time and receive appropriate childcare advice and product recommendations.

[0958] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0959] This invention is a system for monitoring a baby's growth and providing parents with appropriate childcare advice. The system includes means for capturing video of the baby and transmitting it to a server, means for detecting the baby's behavior from the video data using an image analysis algorithm, means for evaluating the baby's growth stage based on the detected behavior using a machine learning model, means for generating childcare advice based on the evaluation results, and means for transmitting and displaying the generated childcare advice on a terminal. It also incorporates an emotion engine that recognizes the user's emotions and has a function to customize childcare advice based on the user's emotional state.

[0960] 1. Data collection and transmission

[0961] terminal

[0962] The device activates its camera and captures video of the baby in real time. The captured video data is compressed and sent to the server via a secure communication protocol.

[0963] 2. Video Analysis

[0964] server

[0965] The server processes the video data received from the terminal. First, it preprocesses the data by removing noise and preparing it for analysis. Next, it uses an image analysis algorithm to detect the baby's actions. For example, it identifies actions such as the baby rolling over or sitting up.

[0966] 3. Evaluation using machine learning

[0967] server

[0968] Based on the detected behavioral data, a machine learning model is used to evaluate the baby's developmental stage. The data obtained in the feature extraction process is input into the model to determine the developmental stage. In addition, future behaviors and potential risk factors are predicted based on the behavioral history.

[0969] 4. Advice generation

[0970] server

[0971] Based on the evaluation and prediction results, specific parenting advice is generated. The generated advice is formatted in a way that is easy for parents to understand.

[0972] 5. Emotion Recognition and Customization

[0973] terminal

[0974] The device is also equipped with a camera and microphone to capture the user's facial expressions and voice tone. This data is sent to an emotion engine to analyze the user's emotional state.

[0975] server

[0976] The emotion engine identifies the user's emotions (e.g., stress, fatigue, exhilaration, etc.) and customizes parenting advice based on the user's emotional state.

[0977] 6. Sending and displaying notifications

[0978] server

[0979] The server sends customized parenting advice to the device.

[0980] Device & User

[0981] The device notifies the user of any advice it receives. The user checks the notification and can view detailed parenting advice within the app.

[0982] Specific example

[0983] Let's take the example of a baby who has just started rolling over in their crib.

[0984] terminal

[0985] The camera captures the baby rolling over and sends this video data to the server.

[0986] server

[0987] The server analyzes the video data and detects when the baby rolls over. The detected data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[0988] terminal

[0989] The device captures the user's facial expressions and tone of voice, and an emotion engine analyzes whether the user is experiencing stress.

[0990] server

[0991] Based on this emotional data, the server generates customized advice such as, "Let's have your partner help you practice together."

[0992] Device & User

[0993] A smartphone app receives this parenting advice and notifies the user. When the user opens the app, they can view customized, specific advice.

[0994] The above describes the detailed configuration for carrying out the invention. This system makes it possible not only to monitor the baby's growth in real time, but also to provide appropriate childcare advice tailored to the parents' emotional state.

[0995] The following describes the processing flow.

[0996] Step 1: Data Capture (Device)

[0997] The device activates the camera and captures the baby's image in real time. The camera continuously records the baby's movements and prepares to send the captured video data to the server in real time.

[0998] Step 2: Data transmission (terminal)

[0999] The device compresses the captured video data and sends it to the server using a secure communication protocol. Encryption technology is also used to ensure data security.

[1000] Step 3: Data reception (server)

[1001] The server receives video data transmitted from the terminal. The received data is temporarily stored in a buffer and then passed on to the analysis process.

[1002] Step 4: Data preprocessing (server)

[1003] The server removes noise from the received video data and prepares it for easier analysis. The pre-processed data is then used in the next analysis step.

[1004] Step 5: Behavior detection (server)

[1005] The server uses image analysis algorithms to detect specific baby behaviors (e.g., rolling over, sitting up, crawling) from video data. The detected behavior data is then categorized by type of behavior.

[1006] Step 6: Feature Extraction (Server)

[1007] The server extracts features from the detected behavioral data. For example, the number of times a cat turns over in its sleep or the duration of a specific behavior may be extracted as features.

[1008] Step 7: Evaluation and Prediction (Server)

[1009] The server applies a machine learning model and uses extracted feature data to assess the baby's developmental stage. Based on behavioral history, it also predicts potential next behaviors and risk factors.

[1010] Step 8: Advice generation (server)

[1011] The server generates specific parenting advice based on the evaluation and prediction results. The generated advice is formatted in a way that is easy for parents to understand.

[1012] Step 9: Emotional Data Capture (Device)

[1013] The device uses its camera and microphone to capture the user's facial expressions and voice tone. This data is then prepared to be sent to the emotion engine.

[1014] Step 10: Emotional Data Analysis (Server)

[1015] The server uses an emotion engine to analyze the user's emotional data. From the analyzed data, it identifies the user's emotional state (e.g., stress, exhilaration, fatigue, etc.).

[1016] Step 11: Customizing parenting advice (server)

[1017] The server customizes parenting advice based on the user's emotional state, as identified by the emotion engine. For example, if the user is feeling stressed, advice such as "get your partner to help you" will be generated.

[1018] Step 12: Send notification (server)

[1019] The server sends customized parenting advice to the device. The sent notifications arrive on the device, such as a smartphone or tablet.

[1020] Step 13: Notification display (device)

[1021] The device notifies the user of childcare advice it has received. Notifications are delivered via pop-up messages and sounds.

[1022] Step 14: View Details (User)

[1023] Users can check notifications on their devices and view detailed parenting advice. Through the app, users can also access past advice and prediction results.

[1024] The above outlines the specific processing steps of the program. This sequence of steps enables a system that monitors the baby's growth in real time and provides appropriate childcare advice tailored to the user's emotional state.

[1025] (Example 2)

[1026] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1027] Conventional infant growth monitoring systems can identify an infant's behavior, but they do not take into account the parent's emotional state, meaning the parenting advice provided may not be appropriate to the parent's mental state. Furthermore, they lacked the ability to predict future behavior and potential risks, in addition to evaluating the infant's developmental stage. This invention aims to solve these problems.

[1028] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1029] In this invention, the server includes means for capturing and transmitting images of the baby to the server; means for detecting the baby's behavior from the image data using an image analysis algorithm; means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model; means for generating childcare advice based on the evaluation results; means for transmitting and displaying the childcare advice on a terminal; means for capturing the user's facial expressions and tone of voice and analyzing their emotional state; and means for customizing the childcare advice based on the emotional state. This makes it possible to accurately monitor the baby's development in real time and provide appropriate childcare advice that takes into account the parents' mental state.

[1030] "Means for capturing video" refers to a camera and its control device that captures video of the baby in real time.

[1031] "Means of sending to the server" refers to a communication device that compresses the captured video data and sends it to a remote server via a secure communication protocol.

[1032] An "image analysis algorithm" refers to a program and method for detecting a baby's behavior based on received video data. Specifically, it includes methods such as object detection and posture estimation.

[1033] A "machine learning model" refers to a data model used to evaluate a baby's developmental stage based on past data. A specific example is a neural network.

[1034] "Means for generating parenting advice" refers to a program that creates appropriate parenting methods and advice for parents based on evaluation results from machine learning models.

[1035] "Means of transmitting and displaying on a device" refers to communication devices and display devices used to transmit and display childcare advice on a device used by a parent (e.g., a smartphone or tablet).

[1036] "Means of capturing facial expressions and voice tone" refers to a camera and microphone, as well as their control device, that record the parent's facial expressions and voice tone in real time in order to understand the parent's emotional state.

[1037] "Means for analyzing emotional states" refers to programs and methods for analyzing a parent's emotions from captured facial expressions and voice tones. Specific examples include emotion recognition algorithms.

[1038] "Methods for customizing parenting advice" refers to a program that adjusts and modifies parenting advice to a situation-appropriate format based on the analyzed emotional state of the parent.

[1039] This invention is a system for monitoring a baby's growth and providing parents with appropriate childcare advice. The system includes means for capturing video of the baby and transmitting it to a server, means for detecting the baby's behavior from the video data using an image analysis algorithm, means for evaluating the baby's growth stage based on the detected behavior using a machine learning model, means for generating childcare advice based on the evaluation results, and means for transmitting and displaying the generated childcare advice on a terminal. It also incorporates an emotion engine that recognizes the user's emotions and has a function to customize childcare advice based on the user's emotional state.

[1040] Data collection and transmission

[1041] terminal

[1042] The device activates its camera and captures video of the baby in real time. For example, if the baby starts rolling over in the crib, it will record that movement. The captured video data is compressed using a standard compression format (e.g., H.264) and sent to the server via a secure communication protocol (e.g., SSL / TLS).

[1043] Video analysis

[1044] server

[1045] The server processes the video data received from the terminal. First, it preprocesses the data using libraries such as OpenCV to remove noise and prepare it for analysis. Next, it detects the baby's actions using image analysis algorithms such as ResNet or YOLO. For example, it detects when the baby rolls over.

[1046] Evaluation using machine learning

[1047] server

[1048] Based on the detected behavioral data, machine learning models such as TensorFlow and PyTorch are used to evaluate the baby's developmental stage. The data obtained in the feature extraction process is input into the model to determine the developmental stage (e.g., the stage of rolling over). In addition, future behaviors and potential risk factors are predicted based on the behavioral history.

[1049] Advice generation

[1050] server

[1051] Based on the evaluation and prediction results, it generates specific parenting advice. For example, it might generate advice such as, "Your baby has started rolling over. Next, try practicing sitting up." The generated advice is formatted in a way that is easy for parents to understand (e.g., text messages or illustrated guides).

[1052] Emotion recognition and customization

[1053] terminal

[1054] The device is also equipped with a camera and microphone to capture the user's facial expressions and voice tone. This data is sent to an emotion engine to analyze the user's emotional state. This process utilizes tools such as Amazon Rekognition and Microsoft Azure's Cognitive Services.

[1055] server

[1056] The emotion engine identifies the user's emotions (e.g., stress, fatigue, exhilaration) and customizes parenting advice based on the user's emotional state. For example, if the user is feeling stressed, advice such as "Ask your partner to help you practice together" might be added.

[1057] Notification sent and displayed

[1058] server

[1059] The server sends customized parenting advice to the device.

[1060] Device & User

[1061] The device notifies the user of any advice it receives. The user checks the notification and views detailed parenting advice in the smartphone app. For example, when the app is opened, specific instructions such as "Your baby has started rolling over. Next, try practicing sitting up with your partner" will be displayed.

[1062] Specific example

[1063] Let's take the example of a baby who has just started rolling over in their crib.

[1064] terminal

[1065] The camera captures the baby rolling over and sends this video data to the server.

[1066] server

[1067] The server analyzes the video data and detects when the baby rolls over. The detected data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[1068] terminal

[1069] The device captures the user's facial expressions and tone of voice, and an emotion engine analyzes whether the user is experiencing stress.

[1070] server

[1071] Based on this emotional data, the server generates customized advice such as, "Let's have your partner help you practice together."

[1072] Device & User

[1073] A smartphone app receives this parenting advice and notifies the user. When the user opens the app, they can view customized, specific advice.

[1074] Example of a prompt

[1075] Please describe how a program works to generate parenting advice for when a baby starts rolling over and display it to stressed parents.

[1076] This system not only allows for real-time monitoring of the baby's development but also provides appropriate parenting advice tailored to the parents' emotional state.

[1077] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1078] Step 1:

[1079] terminal

[1080] The device activates its camera and captures video of the baby in real time. For example, if the baby starts rolling over in the crib, it will record that movement. The captured video data is compressed in H.264 format. The compressed data is sent to the server as input using a secure communication protocol (SSL / TLS). The output is compressed video data.

[1081] Step 2:

[1082] server

[1083] The server analyzes the video data received from the terminal. First, it preprocesses the data using libraries such as OpenCV to remove noise. Using the preprocessed data as input, it identifies the baby's actions using image analysis algorithms such as ResNet and YOLO. Specifically, it detects when the baby is rolling over. The output is the detected action data.

[1084] Step 3:

[1085] server

[1086] The server inputs the behavioral data obtained in the previous step into a machine learning model. TensorFlow or PyTorch is used to evaluate the baby's developmental stage. Based on the extracted features, the neural network determines the developmental stage (e.g., "rolling over"). It also uses the behavioral history to predict future behavior and potential risk factors. The output is the developmental stage and the predicted result.

[1087] Step 4:

[1088] server

[1089] The server generates parenting advice based on the evaluation results obtained in step 3. For example, it might generate specific advice such as, "Your baby has started rolling over. Next, try practicing sitting up." The generated advice is formatted in a way that is easy for parents to understand (text messages or illustrated guides). The output is the formatted parenting advice.

[1090] Step 5:

[1091] terminal

[1092] The device captures the user's facial expressions and voice tone. Using the camera and microphone, it records the user's facial expressions and voice tone in real time. The captured facial and voice data are sent to the emotion engine as input. The output is the captured emotion data.

[1093] Step 6:

[1094] server

[1095] The server analyzes the user's emotional state based on data received by the emotion engine. It uses Amazon Rekognition or Microsoft Azure Cognitive Services to identify emotions such as stress, fatigue, and exhilaration. The emotion analysis results are used as input to customize parenting advice based on the user's emotions. For example, if the user is feeling stressed, the advice might be changed to "Ask your partner to help you practice together." The output is the customized advice.

[1096] Step 7:

[1097] server

[1098] The server sends customized parenting advice to the terminal. The output is the customized parenting advice.

[1099] Device & User

[1100] The device notifies the user of any advice it receives. The user checks the notification and views detailed parenting advice in the smartphone app. Specifically, when the user opens the app, specific instructions are displayed, such as, "Your baby has started rolling over. Next, try practicing sitting up with your partner." The output is the displayed advice.

[1101] (Application Example 2)

[1102] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1103] In modern society, a lack of support for childcare is a significant problem. First-time parents, in particular, often experience a great deal of anxiety and struggle to find appropriate advice and products. Furthermore, baby supply stores face the challenge of providing advice and product recommendations tailored to the individual developmental stages of each baby.

[1104] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing and transmitting images of a baby to the server, means for detecting the baby's behavior from the image data using an image analysis algorithm, means for evaluating the baby's growth stage based on the detected behavior using a machine learning model, means for generating childcare advice based on the evaluation results, means for transmitting and displaying the childcare advice on a terminal, and means for providing childcare advice to a terminal used by a customer in a baby goods store. This enables the provision of appropriate childcare advice in real time according to the growth stage of each baby, and allows parents to choose products with confidence in a baby goods store.

[1105] "Means for capturing images of a baby and sending them to a server" refers to a device that includes a camera for taking images of a baby and a communication interface for sending the captured video data to a central server via a network.

[1106] An "image analysis algorithm" is a set of computational procedures for identifying and extracting specific patterns or movements from video data, and is a software program used to detect a baby's movements.

[1107] A "machine learning model" is an algorithm that learns from data and uses that knowledge to analyze and evaluate new data. Specifically, it is used to evaluate a baby's developmental stage using behavioral data.

[1108] A "means for generating childcare advice" refers to a software program that automatically generates appropriate childcare guidance and suggestions based on evaluation results and provides them to parents.

[1109] "Means of sending and displaying on a device" refers to a device that has the function of sending the generated childcare advice to a user's smartphone, tablet, or other device and displaying it on its screen.

[1110] "Terminals used by customers in baby product stores" refers to information terminals installed in physical stores that sell baby products, which customers use to receive childcare advice.

[1111] "A baby's movements such as rolling over, sitting up, and crawling" refer to specific physical movements that occur during a baby's developmental stages and are important indicators for evaluating their developmental stage.

[1112] "Behavioral history" refers to a record of a baby's past actions and behaviors, and is data used to predict future behavior and potential risk factors.

[1113] "Potential risk factors" refer to dangerous situations or risks that a baby may face in the future, and serve as foundational data to provide preventative advice.

[1114] This invention is a system that monitors a baby's growth in real time and provides parents with appropriate childcare advice. The system functions through a series of processes including video capture, video analysis, machine learning evaluation, advice generation, and notification display. It can also provide childcare advice to customer terminals in baby product stores.

[1115] Data collection and transmission

[1116] The device is equipped with a camera that captures video of the baby in real time. The captured video data is compressed and sent to the server via a secure communication protocol (e.g., HTTPS).

[1117] Video analysis

[1118] The server processes the video data received from the terminal. As a preprocessing step, it removes noise and prepares the data for analysis. Then, it uses OpenCV or similar software to detect the baby's movements (rolling over, sitting up, crawling, etc.) from the video data.

[1119] Evaluation using machine learning

[1120] The server uses a machine learning model (e.g., built using TensorFlow or Keras) to evaluate the baby's developmental stage based on detected behavioral data. The model is input with data obtained during the feature extraction process to determine the developmental stage. In this process, future behaviors and potential risk factors are also predicted based on the behavioral history.

[1121] Advice generation

[1122] The server generates specific parenting advice based on the evaluation and prediction results. The generated advice is formatted in a way that is easy for parents to understand. For example, it might say, "Your baby has started rolling over. Next, try practicing sitting up."

[1123] Notification sent and displayed

[1124] The server sends the generated parenting advice to the terminal. The terminal notifies the user of the received advice and displays the specific content of the advice. In a baby goods store, the advice is displayed on the terminal used by the customer, allowing parents to choose appropriate products in the store.

[1125] Specific example

[1126] Let's take the example of a baby who has just started rolling over in their crib.

[1127] Device: The camera captures the baby rolling over and sends this video data to the server.

[1128] Server: The server analyzes the video data and detects when the baby rolls over. The detected data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[1129] Device: Smartphones and tablets receive this parenting advice and notify the user. When the user opens the app, they can view the specific advice.

[1130] Examples of prompts to input into a generative AI model

[1131] Please create a program for a system that captures a baby's behavior with a camera and provides parenting advice based on that behavior. The advice should include specific product suggestions for parents when they visit a baby goods store.

[1132] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1133] Step 1:

[1134] The device activates the camera and captures video of the baby in real time. The captured video data is compressed and sent to the server via a secure communication protocol (e.g., HTTPS). The input here is real-time video data, and the output is compressed video data.

[1135] Step 2:

[1136] The server processes the video data received from the terminal. First, it preprocesses the data by removing noise and preparing it for analysis. Then, it uses OpenCV or similar software to detect the baby's movements (e.g., rolling over, sitting up, crawling) from the video data. The input is compressed video data, and the output is the detected baby movement data.

[1137] Step 3:

[1138] The server uses a machine learning model based on detected behavioral data to evaluate the baby's developmental stage. Data obtained during the feature extraction process is input into the model to determine the developmental stage. Furthermore, during this process, future behaviors and potential risk factors are predicted based on the behavioral history. The input is the detected behavioral data, and the output is the developmental stage and predicted behaviors.

[1139] Step 4:

[1140] The server generates specific parenting advice based on the evaluation and prediction results. The generated advice is formatted in a way that is easy for parents to understand. For example, it might say, "Your baby has started rolling over. Next, try practicing sitting up." The input is the developmental stage and predicted behavior, and the output is the text of the parenting advice.

[1141] Step 5:

[1142] The server sends the generated childcare advice to the terminal. The input here is the text of the childcare advice, and the output is communication data.

[1143] Step 6:

[1144] The device notifies the user of received advice and displays the specific advice on the app screen. The input here is communication data from the server, and the output is the displayed text of the parenting advice. The user checks the notification and then views the detailed parenting advice within the app.

[1145] Step 7:

[1146] In baby product stores, parenting advice is ultimately displayed on a terminal used by the customer. This allows parents to choose appropriate products within the store. The input is the text of the parenting advice, and the output is the displayed product suggestions.

[1147] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1148] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1149] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1150] [Fourth Embodiment]

[1151] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1152] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1153] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1154] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1155] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1156] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1157] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1158] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1159] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1160] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1161] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1162] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1163] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1164] This invention is a system that monitors a baby's growth and provides parents with appropriate childcare advice. The following describes the system's program processing in natural language, along with specific examples.

[1165] 1. Data collection and transmission

[1166] terminal

[1167] A camera installed in the baby's room captures video of the baby in real time. The captured video data is compressed and sent to a server via a secure communication protocol.

[1168] 2. Video Analysis

[1169] server

[1170] The server processes the video data received from the terminal. First, it preprocesses the data by removing noise and preparing the video data for analysis. Next, it uses an image analysis algorithm (e.g., an object detection algorithm) to detect the baby's actions. Specifically, it identifies actions such as the baby rolling over, sitting up, and crawling.

[1171] 3. Evaluation using machine learning

[1172] server

[1173] The detected behavioral data is used to evaluate the baby's developmental stage. This is done using a pre-trained machine learning model. The data obtained in the feature extraction process is used as input to determine the baby's developmental stage (e.g., starting to roll over, starting to sit up, etc.). In addition, future behaviors and potential risk factors are predicted based on the behavioral history.

[1174] 4. Advice generation

[1175] server

[1176] Based on the evaluation results of a machine learning model, it generates advice and feedback on childcare. For example, it creates specific advice such as, "Now that your baby has started rolling over, it's a good idea to start practicing sitting up next." This advice is formatted in a way that is easy for parents to understand.

[1177] 5. Notifications and Display

[1178] Device & User

[1179] The server sends generated advice and feedback to the device (e.g., a smartphone app). The device notifies the user of the received message using a pop-up notification or sound. When the user opens the app, they can view detailed parenting advice.

[1180] Specific example

[1181] Let's take the example of a baby who has just started rolling over in their crib.

[1182] terminal

[1183] The camera captures the baby rolling over and sends this video data to the server.

[1184] server

[1185] The server analyzes the received video data and detects when the baby rolls over. This motion data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[1186] Device & User

[1187] A smartphone app receives this parenting advice and notifies the user. When the user opens the app, they can view specific advice and take appropriate action according to their baby's development.

[1188] The above describes the detailed configuration for carrying out the invention. This system allows parents to monitor their baby's growth in real time and receive appropriate childcare advice.

[1189] The following describes the processing flow.

[1190] Step 1: Data Capture (Device)

[1191] The device activates the camera and captures the baby's image in real time. The camera continuously records the baby's movements and prepares to send the acquired video data to the server in real time.

[1192] Step 2: Data transmission (terminal)

[1193] The terminal compresses the captured video data and sends it to the server using a secure communication protocol. Data encryption is performed during this process to prevent interception by third parties.

[1194] Step 3: Data reception (server)

[1195] The server receives video data transmitted from the terminal. The received data is temporarily stored in a buffer and then passed on to the analysis process.

[1196] Step 4: Data preprocessing (server)

[1197] The server removes noise from the received video data and extracts only the necessary parts. This improves the accuracy of data analysis.

[1198] Step 5: Behavior detection (server)

[1199] The server uses image analysis algorithms to detect specific baby behaviors (e.g., rolling over, sitting up, crawling) from video data. The analyzed data is then categorized by type of behavior.

[1200] Step 6: Feature Extraction (Server)

[1201] The server extracts features from the detected behavioral data. For example, the number of times a cat turns over in its sleep or the duration of a specific behavior may be extracted as features.

[1202] Step 7: Evaluation and Prediction (Server)

[1203] The server applies a machine learning model and uses the extracted features to evaluate the baby's developmental stage. It also predicts likely future behaviors and potential risks based on the history of behavioral data.

[1204] Step 8: Advice generation (server)

[1205] The server generates parenting advice for parents based on the evaluation and prediction results. The advice is formatted in a way that is easy for parents to understand.

[1206] Step 9: Send notification (server)

[1207] The server generates childcare advice and sends it to the device. The sent notification arrives on the device, which is a smartphone or tablet.

[1208] Step 10: Notification display (device)

[1209] The device notifies the user of childcare advice it has received. The notification is delivered via a pop-up message or sound.

[1210] Step 11: View Details (User)

[1211] Users can check notifications on their devices and view detailed parenting advice. Through the app, users can also access past advice and prediction results.

[1212] The above outlines the specific processing steps of the program. This series of steps enables a system that monitors the baby's growth in real time and provides parents with appropriate childcare advice.

[1213] (Example 1)

[1214] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1215] Traditional childcare support systems often do not provide real-time monitoring of a baby's growth or offer childcare advice, and therefore do not adequately help parents understand their baby's development. Furthermore, they lacked the means to accurately detect a baby's behavior and generate appropriate advice based on their developmental stage.

[1216] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1217] In this invention, the server includes means for capturing and transmitting video of the baby to the server, means for preprocessing by applying a noise reduction filter, means for detecting the baby's actions from the video data using an object detection algorithm, means for extracting the baby's actions data, means for evaluating the baby's developmental stage based on the detected actions using a machine learning model, means for generating childcare advice based on the evaluation results, and means for transmitting, notifying, and displaying the childcare advice to a terminal. This enables parents to understand their baby's developmental status in real time and receive appropriate childcare advice.

[1218] A "noise reduction filter" is an algorithm used to remove random noise from video data.

[1219] An "object detection algorithm" is an algorithm used to detect specific objects within video data, such as the posture or movements of a baby.

[1220] "Behavioral data" refers to information about a baby's specific movements and postures, including the type of movement detected and a timestamp.

[1221] A "machine learning model" is an algorithm that learns behavioral patterns and action stages based on training data, and then uses that knowledge to evaluate and predict new data.

[1222] "Parenting advice" refers to specific suggestions and instructions for parents, generated based on the baby's developmental stage and behavioral data.

[1223] A "server" is a computing device that analyzes video data of babies, detects their behavior, evaluates their developmental stage using machine learning models, and generates parenting advice.

[1224] A "terminal" refers to a device, such as a camera or smartphone, used to capture images of the baby or to receive and display advice sent from a server.

[1225] This invention is a system for monitoring a baby's growth and providing parents with appropriate childcare advice. The system consists of a video capture device, communication means, a server, an analysis algorithm, a machine learning model, and a notification system.

[1226] Hardware and software configuration

[1227] terminal

[1228] The device includes a camera installed in the baby's room. This camera (e.g., an IP camera) captures video of the baby in real time. The video data is encoded in a compressed format such as H.264 and securely transmitted to the server using the SSL / TLS protocol.

[1229] server

[1230] The server receives and processes video data transmitted from the terminal. Specifically, it preprocesses the data by applying a noise reduction filter (e.g., a Gaussian filter), and then analyzes the baby's behavior using object detection algorithms such as OpenCV or YOLO. The baby's behavior data is stored in a database with timestamps.

[1231] The server also uses machine learning models (e.g., TensorFlow, PyTorch) to assess the baby's developmental stage. These models are trained on pre-collected datasets and can automatically evaluate and classify the baby's behavioral patterns. Based on the evaluation results, natural language generation (NLG) algorithms are used to generate specific parenting advice.

[1232] User notification system

[1233] The advice generated by the server is sent to the device (e.g., a smartphone app). The device notifies the user of the received advice via a pop-up notification or sound notification. When the user opens the app, detailed parenting advice is displayed.

[1234] Specific example

[1235] For example, consider a situation where a baby starts rolling over in their crib. A camera captures this movement and sends the video to a server. The server analyzes the video data and detects that the baby is rolling over. A machine learning model evaluates this movement as a developmental stage, specifically "rolling over." The server then generates advice such as, "Your baby has started rolling over. Next, try practicing sitting up," and sends this advice to the device. The device notifies the user of this advice, and when the user opens the app, more detailed advice is displayed.

[1236] Example of a prompt

[1237] An example of a prompt to input into a generative AI model is: "Describe a system that detects behavior from video data of a baby and generates childcare advice. Describe in detail the entire process from preprocessing of input data to notification of advice. Include the names of specific algorithms and software."

[1238] This invention allows parents to monitor their baby's development in real time and receive appropriate childcare advice.

[1239] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1240] Step 1:

[1241] terminal

[1242] A camera in the baby's room captures video. This camera captures the baby's video in real time, frame by frame, and encodes it in a compressed format such as H.264.

[1243] Input: Real-time video data

[1244] Output: Compressed video data

[1245] Step 2:

[1246] terminal

[1247] The video data transmitted from the camera is encrypted using the SSL / TLS protocol and securely sent to the server.

[1248] Input: Compressed video data

[1249] Output: Encrypted video data

[1250] Step 3:

[1251] server

[1252] The server decrypts the encrypted video data received from the terminal. Next, it applies a noise reduction filter (e.g., a Gaussian filter) to preprocess the data. This removes random noise from the video.

[1253] Input: Encrypted video data

[1254] Output: Preprocessed video data

[1255] Step 4:

[1256] server

[1257] Preprocessed video data is divided into frames, and object detection algorithms (e.g., OpenCV, YOLO) are used to detect the baby's actions. For example, actions such as rolling over, sitting up, and crawling are analyzed to identify the baby's posture and movements.

[1258] Input: Preprocessed video data

[1259] Output: Baby behavior data

[1260] Step 5:

[1261] server

[1262] The baby's behavioral data is extracted and stored in a database with timestamps. This data includes the baby's actions and the time they occurred.

[1263] Input: Baby's behavioral data

[1264] Output: Timestamped behavioral data

[1265] Step 6:

[1266] server

[1267] Using machine learning models (e.g., TensorFlow, PyTorch), we evaluate the developmental stage of a baby based on extracted behavioral data. For example, behavioral data is input as features to determine developmental stages such as "the baby has started rolling over." We also predict future behaviors and potential risk factors based on the behavioral history.

[1268] Input: Timestamped behavioral data

[1269] Output: Growth stage evaluation results

[1270] Step 7:

[1271] server

[1272] Based on the results of the developmental stage assessment, a natural language generation (NLG) algorithm is used to generate parenting advice. For example, it can create specific advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[1273] Input: Growth stage evaluation results

[1274] Output: Childcare advice

[1275] Step 8:

[1276] server

[1277] The generated parenting advice is sent to the device (e.g., a smartphone app).

[1278] Input: Childcare advice

[1279] Output: Advice sent to the terminal

[1280] Step 9:

[1281] Device & User

[1282] The device notifies the user of received advice through pop-up notifications and sounds. When the user opens the app, detailed parenting advice is displayed.

[1283] Input: Advice sent to the device

[1284] Output: Advice notified to the user

[1285] The above outlines the specific processing steps of this system.

[1286] (Application Example 1)

[1287] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1288] Systems already exist that appropriately monitor a baby's growth and provide parents with accurate childcare advice. However, these systems typically only provide childcare advice and do not recommend or assist with the purchase of childcare-related products. As a result, there is a challenge in that parents have difficulty choosing appropriate products for their baby's development. Furthermore, there is a lack of integration with virtual stores, making it difficult to purchase childcare-related products in a centralized manner.

[1289] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1290] In this invention, the server includes means for capturing images of a baby and transmitting them to a data server; means for detecting the baby's behavior from the image data using an image analysis algorithm; means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model; means for generating childcare advice based on the evaluation results; means for transmitting and displaying the childcare advice on a user terminal; and means for coordinating with a virtual store to recommend childcare-related products. This makes it possible for parents to not only receive appropriate advice according to their baby's growth, but also to easily purchase products corresponding to that growth stage.

[1291] "Means for capturing images of a baby" refers to a device or method for capturing images of a baby's activities in real time and acquiring that image data.

[1292] "Means of sending to a data server" refers to a device or software that transfers captured video data to a data server via a network.

[1293] An "image analysis algorithm" is a set of computational procedures or programs used to analyze acquired video data and recognize specific actions or objects.

[1294] "Means for detecting a baby's behavior" refers to a device or method that uses an image analysis algorithm to identify a baby's specific movements from video data.

[1295] A "machine learning model" is a computational model that learns from large amounts of data and uses that data to make predictions and classifications about unknown data.

[1296] "Means for evaluating the developmental stage of an infant" refers to a device or method that uses a machine learning model to determine the progress of an infant's development based on their current behavioral data.

[1297] "Means for generating childcare advice" refers to a device or software that automatically creates specific advice and suggestions for parents based on the results of an assessment of the baby's developmental stage.

[1298] "Means of transmitting and displaying on a user terminal" refers to a device or software that transfers and displays the generated childcare advice on a device such as a smartphone or tablet used by the parent.

[1299] "Means of recommending childcare-related products in conjunction with virtual stores" refers to a device or method that links information with virtual stores that provide appropriate products online according to the developmental stage of a baby, and recommends products to parents along with childcare advice.

[1300] This invention is a system for monitoring a baby's growth and providing childcare advice based on that growth. The system includes a camera, a data server, an image analysis algorithm, a machine learning model, a notification system, and integration with a virtual store.

[1301] 1. Data collection and transmission

[1302] The server uses a camera to capture the baby's activity in real time. The captured video data is compressed and sent to the data server via a secure communication protocol. This process uses video processing libraries such as OpenCV.

[1303] 2. Video Analysis

[1304] The server preprocesses the received video data to remove noise. Next, it uses image analysis algorithms to identify the baby's actions. In this process, for example, object detection algorithms are used to analyze actions such as the baby rolling over, sitting up, and crawling.

[1305] 3. Evaluation of Growth Stages

[1306] The server uses a machine learning model to assess the baby's developmental stage based on detected behavioral data. This model utilizes a pre-trained dataset to determine the developmental stage based on the baby's current behavior. Furthermore, it analyzes the behavioral history to predict future behavior and potential risk factors.

[1307] 4. Generating parenting advice

[1308] The server generates parenting advice based on the evaluation results of the machine learning model. For example, it creates specific advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[1309] 5. Notifications and Display

[1310] The server sends the generated advice to the user's device. The user's device (such as a smartphone or tablet) notifies the user of the received message with a pop-up notification or sound, and displays detailed childcare advice within the app.

[1311] 6. Collaboration with virtual stores

[1312] The server integrates information with a virtual store to recommend appropriate products based on the baby's developmental stage. For example, it might provide product recommendations such as, "Your baby has started rolling over. This cushion would be a good next step."

[1313] Specific example

[1314] Consider a scenario where a baby starts rolling over in their crib. A camera captures this movement and sends the video data to a server. The server analyzes the received video and detects the baby's rolling over. Based on this information, a machine learning model evaluates the developmental stage as "rolling over." Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up," and also makes product recommendations such as, "This cushion would be helpful." This information is then sent to the parent's smartphone.

[1315] Example of input prompt text for a generative AI model:

[1316] "Your baby has started rolling over. Next, try practicing sitting up."

[1317] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1318] Step 1:

[1319] The server uses a camera to capture the baby's activities in real time. The input is video data from the camera, and the output is compressed video data. Specifically, the camera films the baby's movements, acquires the video frame by frame, compresses it in JPEG format, and sends the prepared video data to the data server.

[1320] Step 2:

[1321] The server preprocesses the video data it receives. The input is compressed video data, and the output is video data formatted for analysis. Specifically, preprocessing is performed to remove video noise and adjust the resolution to an appropriate level. Video processing libraries such as OpenCV are used for this process.

[1322] Step 3:

[1323] The server identifies the baby's actions from pre-processed video data using an image analysis algorithm. The input is the pre-processed video data, and the output is the detection result of specific baby actions (e.g., rolling over, sitting up, crawling). Specifically, an object detection algorithm identifies the baby's movements, and the result is extracted as data.

[1324] Step 4:

[1325] The server uses a machine learning model to detect behavioral data and evaluate the baby's developmental stage. The input is the baby's behavioral data, and the output is an evaluation of the baby's developmental stage (e.g., rolling over, sitting up). Specifically, it uses a pre-trained dataset to determine the developmental stage based on the current behavioral data.

[1326] Step 5:

[1327] The server generates parenting advice based on the evaluation results of a machine learning model. The input is the evaluation result of the baby's developmental stage, and the output is specific parenting advice. For example, it might generate advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[1328] Step 6:

[1329] The server generates childcare advice and sends it to the user's device. The input is the generated childcare advice, and the output is a notification to the user's device. Specifically, the childcare advice is sent as a push notification or message to the user's smartphone or tablet.

[1330] Step 7:

[1331] The server interacts with a virtual store to recommend childcare-related products. The input is the result of an assessment of the baby's developmental stage, and the output is information about the corresponding product. Specifically, it retrieves products suitable for the baby's growth (e.g., cushions, toys) from the virtual store and generates recommendations for parents.

[1332] Step 8:

[1333] The user's device receives notifications from the server and displays them to the parent. The input is childcare advice and product recommendations sent from the server, and the output is display and notification within the app. Specifically, when the user opens the app, detailed childcare advice and recommended products are displayed for review.

[1334] The above outlines the specific processing steps of the system for implementing the invention. This allows parents to monitor their baby's growth in real time and receive appropriate childcare advice and product recommendations.

[1335] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1336] This invention is a system for monitoring a baby's growth and providing parents with appropriate childcare advice. The system includes means for capturing video of the baby and transmitting it to a server, means for detecting the baby's behavior from the video data using an image analysis algorithm, means for evaluating the baby's growth stage based on the detected behavior using a machine learning model, means for generating childcare advice based on the evaluation results, and means for transmitting and displaying the generated childcare advice on a terminal. It also incorporates an emotion engine that recognizes the user's emotions and has a function to customize childcare advice based on the user's emotional state.

[1337] 1. Data collection and transmission

[1338] terminal

[1339] The device activates its camera and captures video of the baby in real time. The captured video data is compressed and sent to the server via a secure communication protocol.

[1340] 2. Video Analysis

[1341] server

[1342] The server processes the video data received from the terminal. First, it preprocesses the data by removing noise and preparing it for analysis. Next, it uses an image analysis algorithm to detect the baby's actions. For example, it identifies actions such as the baby rolling over or sitting up.

[1343] 3. Evaluation using machine learning

[1344] server

[1345] Based on the detected behavioral data, a machine learning model is used to evaluate the baby's developmental stage. The data obtained in the feature extraction process is input into the model to determine the developmental stage. In addition, future behaviors and potential risk factors are predicted based on the behavioral history.

[1346] 4. Advice generation

[1347] server

[1348] Based on the evaluation and prediction results, specific parenting advice is generated. The generated advice is formatted in a way that is easy for parents to understand.

[1349] 5. Emotion Recognition and Customization

[1350] terminal

[1351] The device is also equipped with a camera and microphone to capture the user's facial expressions and voice tone. This data is sent to an emotion engine to analyze the user's emotional state.

[1352] server

[1353] The emotion engine identifies the user's emotions (e.g., stress, fatigue, exhilaration, etc.) and customizes parenting advice based on the user's emotional state.

[1354] 6. Sending and displaying notifications

[1355] server

[1356] The server sends customized parenting advice to the device.

[1357] Device & User

[1358] The device notifies the user of any advice it receives. The user checks the notification and can view detailed parenting advice within the app.

[1359] Specific example

[1360] Let's take the example of a baby who has just started rolling over in their crib.

[1361] terminal

[1362] The camera captures the baby rolling over and sends this video data to the server.

[1363] server

[1364] The server analyzes the video data and detects when the baby rolls over. The detected data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[1365] terminal

[1366] The device captures the user's facial expressions and tone of voice, and an emotion engine analyzes whether the user is experiencing stress.

[1367] server

[1368] Based on this emotional data, the server generates customized advice such as, "Let's have your partner help you practice together."

[1369] Device & User

[1370] A smartphone app receives this parenting advice and notifies the user. When the user opens the app, they can view customized, specific advice.

[1371] The above describes the detailed configuration for carrying out the invention. This system makes it possible not only to monitor the baby's growth in real time, but also to provide appropriate childcare advice tailored to the parents' emotional state.

[1372] The following describes the processing flow.

[1373] Step 1: Data Capture (Device)

[1374] The device activates the camera and captures the baby's image in real time. The camera continuously records the baby's movements and prepares to send the captured video data to the server in real time.

[1375] Step 2: Data transmission (terminal)

[1376] The device compresses the captured video data and sends it to the server using a secure communication protocol. Encryption technology is also used to ensure data security.

[1377] Step 3: Data reception (server)

[1378] The server receives video data transmitted from the terminal. The received data is temporarily stored in a buffer and then passed on to the analysis process.

[1379] Step 4: Data preprocessing (server)

[1380] The server removes noise from the received video data and prepares it for easier analysis. The pre-processed data is then used in the next analysis step.

[1381] Step 5: Behavior detection (server)

[1382] The server uses image analysis algorithms to detect specific baby behaviors (e.g., rolling over, sitting up, crawling) from video data. The detected behavior data is then categorized by type of behavior.

[1383] Step 6: Feature Extraction (Server)

[1384] The server extracts features from the detected behavioral data. For example, the number of times a cat turns over in its sleep or the duration of a specific behavior may be extracted as features.

[1385] Step 7: Evaluation and Prediction (Server)

[1386] The server applies a machine learning model and uses extracted feature data to assess the baby's developmental stage. Based on behavioral history, it also predicts potential next behaviors and risk factors.

[1387] Step 8: Advice generation (server)

[1388] The server generates specific parenting advice based on the evaluation and prediction results. The generated advice is formatted in a way that is easy for parents to understand.

[1389] Step 9: Emotional Data Capture (Device)

[1390] The device uses its camera and microphone to capture the user's facial expressions and voice tone. This data is then prepared to be sent to the emotion engine.

[1391] Step 10: Emotional Data Analysis (Server)

[1392] The server uses an emotion engine to analyze the user's emotional data. From the analyzed data, it identifies the user's emotional state (e.g., stress, exhilaration, fatigue, etc.).

[1393] Step 11: Customizing parenting advice (server)

[1394] The server customizes parenting advice based on the user's emotional state, as identified by the emotion engine. For example, if the user is feeling stressed, advice such as "get your partner to help you" will be generated.

[1395] Step 12: Send notification (server)

[1396] The server sends customized parenting advice to the device. The sent notifications arrive on the device, such as a smartphone or tablet.

[1397] Step 13: Notification display (device)

[1398] The device notifies the user of childcare advice it has received. Notifications are delivered via pop-up messages and sounds.

[1399] Step 14: View Details (User)

[1400] Users can check notifications on their devices and view detailed parenting advice. Through the app, users can also access past advice and prediction results.

[1401] The above outlines the specific processing steps of the program. This sequence of steps enables a system that monitors the baby's growth in real time and provides appropriate childcare advice tailored to the user's emotional state.

[1402] (Example 2)

[1403] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1404] Conventional infant growth monitoring systems can identify an infant's behavior, but they do not take into account the parent's emotional state, meaning the parenting advice provided may not be appropriate to the parent's mental state. Furthermore, they lacked the ability to predict future behavior and potential risks, in addition to evaluating the infant's developmental stage. This invention aims to solve these problems.

[1405] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1406] In this invention, the server includes means for capturing and transmitting images of the baby to the server; means for detecting the baby's behavior from the image data using an image analysis algorithm; means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model; means for generating childcare advice based on the evaluation results; means for transmitting and displaying the childcare advice on a terminal; means for capturing the user's facial expressions and tone of voice and analyzing their emotional state; and means for customizing the childcare advice based on the emotional state. This makes it possible to accurately monitor the baby's development in real time and provide appropriate childcare advice that takes into account the parents' mental state.

[1407] "Means for capturing video" refers to a camera and its control device that captures video of the baby in real time.

[1408] "Means of sending to the server" refers to a communication device that compresses the captured video data and sends it to a remote server via a secure communication protocol.

[1409] An "image analysis algorithm" refers to a program and method for detecting a baby's behavior based on received video data. Specifically, it includes methods such as object detection and posture estimation.

[1410] A "machine learning model" refers to a data model used to evaluate a baby's developmental stage based on past data. A specific example is a neural network.

[1411] "Means for generating parenting advice" refers to a program that creates appropriate parenting methods and advice for parents based on evaluation results from machine learning models.

[1412] "Means of transmitting and displaying on a device" refers to communication devices and display devices used to transmit and display childcare advice on a device used by a parent (e.g., a smartphone or tablet).

[1413] "Means of capturing facial expressions and voice tone" refers to a camera and microphone, as well as their control device, that record the parent's facial expressions and voice tone in real time in order to understand the parent's emotional state.

[1414] "Means for analyzing emotional states" refers to programs and methods for analyzing a parent's emotions from captured facial expressions and voice tones. Specific examples include emotion recognition algorithms.

[1415] "Methods for customizing parenting advice" refers to a program that adjusts and modifies parenting advice to a situation-appropriate format based on the analyzed emotional state of the parent.

[1416] This invention is a system for monitoring a baby's growth and providing parents with appropriate childcare advice. The system includes means for capturing video of the baby and transmitting it to a server, means for detecting the baby's behavior from the video data using an image analysis algorithm, means for evaluating the baby's growth stage based on the detected behavior using a machine learning model, means for generating childcare advice based on the evaluation results, and means for transmitting and displaying the generated childcare advice on a terminal. It also incorporates an emotion engine that recognizes the user's emotions and has a function to customize childcare advice based on the user's emotional state.

[1417] Data collection and transmission

[1418] terminal

[1419] The device activates its camera and captures video of the baby in real time. For example, if the baby starts rolling over in the crib, it will record that movement. The captured video data is compressed using a standard compression format (e.g., H.264) and sent to the server via a secure communication protocol (e.g., SSL / TLS).

[1420] Video analysis

[1421] server

[1422] The server processes the video data received from the terminal. First, it preprocesses the data using libraries such as OpenCV to remove noise and prepare it for analysis. Next, it detects the baby's actions using image analysis algorithms such as ResNet or YOLO. For example, it detects when the baby rolls over.

[1423] Evaluation using machine learning

[1424] server

[1425] Based on the detected behavioral data, machine learning models such as TensorFlow and PyTorch are used to evaluate the baby's developmental stage. The data obtained in the feature extraction process is input into the model to determine the developmental stage (e.g., the stage of rolling over). In addition, future behaviors and potential risk factors are predicted based on the behavioral history.

[1426] Advice generation

[1427] server

[1428] Based on the evaluation and prediction results, it generates specific parenting advice. For example, it might generate advice such as, "Your baby has started rolling over. Next, try practicing sitting up." The generated advice is formatted in a way that is easy for parents to understand (e.g., text messages or illustrated guides).

[1429] Emotion recognition and customization

[1430] terminal

[1431] The device is also equipped with a camera and microphone to capture the user's facial expressions and voice tone. This data is sent to an emotion engine to analyze the user's emotional state. This process utilizes tools such as Amazon Rekognition and Microsoft Azure's Cognitive Services.

[1432] server

[1433] The emotion engine identifies the user's emotions (e.g., stress, fatigue, exhilaration) and customizes parenting advice based on the user's emotional state. For example, if the user is feeling stressed, advice such as "Ask your partner to help you practice together" might be added.

[1434] Notification sent and displayed

[1435] server

[1436] The server sends customized parenting advice to the device.

[1437] Device & User

[1438] The device notifies the user of any advice it receives. The user checks the notification and views detailed parenting advice in the smartphone app. For example, when the app is opened, specific instructions such as "Your baby has started rolling over. Next, try practicing sitting up with your partner" will be displayed.

[1439] Specific example

[1440] Let's take the example of a baby who has just started rolling over in their crib.

[1441] terminal

[1442] The camera captures the baby rolling over and sends this video data to the server.

[1443] server

[1444] The server analyzes the video data and detects when the baby rolls over. The detected data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[1445] terminal

[1446] The device captures the user's facial expressions and tone of voice, and an emotion engine analyzes whether the user is experiencing stress.

[1447] server

[1448] Based on this emotional data, the server generates customized advice such as, "Let's have your partner help you practice together."

[1449] Device & User

[1450] A smartphone app receives this parenting advice and notifies the user. When the user opens the app, they can view customized, specific advice.

[1451] Example of a prompt

[1452] Please describe how a program works to generate parenting advice for when a baby starts rolling over and display it to stressed parents.

[1453] This system not only allows for real-time monitoring of the baby's development but also provides appropriate parenting advice tailored to the parents' emotional state.

[1454] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1455] Step 1:

[1456] terminal

[1457] The device activates its camera and captures video of the baby in real time. For example, if the baby starts rolling over in the crib, it will record that movement. The captured video data is compressed in H.264 format. The compressed data is sent to the server as input using a secure communication protocol (SSL / TLS). The output is compressed video data.

[1458] Step 2:

[1459] server

[1460] The server analyzes the video data received from the terminal. First, it preprocesses the data using libraries such as OpenCV to remove noise. Using the preprocessed data as input, it identifies the baby's actions using image analysis algorithms such as ResNet and YOLO. Specifically, it detects when the baby is rolling over. The output is the detected action data.

[1461] Step 3:

[1462] server

[1463] The server inputs the behavioral data obtained in the previous step into a machine learning model. TensorFlow or PyTorch is used to evaluate the baby's developmental stage. Based on the extracted features, the neural network determines the developmental stage (e.g., "rolling over"). It also uses the behavioral history to predict future behavior and potential risk factors. The output is the developmental stage and the predicted result.

[1464] Step 4:

[1465] server

[1466] The server generates parenting advice based on the evaluation results obtained in step 3. For example, it might generate specific advice such as, "Your baby has started rolling over. Next, try practicing sitting up." The generated advice is formatted in a way that is easy for parents to understand (text messages or illustrated guides). The output is the formatted parenting advice.

[1467] Step 5:

[1468] terminal

[1469] The device captures the user's facial expressions and voice tone. Using the camera and microphone, it records the user's facial expressions and voice tone in real time. The captured facial and voice data are sent to the emotion engine as input. The output is the captured emotion data.

[1470] Step 6:

[1471] server

[1472] The server analyzes the user's emotional state based on data received by the emotion engine. It uses Amazon Rekognition or Microsoft Azure Cognitive Services to identify emotions such as stress, fatigue, and exhilaration. The emotion analysis results are used as input to customize parenting advice based on the user's emotions. For example, if the user is feeling stressed, the advice might be changed to "Ask your partner to help you practice together." The output is the customized advice.

[1473] Step 7:

[1474] server

[1475] The server sends customized parenting advice to the terminal. The output is the customized parenting advice.

[1476] Device & User

[1477] The device notifies the user of any advice it receives. The user checks the notification and views detailed parenting advice in the smartphone app. Specifically, when the user opens the app, specific instructions are displayed, such as, "Your baby has started rolling over. Next, try practicing sitting up with your partner." The output is the displayed advice.

[1478] (Application Example 2)

[1479] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1480] In modern society, a lack of support for childcare is a significant problem. First-time parents, in particular, often experience a great deal of anxiety and struggle to find appropriate advice and products. Furthermore, baby supply stores face the challenge of providing advice and product recommendations tailored to the individual developmental stages of each baby.

[1481] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing and transmitting images of a baby to the server, means for detecting the baby's behavior from the image data using an image analysis algorithm, means for evaluating the baby's growth stage based on the detected behavior using a machine learning model, means for generating childcare advice based on the evaluation results, means for transmitting and displaying the childcare advice on a terminal, and means for providing childcare advice to a terminal used by a customer in a baby goods store. This enables the provision of appropriate childcare advice in real time according to the growth stage of each baby, and allows parents to choose products with confidence in a baby goods store.

[1482] "Means for capturing images of a baby and sending them to a server" refers to a device that includes a camera for taking images of a baby and a communication interface for sending the captured video data to a central server via a network.

[1483] An "image analysis algorithm" is a set of computational procedures for identifying and extracting specific patterns or movements from video data, and is a software program used to detect a baby's movements.

[1484] A "machine learning model" is an algorithm that learns from data and uses that knowledge to analyze and evaluate new data. Specifically, it is used to evaluate a baby's developmental stage using behavioral data.

[1485] A "means for generating childcare advice" refers to a software program that automatically generates appropriate childcare guidance and suggestions based on evaluation results and provides them to parents.

[1486] "Means of sending and displaying on a device" refers to a device that has the function of sending the generated childcare advice to a user's smartphone, tablet, or other device and displaying it on its screen.

[1487] "Terminals used by customers in baby product stores" refers to information terminals installed in physical stores that sell baby products, which customers use to receive childcare advice.

[1488] "A baby's movements such as rolling over, sitting up, and crawling" refer to specific physical movements that occur during a baby's developmental stages and are important indicators for evaluating their developmental stage.

[1489] "Behavioral history" refers to a record of a baby's past actions and behaviors, and is data used to predict future behavior and potential risk factors.

[1490] "Potential risk factors" refer to dangerous situations or risks that a baby may face in the future, and serve as foundational data to provide preventative advice.

[1491] This invention is a system that monitors a baby's growth in real time and provides parents with appropriate childcare advice. The system functions through a series of processes including video capture, video analysis, machine learning evaluation, advice generation, and notification display. It can also provide childcare advice to customer terminals in baby product stores.

[1492] Data collection and transmission

[1493] The device is equipped with a camera that captures video of the baby in real time. The captured video data is compressed and sent to the server via a secure communication protocol (e.g., HTTPS).

[1494] Video analysis

[1495] The server processes the video data received from the terminal. As a preprocessing step, it removes noise and prepares the data for analysis. Then, it uses OpenCV or similar software to detect the baby's movements (rolling over, sitting up, crawling, etc.) from the video data.

[1496] Evaluation using machine learning

[1497] The server uses a machine learning model (e.g., built using TensorFlow or Keras) to evaluate the baby's developmental stage based on detected behavioral data. The model is input with data obtained during the feature extraction process to determine the developmental stage. In this process, future behaviors and potential risk factors are also predicted based on the behavioral history.

[1498] Advice generation

[1499] The server generates specific parenting advice based on the evaluation and prediction results. The generated advice is formatted in a way that is easy for parents to understand. For example, it might say, "Your baby has started rolling over. Next, try practicing sitting up."

[1500] Notification sent and displayed

[1501] The server sends the generated parenting advice to the terminal. The terminal notifies the user of the received advice and displays the specific content of the advice. In a baby goods store, the advice is displayed on the terminal used by the customer, allowing parents to choose appropriate products in the store.

[1502] Specific example

[1503] Let's take the example of a baby who has just started rolling over in their crib.

[1504] Device: The camera captures the baby rolling over and sends this video data to the server.

[1505] Server: The server analyzes the video data and detects when the baby rolls over. The detected data is input into a machine learning model, which evaluates whether the baby is in a developmental stage (the stage of rolling over). Next, the server generates parenting advice such as, "Your baby has started rolling over. Next, try practicing sitting up."

[1506] Device: Smartphones and tablets receive this parenting advice and notify the user. When the user opens the app, they can view the specific advice.

[1507] Examples of prompts to input into a generative AI model

[1508] Please create a program for a system that captures a baby's behavior with a camera and provides parenting advice based on that behavior. The advice should include specific product suggestions for parents when they visit a baby goods store.

[1509] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1510] Step 1:

[1511] The device activates the camera and captures video of the baby in real time. The captured video data is compressed and sent to the server via a secure communication protocol (e.g., HTTPS). The input here is real-time video data, and the output is compressed video data.

[1512] Step 2:

[1513] The server processes the video data received from the terminal. First, it preprocesses the data by removing noise and preparing it for analysis. Then, it uses OpenCV or similar software to detect the baby's movements (e.g., rolling over, sitting up, crawling) from the video data. The input is compressed video data, and the output is the detected baby movement data.

[1514] Step 3:

[1515] The server uses a machine learning model based on detected behavioral data to evaluate the baby's developmental stage. Data obtained during the feature extraction process is input into the model to determine the developmental stage. Furthermore, during this process, future behaviors and potential risk factors are predicted based on the behavioral history. The input is the detected behavioral data, and the output is the developmental stage and predicted behaviors.

[1516] Step 4:

[1517] The server generates specific parenting advice based on the evaluation and prediction results. The generated advice is formatted in a way that is easy for parents to understand. For example, it might say, "Your baby has started rolling over. Next, try practicing sitting up." The input is the developmental stage and predicted behavior, and the output is the text of the parenting advice.

[1518] Step 5:

[1519] The server sends the generated childcare advice to the terminal. The input here is the text of the childcare advice, and the output is communication data.

[1520] Step 6:

[1521] The device notifies the user of received advice and displays the specific advice on the app screen. The input here is communication data from the server, and the output is the displayed text of the parenting advice. The user checks the notification and then views the detailed parenting advice within the app.

[1522] Step 7:

[1523] In baby product stores, parenting advice is ultimately displayed on a terminal used by the customer. This allows parents to choose appropriate products within the store. The input is the text of the parenting advice, and the output is the displayed product suggestions.

[1524] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1525] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1526] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1527] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1528] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1529] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1530] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1531] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1532] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1533] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1534] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1535] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1536] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1537] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1538] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1539] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1540] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1541] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1542] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1543] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1544] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1545] The following is further disclosed regarding the embodiments described above.

[1546] (Claim 1)

[1547] A means of capturing video of a baby and sending it to a server,

[1548] A means for detecting the baby's behavior from the video data using an image analysis algorithm,

[1549] A means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model,

[1550] A means for generating childcare advice based on the aforementioned evaluation results,

[1551] A means for transmitting and displaying the aforementioned childcare advice on a terminal,

[1552] A system that includes this.

[1553] (Claim 2)

[1554] The system according to claim 1, wherein the means for detecting the baby's actions includes an algorithm for analyzing the baby's movements such as rolling over, sitting up, and crawling.

[1555] (Claim 3)

[1556] The system according to claim 1, wherein the means for evaluating the baby's growth stage includes a machine learning model that predicts future behavior and potential risk factors based on the baby's behavioral history.

[1557] "Example 1"

[1558] (Claim 1)

[1559] A means of capturing video of a baby and sending it to a server,

[1560] A means of preprocessing by applying a noise reduction filter,

[1561] A means for detecting the baby's actions from the video data using an object detection algorithm,

[1562] Methods for extracting baby behavioral data,

[1563] A means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model,

[1564] A means for generating childcare advice based on the aforementioned evaluation results,

[1565] A means for sending the aforementioned childcare advice to a terminal for notification and display,

[1566] A system that includes this.

[1567] (Claim 2)

[1568] The system according to claim 1, wherein the means for detecting the baby's actions includes an algorithm for analyzing the baby's movements such as rolling over, sitting up, and crawling.

[1569] (Claim 3)

[1570] The system according to claim 1, wherein the means for evaluating the growth stage of the baby includes a machine learning model that predicts future behavior and potential risk factors based on past behavioral history.

[1571] "Application Example 1"

[1572] (Claim 1)

[1573] A means of capturing video of a baby and sending it to a data server,

[1574] A means for detecting the baby's behavior from the video data using an image analysis algorithm,

[1575] A means for evaluating the developmental stage of an infant based on the detected behavior using a machine learning model,

[1576] A means for generating childcare advice based on the aforementioned evaluation results,

[1577] A means for transmitting and displaying the aforementioned childcare advice on a user terminal,

[1578] A means of recommending childcare-related products in conjunction with virtual stores,

[1579] A system that includes this.

[1580] (Claim 2)

[1581] The system according to claim 1, wherein the means for detecting the baby's actions includes an algorithm for analyzing the baby's movements such as rolling over, sitting up, and crawling.

[1582] (Claim 3)

[1583] The system according to claim 1, wherein the means for evaluating the developmental stage of the infant includes a machine learning model that predicts future behavior and potential risk factors based on the infant's behavioral history.

[1584] "Example 2 of combining an emotion engine"

[1585] (Claim 1)

[1586] A means of capturing video of a baby and sending it to a server,

[1587] A means for detecting the baby's behavior from the video data using an image analysis algorithm,

[1588] A means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model,

[1589] A means for generating childcare advice based on the aforementioned evaluation results,

[1590] A means for transmitting and displaying the aforementioned childcare advice on a terminal,

[1591] A means of capturing the user's facial expressions and voice tone and analyzing their emotional state,

[1592] A means for customizing parenting advice based on the aforementioned emotional state,

[1593] A system that includes this.

[1594] (Claim 2)

[1595] The system according to claim 1, wherein the means for detecting the baby's actions includes an algorithm for analyzing the baby's movements such as rolling over, sitting up, and crawling.

[1596] (Claim 3)

[1597] The system according to claim 1, wherein the means for evaluating the baby's growth stage includes a machine learning model that predicts future behavior and potential risk factors based on the baby's behavioral history.

[1598] "Application example 2 when combining with an emotional engine"

[1599] (Claim 1)

[1600] A means of capturing video of a baby and sending it to a server,

[1601] A means for detecting the baby's behavior from the video data using an image analysis algorithm,

[1602] A means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model,

[1603] A means for generating childcare advice based on the aforementioned evaluation results,

[1604] A means for transmitting and displaying the aforementioned childcare advice on a terminal,

[1605] A means of providing childcare advice to customers using terminals in a baby goods store,

[1606] A system that includes this.

[1607] (Claim 2)

[1608] The system according to claim 1, wherein the means for detecting the baby's actions includes an algorithm for analyzing the baby's movements such as rolling over, sitting up, and crawling.

[1609] (Claim 3)

[1610] The system according to claim 1, wherein the means for evaluating the baby's growth stage includes a machine learning model that predicts future behavior and potential risk factors based on the baby's behavioral history. [Explanation of symbols]

[1611] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of capturing video of a baby and sending it to a server, A means for detecting the baby's behavior from the video data using an image analysis algorithm, A means for evaluating the baby's developmental stage based on the detected behavior using a machine learning model, A means for generating childcare advice based on the aforementioned evaluation results, A means for transmitting and displaying the aforementioned childcare advice on a terminal, A system that includes this.

2. The system according to claim 1, wherein the means for detecting the baby's actions includes an algorithm for analyzing the baby's movements such as rolling over, sitting up, and crawling.

3. The system according to claim 1, wherein the means for evaluating the baby's growth stage includes a machine learning model that predicts future behavior and potential risk factors based on the baby's behavioral history.

Images