system

Abstract

We provide the system. [Solution] A system comprising: a camera means for detecting the baby's posture and facial condition in real time; a vibration sensor means for detecting the baby's breathing; a temperature and humidity sensor means for measuring the temperature and humidity of the bedroom; an illuminance sensor means for measuring the brightness of the bedroom; a communication means for transmitting data from the camera means, vibration sensor means, temperature and humidity sensor means and illuminance sensor means to a server; an analysis means on the server for monitoring the baby's safe and comfortable environment by analyzing the data; a notification means for notifying the user when an abnormality is detected by the analysis means; and a control means for controlling the bedroom environment based on the analysis means.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern child-rearing, it is very difficult for parents to comprehensively manage the safety and comfort environment of a baby. Although individual monitoring services and devices are provided, there are few systems that can manage them collectively. Furthermore, there is a lack of comprehensive measures to quickly respond to dangerous situations such as a baby lying face down or being covered, or emergencies such as breathing cessation. In addition, environmental adjustments such as temperature, humidity, and illuminance also need to be manually performed, imposing a heavy burden on parents. From these points, there is a problem that it is difficult for parents to always watch over a baby with peace of mind.

Means for Solving the Problems

[0005] This invention provides a system that comprehensively monitors and manages warning notifications for the safety of a baby and the bedroom environment. This system includes the following means:

[0006] 1. A camera system that detects the baby's posture and facial condition in real time.

[0007] 2. A vibration sensor means for detecting the baby's breathing.

[0008] 3. A temperature and humidity sensor means for measuring the temperature and humidity of a bedroom.

[0009] 4. An illuminance sensor means for measuring the brightness of a bedroom.

[0010] 5. Communication means for transmitting data from the camera means, vibration sensor means, temperature and humidity sensor means, and illuminance sensor means to the server.

[0011] 6. Analysis means on the server to monitor the safety and comfort of the baby by analyzing the data.

[0012] 7. Notification means for notifying the user when an abnormality is detected by the analysis means.

[0013] 8. Control means for controlling the bedroom environment based on the analysis means.

[0014] Furthermore, this system also includes the following:

[0015] A learning method that learns individual patterns in conjunction with childcare records, and an advice method that provides timely advice and notifications to the user based on the learning method. This allows parents to manage their baby's condition and environment in one place and respond quickly as needed, enabling them to raise their child with peace of mind.

[0016] A "camera system" is a device that detects the baby's posture and facial condition in real time and acquires video footage of it.

[0017] "The "vibration sensor means" is a device installed on a bed or the like to detect a baby's breathing and sense minute movements."

[0018] "The "temperature and humidity sensor means" is a device for measuring the temperature and humidity in a baby's bedroom."

[0019] "The "illuminance sensor means" is a device for measuring the brightness in a baby's bedroom."

[0020] "The "communication means" is a device for transmitting data from the camera means, vibration sensor means, temperature and humidity sensor means, and illuminance sensor means to a server."

[0021] "The "analysis means" is a device for analyzing the data received by the server and monitoring the safety and comfort environment of the baby."

[0022] "The "notification means" is a device for notifying the user when an abnormality is detected by the analysis means."

[0023] "The "control means" is a device for managing and adjusting the environment of the bedroom based on the analysis means."

[0024] "The "learning means" is a device for learning the individual patterns of the baby in cooperation with the childcare records."

[0025] "The "advice means" is a device for giving timely advice and notifications to the parent based on the learning means."

Brief Description of the Drawings

[0026] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment." [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment." [Figure 3]This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0027] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0028] First, let's explain the terminology used in the following explanation.

[0029] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0030] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0031] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0032] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0033] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0034] [First Embodiment]

[0035] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0036] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0037] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0038] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0039] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0040] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0041] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0042] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0043] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0044] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0045] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0046] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0047] A specific embodiment of the "Baby Safe Watch" of the present invention is described below. This system consists of multiple terminals, a server, and a user (parent), each with its own role.

[0048] System Configuration

[0049] 1. Terminal

[0050] Camera equipment: A camera is installed to detect the baby's posture and facial condition in real time.

[0051] Vibration sensor means: A vibration sensor is installed in the bed to detect the baby's breathing.

[0052] Temperature and humidity sensor means: Install a sensor to measure the temperature and humidity in the bedroom.

[0053] Illuminance sensor means: Install a sensor to measure the brightness of the bedroom.

[0054] Communication means: Equipped with a communication device for transmitting data acquired from the above sensors to a server.

[0055] 2. Server

[0056] Analysis method: The server receives data transmitted from the terminal and performs analysis using advanced algorithms.

[0057] Notification mechanism: The system includes a mechanism for notifying the user according to the analysis results. If an anomaly is detected, an emergency notification will be sent.

[0058] Control mechanism: Based on the analysis results, it sends instructions to the terminal to automatically control environmental factors such as temperature, humidity, and illuminance.

[0059] Learning method: It works in conjunction with a childcare record application to learn the individual patterns of each baby.

[0060] Advice method: Based on learned patterns, it sends timely advice and notifications to the user.

[0061] 3. User

[0062] Using devices such as smartphones and tablets, users can receive notifications and manage their baby's condition and environment through the application.

[0063] Explanation of the program's processing

[0064] Data acquisition and transmission (terminal)

[0065] Terminal: The camera module captures images of the baby and sends them to the server for real-time analysis.

[0066] Terminal: The vibration sensor detects the baby's breathing in real time and sends the data to the server.

[0067] Terminal: Temperature, humidity, and illuminance sensors also measure environmental data and transmit it to the server at regular intervals.

[0068] Data analysis and anomaly detection (server)

[0069] Server: Analyzes received data and monitors the baby's posture, facial expression, breathing patterns, and room environment.

[0070] Server: If an anomaly is detected as a result of the analysis, an emergency notification will be sent to the user's smartphone.

[0071] For example, if the baby turns onto its stomach, an alert is sent immediately.

[0072] An emergency notification will also be issued if breathing stops for 20 seconds or more.

[0073] Environmental control (server)

[0074] Server: Based on the analysis results, it sends instructions to terminals to automatically control air conditioning and lighting in order to maintain the optimal indoor environment.

[0075] If the room temperature exceeds the set range, the server sends a cooling command to the air conditioner.

[0076] If the room is too dark, adjust the lighting appropriately.

[0077] Learning and advice (server)

[0078] Server: It connects with childcare records and learns the baby's daily routines (sleep, eating, elimination, etc.).

[0079] Server: Based on learned data, it sends timely advice and notifications to the user.

[0080] For example, if the system learns that the baby usually goes to bed at 8 PM, it will send a notification 30 minutes beforehand saying, "It's time for the baby to sleep. Please create a quiet environment."

[0081] Specific example

[0082] Common usage examples

[0083] Terminal: The camera detects the baby's posture and sends the data to the server.

[0084] Server: If the analysis determines that everything is normal, it notifies the user that "the baby is safe."

[0085] Terminal: The temperature and humidity sensor detects when the indoor temperature is too high and sends data to the server.

[0086] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[0087] Examples of use in emergencies

[0088] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[0089] Server: The system determines this is an emergency and sends an emergency notification to the parents stating, "There is an abnormality in the baby's breathing."

[0090] User: Receive notifications on your smartphone and respond quickly.

[0091] This concludes the description of the embodiment of the "Baby Safe Watch" of the present invention. By using this system, parents can manage the safety and comfortable childcare environment of their baby in one place, thereby reducing the burden of childcare.

[0092] The following describes the processing flow.

[0093] Step 1:

[0094] Terminal: The camera module acquires real-time video of the baby. It captures 30 frames per second of video and temporarily stores it in local storage.

[0095] Step 2:

[0096] Terminal: Transmits the acquired video to the server via a communication method. The data is transmitted using a secure protocol (e.g., HTTPS).

[0097] Step 3:

[0098] Device: A vibration sensor detects the baby's breathing. The vibration sensor collects data every second and sends it to the server.

[0099] Step 4:

[0100] Terminal: A temperature and humidity sensor measures the temperature and humidity in the bedroom. It acquires data every 10 seconds and sends it to the server.

[0101] Step 5:

[0102] Terminal: An illuminance sensor measures the brightness of the bedroom. It acquires data every 5 seconds and sends it to the server.

[0103] Step 6:

[0104] Server: Analyzes all received data. Analyzes camera footage to recognize the baby's posture and facial expression. Analyzes vibration sensor data to detect breathing patterns. Analyzes temperature, humidity, and illuminance data to evaluate the indoor environment.

[0105] Step 7:

[0106] Server: Detects anomalies based on analysis results. Determines an emergency situation, such as when the baby is lying face down or has stopped breathing.

[0107] Step 8:

[0108] Server: If an anomaly is detected, an emergency notification will be sent to the user. The notification will be sent to the smartphone in the form of a push notification or email.

[0109] Step 9:

[0110] Server: Generates environmental control commands based on the analysis results. For example, if the room temperature exceeds the set range, it sends a cooling command to the air conditioner.

[0111] Step 10:

[0112] Terminal: Receives instructions from the server and controls environmental factors such as air conditioning and lighting. This includes adjusting the air conditioner's temperature setting and changing the brightness of the lights.

[0113] Step 11:

[0114] Server: It works in conjunction with the childcare record application to learn the baby's daily routine. Based on the collected data, it analyzes the baby's sleep, eating, and elimination patterns.

[0115] Step 12:

[0116] Server: Sends advice and notifications to the user based on the learning results. For example, 30 minutes before the usual bedtime, it might notify the user, "It's time for the baby to sleep. Please create a quiet environment."

[0117] The above outlines the specific processing steps for "Baby Safe Watch." This process allows for comprehensive management of your baby's safety and comfortable environment.

[0118] (Example 1)

[0119] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0120] In modern childcare, parents need to constantly monitor their baby's safety, but 24-hour monitoring is physically and psychologically burdensome. Optimizing the indoor environment is also crucial for maintaining the baby's safety and comfort, but manual adjustments are time-consuming. Furthermore, the difficulty in obtaining appropriate advice based on the baby's daily routines leads to decreased efficiency in childcare.

[0121] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0122] In this invention, the server includes a camera means for detecting the baby's posture and facial condition in real time, a vibration sensor means for detecting the baby's breathing, an environmental sensor means for measuring the temperature and humidity of the bedroom, an illuminance sensor means for measuring the brightness of the bedroom, a communication means for transmitting data from the camera means, vibration sensor means, environmental sensor means and illuminance sensor means to the server, an analysis means in the server for monitoring the baby's safe and comfortable environment by analyzing the data, a notification means for notifying the user when the analysis means detects an abnormality, a control means for controlling the bedroom environment based on the analysis means, a learning means for learning individual patterns in conjunction with childcare records, and an advice means for providing timely advice and notifications based on the learned patterns. This makes it possible to automatically and efficiently maintain the baby's safety and comfort and reduce the burden of childcare on parents.

[0123] A "camera system" is a device used to detect the baby's posture and facial condition in real time.

[0124] A "vibration sensor" is a device used to detect a baby's breathing.

[0125] An "environmental sensor" is a device used to measure the temperature and humidity of a bedroom.

[0126] An "illuminance sensor" is a device used to measure the brightness of a bedroom.

[0127] "Communication means" refers to a device for transmitting data acquired from camera means, vibration sensor means, environmental sensor means, and illuminance sensor means to a server.

[0128] "Analysis means" refers to a device that analyzes data transmitted via communication means to monitor the safety and comfort of the baby.

[0129] A "notification means" is a device used to notify the user when an anomaly is detected in the analysis means.

[0130] The "control means" is a device for automatically controlling the bedroom environment based on the analysis means.

[0131] A "learning tool" is a device that works in conjunction with childcare records to learn individual patterns.

[0132] An "advice tool" is a device that provides timely advice and notifications based on learned patterns.

[0133] A specific embodiment of the "Baby Safe Watch" of the present invention is described below. This system consists of multiple terminals, a server, and a user (parent), each with its own role.

[0134] System Configuration

[0135] 1. Terminal

[0136] Camera equipment: A camera is installed to detect the baby's posture and facial condition in real time. For example, it can be used to check if the baby is rolling over or if their face is covered by the blanket.

[0137] Vibration sensor means: A vibration sensor is installed in the bed to detect the baby's breathing. For example, the breathing pattern is obtained from the movement of the baby's chest.

[0138] Environmental sensor means: Install sensors to measure the temperature and humidity in the bedroom. For example, obtain data such as a room temperature of 25 degrees Celsius and a humidity of 50%.

[0139] Illuminance sensor means: Install a sensor to measure the brightness of the bedroom. For example, confirm that the brightness of the room is 400 lux.

[0140] Communication means: The system is equipped with a communication device for transmitting data acquired from the above-mentioned sensors to a server. For example, a Wi-Fi module is used to transmit the data.

[0141] 2. Server

[0142] Analysis method: The server receives data transmitted from the terminal and performs analysis using advanced algorithms. For example, it analyzes the baby's posture, breathing status, and the indoor environment.

[0143] Notification method: The system includes a means to notify the user according to the analysis results. If an abnormality is detected, an emergency notification will be sent. For example, a notification such as "An abnormality has been detected in the baby's breathing" will be sent to the user.

[0144] Control mechanism: Based on the analysis results, it sends instructions to the terminal to automatically control environmental conditions such as temperature, humidity, and illuminance. For example, if the room temperature exceeds the set range, it sends an instruction to the air conditioner to start cooling.

[0145] Learning method: It works in conjunction with a childcare record application to learn the individual patterns of the baby. For example, it learns the baby's daily patterns such as sleep, eating, and elimination.

[0146] Advice methods: Based on learned patterns, the system sends timely advice and notifications to the user. For example, for a baby who has a habit of going to bed at 8 pm, it might send a notification 30 minutes beforehand saying, "It's time for your baby to sleep. Please create a quiet environment."

[0147] 3. User

[0148] Users can receive notifications from the server using devices such as smartphones and tablets, and manage their baby's condition and environment through the application. For example, a user who receives an emergency notification can quickly rush to the baby's bed and check on their baby's condition.

[0149] Specific example

[0150] Common usage examples

[0151] Terminal: The camera detects the baby's posture and sends the data to the server.

[0152] Server: If the analysis determines that everything is normal, it notifies the user that "the baby is safe."

[0153] Terminal: The environmental sensor detects that the indoor temperature is too high and sends data to the server.

[0154] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[0155] Examples of use in emergencies

[0156] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[0157] Server: Determined to be in an emergency, it sends an emergency notification to the user stating, "There is an abnormality in the baby's breathing."

[0158] User: Receives a notification on their smartphone, rushes to the bed immediately, and checks on the baby's condition.

[0159] Example of a prompt

[0160] Example prompt:

[0161] BabySafeWatch is a system that monitors and controls a baby's posture, breathing, and indoor environment. What kind of data does it typically collect, and how does it notify the user?

[0162] How will users be notified in an emergency, and how will they respond?

[0163] This concludes the description of the embodiment of the "Baby Safe Watch" of the present invention. By using this system, parents can manage the safety and comfortable childcare environment of their baby in one place, thereby reducing the burden of childcare.

[0164] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0165] Step 1: Data Acquisition (Device)

[0166] Input: Information about the baby's posture and facial expression, breathing, room temperature, humidity, and light level.

[0167] Specific operation: The camera on the device captures images of the baby. A vibration sensor detects the baby's breathing, an environmental sensor measures the room temperature and humidity, and an illuminance sensor measures the room brightness.

[0168] Output: Acquired video data, respiration data, and environmental data (temperature, humidity, illuminance) are generated.

[0169] Step 2: Data transmission (terminal)

[0170] Input: Data acquired from each sensor (video data, respiratory data, environmental data)

[0171] Specific operation: Data acquired by sensors and cameras is transmitted to a server using communication means. A Wi-Fi module is used as the specific means.

[0172] Output: Data acquired by sensors and cameras is sent to the server.

[0173] Step 3: Data Analysis (Server)

[0174] Input: Data from each sensor sent to the server (video data, respiration data, environmental data)

[0175] Specific operation: The server's analysis system receives the data and uses advanced algorithms to analyze the baby's posture, breathing, and room environment. For example, it analyzes whether the baby's posture is normal and whether the breathing pattern is abnormal.

[0176] Output: Analysis results (determination of normal or abnormal, indoor environment status)

[0177] Step 4: Anomaly detection and notification (server)

[0178] Input: Results of data analysis

[0179] Specific operation: If an anomaly is detected by the server's analysis system, the notification system will inform the user. Specifically, an emergency notification will be sent if the baby is lying face down or if breathing stops for more than 20 seconds.

[0180] Output: An emergency notification is sent to the user.

[0181] Step 5: Environment Control (Server)

[0182] Input: Results of data analysis

[0183] Specific operation: Based on the analysis results, the server's control system sends instructions to the terminal to automatically adjust environmental conditions such as temperature, humidity, and illuminance. For example, if the room temperature exceeds the set range, it sends a cooling command to the air conditioner.

[0184] Output: An instruction for environment adjustment is sent to the terminal.

[0185] Step 6: Data Training (Server)

[0186] Input: Daily life pattern data from a childcare record application.

[0187] Specific operation: The server's learning mechanism works in conjunction with the childcare record application to learn the individual patterns of the baby. For example, it analyzes and learns data such as the baby's sleep, meals, and bowel movements.

[0188] Output: Learning results (insights about babies' daily routines)

[0189] Step 7: Advice and Notification (Server)

[0190] Input: Learning results and data analysis results

[0191] Specific operation: Based on patterns learned by the server's advice system, timely advice and notifications are sent to the user. For example, for a baby who has a habit of going to bed at 8 pm, a notification is sent 30 minutes beforehand saying, "It's time for your baby to sleep. Please create a quiet environment."

[0192] Output: Appropriate advice and notifications are sent to the user.

[0193] (Application Example 1)

[0194] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0195] Ensuring the safety of babies and children and reducing anxiety for parents and guardians while they engage in childcare or shopping activities is crucial. However, current systems lack the functionality to monitor children's safety in stores in real time, in addition to monitoring the baby's environment, and to quickly notify parents and staff in case of abnormalities. Therefore, there is a need to develop a new system that allows parents and guardians to watch over their children with greater peace of mind.

[0196] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0197] In this invention, the server includes an image acquisition means for detecting the baby's posture and facial condition in real time, a vibration detection means for detecting the baby's breathing, a climate data acquisition means for measuring the temperature and humidity of the bedroom, a light intensity sensor means for measuring the brightness of the bedroom, a communication device for transmitting data from the image acquisition means, vibration detection means, climate data acquisition means and light intensity sensor means to a central processing unit, an analysis means in the central processing unit for monitoring the baby's safety and comfort by analyzing the data, a notification means for notifying the user when an abnormality is detected by the analysis means, an environment control means for controlling the bedroom environment based on the analysis means, a means for monitoring the safety of children in the store in real time, and a means for notifying parents or staff when a child enters a specific area. This ensures a safe and comfortable environment for babies, and further enables real-time monitoring of children's safety in the store and a rapid response in the event of an abnormality.

[0198] The "image acquisition means" is a device for detecting the baby's posture and facial condition in real time.

[0199] A "vibration detection device" is a device for detecting a baby's breathing in real time.

[0200] "Climate data acquisition means" refers to a device for measuring the temperature and humidity of a bedroom.

[0201] A "light intensity sensor" is a device used to measure the brightness of a bedroom.

[0202] A "communication device" is a device used to transmit data acquired from various sensors to a central processing unit.

[0203] A "central processing unit" is a device that analyzes received data and monitors the safety and comfort of babies and children.

[0204] An "analysis tool" is a device used to analyze data and detect anomalies.

[0205] A "notification means" is a device that notifies the user when an abnormality is detected in the analysis means.

[0206] An "environmental control means" is a device that controls the bedroom environment based on an analysis means.

[0207] "A means of monitoring the safety of children in stores in real time" refers to a device that monitors the safety of children in stores in real time and notifies the staff if any abnormalities are detected.

[0208] "Means of notifying parents or staff" refers to a device that notifies parents or store staff when a child enters a specific area.

[0209] A specific embodiment of the present invention, "Kids Safe Watch," is described below. The system consists of multiple terminals, a central processing unit, and users, each with its own role.

[0210] System Configuration

[0211] 1. Terminal

[0212] Image acquisition method: Install a camera to detect the child's posture and facial condition in real time.

[0213] Vibration detection method: A vibration sensor is installed on the floor to detect the breathing and position of the baby or child.

[0214] Climate data acquisition method: Install sensors to measure the temperature and humidity in the store.

[0215] Light intensity sensor means: Install a sensor to measure the brightness of the store.

[0216] Communication device: Equipped with a communication device for transmitting data acquired from the above-mentioned sensors to the central processing unit.

[0217] 2. Central Processing Unit

[0218] Analysis method: The central processing unit receives data transmitted from the terminal and performs analysis using advanced algorithms.

[0219] Notification method: The system includes a means to notify the user according to the analysis results. If an anomaly is detected, an emergency notification will be sent.

[0220] Environmental control means: Based on the analysis results, instructions are sent to the terminal to automatically control environmental conditions such as temperature, humidity, and illuminance.

[0221] 3. User

[0222] Using devices such as smartphones and tablets, users can receive notifications and manage their child's status and environment through the application.

[0223] Users can monitor the location and behavior of children within the store and respond quickly as needed.

[0224] Explanation of the program's processing

[0225] Hardware and software to use

[0226] Hardware: Smartphones, tablets, camera modules installed in stores, vibration sensors, temperature and humidity sensors, and illuminance sensors.

[0227] Software: Python (for data analysis and notification implementation), AWS® cloud services (for data storage and processing), Firebase (real-time database and notification service), OpenCV (image analysis library).

[0228] Data acquisition and calculation processing

[0229] 1. Processing camera data

[0230] Using OpenCV, we detect and analyze children's posture and position from camera images.

[0231] The analysis data is sent to AWS in real time.

[0232] 2. Processing of sensor data

[0233] The system periodically acquires data from vibration sensors, temperature and humidity sensors, and illuminance sensors and sends it to AWS.

[0234] 3. Notification function

[0235] Using Firebase, users receive real-time notifications on their smartphones when an anomaly is detected.

[0236] 4. Environmental control

[0237] Data analysis is performed on AWS, and instructions for controlling air conditioners and lighting are sent to the terminal as needed.

[0238] Specific usage examples

[0239] Common usage examples

[0240] Terminal: The camera detects the posture and position of children in the store and transmits the data to the central processing unit.

[0241] Central Processing Unit: If it analyzes the data and determines that it is normal, it notifies the user that "the child is safe."

[0242] Terminal: The temperature and humidity sensor detects if the temperature inside the store is too high and sends data to the central processing unit.

[0243] Central Processing Unit: Sends cooling instructions to the air conditioner and adjusts it to the appropriate temperature.

[0244] Examples of use in emergencies

[0245] Terminal: If the vibration sensor detects that a child has left a designated area, it transmits that data to the central processing unit.

[0246] Central Processing Unit: Determines an emergency and sends an emergency notification to parents or store staff stating, "A child has left a designated area."

[0247] User: Receive notifications on your smartphone and respond quickly.

[0248] Examples of prompts for generative AI models

[0249] Please generate examples of how "Kids Safe Watch" can be used. Describe the details of a system that uses in-store cameras and sensors to manage child safety and notifies parents and store staff in real time if any anomalies are detected. Include specific use cases, such as notifications for when a child leaves the play area or when a child remains in the store after closing time.

[0250] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0251] Step 1:

[0252] Terminal: A camera module is used to detect the child's posture and position. The camera module acquires image data in real time from a designated area within the store. This image data is used as input data. The acquired image data is sent to the central processing unit using the OpenCV library for data preprocessing.

[0253] Step 2:

[0254] Server: The central processing unit receives pre-processed image data. It then uses OpenCV to execute face recognition and posture detection algorithms to extract the child's posture and location information. The extracted posture and location information is generated as output data, and the process proceeds to the next analysis step.

[0255] Step 3:

[0256] Terminal: Uses vibration sensors to acquire vibration data from children's floors and play areas installed in specific areas within the store. The vibration sensors transmit the detected vibration data to the central processing unit in real time. The vibration data is used as input data.

[0257] Step 4:

[0258] Server: The central processing unit receives data from vibration sensors and analyzes it using a rational threshold determination algorithm. Based on the analysis of the vibration data, it determines whether the child has deviated from a specific area. The determination result is generated as output data, and if an anomaly is detected, the process proceeds to the notification step.

[0259] Step 5:

[0260] Terminal: Using a climate data acquisition device, temperature and humidity data inside the store are acquired periodically. This data is used as input data and transmitted to the central processing unit in real time.

[0261] Step 6:

[0262] Server: The central processing unit receives temperature and humidity data transmitted from the climate data acquisition means and performs analysis to determine if it is within an appropriate range. Based on the analysis results, it determines whether the optimal temperature is being maintained. If the temperature deviates from the optimal range, an instruction to adjust using the environmental control means is output.

[0263] Step 7:

[0264] Terminal: Uses a light intensity sensor to acquire brightness (illuminance) data within the store. The brightness data is used as input data and transmitted to the central processing unit in real time.

[0265] Step 8:

[0266] Server: The central processing unit receives data from the light intensity sensor and performs analysis to determine whether the illuminance is within an appropriate range. Based on the analysis results, it determines whether the lighting is maintained within an appropriate range. If the illuminance deviates from the appropriate range, an instruction is output to adjust using the environmental control means.

[0267] Step 9:

[0268] Server: The server comprehensively analyzes all data acquired to date (image data, vibration data, temperature and humidity data, light intensity data) and, if an anomaly is detected, uses a notification system to notify users and store staff in real time. The notification content is generated and sent as output data.

[0269] Step 10:

[0270] Users: Customers or store staff receive notifications on their smartphones or tablets and take prompt action based on the displayed notification content. They review the notification and initiate specific response actions.

[0271] The above outlines the specific processing steps of the system program that implements the application example.

[0272] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0273] The "Baby Safe Watch" system is a comprehensive system for managing a baby's safety and a comfortable childcare environment. It incorporates an emotion engine that recognizes the user's emotions and adjusts notifications and support accordingly. This system consists of multiple terminals, a server, and the user (parent), each with its own specific role.

[0274] System Configuration

[0275] 1. Terminal

[0276] Camera means: Install a camera for detecting the baby's posture and facial condition in real time.

[0277] Vibration sensor means: Install a vibration sensor on the bed to detect the baby's breathing.

[0278] Temperature and humidity sensor means: Install a sensor for measuring the temperature and humidity in the bedroom.

[0279] Illuminance sensor means: Install a sensor for measuring the brightness in the bedroom.

[0280] Communication means: Equip with a communication device for transmitting the data obtained from the above sensors to the server.

[0281] Emotion engine means: Obtain audio and video data to recognize the user's emotion.

[0282] 2. Server

[0283] Analysis means: The server receives the data transmitted from the terminal and performs analysis using advanced algorithms.

[0284] Notification means: Equip with means for notifying the user according to the analysis result. In case of detecting an abnormality, an emergency notification is made.

[0285] Control means: Based on the analysis result, send an instruction to the terminal to automatically control the environment such as temperature, humidity, and illuminance.

[0286] Learning means: Cooperate with the childcare record application to learn the individual patterns of the baby.

[0287] Advice means: Based on the learned patterns, send timely advice and notifications to the user.

[0288] 3. User

[0289] Using devices such as smartphones and tablets, users can receive notifications and manage their baby's condition and environment through the application.

[0290] Explanation of the program's processing

[0291] Data acquisition and transmission (terminal)

[0292] Terminal: The camera module captures images of the baby and sends them to the server for real-time analysis.

[0293] Terminal: The vibration sensor detects the baby's breathing in real time and sends the data to the server.

[0294] Terminal: Temperature, humidity, and illuminance sensors also measure environmental data and transmit it to the server at regular intervals.

[0295] User emotion recognition (device)

[0296] Terminal: The emotion engine acquires the user's voice and video data.

[0297] Terminal: Sends acquired emotion data to the server.

[0298] Data analysis and anomaly detection (server)

[0299] Server: Analyzes received data and monitors the baby's posture, facial expression, breathing patterns, and room environment.

[0300] Server: Further analyzes the user's emotional data obtained by the emotion engine.

[0301] Server: Sends appropriate notifications to the user if an abnormality is detected in the baby or if the user is determined to be experiencing stress.

[0302] Environmental control (server)

[0303] Server: To maintain an optimal indoor environment based on the analysis results, it automatically sends instructions to the terminal to control the air conditioner and lighting.

[0304] Terminal: Adjust the set temperature of the air conditioner or change the brightness of the lighting.

[0305] Learning and Advice (Server)

[0306] Server: Collaborate with the childcare record application to learn the baby's living patterns (such as sleep, meals, excretion, etc.).

[0307] Server: Based on the learned data, it sends timely advice and notifications to the user. Furthermore, it adjusts the content of the advice based on the user's emotional state.

[0308] For example, if it has learned that the baby usually goes to bed at 8 pm, it will send a notification 30 minutes in advance saying "It's time for the baby to go to bed. Let's create a quiet environment." However, if the user is feeling stressed, it will provide additional advice such as "Please take a deep breath and relax."

[0309] Specific Example

[0310] General Usage Example

[0311] Terminal: The camera detects the baby's posture and sends data to the server.

[0312] Terminal: The emotion engine analyzes the user's expression and sends it to the server.

[0313] Server: If it analyzes and determines that it is normal, it will notify the user that "The baby is safe."

[0314] Terminal: The temperature and humidity sensor detects that the indoor temperature is too high and sends data to the server.

[0315] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[0316] Examples of use in emergencies

[0317] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[0318] Server: The system determines this is an emergency and sends an emergency notification to the parents stating, "There is an abnormality in the baby's breathing."

[0319] User: Receive notifications on your smartphone and respond quickly.

[0320] Server: Simultaneously analyzes the user's emotions and, if it determines that the user is experiencing stress, sends a message saying, "Please calm down. We will address this immediately."

[0321] This concludes our explanation of the new implementation, "BabySafeWatch." By combining it with an emotion engine, it is possible to further reduce the burden on the user and provide a higher level of reassurance.

[0322] The following describes the processing flow.

[0323] Step 1:

[0324] Terminal: The camera module captures video of the baby in real time. The video is captured at 30 frames per second and temporarily stored in local storage.

[0325] Step 2:

[0326] Terminal: Sends acquired video data to the server via a communication method. A secure protocol (e.g., HTTPS) is used for transmission.

[0327] Step 3:

[0328] Device: A vibration sensor detects the baby's breathing. It collects data every second and sends it to the server.

[0329] Step 4:

[0330] Terminal: A temperature and humidity sensor measures the temperature and humidity in the bedroom. It acquires data every 10 seconds and sends it to the server.

[0331] Step 5:

[0332] Terminal: An illuminance sensor measures the brightness of the bedroom. It acquires data every 5 seconds and sends it to the server.

[0333] Step 6:

[0334] Terminal: The emotion engine acquires the user's voice and video data. The emotion engine captures the data in real time and temporarily stores it in local storage.

[0335] Step 7:

[0336] Terminal: Sends acquired emotional data to the server via communication means. The emotional data is also transmitted using a secure protocol.

[0337] Step 8:

[0338] Server: Analyzes all received data. It analyzes the baby's posture and facial condition from camera footage and detects breathing patterns from vibration sensor data. It analyzes temperature, humidity, and illuminance data to evaluate the indoor environment.

[0339] Step 9:

[0340] Server: Analyzes user emotion data acquired by the emotion engine. Evaluates the user's emotional state based on voice tone and facial expression data.

[0341] Step 10:

[0342] Server: Detects anomalies based on analysis results. Determines an emergency situation, such as when the baby is lying face down, breathing has stopped, or the user is experiencing high levels of stress.

[0343] Step 11:

[0344] Server: If an anomaly is detected, an emergency notification will be sent to the user. The notification will be sent to the smartphone via push notification or email. If the user is feeling stressed, a message encouraging relaxation will also be included.

[0345] Step 12:

[0346] Server: Generates environmental control commands based on analysis results. If the room temperature exceeds the set range, it sends a cooling command to the air conditioner. If the room is too dark, it adjusts the lighting appropriately.

[0347] Step 13:

[0348] Terminal: Receives instructions from the server and controls environmental factors such as air conditioning and lighting. This includes adjusting the air conditioner's temperature setting and changing the brightness of the lights.

[0349] Step 14:

[0350] Server: It works in conjunction with the childcare record application to learn the baby's daily routines (sleep, eating, elimination, etc.). Based on the collected data, it analyzes the baby's normal behavioral patterns.

[0351] Step 15:

[0352] Server: Sends advice and notifications to the user based on learning results. For example, 30 minutes before the usual bedtime, it will notify the user, "It's time for the baby to sleep. Create a quiet environment," but if the user is feeling stressed, it will offer additional advice such as, "Take a deep breath and relax."

[0353] The above outlines the specific processing steps of "Baby Safe Watch." This process allows for the comprehensive management of the baby's safety and comfortable environment, and provides support tailored to the user's emotional state.

[0354] (Example 2)

[0355] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0356] Conventional childcare monitoring systems are limited to monitoring the baby's physical condition and do not take into account the emotional state of the user (parent), resulting in a failure to alleviate the stress and burden of childcare. Furthermore, there are challenges in the accuracy and reliability of systems for promptly responding to detected abnormalities.

[0357] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0358] In this invention, the server includes analysis means for monitoring the baby's safe and comfortable environment, emotion recognition means for recognizing the user's emotions, notification means for detecting anomalies or providing notifications in accordance with the user's emotions, and environment control means for controlling the bedroom environment. This allows for real-time monitoring of the baby's safe and comfortable environment while responding to the user's emotional state, thereby reducing stress and burden from childcare and enabling quick and appropriate responses in emergencies.

[0359] "Video acquisition means" refers to devices or modules for capturing a baby's posture and facial condition in real time.

[0360] A "vibration detection device" is a sensor used to detect the baby's breathing and body movements.

[0361] The "environmental sensor means" refers to a sensor used to measure the temperature and humidity of a bedroom.

[0362] The "illuminance detection means" is a sensor used to measure the brightness of a bedroom.

[0363] "Communication means" refers to devices or interfaces for transmitting data collected from image acquisition means, vibration detection means, environmental sensor means, and illuminance detection means to a server.

[0364] The "analysis means" refers to a function on the server that analyzes acquired data and monitors the baby's safety and comfortable environment.

[0365] "Emotion recognition means" refers to a function that acquires the user's voice and video and analyzes their emotional state.

[0366] A "notification method" is a means of notifying the user of important information or warnings based on analysis results or emotion recognition results.

[0367] The "environmental control means" is a function that issues instructions to automatically adjust the environment of the bedroom, such as temperature, humidity, and illuminance, based on the analysis results.

[0368] The "learning tool" refers to a function that works in conjunction with the childcare record application to learn the individual daily routines of babies.

[0369] An "advice tool" is a means of providing users with timely and appropriate advice based on learned patterns and the user's emotional state.

[0370] The present invention, "a system for monitoring a baby's safe and comfortable environment," includes the following configuration and operation:

[0371] System Configuration

[0372] terminal

[0373] Means of acquiring video:

[0374] A camera module is used to capture the baby's posture and facial expression in real time.

[0375] Specific example: A baby monitor camera captures the movements and facial expressions of a baby lying down.

[0376] Vibration detection means:

[0377] A vibration sensor is used to detect the baby's breathing and is placed under the bed.

[0378] Specific example: Continuously track the up-and-down movement of a baby's chest and collect data.

[0379] Environmental sensor means:

[0380] Temperature and humidity sensors are used to measure the environmental conditions in the bedroom.

[0381] Specific example: When the indoor temperature reaches 28 degrees Celsius, the sensor detects this and provides data.

[0382] Illuminance detection means:

[0383] Use an illuminance sensor to measure the brightness of the bedroom.

[0384] Specific example: When the brightness of the bedroom lighting exceeds a certain level, the data is recorded.

[0385] Means of communication:

[0386] The system transmits data collected from the camera module, vibration sensor, temperature and humidity sensor, and illuminance sensor to the server.

[0387] Specific example: Real-time video and sensor data are transmitted to a server via wireless communication.

[0388] server

[0389] Analysis method:

[0390] The server uses advanced algorithms to analyze the data sent from the terminal.

[0391] Specific example: Analyze data to monitor the baby's posture, facial expression, breathing patterns, and indoor environment.

[0392] Emotion recognition means:

[0393] It acquires audio and video data and recognizes the user's emotions.

[0394] Specific example: While a user is comforting a baby, their facial expressions and voice are analyzed to determine their emotional state.

[0395] Notification method:

[0396] If an anomaly is detected or if it is determined that the user is experiencing stress, an appropriate notification will be sent to the user.

[0397] Specific example: If a baby's breathing stops for 20 seconds, an emergency notification will be sent stating, "An abnormality has occurred in the baby's breathing."

[0398] Environmental control means:

[0399] Based on the analysis results, the system sends instructions to the terminal to automatically control the settings of the air conditioner and lighting in order to maintain the optimal indoor environment.

[0400] Specific example: If the room temperature is determined to be too high, a command is sent to the air conditioner to start cooling.

[0401] Learning methods:

[0402] It connects with a childcare record application to learn the baby's daily routine.

[0403] Specific example: Determine the baby's bedtime pattern, such as going to sleep at 8 PM every night.

[0404] Advice methods:

[0405] Based on learned patterns, it sends timely advice and notifications to the user. Furthermore, it provides advice based on the user's emotional state through emotion recognition.

[0406] Specific example: When it's time for the baby to sleep, the system will notify the user with "It's time for the baby to sleep. Let's create a quiet environment," and if the user is feeling stressed, it will offer additional advice such as "Take a deep breath and relax."

[0407] Example of a prompt

[0408] "Please explain the procedure for a program that sends a notification if a baby cries for an extended period of time."

[0409] "Please explain how to handle situations when users are experiencing high levels of stress."

[0410] This invention makes it possible to ensure a safe and comfortable environment for the baby, as well as reduce the mental burden on the user. The system uses various sensors and advanced analytical means to monitor the baby's and user's condition in real time and provide appropriate responses.

[0411] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0412] Step 1: Data acquisition and transmission (device)

[0413] Specific operation: Each sensor in the terminal (video acquisition means, vibration detection means, environmental sensor means, illuminance detection means) collects its own data. The video acquisition means captures video of the baby, and the vibration detection means monitors the baby's breathing. The environmental sensor means measures room temperature and humidity, and the illuminance detection means measures the brightness of the room.

[0414] Inputs: Baby's posture, facial expression, breathing, room temperature, humidity, and brightness.

[0415] Data processing and calculation: Each sensor collects data in real time and transmits it to the server via communication means.

[0416] Output: Various sensor data transferred to the server

[0417] Step 2: User emotion recognition (device)

[0418] Specific operation: The device's emotion recognition system captures the user's voice and video, acquiring voice tone and facial expression data. This data is sent to a server in real time for emotion analysis.

[0419] Input: User's voice data, video data

[0420] Data processing and calculation: The emotion recognition system collects data to analyze the user's emotions based on an emotion model and sends it to the server.

[0421] Output: Audio and video data sent to the server

[0422] Step 3: Data analysis and anomaly detection (server)

[0423] Specific operation: The server analyzes the received data and uses advanced algorithms to monitor the baby's posture, facial expression, breathing patterns, and room environment. Simultaneously, it analyzes the user's emotional state.

[0424] Input: Sensor data, audio data, and video data transmitted from the device.

[0425] Data processing and calculation: The server analyzes the data using analytical algorithms and detects anomalies. This includes determining whether the baby has stopped breathing or whether the user is in a high-stress state.

[0426] Output: Anomaly detection results, user emotional state analysis results

[0427] Step 4: Notification (Server)

[0428] Specific operation: If a server anomaly is detected, or if a user is experiencing stress, an emergency notification will be sent to the user using a notification system. The notification will arrive on a device such as a smartphone or tablet.

[0429] Input: Anomaly detection results, user emotional state analysis results

[0430] Data processing and calculation: When an anomaly is detected, immediately prepare a notification and send it to the user's device. Customize the message content according to the type of anomaly and the emotional state of the user.

[0431] Output: Notification message to the user

[0432] Step 5: Environment Control (Server)

[0433] Specific operation: Based on the analysis results, the server sends instructions to the terminal to control the settings of the air conditioner and lighting in order to maintain an optimal environment (temperature, humidity, brightness) in the bedroom.

[0434] Input: Baby's condition data, indoor environment data

[0435] Data processing and calculation: The server calculates the settings necessary to maintain a comfortable environment for the baby and sends instructions to the terminal.

[0436] Output: Control instructions for air conditioners and lighting.

[0437] Step 6: Learning and Advice (Server)

[0438] Specific operation: The server interacts with the childcare record application to learn the baby's daily routines (sleep, meals, elimination, etc.). Based on the learned patterns, it sends timely advice and notifications to the user.

[0439] Input: Childcare record data, past sensor data

[0440] Data Processing and Calculation: Analyze data from childcare record applications to analyze the baby's daily routines. Use the analysis results to generate appropriate advice.

[0441] Output: Advice messages, notification messages

[0442] The above outlines the processing flow of this system's program. By collecting and analyzing data in real time, this system provides comprehensive support to enhance the safety of babies and the comfort of users.

[0443] (Application Example 2)

[0444] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0445] Traditional systems for managing the safety and comfort of babies could monitor the baby's condition but failed to consider the user's emotions. Similarly, in physical stores, it was difficult to monitor customer emotions and behavior in real time and implement appropriate responses and environmental controls. This resulted in insufficient improvement in the quality of childcare and customer service, sometimes leading to delays in providing optimal support.

[0446] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an analysis means, a notification means, a control means, an automatic notification means, a customer emotion notification means, and a store environment control means. This enables monitoring of the baby's safety and comfort, as well as responses to customer emotions and automatic control of the store environment.

[0447] A "camera system" is a device used to detect the posture and facial condition of babies or customers in real time.

[0448] A "vibration sensor" is a device used to detect a baby's breathing.

[0449] A "temperature and humidity sensor means" is a sensor for measuring the temperature and humidity in a bedroom or store.

[0450] An "illuminance sensor means" is a sensor used to measure the brightness inside a bedroom or store.

[0451] "Communication means" refers to means for transmitting data from the camera means, vibration sensor means, temperature and humidity sensor means, and illuminance sensor means to the server.

[0452] "Analysis means" refers to means for monitoring the safety and comfort of babies and customers by analyzing the aforementioned data.

[0453] "Notification means" refers to a means for notifying a user or staff member when an abnormality is detected by the analysis means.

[0454] "Control means" refers to means for controlling the environment of a bedroom or store based on the analysis means.

[0455] An "emotion engine" is a means of detecting customer emotions in real time.

[0456] A "customer sentiment notification system" is a means of analyzing customer sentiment data and notifying staff of appropriate responses.

[0457] "Store environment control means" refers to means for automatically adjusting the store environment based on customer sentiment data.

[0458] "Learning methods" refer to techniques for learning individual patterns by linking them with the baby's care records.

[0459] "Advice means" refers to means for providing timely advice and notifications to the user based on the learning means.

[0460] This invention is a system for managing the safety and comfort of babies. Furthermore, the same technology can be used to monitor customer emotions and behavior in real time in physical stores, thereby improving customer service.

[0461] System Configuration

[0462] 1. Terminal

[0463] Camera System: This camera is used to detect the posture and facial condition of babies and customers in real time. The video data acquired from the camera is transmitted to the server described below.

[0464] Vibration sensor: This sensor is used to detect the baby's breathing. The vibration data is transmitted to a server.

[0465] Temperature and humidity sensor means: A sensor for measuring the temperature and humidity in a bedroom or store. This data is also transmitted to the server.

[0466] Illuminance sensor: This is a sensor used to measure the brightness in a bedroom or store. This data is also transmitted to the server.

[0467] Communication method: This is a communication device used to transmit data acquired from the above-mentioned sensors to the server. For example, Wi-Fi modules or Bluetooth are used.

[0468] Emotion Engine: A device that acquires customer voice and video data and detects customer emotions in real time.

[0469] 2. Server

[0470] Analysis method: This method involves analyzing data transmitted from terminals on a server to monitor the safety and comfort of babies and customers. Specifically, it utilizes image analysis and data mining techniques.

[0471] Notification method: This is a means of notifying users or store staff when an anomaly is detected by the analysis method. Notifications are made via devices such as smartphones and tablets.

[0472] Control means: A means for transmitting instructions to a terminal to control the environment of a bedroom or store based on analysis means. For example, it can be used to adjust the temperature of an air conditioner or change the brightness of lighting.

[0473] Customer sentiment notification method: This method analyzes customer sentiment data and notifies staff of appropriate responses. For example, if a customer expresses dissatisfaction, a notification is sent to staff prompting a quick response.

[0474] Store environment control means: This means automatically adjusting the store environment based on customer emotion data. For example, it may change the store temperature or background music.

[0475] 3. User

[0476] User Devices: Users or store staff can receive notifications using devices such as smartphones and tablets, and manage the baby's condition, environment, and customer's emotional state through the application. They can also receive timely advice and notifications provided by the system.

[0477] Specific example

[0478] Common usage examples

[0479] Terminal: The camera detects the baby's posture and sends the data to the server.

[0480] Terminal: The emotion engine analyzes the customer's facial expressions and sends them to the server.

[0481] Server: If the server analyzes the data and determines that everything is normal, it notifies the user or store staff with messages such as "The baby is safe" or "The customer is satisfied."

[0482] Terminal: The temperature and humidity sensor detects when the indoor temperature is too high and sends data to the server.

[0483] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[0484] Example of a prompt

[0485] Application: Design a smart store watch application to enhance the customer experience in physical stores. Using cameras and sensors, monitor customer emotions and behavior in real time, automating all responses and environmental controls. For example, if a customer appears distressed, quickly notify staff and adjust settings such as air conditioning or music to create a more relaxing atmosphere.

[0486] Examples of use in emergencies

[0487] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[0488] Server: The system determines this is an emergency and sends an emergency notification to the parents stating, "There is an abnormality in the baby's breathing."

[0489] User: Receive notifications on your smartphone and respond quickly.

[0490] Server: Simultaneously analyzes the user's emotions and, if it determines that the user is experiencing stress, sends a message saying, "Please calm down. We will address this immediately."

[0491] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0492] Step 1:

[0493] The device uses a camera to detect the posture and facial features of babies and customers in real time and acquire video data. The input is camera footage, and the output is the detected face and posture data. Specifically, it uses OpenCV and the dlib library to capture video and execute a face detection algorithm.

[0494] Step 2:

[0495] The device uses a vibration sensor to detect the baby's breathing. The input is vibration data from the sensor, and the output is data indicating the presence of breathing. The sensor data is acquired at regular intervals and processed into data for analyzing the breathing pattern.

[0496] Step 3:

[0497] The terminal uses temperature and humidity sensors to measure and acquire data on the temperature and humidity in a bedroom or store. The input is the reading from the temperature and humidity sensors, and the output is accurate temperature and humidity data for the room. This data is used as foundational data for analysis and control described later.

[0498] Step 4:

[0499] The terminal uses an illuminance sensor to measure the brightness in a bedroom or store and acquire data. The input is the illuminance sensor reading, and the output is the room's illuminance data. The measured data is sent to the server.

[0500] Step 5:

[0501] The terminal uses communication methods to transmit data acquired from the camera, vibration sensor, temperature and humidity sensor, and illuminance sensor to the server. The input is the data acquired from each sensor, and the output is the data transferred to the server. Wi-Fi, Bluetooth, and other technologies are used for communication.

[0502] Step 6:

[0503] The server uses analysis tools to analyze the transmitted data and monitor the safety and comfort of babies and customers. The input is sensor data transmitted from the terminal, and the output is the analysis results. Specifically, it uses image analysis algorithms and machine learning models to detect various anomaly patterns.

[0504] Step 7:

[0505] The server uses a notification system to notify users or store staff if it detects an anomaly in the analysis process. The input is the analysis result, and the output is a notification message. This notification is sent via a push notification system to smartphones and tablets.

[0506] Step 8:

[0507] The server uses control means to send instructions to the terminal to control the environment of a bedroom or store based on analysis means. The input is the analysis result, and the output is the environment control instruction. For example, it may issue instructions to adjust the set temperature of an air conditioner or the brightness of lighting.

[0508] Step 9:

[0509] The terminal uses an emotion engine to acquire customer voice and video data and detect the customer's emotions. The input is voice and video data, and the output is analyzed emotion data. An emotion recognition model is used to detect changes in emotion, and this data is sent to the server.

[0510] Step 10:

[0511] The server uses a customer sentiment notification system to analyze customer sentiment data and notify staff of appropriate actions. The input is sentiment data sent from the terminal, and the output is a notification to staff. The notification content changes according to the customer's status, prompting a quick response for dissatisfied customers.

[0512] Step 11:

[0513] The server uses store environment control means to automatically adjust the store environment based on customer emotion data. The input is analyzed emotion data, and the output is environment control instructions. For example, it adjusts the store temperature and background music.

[0514] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0515] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0516] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0517] [Second Embodiment]

[0518] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0519] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0520] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0521] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0522] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0523] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0524] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0525] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0526] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0527] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0528] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0529] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0530] A specific embodiment of the "Baby Safe Watch" of the present invention is described below. This system consists of multiple terminals, a server, and a user (parent), each with its own role.

[0531] System Configuration

[0532] 1. Terminal

[0533] Camera equipment: A camera is installed to detect the baby's posture and facial condition in real time.

[0534] Vibration sensor means: A vibration sensor is installed in the bed to detect the baby's breathing.

[0535] Temperature and humidity sensor means: Install a sensor to measure the temperature and humidity in the bedroom.

[0536] Illuminance sensor means: Install a sensor to measure the brightness of the bedroom.

[0537] Communication means: Equipped with a communication device for transmitting data acquired from the above sensors to a server.

[0538] 2. Server

[0539] Analysis method: The server receives data transmitted from the terminal and performs analysis using advanced algorithms.

[0540] Notification mechanism: The system includes a mechanism for notifying the user according to the analysis results. If an anomaly is detected, an emergency notification will be sent.

[0541] Control mechanism: Based on the analysis results, it sends instructions to the terminal to automatically control environmental factors such as temperature, humidity, and illuminance.

[0542] Learning method: It works in conjunction with a childcare record application to learn the individual patterns of each baby.

[0543] Advice method: Based on learned patterns, it sends timely advice and notifications to the user.

[0544] 3. User

[0545] Using devices such as smartphones and tablets, users can receive notifications and manage their baby's condition and environment through the application.

[0546] Explanation of the program's processing

[0547] Data acquisition and transmission (terminal)

[0548] Terminal: The camera module captures images of the baby and sends them to the server for real-time analysis.

[0549] Terminal: The vibration sensor detects the baby's breathing in real time and sends the data to the server.

[0550] Terminal: Temperature, humidity, and illuminance sensors also measure environmental data and transmit it to the server at regular intervals.

[0551] Data analysis and anomaly detection (server)

[0552] Server: Analyzes received data and monitors the baby's posture, facial expression, breathing patterns, and room environment.

[0553] Server: If an anomaly is detected as a result of the analysis, an emergency notification will be sent to the user's smartphone.

[0554] For example, if the baby turns onto its stomach, an alert is sent immediately.

[0555] An emergency notification will also be issued if breathing stops for 20 seconds or more.

[0556] Environmental control (server)

[0557] Server: Based on the analysis results, it sends instructions to terminals to automatically control air conditioning and lighting in order to maintain the optimal indoor environment.

[0558] If the room temperature exceeds the set range, the server sends a cooling command to the air conditioner.

[0559] If the room is too dark, adjust the lighting appropriately.

[0560] Learning and advice (server)

[0561] Server: It connects with childcare records and learns the baby's daily routines (sleep, eating, elimination, etc.).

[0562] Server: Based on learned data, it sends timely advice and notifications to the user.

[0563] For example, if the system learns that the baby usually goes to bed at 8 PM, it will send a notification 30 minutes beforehand saying, "It's time for the baby to sleep. Please create a quiet environment."

[0564] Specific example

[0565] Common usage examples

[0566] Terminal: The camera detects the baby's posture and sends the data to the server.

[0567] Server: If the analysis determines that everything is normal, it notifies the user that "the baby is safe."

[0568] Terminal: The temperature and humidity sensor detects when the indoor temperature is too high and sends data to the server.

[0569] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[0570] Examples of use in emergencies

[0571] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[0572] Server: The system determines this is an emergency and sends an emergency notification to the parents stating, "There is an abnormality in the baby's breathing."

[0573] User: Receive notifications on your smartphone and respond quickly.

[0574] This concludes the description of the embodiment of the "Baby Safe Watch" of the present invention. By using this system, parents can manage the safety and comfortable childcare environment of their baby in one place, thereby reducing the burden of childcare.

[0575] The following describes the processing flow.

[0576] Step 1:

[0577] Terminal: The camera module acquires real-time video of the baby. It captures 30 frames per second of video and temporarily stores it in local storage.

[0578] Step 2:

[0579] Terminal: Transmits the acquired video to the server via a communication method. The data is transmitted using a secure protocol (e.g., HTTPS).

[0580] Step 3:

[0581] Device: A vibration sensor detects the baby's breathing. The vibration sensor collects data every second and sends it to the server.

[0582] Step 4:

[0583] Terminal: A temperature and humidity sensor measures the temperature and humidity in the bedroom. It acquires data every 10 seconds and sends it to the server.

[0584] Step 5:

[0585] Terminal: An illuminance sensor measures the brightness of the bedroom. It acquires data every 5 seconds and sends it to the server.

[0586] Step 6:

[0587] Server: Analyzes all received data. Analyzes camera footage to recognize the baby's posture and facial expression. Analyzes vibration sensor data to detect breathing patterns. Analyzes temperature, humidity, and illuminance data to evaluate the indoor environment.

[0588] Step 7:

[0589] Server: Detects anomalies based on analysis results. Determines an emergency situation, such as when the baby is lying face down or has stopped breathing.

[0590] Step 8:

[0591] Server: If an anomaly is detected, an emergency notification will be sent to the user. The notification will be sent to the smartphone in the form of a push notification or email.

[0592] Step 9:

[0593] Server: Generates environmental control commands based on the analysis results. For example, if the room temperature exceeds the set range, it sends a cooling command to the air conditioner.

[0594] Step 10:

[0595] Terminal: Receives instructions from the server and controls environmental factors such as air conditioning and lighting. This includes adjusting the air conditioner's temperature setting and changing the brightness of the lights.

[0596] Step 11:

[0597] Server: It works in conjunction with the childcare record application to learn the baby's daily routine. Based on the collected data, it analyzes the baby's sleep, eating, and elimination patterns.

[0598] Step 12:

[0599] Server: Sends advice and notifications to the user based on the learning results. For example, 30 minutes before the usual bedtime, it might notify the user, "It's time for the baby to sleep. Please create a quiet environment."

[0600] The above outlines the specific processing steps for "Baby Safe Watch." This process allows for comprehensive management of your baby's safety and comfortable environment.

[0601] (Example 1)

[0602] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0603] In modern childcare, parents need to constantly monitor their baby's safety, but 24-hour monitoring is physically and psychologically burdensome. Optimizing the indoor environment is also crucial for maintaining the baby's safety and comfort, but manual adjustments are time-consuming. Furthermore, the difficulty in obtaining appropriate advice based on the baby's daily routines leads to decreased efficiency in childcare.

[0604] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0605] In this invention, the server includes a camera means for detecting the baby's posture and facial condition in real time, a vibration sensor means for detecting the baby's breathing, an environmental sensor means for measuring the temperature and humidity of the bedroom, an illuminance sensor means for measuring the brightness of the bedroom, a communication means for transmitting data from the camera means, vibration sensor means, environmental sensor means and illuminance sensor means to the server, an analysis means in the server for monitoring the baby's safe and comfortable environment by analyzing the data, a notification means for notifying the user when the analysis means detects an abnormality, a control means for controlling the bedroom environment based on the analysis means, a learning means for learning individual patterns in conjunction with childcare records, and an advice means for providing timely advice and notifications based on the learned patterns. This makes it possible to automatically and efficiently maintain the baby's safety and comfort and reduce the burden of childcare on parents.

[0606] A "camera system" is a device used to detect the baby's posture and facial condition in real time.

[0607] A "vibration sensor" is a device used to detect a baby's breathing.

[0608] An "environmental sensor" is a device used to measure the temperature and humidity of a bedroom.

[0609] An "illuminance sensor" is a device used to measure the brightness of a bedroom.

[0610] "Communication means" refers to a device for transmitting data acquired from camera means, vibration sensor means, environmental sensor means, and illuminance sensor means to a server.

[0611] "Analysis means" refers to a device that analyzes data transmitted via communication means to monitor the safety and comfort of the baby.

[0612] A "notification means" is a device used to notify the user when an anomaly is detected in the analysis means.

[0613] The "control means" is a device for automatically controlling the bedroom environment based on the analysis means.

[0614] A "learning tool" is a device that works in conjunction with childcare records to learn individual patterns.

[0615] An "advice tool" is a device that provides timely advice and notifications based on learned patterns.

[0616] A specific embodiment of the "Baby Safe Watch" of the present invention is described below. This system consists of multiple terminals, a server, and a user (parent), each with its own role.

[0617] System Configuration

[0618] 1. Terminal

[0619] Camera equipment: A camera is installed to detect the baby's posture and facial condition in real time. For example, it can be used to check if the baby is rolling over or if their face is covered by the blanket.

[0620] Vibration sensor means: A vibration sensor is installed in the bed to detect the baby's breathing. For example, the breathing pattern is obtained from the movement of the baby's chest.

[0621] Environmental sensor means: Install sensors to measure the temperature and humidity in the bedroom. For example, obtain data such as a room temperature of 25 degrees Celsius and a humidity of 50%.

[0622] Illuminance sensor means: Install a sensor to measure the brightness of the bedroom. For example, confirm that the brightness of the room is 400 lux.

[0623] Communication means: The system is equipped with a communication device for transmitting data acquired from the above-mentioned sensors to a server. For example, a Wi-Fi module is used to transmit the data.

[0624] 2. Server

[0625] Analysis method: The server receives data transmitted from the terminal and performs analysis using advanced algorithms. For example, it analyzes the baby's posture, breathing status, and the indoor environment.

[0626] Notification method: The system includes a means to notify the user according to the analysis results. If an abnormality is detected, an emergency notification will be sent. For example, a notification such as "An abnormality has been detected in the baby's breathing" will be sent to the user.

[0627] Control mechanism: Based on the analysis results, it sends instructions to the terminal to automatically control environmental conditions such as temperature, humidity, and illuminance. For example, if the room temperature exceeds the set range, it sends an instruction to the air conditioner to start cooling.

[0628] Learning method: It works in conjunction with a childcare record application to learn the individual patterns of the baby. For example, it learns the baby's daily patterns such as sleep, eating, and elimination.

[0629] Advice methods: Based on learned patterns, the system sends timely advice and notifications to the user. For example, for a baby who has a habit of going to bed at 8 pm, it might send a notification 30 minutes beforehand saying, "It's time for your baby to sleep. Please create a quiet environment."

[0630] 3. User

[0631] Users can receive notifications from the server using devices such as smartphones and tablets, and manage their baby's condition and environment through the application. For example, a user who receives an emergency notification can quickly rush to the baby's bed and check on their baby's condition.

[0632] Specific example

[0633] Common usage examples

[0634] Terminal: The camera detects the baby's posture and sends the data to the server.

[0635] Server: If the analysis determines that everything is normal, it notifies the user that "the baby is safe."

[0636] Terminal: The environmental sensor detects that the indoor temperature is too high and sends data to the server.

[0637] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[0638] Examples of use in emergencies

[0639] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[0640] Server: Determined to be in an emergency, it sends an emergency notification to the user stating, "There is an abnormality in the baby's breathing."

[0641] User: Receives a notification on their smartphone, rushes to the bed immediately, and checks on the baby's condition.

[0642] Example of a prompt

[0643] Example prompt:

[0644] BabySafeWatch is a system that monitors and controls a baby's posture, breathing, and indoor environment. What kind of data does it typically collect, and how does it notify the user?

[0645] How will users be notified in an emergency, and how will they respond?

[0646] This concludes the description of the embodiment of the "Baby Safe Watch" of the present invention. By using this system, parents can manage the safety and comfortable childcare environment of their baby in one place, thereby reducing the burden of childcare.

[0647] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0648] Step 1: Data Acquisition (Device)

[0649] Input: Information about the baby's posture and facial expression, breathing, room temperature, humidity, and light level.

[0650] Specific operation: The camera on the device captures images of the baby. A vibration sensor detects the baby's breathing, an environmental sensor measures the room temperature and humidity, and an illuminance sensor measures the room brightness.

[0651] Output: Acquired video data, respiration data, and environmental data (temperature, humidity, illuminance) are generated.

[0652] Step 2: Data transmission (terminal)

[0653] Input: Data acquired from each sensor (video data, respiratory data, environmental data)

[0654] Specific operation: Data acquired by sensors and cameras is transmitted to a server using communication means. A Wi-Fi module is used as the specific means.

[0655] Output: Data acquired by sensors and cameras is sent to the server.

[0656] Step 3: Data Analysis (Server)

[0657] Input: Data from each sensor sent to the server (video data, respiration data, environmental data)

[0658] Specific operation: The server's analysis system receives the data and uses advanced algorithms to analyze the baby's posture, breathing, and room environment. For example, it analyzes whether the baby's posture is normal and whether the breathing pattern is abnormal.

[0659] Output: Analysis results (determination of normal or abnormal, indoor environment status)

[0660] Step 4: Anomaly detection and notification (server)

[0661] Input: Results of data analysis

[0662] Specific operation: If an anomaly is detected by the server's analysis system, the notification system will inform the user. Specifically, an emergency notification will be sent if the baby is lying face down or if breathing stops for more than 20 seconds.

[0663] Output: An emergency notification is sent to the user.

[0664] Step 5: Environment Control (Server)

[0665] Input: Results of data analysis

[0666] Specific operation: Based on the analysis results, the server's control system sends instructions to the terminal to automatically adjust environmental conditions such as temperature, humidity, and illuminance. For example, if the room temperature exceeds the set range, it sends a cooling command to the air conditioner.

[0667] Output: An instruction for environment adjustment is sent to the terminal.

[0668] Step 6: Data Training (Server)

[0669] Input: Daily life pattern data from a childcare record application.

[0670] Specific operation: The server's learning mechanism works in conjunction with the childcare record application to learn the individual patterns of the baby. For example, it analyzes and learns data such as the baby's sleep, meals, and bowel movements.

[0671] Output: Learning results (insights about babies' daily routines)

[0672] Step 7: Advice and Notification (Server)

[0673] Input: Learning results and data analysis results

[0674] Specific operation: Based on patterns learned by the server's advice system, timely advice and notifications are sent to the user. For example, for a baby who has a habit of going to bed at 8 pm, a notification is sent 30 minutes beforehand saying, "It's time for your baby to sleep. Please create a quiet environment."

[0675] Output: Appropriate advice and notifications are sent to the user.

[0676] (Application Example 1)

[0677] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0678] Ensuring the safety of babies and children and reducing anxiety for parents and guardians while they engage in childcare or shopping activities is crucial. However, current systems lack the functionality to monitor children's safety in stores in real time, in addition to monitoring the baby's environment, and to quickly notify parents and staff in case of abnormalities. Therefore, there is a need to develop a new system that allows parents and guardians to watch over their children with greater peace of mind.

[0679] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0680] In this invention, the server includes an image acquisition means for detecting the baby's posture and facial condition in real time, a vibration detection means for detecting the baby's breathing, a climate data acquisition means for measuring the temperature and humidity of the bedroom, a light intensity sensor means for measuring the brightness of the bedroom, a communication device for transmitting data from the image acquisition means, vibration detection means, climate data acquisition means and light intensity sensor means to a central processing unit, an analysis means in the central processing unit for monitoring the baby's safety and comfort by analyzing the data, a notification means for notifying the user when an abnormality is detected by the analysis means, an environment control means for controlling the bedroom environment based on the analysis means, a means for monitoring the safety of children in the store in real time, and a means for notifying parents or staff when a child enters a specific area. This ensures a safe and comfortable environment for babies, and further enables real-time monitoring of children's safety in the store and a rapid response in the event of an abnormality.

[0681] The "image acquisition means" is a device for detecting the baby's posture and facial condition in real time.

[0682] A "vibration detection device" is a device for detecting a baby's breathing in real time.

[0683] "Climate data acquisition means" refers to a device for measuring the temperature and humidity of a bedroom.

[0684] A "light intensity sensor" is a device used to measure the brightness of a bedroom.

[0685] A "communication device" is a device used to transmit data acquired from various sensors to a central processing unit.

[0686] A "central processing unit" is a device that analyzes received data and monitors the safety and comfort of babies and children.

[0687] An "analysis tool" is a device used to analyze data and detect anomalies.

[0688] A "notification means" is a device that notifies the user when an abnormality is detected in the analysis means.

[0689] An "environmental control means" is a device that controls the bedroom environment based on an analysis means.

[0690] "A means of monitoring the safety of children in stores in real time" refers to a device that monitors the safety of children in stores in real time and notifies the staff if any abnormalities are detected.

[0691] "Means of notifying parents or staff" refers to a device that notifies parents or store staff when a child enters a specific area.

[0692] A specific embodiment of the present invention, "Kids Safe Watch," is described below. The system consists of multiple terminals, a central processing unit, and users, each with its own role.

[0693] System Configuration

[0694] 1. Terminal

[0695] Image acquisition method: Install a camera to detect the child's posture and facial condition in real time.

[0696] Vibration detection method: A vibration sensor is installed on the floor to detect the breathing and position of the baby or child.

[0697] Climate data acquisition method: Install sensors to measure the temperature and humidity in the store.

[0698] Light intensity sensor means: Install a sensor to measure the brightness of the store.

[0699] Communication device: Equipped with a communication device for transmitting data acquired from the above-mentioned sensors to the central processing unit.

[0700] 2. Central Processing Unit

[0701] Analysis method: The central processing unit receives data transmitted from the terminal and performs analysis using advanced algorithms.

[0702] Notification method: The system includes a means to notify the user according to the analysis results. If an anomaly is detected, an emergency notification will be sent.

[0703] Environmental control means: Based on the analysis results, instructions are sent to the terminal to automatically control environmental conditions such as temperature, humidity, and illuminance.

[0704] 3. User

[0705] Using devices such as smartphones and tablets, users can receive notifications and manage their child's status and environment through the application.

[0706] Users can monitor the location and behavior of children within the store and respond quickly as needed.

[0707] Explanation of the program's processing

[0708] Hardware and software to use

[0709] Hardware: Smartphones, tablets, camera modules installed in stores, vibration sensors, temperature and humidity sensors, and illuminance sensors.

[0710] Software: Python (for data analysis and notification implementation), AWS cloud services (for data storage and processing), Firebase (real-time database and notification service), OpenCV (image analysis library).

[0711] Data acquisition and calculation processing

[0712] 1. Processing camera data

[0713] Using OpenCV, we detect and analyze children's posture and position from camera images.

[0714] The analysis data is sent to AWS in real time.

[0715] 2. Processing of sensor data

[0716] The system periodically acquires data from vibration sensors, temperature and humidity sensors, and illuminance sensors and sends it to AWS.

[0717] 3. Notification function

[0718] Using Firebase, users receive real-time notifications on their smartphones when an anomaly is detected.

[0719] 4. Environmental control

[0720] Data analysis is performed on AWS, and instructions for controlling air conditioners and lighting are sent to the terminal as needed.

[0721] Specific usage examples

[0722] Common usage examples

[0723] Terminal: The camera detects the posture and position of children in the store and transmits the data to the central processing unit.

[0724] Central Processing Unit: If it analyzes the data and determines that it is normal, it notifies the user that "the child is safe."

[0725] Terminal: The temperature and humidity sensor detects if the temperature inside the store is too high and sends data to the central processing unit.

[0726] Central Processing Unit: Sends cooling instructions to the air conditioner and adjusts it to the appropriate temperature.

[0727] Examples of use in emergencies

[0728] Terminal: If the vibration sensor detects that a child has left a designated area, it transmits that data to the central processing unit.

[0729] Central Processing Unit: Determines an emergency and sends an emergency notification to parents or store staff stating, "A child has left a designated area."

[0730] User: Receive notifications on your smartphone and respond quickly.

[0731] Examples of prompts for generative AI models

[0732] Please generate examples of how "Kids Safe Watch" can be used. Describe the details of a system that uses in-store cameras and sensors to manage child safety and notifies parents and store staff in real time if any anomalies are detected. Include specific use cases, such as notifications for when a child leaves the play area or when a child remains in the store after closing time.

[0733] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0734] Step 1:

[0735] Terminal: A camera module is used to detect the child's posture and position. The camera module acquires image data in real time from a designated area within the store. This image data is used as input data. The acquired image data is sent to the central processing unit using the OpenCV library for data preprocessing.

[0736] Step 2:

[0737] Server: The central processing unit receives pre-processed image data. It then uses OpenCV to execute face recognition and posture detection algorithms to extract the child's posture and location information. The extracted posture and location information is generated as output data, and the process proceeds to the next analysis step.

[0738] Step 3:

[0739] Terminal: Uses vibration sensors to acquire vibration data from children's floors and play areas installed in specific areas within the store. The vibration sensors transmit the detected vibration data to the central processing unit in real time. The vibration data is used as input data.

[0740] Step 4:

[0741] Server: The central processing unit receives data from vibration sensors and analyzes it using a rational threshold determination algorithm. Based on the analysis of the vibration data, it determines whether the child has deviated from a specific area. The determination result is generated as output data, and if an anomaly is detected, the process proceeds to the notification step.

[0742] Step 5:

[0743] Terminal: Using a climate data acquisition device, temperature and humidity data inside the store are acquired periodically. This data is used as input data and transmitted to the central processing unit in real time.

[0744] Step 6:

[0745] Server: The central processing unit receives temperature and humidity data transmitted from the climate data acquisition means and performs analysis to determine if it is within an appropriate range. Based on the analysis results, it determines whether the optimal temperature is being maintained. If the temperature deviates from the optimal range, an instruction to adjust using the environmental control means is output.

[0746] Step 7:

[0747] Terminal: Uses a light intensity sensor to acquire brightness (illuminance) data within the store. The brightness data is used as input data and transmitted to the central processing unit in real time.

[0748] Step 8:

[0749] Server: The central processing unit receives data from the light intensity sensor and performs analysis to determine whether the illuminance is within an appropriate range. Based on the analysis results, it determines whether the lighting is maintained within an appropriate range. If the illuminance deviates from the appropriate range, an instruction is output to adjust using the environmental control means.

[0750] Step 9:

[0751] Server: The server comprehensively analyzes all data acquired to date (image data, vibration data, temperature and humidity data, light intensity data) and, if an anomaly is detected, uses a notification system to notify users and store staff in real time. The notification content is generated and sent as output data.

[0752] Step 10:

[0753] Users: Customers or store staff receive notifications on their smartphones or tablets and take prompt action based on the displayed notification content. They review the notification and initiate specific response actions.

[0754] The above outlines the specific processing steps of the system program that implements the application example.

[0755] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0756] The "Baby Safe Watch" system is a comprehensive system for managing a baby's safety and a comfortable childcare environment. It incorporates an emotion engine that recognizes the user's emotions and adjusts notifications and support accordingly. This system consists of multiple terminals, a server, and the user (parent), each with its own specific role.

[0757] System Configuration

[0758] 1. Terminal

[0759] Camera equipment: A camera is installed to detect the baby's posture and facial condition in real time.

[0760] Vibration sensor means: A vibration sensor is installed in the bed to detect the baby's breathing.

[0761] Temperature and humidity sensor means: Install a sensor to measure the temperature and humidity in the bedroom.

[0762] Illuminance sensor means: Install a sensor to measure the brightness of the bedroom.

[0763] Communication means: Equipped with a communication device for transmitting data acquired from the above sensors to a server.

[0764] Emotion Engine Method: Acquires audio and video data to recognize the user's emotions.

[0765] 2. Server

[0766] Analysis method: The server receives data transmitted from the terminal and performs analysis using advanced algorithms.

[0767] Notification mechanism: The system includes a mechanism for notifying the user according to the analysis results. If an anomaly is detected, an emergency notification will be sent.

[0768] Control mechanism: Based on the analysis results, it sends instructions to the terminal to automatically control environmental factors such as temperature, humidity, and illuminance.

[0769] Learning method: It works in conjunction with a childcare record application to learn the individual patterns of each baby.

[0770] Advice method: Based on learned patterns, it sends timely advice and notifications to the user.

[0771] 3. User

[0772] Using devices such as smartphones and tablets, users can receive notifications and manage their baby's condition and environment through the application.

[0773] Explanation of the program's processing

[0774] Data acquisition and transmission (terminal)

[0775] Terminal: The camera module captures images of the baby and sends them to the server for real-time analysis.

[0776] Terminal: The vibration sensor detects the baby's breathing in real time and sends the data to the server.

[0777] Terminal: Temperature, humidity, and illuminance sensors also measure environmental data and transmit it to the server at regular intervals.

[0778] User emotion recognition (device)

[0779] Terminal: The emotion engine acquires the user's voice and video data.

[0780] Terminal: Sends acquired emotion data to the server.

[0781] Data analysis and anomaly detection (server)

[0782] Server: Analyzes received data and monitors the baby's posture, facial expression, breathing patterns, and room environment.

[0783] Server: Further analyzes the user's emotional data obtained by the emotion engine.

[0784] Server: Sends appropriate notifications to the user if an abnormality is detected in the baby or if the user is determined to be experiencing stress.

[0785] Environmental control (server)

[0786] Server: Based on the analysis results, it sends instructions to terminals to automatically control air conditioning and lighting in order to maintain the optimal indoor environment.

[0787] Terminal: Adjust the air conditioner temperature setting or change the brightness of the lights.

[0788] Learning and advice (server)

[0789] Server: Works in conjunction with the childcare record application to learn the baby's daily routine (sleep, meals, elimination, etc.).

[0790] Server: Based on learned data, it sends timely advice and notifications to the user. Furthermore, it adjusts the content of the advice based on the user's emotional state.

[0791] For example, if the system learns that the baby usually goes to bed at 8 PM, it will notify the user 30 minutes beforehand with the message, "It's time for your baby to sleep. Please create a quiet environment." However, if the user is feeling stressed, it will offer additional advice such as, "Take a deep breath and relax."

[0792] Specific example

[0793] Common usage examples

[0794] Terminal: The camera detects the baby's posture and sends the data to the server.

[0795] Terminal: The emotion engine analyzes the user's facial expressions and sends them to the server.

[0796] Server: If the analysis determines that everything is normal, it notifies the user that "the baby is safe."

[0797] Terminal: The temperature and humidity sensor detects when the indoor temperature is too high and sends data to the server.

[0798] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[0799] Examples of use in emergencies

[0800] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[0801] Server: The system determines this is an emergency and sends an emergency notification to the parents stating, "There is an abnormality in the baby's breathing."

[0802] User: Receive notifications on your smartphone and respond quickly.

[0803] Server: Simultaneously analyzes the user's emotions and, if it determines that the user is experiencing stress, sends a message saying, "Please calm down. We will address this immediately."

[0804] This concludes our explanation of the new implementation, "BabySafeWatch." By combining it with an emotion engine, it is possible to further reduce the burden on the user and provide a higher level of reassurance.

[0805] The following describes the processing flow.

[0806] Step 1:

[0807] Terminal: The camera module captures video of the baby in real time. The video is captured at 30 frames per second and temporarily stored in local storage.

[0808] Step 2:

[0809] Terminal: Sends acquired video data to the server via a communication method. A secure protocol (e.g., HTTPS) is used for transmission.

[0810] Step 3:

[0811] Device: A vibration sensor detects the baby's breathing. It collects data every second and sends it to the server.

[0812] Step 4:

[0813] Terminal: A temperature and humidity sensor measures the temperature and humidity in the bedroom. It acquires data every 10 seconds and sends it to the server.

[0814] Step 5:

[0815] Terminal: An illuminance sensor measures the brightness of the bedroom. It acquires data every 5 seconds and sends it to the server.

[0816] Step 6:

[0817] Terminal: The emotion engine acquires the user's voice and video data. The emotion engine captures the data in real time and temporarily stores it in local storage.

[0818] Step 7:

[0819] Terminal: Sends acquired emotional data to the server via communication means. The emotional data is also transmitted using a secure protocol.

[0820] Step 8:

[0821] Server: Analyzes all received data. It analyzes the baby's posture and facial condition from camera footage and detects breathing patterns from vibration sensor data. It analyzes temperature, humidity, and illuminance data to evaluate the indoor environment.

[0822] Step 9:

[0823] Server: Analyzes user emotion data acquired by the emotion engine. Evaluates the user's emotional state based on voice tone and facial expression data.

[0824] Step 10:

[0825] Server: Detects anomalies based on analysis results. Determines an emergency situation, such as when the baby is lying face down, breathing has stopped, or the user is experiencing high levels of stress.

[0826] Step 11:

[0827] Server: If an anomaly is detected, an emergency notification will be sent to the user. The notification will be sent to the smartphone via push notification or email. If the user is feeling stressed, a message encouraging relaxation will also be included.

[0828] Step 12:

[0829] Server: Generates environmental control commands based on analysis results. If the room temperature exceeds the set range, it sends a cooling command to the air conditioner. If the room is too dark, it adjusts the lighting appropriately.

[0830] Step 13:

[0831] Terminal: Receives instructions from the server and controls environmental factors such as air conditioning and lighting. This includes adjusting the air conditioner's temperature setting and changing the brightness of the lights.

[0832] Step 14:

[0833] Server: It works in conjunction with the childcare record application to learn the baby's daily routines (sleep, eating, elimination, etc.). Based on the collected data, it analyzes the baby's normal behavioral patterns.

[0834] Step 15:

[0835] Server: Sends advice and notifications to the user based on learning results. For example, 30 minutes before the usual bedtime, it will notify the user, "It's time for the baby to sleep. Create a quiet environment," but if the user is feeling stressed, it will offer additional advice such as, "Take a deep breath and relax."

[0836] The above outlines the specific processing steps of "Baby Safe Watch." This process allows for the comprehensive management of the baby's safety and comfortable environment, and provides support tailored to the user's emotional state.

[0837] (Example 2)

[0838] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0839] Conventional childcare monitoring systems are limited to monitoring the baby's physical condition and do not take into account the emotional state of the user (parent), resulting in a failure to alleviate the stress and burden of childcare. Furthermore, there are challenges in the accuracy and reliability of systems for promptly responding to detected abnormalities.

[0840] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0841] In this invention, the server includes analysis means for monitoring the baby's safe and comfortable environment, emotion recognition means for recognizing the user's emotions, notification means for detecting anomalies or providing notifications in accordance with the user's emotions, and environment control means for controlling the bedroom environment. This allows for real-time monitoring of the baby's safe and comfortable environment while responding to the user's emotional state, thereby reducing stress and burden from childcare and enabling quick and appropriate responses in emergencies.

[0842] "Video acquisition means" refers to devices or modules for capturing a baby's posture and facial condition in real time.

[0843] A "vibration detection device" is a sensor used to detect the baby's breathing and body movements.

[0844] The "environmental sensor means" refers to a sensor used to measure the temperature and humidity of a bedroom.

[0845] The "illuminance detection means" is a sensor used to measure the brightness of a bedroom.

[0846] "Communication means" refers to devices or interfaces for transmitting data collected from image acquisition means, vibration detection means, environmental sensor means, and illuminance detection means to a server.

[0847] The "analysis means" refers to a function on the server that analyzes acquired data and monitors the baby's safety and comfortable environment.

[0848] "Emotion recognition means" refers to a function that acquires the user's voice and video and analyzes their emotional state.

[0849] A "notification method" is a means of notifying the user of important information or warnings based on analysis results or emotion recognition results.

[0850] The "environmental control means" is a function that issues instructions to automatically adjust the environment of the bedroom, such as temperature, humidity, and illuminance, based on the analysis results.

[0851] The "learning tool" refers to a function that works in conjunction with the childcare record application to learn the individual daily routines of babies.

[0852] An "advice tool" is a means of providing users with timely and appropriate advice based on learned patterns and the user's emotional state.

[0853] The present invention, "a system for monitoring a baby's safe and comfortable environment," includes the following configuration and operation:

[0854] System Configuration

[0855] terminal

[0856] Means of acquiring video:

[0857] A camera module is used to capture the baby's posture and facial expression in real time.

[0858] Specific example: A baby monitor camera captures the movements and facial expressions of a baby lying down.

[0859] Vibration detection means:

[0860] A vibration sensor is used to detect the baby's breathing and is placed under the bed.

[0861] Specific example: Continuously track the up-and-down movement of a baby's chest and collect data.

[0862] Environmental sensor means:

[0863] Temperature and humidity sensors are used to measure the environmental conditions in the bedroom.

[0864] Specific example: When the indoor temperature reaches 28 degrees Celsius, the sensor detects this and provides data.

[0865] Illuminance detection means:

[0866] Use an illuminance sensor to measure the brightness of the bedroom.

[0867] Specific example: When the brightness of the bedroom lighting exceeds a certain level, the data is recorded.

[0868] Means of communication:

[0869] The system transmits data collected from the camera module, vibration sensor, temperature and humidity sensor, and illuminance sensor to the server.

[0870] Specific example: Real-time video and sensor data are transmitted to a server via wireless communication.

[0871] server

[0872] Analysis method:

[0873] The server uses advanced algorithms to analyze the data sent from the terminal.

[0874] Specific example: Analyze data to monitor the baby's posture, facial expression, breathing patterns, and indoor environment.

[0875] Emotion recognition means:

[0876] It acquires audio and video data and recognizes the user's emotions.

[0877] Specific example: While a user is comforting a baby, their facial expressions and voice are analyzed to determine their emotional state.

[0878] Notification method:

[0879] If an anomaly is detected or if it is determined that the user is experiencing stress, an appropriate notification will be sent to the user.

[0880] Specific example: If a baby's breathing stops for 20 seconds, an emergency notification will be sent stating, "An abnormality has occurred in the baby's breathing."

[0881] Environmental control means:

[0882] Based on the analysis results, the system sends instructions to the terminal to automatically control the settings of the air conditioner and lighting in order to maintain the optimal indoor environment.

[0883] Specific example: If the room temperature is determined to be too high, a command is sent to the air conditioner to start cooling.

[0884] Learning methods:

[0885] It connects with a childcare record application to learn the baby's daily routine.

[0886] Specific example: Determine the baby's bedtime pattern, such as going to sleep at 8 PM every night.

[0887] Advice methods:

[0888] Based on learned patterns, it sends timely advice and notifications to the user. Furthermore, it provides advice based on the user's emotional state through emotion recognition.

[0889] Specific example: When it's time for the baby to sleep, the system will notify the user with "It's time for the baby to sleep. Let's create a quiet environment," and if the user is feeling stressed, it will offer additional advice such as "Take a deep breath and relax."

[0890] Example of a prompt

[0891] "Please explain the procedure for a program that sends a notification if a baby cries for an extended period of time."

[0892] "Please explain how to handle situations when users are experiencing high levels of stress."

[0893] This invention makes it possible to ensure a safe and comfortable environment for the baby, as well as reduce the mental burden on the user. The system uses various sensors and advanced analytical means to monitor the baby's and user's condition in real time and provide appropriate responses.

[0894] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0895] Step 1: Data acquisition and transmission (device)

[0896] Specific operation: Each sensor in the terminal (video acquisition means, vibration detection means, environmental sensor means, illuminance detection means) collects its own data. The video acquisition means captures video of the baby, and the vibration detection means monitors the baby's breathing. The environmental sensor means measures room temperature and humidity, and the illuminance detection means measures the brightness of the room.

[0897] Inputs: Baby's posture, facial expression, breathing, room temperature, humidity, and brightness.

[0898] Data processing and calculation: Each sensor collects data in real time and transmits it to the server via communication means.

[0899] Output: Various sensor data transferred to the server

[0900] Step 2: User emotion recognition (device)

[0901] Specific operation: The device's emotion recognition system captures the user's voice and video, acquiring voice tone and facial expression data. This data is sent to a server in real time for emotion analysis.

[0902] Input: User's voice data, video data

[0903] Data processing and calculation: The emotion recognition system collects data to analyze the user's emotions based on an emotion model and sends it to the server.

[0904] Output: Audio and video data sent to the server

[0905] Step 3: Data analysis and anomaly detection (server)

[0906] Specific operation: The server analyzes the received data and uses advanced algorithms to monitor the baby's posture, facial expression, breathing patterns, and room environment. Simultaneously, it analyzes the user's emotional state.

[0907] Input: Sensor data, audio data, and video data transmitted from the device.

[0908] Data processing and calculation: The server analyzes the data using analytical algorithms and detects anomalies. This includes determining whether the baby has stopped breathing or whether the user is in a high-stress state.

[0909] Output: Anomaly detection results, user emotional state analysis results

[0910] Step 4: Notification (Server)

[0911] Specific operation: If a server anomaly is detected, or if a user is experiencing stress, an emergency notification will be sent to the user using a notification system. The notification will arrive on a device such as a smartphone or tablet.

[0912] Input: Anomaly detection results, user emotional state analysis results

[0913] Data processing and calculation: When an anomaly is detected, immediately prepare a notification and send it to the user's device. Customize the message content according to the type of anomaly and the emotional state of the user.

[0914] Output: Notification message to the user

[0915] Step 5: Environment Control (Server)

[0916] Specific operation: Based on the analysis results, the server sends instructions to the terminal to control the settings of the air conditioner and lighting in order to maintain an optimal environment (temperature, humidity, brightness) in the bedroom.

[0917] Input: Baby's condition data, indoor environment data

[0918] Data processing and calculation: The server calculates the settings necessary to maintain a comfortable environment for the baby and sends instructions to the terminal.

[0919] Output: Control instructions for air conditioners and lighting.

[0920] Step 6: Learning and Advice (Server)

[0921] Specific operation: The server interacts with the childcare record application to learn the baby's daily routines (sleep, meals, elimination, etc.). Based on the learned patterns, it sends timely advice and notifications to the user.

[0922] Input: Childcare record data, past sensor data

[0923] Data Processing and Calculation: Analyze data from childcare record applications to analyze the baby's daily routines. Use the analysis results to generate appropriate advice.

[0924] Output: Advice messages, notification messages

[0925] The above outlines the processing flow of this system's program. By collecting and analyzing data in real time, this system provides comprehensive support to enhance the safety of babies and the comfort of users.

[0926] (Application Example 2)

[0927] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0928] Traditional systems for managing the safety and comfort of babies could monitor the baby's condition but failed to consider the user's emotions. Similarly, in physical stores, it was difficult to monitor customer emotions and behavior in real time and implement appropriate responses and environmental controls. This resulted in insufficient improvement in the quality of childcare and customer service, sometimes leading to delays in providing optimal support.

[0929] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an analysis means, a notification means, a control means, an automatic notification means, a customer emotion notification means, and a store environment control means. This enables monitoring of the baby's safety and comfort, as well as responses to customer emotions and automatic control of the store environment.

[0930] A "camera system" is a device used to detect the posture and facial condition of babies or customers in real time.

[0931] A "vibration sensor" is a device used to detect a baby's breathing.

[0932] A "temperature and humidity sensor means" is a sensor for measuring the temperature and humidity in a bedroom or store.

[0933] An "illuminance sensor means" is a sensor used to measure the brightness inside a bedroom or store.

[0934] "Communication means" refers to means for transmitting data from the camera means, vibration sensor means, temperature and humidity sensor means, and illuminance sensor means to the server.

[0935] "Analysis means" refers to means for monitoring the safety and comfort of babies and customers by analyzing the aforementioned data.

[0936] "Notification means" refers to a means for notifying a user or staff member when an abnormality is detected by the analysis means.

[0937] "Control means" refers to means for controlling the environment of a bedroom or store based on the analysis means.

[0938] An "emotion engine" is a means of detecting customer emotions in real time.

[0939] A "customer sentiment notification system" is a means of analyzing customer sentiment data and notifying staff of appropriate responses.

[0940] "Store environment control means" refers to means for automatically adjusting the store environment based on customer sentiment data.

[0941] "Learning methods" refer to techniques for learning individual patterns by linking them with the baby's care records.

[0942] "Advice means" refers to means for providing timely advice and notifications to the user based on the learning means.

[0943] This invention is a system for managing the safety and comfort of babies. Furthermore, the same technology can be used to monitor customer emotions and behavior in real time in physical stores, thereby improving customer service.

[0944] System Configuration

[0945] 1. Terminal

[0946] Camera System: This camera is used to detect the posture and facial condition of babies and customers in real time. The video data acquired from the camera is transmitted to the server described below.

[0947] Vibration sensor: This sensor is used to detect the baby's breathing. The vibration data is transmitted to a server.

[0948] Temperature and humidity sensor means: A sensor for measuring the temperature and humidity in a bedroom or store. This data is also transmitted to the server.

[0949] Illuminance sensor: This is a sensor used to measure the brightness in a bedroom or store. This data is also transmitted to the server.

[0950] Communication method: This is a communication device used to transmit data acquired from the above-mentioned sensors to the server. For example, Wi-Fi modules or Bluetooth are used.

[0951] Emotion Engine: A device that acquires customer voice and video data and detects customer emotions in real time.

[0952] 2. Server

[0953] Analysis method: This method involves analyzing data transmitted from terminals on a server to monitor the safety and comfort of babies and customers. Specifically, it utilizes image analysis and data mining techniques.

[0954] Notification method: This is a means of notifying users or store staff when an anomaly is detected by the analysis method. Notifications are made via devices such as smartphones and tablets.

[0955] Control means: A means for transmitting instructions to a terminal to control the environment of a bedroom or store based on analysis means. For example, it can be used to adjust the temperature of an air conditioner or change the brightness of lighting.

[0956] Customer sentiment notification method: This method analyzes customer sentiment data and notifies staff of appropriate responses. For example, if a customer expresses dissatisfaction, a notification is sent to staff prompting a quick response.

[0957] Store environment control means: This means automatically adjusting the store environment based on customer emotion data. For example, it may change the store temperature or background music.

[0958] 3. User

[0959] User Devices: Users or store staff can receive notifications using devices such as smartphones and tablets, and manage the baby's condition, environment, and customer's emotional state through the application. They can also receive timely advice and notifications provided by the system.

[0960] Specific example

[0961] Common usage examples

[0962] Terminal: The camera detects the baby's posture and sends the data to the server.

[0963] Terminal: The emotion engine analyzes the customer's facial expressions and sends them to the server.

[0964] Server: If the server analyzes the data and determines that everything is normal, it notifies the user or store staff with messages such as "The baby is safe" or "The customer is satisfied."

[0965] Terminal: The temperature and humidity sensor detects when the indoor temperature is too high and sends data to the server.

[0966] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[0967] Example of a prompt

[0968] Application: Design a smart store watch application to enhance the customer experience in physical stores. Using cameras and sensors, monitor customer emotions and behavior in real time, automating all responses and environmental controls. For example, if a customer appears distressed, quickly notify staff and adjust settings such as air conditioning or music to create a more relaxing atmosphere.

[0969] Examples of use in emergencies

[0970] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[0971] Server: The system determines this is an emergency and sends an emergency notification to the parents stating, "There is an abnormality in the baby's breathing."

[0972] User: Receive notifications on your smartphone and respond quickly.

[0973] Server: Simultaneously analyzes the user's emotions and, if it determines that the user is experiencing stress, sends a message saying, "Please calm down. We will address this immediately."

[0974] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0975] Step 1:

[0976] The device uses a camera to detect the posture and facial features of babies and customers in real time and acquire video data. The input is camera footage, and the output is the detected face and posture data. Specifically, it uses OpenCV and the dlib library to capture video and execute a face detection algorithm.

[0977] Step 2:

[0978] The device uses a vibration sensor to detect the baby's breathing. The input is vibration data from the sensor, and the output is data indicating the presence of breathing. The sensor data is acquired at regular intervals and processed into data for analyzing the breathing pattern.

[0979] Step 3:

[0980] The terminal uses temperature and humidity sensors to measure and acquire data on the temperature and humidity in a bedroom or store. The input is the reading from the temperature and humidity sensors, and the output is accurate temperature and humidity data for the room. This data is used as foundational data for analysis and control described later.

[0981] Step 4:

[0982] The terminal uses an illuminance sensor to measure the brightness in a bedroom or store and acquire data. The input is the illuminance sensor reading, and the output is the room's illuminance data. The measured data is sent to the server.

[0983] Step 5:

[0984] The terminal uses communication methods to transmit data acquired from the camera, vibration sensor, temperature and humidity sensor, and illuminance sensor to the server. The input is the data acquired from each sensor, and the output is the data transferred to the server. Wi-Fi, Bluetooth, and other technologies are used for communication.

[0985] Step 6:

[0986] The server uses analysis tools to analyze the transmitted data and monitor the safety and comfort of babies and customers. The input is sensor data transmitted from the terminal, and the output is the analysis results. Specifically, it uses image analysis algorithms and machine learning models to detect various anomaly patterns.

[0987] Step 7:

[0988] The server uses a notification system to notify users or store staff if it detects an anomaly in the analysis process. The input is the analysis result, and the output is a notification message. This notification is sent via a push notification system to smartphones and tablets.

[0989] Step 8:

[0990] The server uses control means to send instructions to the terminal to control the environment of a bedroom or store based on analysis means. The input is the analysis result, and the output is the environment control instruction. For example, it may issue instructions to adjust the set temperature of an air conditioner or the brightness of lighting.

[0991] Step 9:

[0992] The terminal uses an emotion engine to acquire customer voice and video data and detect the customer's emotions. The input is voice and video data, and the output is analyzed emotion data. An emotion recognition model is used to detect changes in emotion, and this data is sent to the server.

[0993] Step 10:

[0994] The server uses a customer sentiment notification system to analyze customer sentiment data and notify staff of appropriate actions. The input is sentiment data sent from the terminal, and the output is a notification to staff. The notification content changes according to the customer's status, prompting a quick response for dissatisfied customers.

[0995] Step 11:

[0996] The server uses store environment control means to automatically adjust the store environment based on customer emotion data. The input is analyzed emotion data, and the output is environment control instructions. For example, it adjusts the store temperature and background music.

[0997] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0998] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0999] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1000] [Third Embodiment]

[1001] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1002] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1003] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1004] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1005] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1006] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1007] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1008] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1009] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1010] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1011] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1012] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1013] A specific embodiment of the "Baby Safe Watch" of the present invention is described below. This system consists of multiple terminals, a server, and a user (parent), each with its own role.

[1014] System Configuration

[1015] 1. Terminal

[1016] Camera equipment: A camera is installed to detect the baby's posture and facial condition in real time.

[1017] Vibration sensor means: A vibration sensor is installed in the bed to detect the baby's breathing.

[1018] Temperature and humidity sensor means: Install a sensor to measure the temperature and humidity in the bedroom.

[1019] Illuminance sensor means: Install a sensor to measure the brightness of the bedroom.

[1020] Communication means: Equipped with a communication device for transmitting data acquired from the above sensors to a server.

[1021] 2. Server

[1022] Analysis method: The server receives data transmitted from the terminal and performs analysis using advanced algorithms.

[1023] Notification mechanism: The system includes a mechanism for notifying the user according to the analysis results. If an anomaly is detected, an emergency notification will be sent.

[1024] Control mechanism: Based on the analysis results, it sends instructions to the terminal to automatically control environmental factors such as temperature, humidity, and illuminance.

[1025] Learning method: It works in conjunction with a childcare record application to learn the individual patterns of each baby.

[1026] Advice method: Based on learned patterns, it sends timely advice and notifications to the user.

[1027] 3. User

[1028] Using devices such as smartphones and tablets, users can receive notifications and manage their baby's condition and environment through the application.

[1029] Explanation of the program's processing

[1030] Data acquisition and transmission (terminal)

[1031] Terminal: The camera module captures images of the baby and sends them to the server for real-time analysis.

[1032] Terminal: The vibration sensor detects the baby's breathing in real time and sends the data to the server.

[1033] Terminal: Temperature, humidity, and illuminance sensors also measure environmental data and transmit it to the server at regular intervals.

[1034] Data analysis and anomaly detection (server)

[1035] Server: Analyzes received data and monitors the baby's posture, facial expression, breathing patterns, and room environment.

[1036] Server: If an anomaly is detected as a result of the analysis, an emergency notification will be sent to the user's smartphone.

[1037] For example, if the baby turns onto its stomach, an alert is sent immediately.

[1038] An emergency notification will also be issued if breathing stops for 20 seconds or more.

[1039] Environmental control (server)

[1040] Server: Based on the analysis results, it sends instructions to terminals to automatically control air conditioning and lighting in order to maintain the optimal indoor environment.

[1041] If the room temperature exceeds the set range, the server sends a cooling command to the air conditioner.

[1042] If the room is too dark, adjust the lighting appropriately.

[1043] Learning and advice (server)

[1044] Server: It connects with childcare records and learns the baby's daily routines (sleep, eating, elimination, etc.).

[1045] Server: Based on learned data, it sends timely advice and notifications to the user.

[1046] For example, if the system learns that the baby usually goes to bed at 8 PM, it will send a notification 30 minutes beforehand saying, "It's time for the baby to sleep. Please create a quiet environment."

[1047] Specific example

[1048] Common usage examples

[1049] Terminal: The camera detects the baby's posture and sends the data to the server.

[1050] Server: If the analysis determines that everything is normal, it notifies the user that "the baby is safe."

[1051] Terminal: The temperature and humidity sensor detects when the indoor temperature is too high and sends data to the server.

[1052] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[1053] Examples of use in emergencies

[1054] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[1055] Server: The system determines this is an emergency and sends an emergency notification to the parents stating, "There is an abnormality in the baby's breathing."

[1056] User: Receive notifications on your smartphone and respond quickly.

[1057] This concludes the description of the embodiment of the "Baby Safe Watch" of the present invention. By using this system, parents can manage the safety and comfortable childcare environment of their baby in one place, thereby reducing the burden of childcare.

[1058] The following describes the processing flow.

[1059] Step 1:

[1060] Terminal: The camera module acquires real-time video of the baby. It captures 30 frames per second of video and temporarily stores it in local storage.

[1061] Step 2:

[1062] Terminal: Transmits the acquired video to the server via a communication method. The data is transmitted using a secure protocol (e.g., HTTPS).

[1063] Step 3:

[1064] Device: A vibration sensor detects the baby's breathing. The vibration sensor collects data every second and sends it to the server.

[1065] Step 4:

[1066] Terminal: A temperature and humidity sensor measures the temperature and humidity in the bedroom. It acquires data every 10 seconds and sends it to the server.

[1067] Step 5:

[1068] Terminal: An illuminance sensor measures the brightness of the bedroom. It acquires data every 5 seconds and sends it to the server.

[1069] Step 6:

[1070] Server: Analyzes all received data. Analyzes camera footage to recognize the baby's posture and facial expression. Analyzes vibration sensor data to detect breathing patterns. Analyzes temperature, humidity, and illuminance data to evaluate the indoor environment.

[1071] Step 7:

[1072] Server: Detects anomalies based on analysis results. Determines an emergency situation, such as when the baby is lying face down or has stopped breathing.

[1073] Step 8:

[1074] Server: If an anomaly is detected, an emergency notification will be sent to the user. The notification will be sent to the smartphone in the form of a push notification or email.

[1075] Step 9:

[1076] Server: Generates environmental control commands based on the analysis results. For example, if the room temperature exceeds the set range, it sends a cooling command to the air conditioner.

[1077] Step 10:

[1078] Terminal: Receives instructions from the server and controls environmental factors such as air conditioning and lighting. This includes adjusting the air conditioner's temperature setting and changing the brightness of the lights.

[1079] Step 11:

[1080] Server: It works in conjunction with the childcare record application to learn the baby's daily routine. Based on the collected data, it analyzes the baby's sleep, eating, and elimination patterns.

[1081] Step 12:

[1082] Server: Sends advice and notifications to the user based on the learning results. For example, 30 minutes before the usual bedtime, it might notify the user, "It's time for the baby to sleep. Please create a quiet environment."

[1083] The above outlines the specific processing steps for "Baby Safe Watch." This process allows for comprehensive management of your baby's safety and comfortable environment.

[1084] (Example 1)

[1085] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1086] In modern childcare, parents need to constantly monitor their baby's safety, but 24-hour monitoring is physically and psychologically burdensome. Optimizing the indoor environment is also crucial for maintaining the baby's safety and comfort, but manual adjustments are time-consuming. Furthermore, the difficulty in obtaining appropriate advice based on the baby's daily routines leads to decreased efficiency in childcare.

[1087] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1088] In this invention, the server includes a camera means for detecting the baby's posture and facial condition in real time, a vibration sensor means for detecting the baby's breathing, an environmental sensor means for measuring the temperature and humidity of the bedroom, an illuminance sensor means for measuring the brightness of the bedroom, a communication means for transmitting data from the camera means, vibration sensor means, environmental sensor means and illuminance sensor means to the server, an analysis means in the server for monitoring the baby's safe and comfortable environment by analyzing the data, a notification means for notifying the user when the analysis means detects an abnormality, a control means for controlling the bedroom environment based on the analysis means, a learning means for learning individual patterns in conjunction with childcare records, and an advice means for providing timely advice and notifications based on the learned patterns. This makes it possible to automatically and efficiently maintain the baby's safety and comfort and reduce the burden of childcare on parents.

[1089] A "camera system" is a device used to detect the baby's posture and facial condition in real time.

[1090] A "vibration sensor" is a device used to detect a baby's breathing.

[1091] An "environmental sensor" is a device used to measure the temperature and humidity of a bedroom.

[1092] An "illuminance sensor" is a device used to measure the brightness of a bedroom.

[1093] "Communication means" refers to a device for transmitting data acquired from camera means, vibration sensor means, environmental sensor means, and illuminance sensor means to a server.

[1094] "Analysis means" refers to a device that analyzes data transmitted via communication means to monitor the safety and comfort of the baby.

[1095] A "notification means" is a device used to notify the user when an anomaly is detected in the analysis means.

[1096] The "control means" is a device for automatically controlling the bedroom environment based on the analysis means.

[1097] A "learning tool" is a device that works in conjunction with childcare records to learn individual patterns.

[1098] An "advice tool" is a device that provides timely advice and notifications based on learned patterns.

[1099] A specific embodiment of the "Baby Safe Watch" of the present invention is described below. This system consists of multiple terminals, a server, and a user (parent), each with its own role.

[1100] System Configuration

[1101] 1. Terminal

[1102] Camera equipment: A camera is installed to detect the baby's posture and facial condition in real time. For example, it can be used to check if the baby is rolling over or if their face is covered by the blanket.

[1103] Vibration sensor means: A vibration sensor is installed in the bed to detect the baby's breathing. For example, the breathing pattern is obtained from the movement of the baby's chest.

[1104] Environmental sensor means: Install sensors to measure the temperature and humidity in the bedroom. For example, obtain data such as a room temperature of 25 degrees Celsius and a humidity of 50%.

[1105] Illuminance sensor means: Install a sensor to measure the brightness of the bedroom. For example, confirm that the brightness of the room is 400 lux.

[1106] Communication means: The system is equipped with a communication device for transmitting data acquired from the above-mentioned sensors to a server. For example, a Wi-Fi module is used to transmit the data.

[1107] 2. Server

[1108] Analysis method: The server receives data transmitted from the terminal and performs analysis using advanced algorithms. For example, it analyzes the baby's posture, breathing status, and the indoor environment.

[1109] Notification method: The system includes a means to notify the user according to the analysis results. If an abnormality is detected, an emergency notification will be sent. For example, a notification such as "An abnormality has been detected in the baby's breathing" will be sent to the user.

[1110] Control mechanism: Based on the analysis results, it sends instructions to the terminal to automatically control environmental conditions such as temperature, humidity, and illuminance. For example, if the room temperature exceeds the set range, it sends an instruction to the air conditioner to start cooling.

[1111] Learning method: It works in conjunction with a childcare record application to learn the individual patterns of the baby. For example, it learns the baby's daily patterns such as sleep, eating, and elimination.

[1112] Advice methods: Based on learned patterns, the system sends timely advice and notifications to the user. For example, for a baby who has a habit of going to bed at 8 pm, it might send a notification 30 minutes beforehand saying, "It's time for your baby to sleep. Please create a quiet environment."

[1113] 3. User

[1114] Users can receive notifications from the server using devices such as smartphones and tablets, and manage their baby's condition and environment through the application. For example, a user who receives an emergency notification can quickly rush to the baby's bed and check on their baby's condition.

[1115] Specific example

[1116] Common usage examples

[1117] Terminal: The camera detects the baby's posture and sends the data to the server.

[1118] Server: If the analysis determines that everything is normal, it notifies the user that "the baby is safe."

[1119] Terminal: The environmental sensor detects that the indoor temperature is too high and sends data to the server.

[1120] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[1121] Examples of use in emergencies

[1122] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[1123] Server: Determined to be in an emergency, it sends an emergency notification to the user stating, "There is an abnormality in the baby's breathing."

[1124] User: Receives a notification on their smartphone, rushes to the bed immediately, and checks on the baby's condition.

[1125] Example of a prompt

[1126] Example prompt:

[1127] BabySafeWatch is a system that monitors and controls a baby's posture, breathing, and indoor environment. What kind of data does it typically collect, and how does it notify the user?

[1128] How will users be notified in an emergency, and how will they respond?

[1129] This concludes the description of the embodiment of the "Baby Safe Watch" of the present invention. By using this system, parents can manage the safety and comfortable childcare environment of their baby in one place, thereby reducing the burden of childcare.

[1130] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1131] Step 1: Data Acquisition (Device)

[1132] Input: Information about the baby's posture and facial expression, breathing, room temperature, humidity, and light level.

[1133] Specific operation: The camera on the device captures images of the baby. A vibration sensor detects the baby's breathing, an environmental sensor measures the room temperature and humidity, and an illuminance sensor measures the room brightness.

[1134] Output: Acquired video data, respiration data, and environmental data (temperature, humidity, illuminance) are generated.

[1135] Step 2: Data transmission (terminal)

[1136] Input: Data acquired from each sensor (video data, respiratory data, environmental data)

[1137] Specific operation: Data acquired by sensors and cameras is transmitted to a server using communication means. A Wi-Fi module is used as the specific means.

[1138] Output: Data acquired by sensors and cameras is sent to the server.

[1139] Step 3: Data Analysis (Server)

[1140] Input: Data from each sensor sent to the server (video data, respiration data, environmental data)

[1141] Specific operation: The server's analysis system receives the data and uses advanced algorithms to analyze the baby's posture, breathing, and room environment. For example, it analyzes whether the baby's posture is normal and whether the breathing pattern is abnormal.

[1142] Output: Analysis results (determination of normal or abnormal, indoor environment status)

[1143] Step 4: Anomaly detection and notification (server)

[1144] Input: Results of data analysis

[1145] Specific operation: If an anomaly is detected by the server's analysis system, the notification system will inform the user. Specifically, an emergency notification will be sent if the baby is lying face down or if breathing stops for more than 20 seconds.

[1146] Output: An emergency notification is sent to the user.

[1147] Step 5: Environment Control (Server)

[1148] Input: Results of data analysis

[1149] Specific operation: Based on the analysis results, the server's control system sends instructions to the terminal to automatically adjust environmental conditions such as temperature, humidity, and illuminance. For example, if the room temperature exceeds the set range, it sends a cooling command to the air conditioner.

[1150] Output: An instruction for environment adjustment is sent to the terminal.

[1151] Step 6: Data Training (Server)

[1152] Input: Daily life pattern data from a childcare record application.

[1153] Specific operation: The server's learning mechanism works in conjunction with the childcare record application to learn the individual patterns of the baby. For example, it analyzes and learns data such as the baby's sleep, meals, and bowel movements.

[1154] Output: Learning results (insights about babies' daily routines)

[1155] Step 7: Advice and Notification (Server)

[1156] Input: Learning results and data analysis results

[1157] Specific operation: Based on patterns learned by the server's advice system, timely advice and notifications are sent to the user. For example, for a baby who has a habit of going to bed at 8 pm, a notification is sent 30 minutes beforehand saying, "It's time for your baby to sleep. Please create a quiet environment."

[1158] Output: Appropriate advice and notifications are sent to the user.

[1159] (Application Example 1)

[1160] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1161] Ensuring the safety of babies and children and reducing anxiety for parents and guardians while they engage in childcare or shopping activities is crucial. However, current systems lack the functionality to monitor children's safety in stores in real time, in addition to monitoring the baby's environment, and to quickly notify parents and staff in case of abnormalities. Therefore, there is a need to develop a new system that allows parents and guardians to watch over their children with greater peace of mind.

[1162] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1163] In this invention, the server includes an image acquisition means for detecting the baby's posture and facial condition in real time, a vibration detection means for detecting the baby's breathing, a climate data acquisition means for measuring the temperature and humidity of the bedroom, a light intensity sensor means for measuring the brightness of the bedroom, a communication device for transmitting data from the image acquisition means, vibration detection means, climate data acquisition means and light intensity sensor means to a central processing unit, an analysis means in the central processing unit for monitoring the baby's safety and comfort by analyzing the data, a notification means for notifying the user when an abnormality is detected by the analysis means, an environment control means for controlling the bedroom environment based on the analysis means, a means for monitoring the safety of children in the store in real time, and a means for notifying parents or staff when a child enters a specific area. This ensures a safe and comfortable environment for babies, and further enables real-time monitoring of children's safety in the store and a rapid response in the event of an abnormality.

[1164] The "image acquisition means" is a device for detecting the baby's posture and facial condition in real time.

[1165] A "vibration detection device" is a device for detecting a baby's breathing in real time.

[1166] "Climate data acquisition means" refers to a device for measuring the temperature and humidity of a bedroom.

[1167] A "light intensity sensor" is a device used to measure the brightness of a bedroom.

[1168] A "communication device" is a device used to transmit data acquired from various sensors to a central processing unit.

[1169] A "central processing unit" is a device that analyzes received data and monitors the safety and comfort of babies and children.

[1170] An "analysis tool" is a device used to analyze data and detect anomalies.

[1171] A "notification means" is a device that notifies the user when an abnormality is detected in the analysis means.

[1172] An "environmental control means" is a device that controls the bedroom environment based on an analysis means.

[1173] "A means of monitoring the safety of children in stores in real time" refers to a device that monitors the safety of children in stores in real time and notifies the staff if any abnormalities are detected.

[1174] "Means of notifying parents or staff" refers to a device that notifies parents or store staff when a child enters a specific area.

[1175] A specific embodiment of the present invention, "Kids Safe Watch," is described below. The system consists of multiple terminals, a central processing unit, and users, each with its own role.

[1176] System Configuration

[1177] 1. Terminal

[1178] Image acquisition method: Install a camera to detect the child's posture and facial condition in real time.

[1179] Vibration detection method: A vibration sensor is installed on the floor to detect the breathing and position of the baby or child.

[1180] Climate data acquisition method: Install sensors to measure the temperature and humidity in the store.

[1181] Light intensity sensor means: Install a sensor to measure the brightness of the store.

[1182] Communication device: Equipped with a communication device for transmitting data acquired from the above-mentioned sensors to the central processing unit.

[1183] 2. Central Processing Unit

[1184] Analysis method: The central processing unit receives data transmitted from the terminal and performs analysis using advanced algorithms.

[1185] Notification method: The system includes a means to notify the user according to the analysis results. If an anomaly is detected, an emergency notification will be sent.

[1186] Environmental control means: Based on the analysis results, instructions are sent to the terminal to automatically control environmental conditions such as temperature, humidity, and illuminance.

[1187] 3. User

[1188] Using devices such as smartphones and tablets, users can receive notifications and manage their child's status and environment through the application.

[1189] Users can monitor the location and behavior of children within the store and respond quickly as needed.

[1190] Explanation of the program's processing

[1191] Hardware and software to use

[1192] Hardware: Smartphones, tablets, camera modules installed in stores, vibration sensors, temperature and humidity sensors, and illuminance sensors.

[1193] Software: Python (for data analysis and notification implementation), AWS cloud services (for data storage and processing), Firebase (real-time database and notification service), OpenCV (image analysis library).

[1194] Data acquisition and calculation processing

[1195] 1. Processing camera data

[1196] Using OpenCV, we detect and analyze children's posture and position from camera images.

[1197] The analysis data is sent to AWS in real time.

[1198] 2. Processing of sensor data

[1199] The system periodically acquires data from vibration sensors, temperature and humidity sensors, and illuminance sensors and sends it to AWS.

[1200] 3. Notification function

[1201] Using Firebase, users receive real-time notifications on their smartphones when an anomaly is detected.

[1202] 4. Environmental control

[1203] Data analysis is performed on AWS, and instructions for controlling air conditioners and lighting are sent to the terminal as needed.

[1204] Specific usage examples

[1205] Common usage examples

[1206] Terminal: The camera detects the posture and position of children in the store and transmits the data to the central processing unit.

[1207] Central Processing Unit: If it analyzes the data and determines that it is normal, it notifies the user that "the child is safe."

[1208] Terminal: The temperature and humidity sensor detects if the temperature inside the store is too high and sends data to the central processing unit.

[1209] Central Processing Unit: Sends cooling instructions to the air conditioner and adjusts it to the appropriate temperature.

[1210] Examples of use in emergencies

[1211] Terminal: If the vibration sensor detects that a child has left a designated area, it transmits that data to the central processing unit.

[1212] Central Processing Unit: Determines an emergency and sends an emergency notification to parents or store staff stating, "A child has left a designated area."

[1213] User: Receive notifications on your smartphone and respond quickly.

[1214] Examples of prompts for generative AI models

[1215] Please generate examples of how "Kids Safe Watch" can be used. Describe the details of a system that uses in-store cameras and sensors to manage child safety and notifies parents and store staff in real time if any anomalies are detected. Include specific use cases, such as notifications for when a child leaves the play area or when a child remains in the store after closing time.

[1216] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1217] Step 1:

[1218] Terminal: A camera module is used to detect the child's posture and position. The camera module acquires image data in real time from a designated area within the store. This image data is used as input data. The acquired image data is sent to the central processing unit using the OpenCV library for data preprocessing.

[1219] Step 2:

[1220] Server: The central processing unit receives pre-processed image data. It then uses OpenCV to execute face recognition and posture detection algorithms to extract the child's posture and location information. The extracted posture and location information is generated as output data, and the process proceeds to the next analysis step.

[1221] Step 3:

[1222] Terminal: Uses vibration sensors to acquire vibration data from children's floors and play areas installed in specific areas within the store. The vibration sensors transmit the detected vibration data to the central processing unit in real time. The vibration data is used as input data.

[1223] Step 4:

[1224] Server: The central processing unit receives data from vibration sensors and analyzes it using a rational threshold determination algorithm. Based on the analysis of the vibration data, it determines whether the child has deviated from a specific area. The determination result is generated as output data, and if an anomaly is detected, the process proceeds to the notification step.

[1225] Step 5:

[1226] Terminal: Using a climate data acquisition device, temperature and humidity data inside the store are acquired periodically. This data is used as input data and transmitted to the central processing unit in real time.

[1227] Step 6:

[1228] Server: The central processing unit receives temperature and humidity data transmitted from the climate data acquisition means and performs analysis to determine if it is within an appropriate range. Based on the analysis results, it determines whether the optimal temperature is being maintained. If the temperature deviates from the optimal range, an instruction to adjust using the environmental control means is output.

[1229] Step 7:

[1230] Terminal: Uses a light intensity sensor to acquire brightness (illuminance) data within the store. The brightness data is used as input data and transmitted to the central processing unit in real time.

[1231] Step 8:

[1232] Server: The central processing unit receives data from the light intensity sensor and performs analysis to determine whether the illuminance is within an appropriate range. Based on the analysis results, it determines whether the lighting is maintained within an appropriate range. If the illuminance deviates from the appropriate range, an instruction is output to adjust using the environmental control means.

[1233] Step 9:

[1234] Server: The server comprehensively analyzes all data acquired to date (image data, vibration data, temperature and humidity data, light intensity data) and, if an anomaly is detected, uses a notification system to notify users and store staff in real time. The notification content is generated and sent as output data.

[1235] Step 10:

[1236] Users: Customers or store staff receive notifications on their smartphones or tablets and take prompt action based on the displayed notification content. They review the notification and initiate specific response actions.

[1237] The above outlines the specific processing steps of the system program that implements the application example.

[1238] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1239] The "Baby Safe Watch" system is a comprehensive system for managing a baby's safety and a comfortable childcare environment. It incorporates an emotion engine that recognizes the user's emotions and adjusts notifications and support accordingly. This system consists of multiple terminals, a server, and the user (parent), each with its own specific role.

[1240] System Configuration

[1241] 1. Terminal

[1242] Camera equipment: A camera is installed to detect the baby's posture and facial condition in real time.

[1243] Vibration sensor means: A vibration sensor is installed in the bed to detect the baby's breathing.

[1244] Temperature and humidity sensor means: Install a sensor to measure the temperature and humidity in the bedroom.

[1245] Illuminance sensor means: Install a sensor to measure the brightness of the bedroom.

[1246] Communication means: Equipped with a communication device for transmitting data acquired from the above sensors to a server.

[1247] Emotion Engine Method: Acquires audio and video data to recognize the user's emotions.

[1248] 2. Server

[1249] Analysis method: The server receives data transmitted from the terminal and performs analysis using advanced algorithms.

[1250] Notification mechanism: The system includes a mechanism for notifying the user according to the analysis results. If an anomaly is detected, an emergency notification will be sent.

[1251] Control mechanism: Based on the analysis results, it sends instructions to the terminal to automatically control environmental factors such as temperature, humidity, and illuminance.

[1252] Learning method: It works in conjunction with a childcare record application to learn the individual patterns of each baby.

[1253] Advice method: Based on learned patterns, it sends timely advice and notifications to the user.

[1254] 3. User

[1255] Using devices such as smartphones and tablets, users can receive notifications and manage their baby's condition and environment through the application.

[1256] Explanation of the program's processing

[1257] Data acquisition and transmission (terminal)

[1258] Terminal: The camera module captures images of the baby and sends them to the server for real-time analysis.

[1259] Terminal: The vibration sensor detects the baby's breathing in real time and sends the data to the server.

[1260] Terminal: Temperature, humidity, and illuminance sensors also measure environmental data and transmit it to the server at regular intervals.

[1261] User emotion recognition (device)

[1262] Terminal: The emotion engine acquires the user's voice and video data.

[1263] Terminal: Sends acquired emotion data to the server.

[1264] Data analysis and anomaly detection (server)

[1265] Server: Analyzes received data and monitors the baby's posture, facial expression, breathing patterns, and room environment.

[1266] Server: Further analyzes the user's emotional data obtained by the emotion engine.

[1267] Server: Sends appropriate notifications to the user if an abnormality is detected in the baby or if the user is determined to be experiencing stress.

[1268] Environmental control (server)

[1269] Server: Based on the analysis results, it sends instructions to terminals to automatically control air conditioning and lighting in order to maintain the optimal indoor environment.

[1270] Terminal: Adjust the air conditioner temperature setting or change the brightness of the lights.

[1271] Learning and advice (server)

[1272] Server: Works in conjunction with the childcare record application to learn the baby's daily routine (sleep, meals, elimination, etc.).

[1273] Server: Based on learned data, it sends timely advice and notifications to the user. Furthermore, it adjusts the content of the advice based on the user's emotional state.

[1274] For example, if the system learns that the baby usually goes to bed at 8 PM, it will notify the user 30 minutes beforehand with the message, "It's time for your baby to sleep. Please create a quiet environment." However, if the user is feeling stressed, it will offer additional advice such as, "Take a deep breath and relax."

[1275] Specific example

[1276] Common usage examples

[1277] Terminal: The camera detects the baby's posture and sends the data to the server.

[1278] Terminal: The emotion engine analyzes the user's facial expressions and sends them to the server.

[1279] Server: If the analysis determines that everything is normal, it notifies the user that "the baby is safe."

[1280] Terminal: The temperature and humidity sensor detects when the indoor temperature is too high and sends data to the server.

[1281] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[1282] Examples of use in emergencies

[1283] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[1284] Server: The system determines this is an emergency and sends an emergency notification to the parents stating, "There is an abnormality in the baby's breathing."

[1285] User: Receive notifications on your smartphone and respond quickly.

[1286] Server: Simultaneously analyzes the user's emotions and, if it determines that the user is experiencing stress, sends a message saying, "Please calm down. We will address this immediately."

[1287] This concludes our explanation of the new implementation, "BabySafeWatch." By combining it with an emotion engine, it is possible to further reduce the burden on the user and provide a higher level of reassurance.

[1288] The following describes the processing flow.

[1289] Step 1:

[1290] Terminal: The camera module captures video of the baby in real time. The video is captured at 30 frames per second and temporarily stored in local storage.

[1291] Step 2:

[1292] Terminal: Sends acquired video data to the server via a communication method. A secure protocol (e.g., HTTPS) is used for transmission.

[1293] Step 3:

[1294] Device: A vibration sensor detects the baby's breathing. It collects data every second and sends it to the server.

[1295] Step 4:

[1296] Terminal: A temperature and humidity sensor measures the temperature and humidity in the bedroom. It acquires data every 10 seconds and sends it to the server.

[1297] Step 5:

[1298] Terminal: An illuminance sensor measures the brightness of the bedroom. It acquires data every 5 seconds and sends it to the server.

[1299] Step 6:

[1300] Terminal: The emotion engine acquires the user's voice and video data. The emotion engine captures the data in real time and temporarily stores it in local storage.

[1301] Step 7:

[1302] Terminal: Sends acquired emotional data to the server via communication means. The emotional data is also transmitted using a secure protocol.

[1303] Step 8:

[1304] Server: Analyzes all received data. It analyzes the baby's posture and facial condition from camera footage and detects breathing patterns from vibration sensor data. It analyzes temperature, humidity, and illuminance data to evaluate the indoor environment.

[1305] Step 9:

[1306] Server: Analyzes user emotion data acquired by the emotion engine. Evaluates the user's emotional state based on voice tone and facial expression data.

[1307] Step 10:

[1308] Server: Detects anomalies based on analysis results. Determines an emergency situation, such as when the baby is lying face down, breathing has stopped, or the user is experiencing high levels of stress.

[1309] Step 11:

[1310] Server: If an anomaly is detected, an emergency notification will be sent to the user. The notification will be sent to the smartphone via push notification or email. If the user is feeling stressed, a message encouraging relaxation will also be included.

[1311] Step 12:

[1312] Server: Generates environmental control commands based on analysis results. If the room temperature exceeds the set range, it sends a cooling command to the air conditioner. If the room is too dark, it adjusts the lighting appropriately.

[1313] Step 13:

[1314] Terminal: Receives instructions from the server and controls environmental factors such as air conditioning and lighting. This includes adjusting the air conditioner's temperature setting and changing the brightness of the lights.

[1315] Step 14:

[1316] Server: It works in conjunction with the childcare record application to learn the baby's daily routines (sleep, eating, elimination, etc.). Based on the collected data, it analyzes the baby's normal behavioral patterns.

[1317] Step 15:

[1318] Server: Sends advice and notifications to the user based on learning results. For example, 30 minutes before the usual bedtime, it will notify the user, "It's time for the baby to sleep. Create a quiet environment," but if the user is feeling stressed, it will offer additional advice such as, "Take a deep breath and relax."

[1319] The above outlines the specific processing steps of "Baby Safe Watch." This process allows for the comprehensive management of the baby's safety and comfortable environment, and provides support tailored to the user's emotional state.

[1320] (Example 2)

[1321] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1322] Conventional childcare monitoring systems are limited to monitoring the baby's physical condition and do not take into account the emotional state of the user (parent), resulting in a failure to alleviate the stress and burden of childcare. Furthermore, there are challenges in the accuracy and reliability of systems for promptly responding to detected abnormalities.

[1323] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1324] In this invention, the server includes analysis means for monitoring the baby's safe and comfortable environment, emotion recognition means for recognizing the user's emotions, notification means for detecting anomalies or providing notifications in accordance with the user's emotions, and environment control means for controlling the bedroom environment. This allows for real-time monitoring of the baby's safe and comfortable environment while responding to the user's emotional state, thereby reducing stress and burden from childcare and enabling quick and appropriate responses in emergencies.

[1325] "Video acquisition means" refers to devices or modules for capturing a baby's posture and facial condition in real time.

[1326] A "vibration detection device" is a sensor used to detect the baby's breathing and body movements.

[1327] The "environmental sensor means" refers to a sensor used to measure the temperature and humidity of a bedroom.

[1328] The "illuminance detection means" is a sensor used to measure the brightness of a bedroom.

[1329] "Communication means" refers to devices or interfaces for transmitting data collected from image acquisition means, vibration detection means, environmental sensor means, and illuminance detection means to a server.

[1330] The "analysis means" refers to a function on the server that analyzes acquired data and monitors the baby's safety and comfortable environment.

[1331] "Emotion recognition means" refers to a function that acquires the user's voice and video and analyzes their emotional state.

[1332] A "notification method" is a means of notifying the user of important information or warnings based on analysis results or emotion recognition results.

[1333] The "environmental control means" is a function that issues instructions to automatically adjust the environment of the bedroom, such as temperature, humidity, and illuminance, based on the analysis results.

[1334] The "learning tool" refers to a function that works in conjunction with the childcare record application to learn the individual daily routines of babies.

[1335] An "advice tool" is a means of providing users with timely and appropriate advice based on learned patterns and the user's emotional state.

[1336] The present invention, "a system for monitoring a baby's safe and comfortable environment," includes the following configuration and operation:

[1337] System Configuration

[1338] terminal

[1339] Means of acquiring video:

[1340] A camera module is used to capture the baby's posture and facial expression in real time.

[1341] Specific example: A baby monitor camera captures the movements and facial expressions of a baby lying down.

[1342] Vibration detection means:

[1343] A vibration sensor is used to detect the baby's breathing and is placed under the bed.

[1344] Specific example: Continuously track the up-and-down movement of a baby's chest and collect data.

[1345] Environmental sensor means:

[1346] Temperature and humidity sensors are used to measure the environmental conditions in the bedroom.

[1347] Specific example: When the indoor temperature reaches 28 degrees Celsius, the sensor detects this and provides data.

[1348] Illuminance detection means:

[1349] Use an illuminance sensor to measure the brightness of the bedroom.

[1350] Specific example: When the brightness of the bedroom lighting exceeds a certain level, the data is recorded.

[1351] Means of communication:

[1352] The system transmits data collected from the camera module, vibration sensor, temperature and humidity sensor, and illuminance sensor to the server.

[1353] Specific example: Real-time video and sensor data are transmitted to a server via wireless communication.

[1354] server

[1355] Analysis method:

[1356] The server uses advanced algorithms to analyze the data sent from the terminal.

[1357] Specific example: Analyze data to monitor the baby's posture, facial expression, breathing patterns, and indoor environment.

[1358] Emotion recognition means:

[1359] It acquires audio and video data and recognizes the user's emotions.

[1360] Specific example: While a user is comforting a baby, their facial expressions and voice are analyzed to determine their emotional state.

[1361] Notification method:

[1362] If an anomaly is detected or if it is determined that the user is experiencing stress, an appropriate notification will be sent to the user.

[1363] Specific example: If a baby's breathing stops for 20 seconds, an emergency notification will be sent stating, "An abnormality has occurred in the baby's breathing."

[1364] Environmental control means:

[1365] Based on the analysis results, the system sends instructions to the terminal to automatically control the settings of the air conditioner and lighting in order to maintain the optimal indoor environment.

[1366] Specific example: If the room temperature is determined to be too high, a command is sent to the air conditioner to start cooling.

[1367] Learning methods:

[1368] It connects with a childcare record application to learn the baby's daily routine.

[1369] Specific example: Determine the baby's bedtime pattern, such as going to sleep at 8 PM every night.

[1370] Advice methods:

[1371] Based on learned patterns, it sends timely advice and notifications to the user. Furthermore, it provides advice based on the user's emotional state through emotion recognition.

[1372] Specific example: When it's time for the baby to sleep, the system will notify the user with "It's time for the baby to sleep. Let's create a quiet environment," and if the user is feeling stressed, it will offer additional advice such as "Take a deep breath and relax."

[1373] Example of a prompt

[1374] "Please explain the procedure for a program that sends a notification if a baby cries for an extended period of time."

[1375] "Please explain how to handle situations when users are experiencing high levels of stress."

[1376] This invention makes it possible to ensure a safe and comfortable environment for the baby, as well as reduce the mental burden on the user. The system uses various sensors and advanced analytical means to monitor the baby's and user's condition in real time and provide appropriate responses.

[1377] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1378] Step 1: Data acquisition and transmission (device)

[1379] Specific operation: Each sensor in the terminal (video acquisition means, vibration detection means, environmental sensor means, illuminance detection means) collects its own data. The video acquisition means captures video of the baby, and the vibration detection means monitors the baby's breathing. The environmental sensor means measures room temperature and humidity, and the illuminance detection means measures the brightness of the room.

[1380] Inputs: Baby's posture, facial expression, breathing, room temperature, humidity, and brightness.

[1381] Data processing and calculation: Each sensor collects data in real time and transmits it to the server via communication means.

[1382] Output: Various sensor data transferred to the server

[1383] Step 2: User emotion recognition (device)

[1384] Specific operation: The device's emotion recognition system captures the user's voice and video, acquiring voice tone and facial expression data. This data is sent to a server in real time for emotion analysis.

[1385] Input: User's voice data, video data

[1386] Data processing and calculation: The emotion recognition system collects data to analyze the user's emotions based on an emotion model and sends it to the server.

[1387] Output: Audio and video data sent to the server

[1388] Step 3: Data analysis and anomaly detection (server)

[1389] Specific operation: The server analyzes the received data and uses advanced algorithms to monitor the baby's posture, facial expression, breathing patterns, and room environment. Simultaneously, it analyzes the user's emotional state.

[1390] Input: Sensor data, audio data, and video data transmitted from the device.

[1391] Data processing and calculation: The server analyzes the data using analytical algorithms and detects anomalies. This includes determining whether the baby has stopped breathing or whether the user is in a high-stress state.

[1392] Output: Anomaly detection results, user emotional state analysis results

[1393] Step 4: Notification (Server)

[1394] Specific operation: If a server anomaly is detected, or if a user is experiencing stress, an emergency notification will be sent to the user using a notification system. The notification will arrive on a device such as a smartphone or tablet.

[1395] Input: Anomaly detection results, user emotional state analysis results

[1396] Data processing and calculation: When an anomaly is detected, immediately prepare a notification and send it to the user's device. Customize the message content according to the type of anomaly and the emotional state of the user.

[1397] Output: Notification message to the user

[1398] Step 5: Environment Control (Server)

[1399] Specific operation: Based on the analysis results, the server sends instructions to the terminal to control the settings of the air conditioner and lighting in order to maintain an optimal environment (temperature, humidity, brightness) in the bedroom.

[1400] Input: Baby's condition data, indoor environment data

[1401] Data processing and calculation: The server calculates the settings necessary to maintain a comfortable environment for the baby and sends instructions to the terminal.

[1402] Output: Control instructions for air conditioners and lighting.

[1403] Step 6: Learning and Advice (Server)

[1404] Specific operation: The server interacts with the childcare record application to learn the baby's daily routines (sleep, meals, elimination, etc.). Based on the learned patterns, it sends timely advice and notifications to the user.

[1405] Input: Childcare record data, past sensor data

[1406] Data Processing and Calculation: Analyze data from childcare record applications to analyze the baby's daily routines. Use the analysis results to generate appropriate advice.

[1407] Output: Advice messages, notification messages

[1408] The above outlines the processing flow of this system's program. By collecting and analyzing data in real time, this system provides comprehensive support to enhance the safety of babies and the comfort of users.

[1409] (Application Example 2)

[1410] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1411] Traditional systems for managing the safety and comfort of babies could monitor the baby's condition but failed to consider the user's emotions. Similarly, in physical stores, it was difficult to monitor customer emotions and behavior in real time and implement appropriate responses and environmental controls. This resulted in insufficient improvement in the quality of childcare and customer service, sometimes leading to delays in providing optimal support.

[1412] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an analysis means, a notification means, a control means, an automatic notification means, a customer emotion notification means, and a store environment control means. This enables monitoring of the baby's safety and comfort, as well as responses to customer emotions and automatic control of the store environment.

[1413] A "camera system" is a device used to detect the posture and facial condition of babies or customers in real time.

[1414] A "vibration sensor" is a device used to detect a baby's breathing.

[1415] A "temperature and humidity sensor means" is a sensor for measuring the temperature and humidity in a bedroom or store.

[1416] An "illuminance sensor means" is a sensor used to measure the brightness inside a bedroom or store.

[1417] "Communication means" refers to means for transmitting data from the camera means, vibration sensor means, temperature and humidity sensor means, and illuminance sensor means to the server.

[1418] "Analysis means" refers to means for monitoring the safety and comfort of babies and customers by analyzing the aforementioned data.

[1419] "Notification means" refers to a means for notifying a user or staff member when an abnormality is detected by the analysis means.

[1420] "Control means" refers to means for controlling the environment of a bedroom or store based on the analysis means.

[1421] An "emotion engine" is a means of detecting customer emotions in real time.

[1422] A "customer sentiment notification system" is a means of analyzing customer sentiment data and notifying staff of appropriate responses.

[1423] "Store environment control means" refers to means for automatically adjusting the store environment based on customer sentiment data.

[1424] "Learning methods" refer to techniques for learning individual patterns by linking them with the baby's care records.

[1425] "Advice means" refers to means for providing timely advice and notifications to the user based on the learning means.

[1426] This invention is a system for managing the safety and comfort of babies. Furthermore, the same technology can be used to monitor customer emotions and behavior in real time in physical stores, thereby improving customer service.

[1427] System Configuration

[1428] 1. Terminal

[1429] Camera System: This camera is used to detect the posture and facial condition of babies and customers in real time. The video data acquired from the camera is transmitted to the server described below.

[1430] Vibration sensor: This sensor is used to detect the baby's breathing. The vibration data is transmitted to a server.

[1431] Temperature and humidity sensor means: A sensor for measuring the temperature and humidity in a bedroom or store. This data is also transmitted to the server.

[1432] Illuminance sensor: This is a sensor used to measure the brightness in a bedroom or store. This data is also transmitted to the server.

[1433] Communication method: This is a communication device used to transmit data acquired from the above-mentioned sensors to the server. For example, Wi-Fi modules or Bluetooth are used.

[1434] Emotion Engine: A device that acquires customer voice and video data and detects customer emotions in real time.

[1435] 2. Server

[1436] Analysis method: This method involves analyzing data transmitted from terminals on a server to monitor the safety and comfort of babies and customers. Specifically, it utilizes image analysis and data mining techniques.

[1437] Notification method: This is a means of notifying users or store staff when an anomaly is detected by the analysis method. Notifications are made via devices such as smartphones and tablets.

[1438] Control means: A means for transmitting instructions to a terminal to control the environment of a bedroom or store based on analysis means. For example, it can be used to adjust the temperature of an air conditioner or change the brightness of lighting.

[1439] Customer sentiment notification method: This method analyzes customer sentiment data and notifies staff of appropriate responses. For example, if a customer expresses dissatisfaction, a notification is sent to staff prompting a quick response.

[1440] Store environment control means: This means automatically adjusting the store environment based on customer emotion data. For example, it may change the store temperature or background music.

[1441] 3. User

[1442] User Devices: Users or store staff can receive notifications using devices such as smartphones and tablets, and manage the baby's condition, environment, and customer's emotional state through the application. They can also receive timely advice and notifications provided by the system.

[1443] Specific example

[1444] Common usage examples

[1445] Terminal: The camera detects the baby's posture and sends the data to the server.

[1446] Terminal: The emotion engine analyzes the customer's facial expressions and sends them to the server.

[1447] Server: If the server analyzes the data and determines that everything is normal, it notifies the user or store staff with messages such as "The baby is safe" or "The customer is satisfied."

[1448] Terminal: The temperature and humidity sensor detects when the indoor temperature is too high and sends data to the server.

[1449] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[1450] Example of a prompt

[1451] Application: Design a smart store watch application to enhance the customer experience in physical stores. Using cameras and sensors, monitor customer emotions and behavior in real time, automating all responses and environmental controls. For example, if a customer appears distressed, quickly notify staff and adjust settings such as air conditioning or music to create a more relaxing atmosphere.

[1452] Examples of use in emergencies

[1453] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[1454] Server: The system determines this is an emergency and sends an emergency notification to the parents stating, "There is an abnormality in the baby's breathing."

[1455] User: Receive notifications on your smartphone and respond quickly.

[1456] Server: Simultaneously analyzes the user's emotions and, if it determines that the user is experiencing stress, sends a message saying, "Please calm down. We will address this immediately."

[1457] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1458] Step 1:

[1459] The device uses a camera to detect the posture and facial features of babies and customers in real time and acquire video data. The input is camera footage, and the output is the detected face and posture data. Specifically, it uses OpenCV and the dlib library to capture video and execute a face detection algorithm.

[1460] Step 2:

[1461] The device uses a vibration sensor to detect the baby's breathing. The input is vibration data from the sensor, and the output is data indicating the presence of breathing. The sensor data is acquired at regular intervals and processed into data for analyzing the breathing pattern.

[1462] Step 3:

[1463] The terminal uses temperature and humidity sensors to measure and acquire data on the temperature and humidity in a bedroom or store. The input is the reading from the temperature and humidity sensors, and the output is accurate temperature and humidity data for the room. This data is used as foundational data for analysis and control described later.

[1464] Step 4:

[1465] The terminal uses an illuminance sensor to measure the brightness in a bedroom or store and acquire data. The input is the illuminance sensor reading, and the output is the room's illuminance data. The measured data is sent to the server.

[1466] Step 5:

[1467] The terminal uses communication methods to transmit data acquired from the camera, vibration sensor, temperature and humidity sensor, and illuminance sensor to the server. The input is the data acquired from each sensor, and the output is the data transferred to the server. Wi-Fi, Bluetooth, and other technologies are used for communication.

[1468] Step 6:

[1469] The server uses analysis tools to analyze the transmitted data and monitor the safety and comfort of babies and customers. The input is sensor data transmitted from the terminal, and the output is the analysis results. Specifically, it uses image analysis algorithms and machine learning models to detect various anomaly patterns.

[1470] Step 7:

[1471] The server uses a notification system to notify users or store staff if it detects an anomaly in the analysis process. The input is the analysis result, and the output is a notification message. This notification is sent via a push notification system to smartphones and tablets.

[1472] Step 8:

[1473] The server uses control means to send instructions to the terminal to control the environment of a bedroom or store based on analysis means. The input is the analysis result, and the output is the environment control instruction. For example, it may issue instructions to adjust the set temperature of an air conditioner or the brightness of lighting.

[1474] Step 9:

[1475] The terminal uses an emotion engine to acquire customer voice and video data and detect the customer's emotions. The input is voice and video data, and the output is analyzed emotion data. An emotion recognition model is used to detect changes in emotion, and this data is sent to the server.

[1476] Step 10:

[1477] The server uses a customer sentiment notification system to analyze customer sentiment data and notify staff of appropriate actions. The input is sentiment data sent from the terminal, and the output is a notification to staff. The notification content changes according to the customer's status, prompting a quick response for dissatisfied customers.

[1478] Step 11:

[1479] The server uses store environment control means to automatically adjust the store environment based on customer emotion data. The input is analyzed emotion data, and the output is environment control instructions. For example, it adjusts the store temperature and background music.

[1480] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1481] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1482] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1483] [Fourth Embodiment]

[1484] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1485] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1486] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1487] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1488] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1489] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1490] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1491] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1492] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1493] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1494] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1495] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1496] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1497] A specific embodiment of the "Baby Safe Watch" of the present invention is described below. This system consists of multiple terminals, a server, and a user (parent), each with its own role.

[1498] System Configuration

[1499] 1. Terminal

[1500] Camera equipment: A camera is installed to detect the baby's posture and facial condition in real time.

[1501] Vibration sensor means: A vibration sensor is installed in the bed to detect the baby's breathing.

[1502] Temperature and humidity sensor means: Install a sensor to measure the temperature and humidity in the bedroom.

[1503] Illuminance sensor means: Install a sensor to measure the brightness of the bedroom.

[1504] Communication means: Equipped with a communication device for transmitting data acquired from the above sensors to a server.

[1505] 2. Server

[1506] Analysis method: The server receives data transmitted from the terminal and performs analysis using advanced algorithms.

[1507] Notification mechanism: The system includes a mechanism for notifying the user according to the analysis results. If an anomaly is detected, an emergency notification will be sent.

[1508] Control mechanism: Based on the analysis results, it sends instructions to the terminal to automatically control environmental factors such as temperature, humidity, and illuminance.

[1509] Learning method: It works in conjunction with a childcare record application to learn the individual patterns of each baby.

[1510] Advice method: Based on learned patterns, it sends timely advice and notifications to the user.

[1511] 3. User

[1512] Using devices such as smartphones and tablets, users can receive notifications and manage their baby's condition and environment through the application.

[1513] Explanation of the program's processing

[1514] Data acquisition and transmission (terminal)

[1515] Terminal: The camera module captures images of the baby and sends them to the server for real-time analysis.

[1516] Terminal: The vibration sensor detects the baby's breathing in real time and sends the data to the server.

[1517] Terminal: Temperature, humidity, and illuminance sensors also measure environmental data and transmit it to the server at regular intervals.

[1518] Data analysis and anomaly detection (server)

[1519] Server: Analyzes received data and monitors the baby's posture, facial expression, breathing patterns, and room environment.

[1520] Server: If an anomaly is detected as a result of the analysis, an emergency notification will be sent to the user's smartphone.

[1521] For example, if the baby turns onto its stomach, an alert is sent immediately.

[1522] An emergency notification will also be issued if breathing stops for 20 seconds or more.

[1523] Environmental control (server)

[1524] Server: Based on the analysis results, it sends instructions to terminals to automatically control air conditioning and lighting in order to maintain the optimal indoor environment.

[1525] If the room temperature exceeds the set range, the server sends a cooling command to the air conditioner.

[1526] If the room is too dark, adjust the lighting appropriately.

[1527] Learning and advice (server)

[1528] Server: It connects with childcare records and learns the baby's daily routines (sleep, eating, elimination, etc.).

[1529] Server: Based on learned data, it sends timely advice and notifications to the user.

[1530] For example, if the system learns that the baby usually goes to bed at 8 PM, it will send a notification 30 minutes beforehand saying, "It's time for the baby to sleep. Please create a quiet environment."

[1531] Specific example

[1532] Common usage examples

[1533] Terminal: The camera detects the baby's posture and sends the data to the server.

[1534] Server: If the analysis determines that everything is normal, it notifies the user that "the baby is safe."

[1535] Terminal: The temperature and humidity sensor detects when the indoor temperature is too high and sends data to the server.

[1536] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[1537] Examples of use in emergencies

[1538] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[1539] Server: The system determines this is an emergency and sends an emergency notification to the parents stating, "There is an abnormality in the baby's breathing."

[1540] User: Receive notifications on your smartphone and respond quickly.

[1541] This concludes the description of the embodiment of the "Baby Safe Watch" of the present invention. By using this system, parents can manage the safety and comfortable childcare environment of their baby in one place, thereby reducing the burden of childcare.

[1542] The following describes the processing flow.

[1543] Step 1:

[1544] Terminal: The camera module acquires real-time video of the baby. It captures 30 frames per second of video and temporarily stores it in local storage.

[1545] Step 2:

[1546] Terminal: Transmits the acquired video to the server via a communication method. The data is transmitted using a secure protocol (e.g., HTTPS).

[1547] Step 3:

[1548] Device: A vibration sensor detects the baby's breathing. The vibration sensor collects data every second and sends it to the server.

[1549] Step 4:

[1550] Terminal: A temperature and humidity sensor measures the temperature and humidity in the bedroom. It acquires data every 10 seconds and sends it to the server.

[1551] Step 5:

[1552] Terminal: An illuminance sensor measures the brightness of the bedroom. It acquires data every 5 seconds and sends it to the server.

[1553] Step 6:

[1554] Server: Analyzes all received data. Analyzes camera footage to recognize the baby's posture and facial expression. Analyzes vibration sensor data to detect breathing patterns. Analyzes temperature, humidity, and illuminance data to evaluate the indoor environment.

[1555] Step 7:

[1556] Server: Detects anomalies based on analysis results. Determines an emergency situation, such as when the baby is lying face down or has stopped breathing.

[1557] Step 8:

[1558] Server: If an anomaly is detected, an emergency notification will be sent to the user. The notification will be sent to the smartphone in the form of a push notification or email.

[1559] Step 9:

[1560] Server: Generates environmental control commands based on the analysis results. For example, if the room temperature exceeds the set range, it sends a cooling command to the air conditioner.

[1561] Step 10:

[1562] Terminal: Receives instructions from the server and controls environmental factors such as air conditioning and lighting. This includes adjusting the air conditioner's temperature setting and changing the brightness of the lights.

[1563] Step 11:

[1564] Server: It works in conjunction with the childcare record application to learn the baby's daily routine. Based on the collected data, it analyzes the baby's sleep, eating, and elimination patterns.

[1565] Step 12:

[1566] Server: Sends advice and notifications to the user based on the learning results. For example, 30 minutes before the usual bedtime, it might notify the user, "It's time for the baby to sleep. Please create a quiet environment."

[1567] The above outlines the specific processing steps for "Baby Safe Watch." This process allows for comprehensive management of your baby's safety and comfortable environment.

[1568] (Example 1)

[1569] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1570] In modern childcare, parents need to constantly monitor their baby's safety, but 24-hour monitoring is physically and psychologically burdensome. Optimizing the indoor environment is also crucial for maintaining the baby's safety and comfort, but manual adjustments are time-consuming. Furthermore, the difficulty in obtaining appropriate advice based on the baby's daily routines leads to decreased efficiency in childcare.

[1571] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1572] In this invention, the server includes a camera means for detecting the baby's posture and facial condition in real time, a vibration sensor means for detecting the baby's breathing, an environmental sensor means for measuring the temperature and humidity of the bedroom, an illuminance sensor means for measuring the brightness of the bedroom, a communication means for transmitting data from the camera means, vibration sensor means, environmental sensor means and illuminance sensor means to the server, an analysis means in the server for monitoring the baby's safe and comfortable environment by analyzing the data, a notification means for notifying the user when the analysis means detects an abnormality, a control means for controlling the bedroom environment based on the analysis means, a learning means for learning individual patterns in conjunction with childcare records, and an advice means for providing timely advice and notifications based on the learned patterns. This makes it possible to automatically and efficiently maintain the baby's safety and comfort and reduce the burden of childcare on parents.

[1573] A "camera system" is a device used to detect the baby's posture and facial condition in real time.

[1574] A "vibration sensor" is a device used to detect a baby's breathing.

[1575] An "environmental sensor" is a device used to measure the temperature and humidity of a bedroom.

[1576] An "illuminance sensor" is a device used to measure the brightness of a bedroom.

[1577] "Communication means" refers to a device for transmitting data acquired from camera means, vibration sensor means, environmental sensor means, and illuminance sensor means to a server.

[1578] "Analysis means" refers to a device that analyzes data transmitted via communication means to monitor the safety and comfort of the baby.

[1579] A "notification means" is a device used to notify the user when an anomaly is detected in the analysis means.

[1580] The "control means" is a device for automatically controlling the bedroom environment based on the analysis means.

[1581] A "learning tool" is a device that works in conjunction with childcare records to learn individual patterns.

[1582] An "advice tool" is a device that provides timely advice and notifications based on learned patterns.

[1583] A specific embodiment of the "Baby Safe Watch" of the present invention is described below. This system consists of multiple terminals, a server, and a user (parent), each with its own role.

[1584] System Configuration

[1585] 1. Terminal

[1586] Camera equipment: A camera is installed to detect the baby's posture and facial condition in real time. For example, it can be used to check if the baby is rolling over or if their face is covered by the blanket.

[1587] Vibration sensor means: A vibration sensor is installed in the bed to detect the baby's breathing. For example, the breathing pattern is obtained from the movement of the baby's chest.

[1588] Environmental sensor means: Install sensors to measure the temperature and humidity in the bedroom. For example, obtain data such as a room temperature of 25 degrees Celsius and a humidity of 50%.

[1589] Illuminance sensor means: Install a sensor to measure the brightness of the bedroom. For example, confirm that the brightness of the room is 400 lux.

[1590] Communication means: The system is equipped with a communication device for transmitting data acquired from the above-mentioned sensors to a server. For example, a Wi-Fi module is used to transmit the data.

[1591] 2. Server

[1592] Analysis method: The server receives data transmitted from the terminal and performs analysis using advanced algorithms. For example, it analyzes the baby's posture, breathing status, and the indoor environment.

[1593] Notification method: The system includes a means to notify the user according to the analysis results. If an abnormality is detected, an emergency notification will be sent. For example, a notification such as "An abnormality has been detected in the baby's breathing" will be sent to the user.

[1594] Control mechanism: Based on the analysis results, it sends instructions to the terminal to automatically control environmental conditions such as temperature, humidity, and illuminance. For example, if the room temperature exceeds the set range, it sends an instruction to the air conditioner to start cooling.

[1595] Learning method: It works in conjunction with a childcare record application to learn the individual patterns of the baby. For example, it learns the baby's daily patterns such as sleep, eating, and elimination.

[1596] Advice methods: Based on learned patterns, the system sends timely advice and notifications to the user. For example, for a baby who has a habit of going to bed at 8 pm, it might send a notification 30 minutes beforehand saying, "It's time for your baby to sleep. Please create a quiet environment."

[1597] 3. User

[1598] Users can receive notifications from the server using devices such as smartphones and tablets, and manage their baby's condition and environment through the application. For example, a user who receives an emergency notification can quickly rush to the baby's bed and check on their baby's condition.

[1599] Specific example

[1600] Common usage examples

[1601] Terminal: The camera detects the baby's posture and sends the data to the server.

[1602] Server: If the analysis determines that everything is normal, it notifies the user that "the baby is safe."

[1603] Terminal: The environmental sensor detects that the indoor temperature is too high and sends data to the server.

[1604] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[1605] Examples of use in emergencies

[1606] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[1607] Server: Determined to be in an emergency, it sends an emergency notification to the user stating, "There is an abnormality in the baby's breathing."

[1608] User: Receives a notification on their smartphone, rushes to the bed immediately, and checks on the baby's condition.

[1609] Example of a prompt

[1610] Example prompt:

[1611] BabySafeWatch is a system that monitors and controls a baby's posture, breathing, and indoor environment. What kind of data does it typically collect, and how does it notify the user?

[1612] How will users be notified in an emergency, and how will they respond?

[1613] This concludes the description of the embodiment of the "Baby Safe Watch" of the present invention. By using this system, parents can manage the safety and comfortable childcare environment of their baby in one place, thereby reducing the burden of childcare.

[1614] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1615] Step 1: Data Acquisition (Device)

[1616] Input: Information about the baby's posture and facial expression, breathing, room temperature, humidity, and light level.

[1617] Specific operation: The camera on the device captures images of the baby. A vibration sensor detects the baby's breathing, an environmental sensor measures the room temperature and humidity, and an illuminance sensor measures the room brightness.

[1618] Output: Acquired video data, respiration data, and environmental data (temperature, humidity, illuminance) are generated.

[1619] Step 2: Data transmission (terminal)

[1620] Input: Data acquired from each sensor (video data, respiratory data, environmental data)

[1621] Specific operation: Data acquired by sensors and cameras is transmitted to a server using communication means. A Wi-Fi module is used as the specific means.

[1622] Output: Data acquired by sensors and cameras is sent to the server.

[1623] Step 3: Data Analysis (Server)

[1624] Input: Data from each sensor sent to the server (video data, respiration data, environmental data)

[1625] Specific operation: The server's analysis system receives the data and uses advanced algorithms to analyze the baby's posture, breathing, and room environment. For example, it analyzes whether the baby's posture is normal and whether the breathing pattern is abnormal.

[1626] Output: Analysis results (determination of normal or abnormal, indoor environment status)

[1627] Step 4: Anomaly detection and notification (server)

[1628] Input: Results of data analysis

[1629] Specific operation: If an anomaly is detected by the server's analysis system, the notification system will inform the user. Specifically, an emergency notification will be sent if the baby is lying face down or if breathing stops for more than 20 seconds.

[1630] Output: An emergency notification is sent to the user.

[1631] Step 5: Environment Control (Server)

[1632] Input: Results of data analysis

[1633] Specific operation: Based on the analysis results, the server's control system sends instructions to the terminal to automatically adjust environmental conditions such as temperature, humidity, and illuminance. For example, if the room temperature exceeds the set range, it sends a cooling command to the air conditioner.

[1634] Output: An instruction for environment adjustment is sent to the terminal.

[1635] Step 6: Data Training (Server)

[1636] Input: Daily life pattern data from a childcare record application.

[1637] Specific operation: The server's learning mechanism works in conjunction with the childcare record application to learn the individual patterns of the baby. For example, it analyzes and learns data such as the baby's sleep, meals, and bowel movements.

[1638] Output: Learning results (insights about babies' daily routines)

[1639] Step 7: Advice and Notification (Server)

[1640] Input: Learning results and data analysis results

[1641] Specific operation: Based on patterns learned by the server's advice system, timely advice and notifications are sent to the user. For example, for a baby who has a habit of going to bed at 8 pm, a notification is sent 30 minutes beforehand saying, "It's time for your baby to sleep. Please create a quiet environment."

[1642] Output: Appropriate advice and notifications are sent to the user.

[1643] (Application Example 1)

[1644] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1645] Ensuring the safety of babies and children and reducing anxiety for parents and guardians while they engage in childcare or shopping activities is crucial. However, current systems lack the functionality to monitor children's safety in stores in real time, in addition to monitoring the baby's environment, and to quickly notify parents and staff in case of abnormalities. Therefore, there is a need to develop a new system that allows parents and guardians to watch over their children with greater peace of mind.

[1646] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1647] In this invention, the server includes an image acquisition means for detecting the baby's posture and facial condition in real time, a vibration detection means for detecting the baby's breathing, a climate data acquisition means for measuring the temperature and humidity of the bedroom, a light intensity sensor means for measuring the brightness of the bedroom, a communication device for transmitting data from the image acquisition means, vibration detection means, climate data acquisition means and light intensity sensor means to a central processing unit, an analysis means in the central processing unit for monitoring the baby's safety and comfort by analyzing the data, a notification means for notifying the user when an abnormality is detected by the analysis means, an environment control means for controlling the bedroom environment based on the analysis means, a means for monitoring the safety of children in the store in real time, and a means for notifying parents or staff when a child enters a specific area. This ensures a safe and comfortable environment for babies, and further enables real-time monitoring of children's safety in the store and a rapid response in the event of an abnormality.

[1648] The "image acquisition means" is a device for detecting the baby's posture and facial condition in real time.

[1649] A "vibration detection device" is a device for detecting a baby's breathing in real time.

[1650] "Climate data acquisition means" refers to a device for measuring the temperature and humidity of a bedroom.

[1651] A "light intensity sensor" is a device used to measure the brightness of a bedroom.

[1652] A "communication device" is a device used to transmit data acquired from various sensors to a central processing unit.

[1653] A "central processing unit" is a device that analyzes received data and monitors the safety and comfort of babies and children.

[1654] An "analysis tool" is a device used to analyze data and detect anomalies.

[1655] A "notification means" is a device that notifies the user when an abnormality is detected in the analysis means.

[1656] An "environmental control means" is a device that controls the bedroom environment based on an analysis means.

[1657] "A means of monitoring the safety of children in stores in real time" refers to a device that monitors the safety of children in stores in real time and notifies the staff if any abnormalities are detected.

[1658] "Means of notifying parents or staff" refers to a device that notifies parents or store staff when a child enters a specific area.

[1659] A specific embodiment of the present invention, "Kids Safe Watch," is described below. The system consists of multiple terminals, a central processing unit, and users, each with its own role.

[1660] System Configuration

[1661] 1. Terminal

[1662] Image acquisition method: Install a camera to detect the child's posture and facial condition in real time.

[1663] Vibration detection method: A vibration sensor is installed on the floor to detect the breathing and position of the baby or child.

[1664] Climate data acquisition method: Install sensors to measure the temperature and humidity in the store.

[1665] Light intensity sensor means: Install a sensor to measure the brightness of the store.

[1666] Communication device: Equipped with a communication device for transmitting data acquired from the above-mentioned sensors to the central processing unit.

[1667] 2. Central Processing Unit

[1668] Analysis method: The central processing unit receives data transmitted from the terminal and performs analysis using advanced algorithms.

[1669] Notification method: The system includes a means to notify the user according to the analysis results. If an anomaly is detected, an emergency notification will be sent.

[1670] Environmental control means: Based on the analysis results, instructions are sent to the terminal to automatically control environmental conditions such as temperature, humidity, and illuminance.

[1671] 3. User

[1672] Using devices such as smartphones and tablets, users can receive notifications and manage their child's status and environment through the application.

[1673] Users can monitor the location and behavior of children within the store and respond quickly as needed.

[1674] Explanation of the program's processing

[1675] Hardware and software to use

[1676] Hardware: Smartphones, tablets, camera modules installed in stores, vibration sensors, temperature and humidity sensors, and illuminance sensors.

[1677] Software: Python (for data analysis and notification implementation), AWS cloud services (for data storage and processing), Firebase (real-time database and notification service), OpenCV (image analysis library).

[1678] Data acquisition and calculation processing

[1679] 1. Processing camera data

[1680] Using OpenCV, we detect and analyze children's posture and position from camera images.

[1681] The analysis data is sent to AWS in real time.

[1682] 2. Processing of sensor data

[1683] The system periodically acquires data from vibration sensors, temperature and humidity sensors, and illuminance sensors and sends it to AWS.

[1684] 3. Notification function

[1685] Using Firebase, users receive real-time notifications on their smartphones when an anomaly is detected.

[1686] 4. Environmental control

[1687] Data analysis is performed on AWS, and instructions for controlling air conditioners and lighting are sent to the terminal as needed.

[1688] Specific usage examples

[1689] Common usage examples

[1690] Terminal: The camera detects the posture and position of children in the store and transmits the data to the central processing unit.

[1691] Central Processing Unit: If it analyzes the data and determines that it is normal, it notifies the user that "the child is safe."

[1692] Terminal: The temperature and humidity sensor detects if the temperature inside the store is too high and sends data to the central processing unit.

[1693] Central Processing Unit: Sends cooling instructions to the air conditioner and adjusts it to the appropriate temperature.

[1694] Examples of use in emergencies

[1695] Terminal: If the vibration sensor detects that a child has left a designated area, it transmits that data to the central processing unit.

[1696] Central Processing Unit: Determines an emergency and sends an emergency notification to parents or store staff stating, "A child has left a designated area."

[1697] User: Receive notifications on your smartphone and respond quickly.

[1698] Examples of prompts for generative AI models

[1699] Please generate examples of how "Kids Safe Watch" can be used. Describe the details of a system that uses in-store cameras and sensors to manage child safety and notifies parents and store staff in real time if any anomalies are detected. Include specific use cases, such as notifications for when a child leaves the play area or when a child remains in the store after closing time.

[1700] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1701] Step 1:

[1702] Terminal: A camera module is used to detect the child's posture and position. The camera module acquires image data in real time from a designated area within the store. This image data is used as input data. The acquired image data is sent to the central processing unit using the OpenCV library for data preprocessing.

[1703] Step 2:

[1704] Server: The central processing unit receives pre-processed image data. It then uses OpenCV to execute face recognition and posture detection algorithms to extract the child's posture and location information. The extracted posture and location information is generated as output data, and the process proceeds to the next analysis step.

[1705] Step 3:

[1706] Terminal: Uses vibration sensors to acquire vibration data from children's floors and play areas installed in specific areas within the store. The vibration sensors transmit the detected vibration data to the central processing unit in real time. The vibration data is used as input data.

[1707] Step 4:

[1708] Server: The central processing unit receives data from vibration sensors and analyzes it using a rational threshold determination algorithm. Based on the analysis of the vibration data, it determines whether the child has deviated from a specific area. The determination result is generated as output data, and if an anomaly is detected, the process proceeds to the notification step.

[1709] Step 5:

[1710] Terminal: Using a climate data acquisition device, temperature and humidity data inside the store are acquired periodically. This data is used as input data and transmitted to the central processing unit in real time.

[1711] Step 6:

[1712] Server: The central processing unit receives temperature and humidity data transmitted from the climate data acquisition means and performs analysis to determine if it is within an appropriate range. Based on the analysis results, it determines whether the optimal temperature is being maintained. If the temperature deviates from the optimal range, an instruction to adjust using the environmental control means is output.

[1713] Step 7:

[1714] Terminal: Uses a light intensity sensor to acquire brightness (illuminance) data within the store. The brightness data is used as input data and transmitted to the central processing unit in real time.

[1715] Step 8:

[1716] Server: The central processing unit receives data from the light intensity sensor and performs analysis to determine whether the illuminance is within an appropriate range. Based on the analysis results, it determines whether the lighting is maintained within an appropriate range. If the illuminance deviates from the appropriate range, an instruction is output to adjust using the environmental control means.

[1717] Step 9:

[1718] Server: The server comprehensively analyzes all data acquired to date (image data, vibration data, temperature and humidity data, light intensity data) and, if an anomaly is detected, uses a notification system to notify users and store staff in real time. The notification content is generated and sent as output data.

[1719] Step 10:

[1720] Users: Customers or store staff receive notifications on their smartphones or tablets and take prompt action based on the displayed notification content. They review the notification and initiate specific response actions.

[1721] The above outlines the specific processing steps of the system program that implements the application example.

[1722] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1723] The "Baby Safe Watch" system is a comprehensive system for managing a baby's safety and a comfortable childcare environment. It incorporates an emotion engine that recognizes the user's emotions and adjusts notifications and support accordingly. This system consists of multiple terminals, a server, and the user (parent), each with its own specific role.

[1724] System Configuration

[1725] 1. Terminal

[1726] Camera equipment: A camera is installed to detect the baby's posture and facial condition in real time.

[1727] Vibration sensor means: A vibration sensor is installed in the bed to detect the baby's breathing.

[1728] Temperature and humidity sensor means: Install a sensor to measure the temperature and humidity in the bedroom.

[1729] Illuminance sensor means: Install a sensor to measure the brightness of the bedroom.

[1730] Communication means: Equipped with a communication device for transmitting data acquired from the above sensors to a server.

[1731] Emotion Engine Method: Acquires audio and video data to recognize the user's emotions.

[1732] 2. Server

[1733] Analysis method: The server receives data transmitted from the terminal and performs analysis using advanced algorithms.

[1734] Notification mechanism: The system includes a mechanism for notifying the user according to the analysis results. If an anomaly is detected, an emergency notification will be sent.

[1735] Control mechanism: Based on the analysis results, it sends instructions to the terminal to automatically control environmental factors such as temperature, humidity, and illuminance.

[1736] Learning method: It works in conjunction with a childcare record application to learn the individual patterns of each baby.

[1737] Advice method: Based on learned patterns, it sends timely advice and notifications to the user.

[1738] 3. User

[1739] Using devices such as smartphones and tablets, users can receive notifications and manage their baby's condition and environment through the application.

[1740] Explanation of the program's processing

[1741] Data acquisition and transmission (terminal)

[1742] Terminal: The camera module captures images of the baby and sends them to the server for real-time analysis.

[1743] Terminal: The vibration sensor detects the baby's breathing in real time and sends the data to the server.

[1744] Terminal: Temperature, humidity, and illuminance sensors also measure environmental data and transmit it to the server at regular intervals.

[1745] User emotion recognition (device)

[1746] Terminal: The emotion engine acquires the user's voice and video data.

[1747] Terminal: Sends acquired emotion data to the server.

[1748] Data analysis and anomaly detection (server)

[1749] Server: Analyzes received data and monitors the baby's posture, facial expression, breathing patterns, and room environment.

[1750] Server: Further analyzes the user's emotional data obtained by the emotion engine.

[1751] Server: Sends appropriate notifications to the user if an abnormality is detected in the baby or if the user is determined to be experiencing stress.

[1752] Environmental control (server)

[1753] Server: Based on the analysis results, it sends instructions to terminals to automatically control air conditioning and lighting in order to maintain the optimal indoor environment.

[1754] Terminal: Adjust the air conditioner temperature setting or change the brightness of the lights.

[1755] Learning and advice (server)

[1756] Server: Works in conjunction with the childcare record application to learn the baby's daily routine (sleep, meals, elimination, etc.).

[1757] Server: Based on learned data, it sends timely advice and notifications to the user. Furthermore, it adjusts the content of the advice based on the user's emotional state.

[1758] For example, if the system learns that the baby usually goes to bed at 8 PM, it will notify the user 30 minutes beforehand with the message, "It's time for your baby to sleep. Please create a quiet environment." However, if the user is feeling stressed, it will offer additional advice such as, "Take a deep breath and relax."

[1759] Specific example

[1760] Common usage examples

[1761] Terminal: The camera detects the baby's posture and sends the data to the server.

[1762] Terminal: The emotion engine analyzes the user's facial expressions and sends them to the server.

[1763] Server: If the analysis determines that everything is normal, it notifies the user that "the baby is safe."

[1764] Terminal: The temperature and humidity sensor detects when the indoor temperature is too high and sends data to the server.

[1765] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[1766] Examples of use in emergencies

[1767] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[1768] Server: The system determines this is an emergency and sends an emergency notification to the parents stating, "There is an abnormality in the baby's breathing."

[1769] User: Receive notifications on your smartphone and respond quickly.

[1770] Server: Simultaneously analyzes the user's emotions and, if it determines that the user is experiencing stress, sends a message saying, "Please calm down. We will address this immediately."

[1771] This concludes our explanation of the new implementation, "BabySafeWatch." By combining it with an emotion engine, it is possible to further reduce the burden on the user and provide a higher level of reassurance.

[1772] The following describes the processing flow.

[1773] Step 1:

[1774] Terminal: The camera module captures video of the baby in real time. The video is captured at 30 frames per second and temporarily stored in local storage.

[1775] Step 2:

[1776] Terminal: Sends acquired video data to the server via a communication method. A secure protocol (e.g., HTTPS) is used for transmission.

[1777] Step 3:

[1778] Device: A vibration sensor detects the baby's breathing. It collects data every second and sends it to the server.

[1779] Step 4:

[1780] Terminal: A temperature and humidity sensor measures the temperature and humidity in the bedroom. It acquires data every 10 seconds and sends it to the server.

[1781] Step 5:

[1782] Terminal: An illuminance sensor measures the brightness of the bedroom. It acquires data every 5 seconds and sends it to the server.

[1783] Step 6:

[1784] Terminal: The emotion engine acquires the user's voice and video data. The emotion engine captures the data in real time and temporarily stores it in local storage.

[1785] Step 7:

[1786] Terminal: Sends acquired emotional data to the server via communication means. The emotional data is also transmitted using a secure protocol.

[1787] Step 8:

[1788] Server: Analyzes all received data. It analyzes the baby's posture and facial condition from camera footage and detects breathing patterns from vibration sensor data. It analyzes temperature, humidity, and illuminance data to evaluate the indoor environment.

[1789] Step 9:

[1790] Server: Analyzes user emotion data acquired by the emotion engine. Evaluates the user's emotional state based on voice tone and facial expression data.

[1791] Step 10:

[1792] Server: Detects anomalies based on analysis results. Determines an emergency situation, such as when the baby is lying face down, breathing has stopped, or the user is experiencing high levels of stress.

[1793] Step 11:

[1794] Server: If an anomaly is detected, an emergency notification will be sent to the user. The notification will be sent to the smartphone via push notification or email. If the user is feeling stressed, a message encouraging relaxation will also be included.

[1795] Step 12:

[1796] Server: Generates environmental control commands based on analysis results. If the room temperature exceeds the set range, it sends a cooling command to the air conditioner. If the room is too dark, it adjusts the lighting appropriately.

[1797] Step 13:

[1798] Terminal: Receives instructions from the server and controls environmental factors such as air conditioning and lighting. This includes adjusting the air conditioner's temperature setting and changing the brightness of the lights.

[1799] Step 14:

[1800] Server: It works in conjunction with the childcare record application to learn the baby's daily routines (sleep, eating, elimination, etc.). Based on the collected data, it analyzes the baby's normal behavioral patterns.

[1801] Step 15:

[1802] Server: Sends advice and notifications to the user based on learning results. For example, 30 minutes before the usual bedtime, it will notify the user, "It's time for the baby to sleep. Create a quiet environment," but if the user is feeling stressed, it will offer additional advice such as, "Take a deep breath and relax."

[1803] The above outlines the specific processing steps of "Baby Safe Watch." This process allows for the comprehensive management of the baby's safety and comfortable environment, and provides support tailored to the user's emotional state.

[1804] (Example 2)

[1805] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1806] Conventional childcare monitoring systems are limited to monitoring the baby's physical condition and do not take into account the emotional state of the user (parent), resulting in a failure to alleviate the stress and burden of childcare. Furthermore, there are challenges in the accuracy and reliability of systems for promptly responding to detected abnormalities.

[1807] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1808] In this invention, the server includes analysis means for monitoring the baby's safe and comfortable environment, emotion recognition means for recognizing the user's emotions, notification means for detecting anomalies or providing notifications in accordance with the user's emotions, and environment control means for controlling the bedroom environment. This allows for real-time monitoring of the baby's safe and comfortable environment while responding to the user's emotional state, thereby reducing stress and burden from childcare and enabling quick and appropriate responses in emergencies.

[1809] "Video acquisition means" refers to devices or modules for capturing a baby's posture and facial condition in real time.

[1810] A "vibration detection device" is a sensor used to detect the baby's breathing and body movements.

[1811] The "environmental sensor means" refers to a sensor used to measure the temperature and humidity of a bedroom.

[1812] The "illuminance detection means" is a sensor used to measure the brightness of a bedroom.

[1813] "Communication means" refers to devices or interfaces for transmitting data collected from image acquisition means, vibration detection means, environmental sensor means, and illuminance detection means to a server.

[1814] The "analysis means" refers to a function on the server that analyzes acquired data and monitors the baby's safety and comfortable environment.

[1815] "Emotion recognition means" refers to a function that acquires the user's voice and video and analyzes their emotional state.

[1816] A "notification method" is a means of notifying the user of important information or warnings based on analysis results or emotion recognition results.

[1817] The "environmental control means" is a function that issues instructions to automatically adjust the environment of the bedroom, such as temperature, humidity, and illuminance, based on the analysis results.

[1818] The "learning tool" refers to a function that works in conjunction with the childcare record application to learn the individual daily routines of babies.

[1819] An "advice tool" is a means of providing users with timely and appropriate advice based on learned patterns and the user's emotional state.

[1820] The present invention, "a system for monitoring a baby's safe and comfortable environment," includes the following configuration and operation:

[1821] System Configuration

[1822] terminal

[1823] Means of acquiring video:

[1824] A camera module is used to capture the baby's posture and facial expression in real time.

[1825] Specific example: A baby monitor camera captures the movements and facial expressions of a baby lying down.

[1826] Vibration detection means:

[1827] A vibration sensor is used to detect the baby's breathing and is placed under the bed.

[1828] Specific example: Continuously track the up-and-down movement of a baby's chest and collect data.

[1829] Environmental sensor means:

[1830] Temperature and humidity sensors are used to measure the environmental conditions in the bedroom.

[1831] Specific example: When the indoor temperature reaches 28 degrees Celsius, the sensor detects this and provides data.

[1832] Illuminance detection means:

[1833] Use an illuminance sensor to measure the brightness of the bedroom.

[1834] Specific example: When the brightness of the bedroom lighting exceeds a certain level, the data is recorded.

[1835] Means of communication:

[1836] The system transmits data collected from the camera module, vibration sensor, temperature and humidity sensor, and illuminance sensor to the server.

[1837] Specific example: Real-time video and sensor data are transmitted to a server via wireless communication.

[1838] server

[1839] Analysis method:

[1840] The server uses advanced algorithms to analyze the data sent from the terminal.

[1841] Specific example: Analyze data to monitor the baby's posture, facial expression, breathing patterns, and indoor environment.

[1842] Emotion recognition means:

[1843] It acquires audio and video data and recognizes the user's emotions.

[1844] Specific example: While a user is comforting a baby, their facial expressions and voice are analyzed to determine their emotional state.

[1845] Notification method:

[1846] If an anomaly is detected or if it is determined that the user is experiencing stress, an appropriate notification will be sent to the user.

[1847] Specific example: If a baby's breathing stops for 20 seconds, an emergency notification will be sent stating, "An abnormality has occurred in the baby's breathing."

[1848] Environmental control means:

[1849] Based on the analysis results, the system sends instructions to the terminal to automatically control the settings of the air conditioner and lighting in order to maintain the optimal indoor environment.

[1850] Specific example: If the room temperature is determined to be too high, a command is sent to the air conditioner to start cooling.

[1851] Learning methods:

[1852] It connects with a childcare record application to learn the baby's daily routine.

[1853] Specific example: Determine the baby's bedtime pattern, such as going to sleep at 8 PM every night.

[1854] Advice methods:

[1855] Based on learned patterns, it sends timely advice and notifications to the user. Furthermore, it provides advice based on the user's emotional state through emotion recognition.

[1856] Specific example: When it's time for the baby to sleep, the system will notify the user with "It's time for the baby to sleep. Let's create a quiet environment," and if the user is feeling stressed, it will offer additional advice such as "Take a deep breath and relax."

[1857] Example of a prompt

[1858] "Please explain the procedure for a program that sends a notification if a baby cries for an extended period of time."

[1859] "Please explain how to handle situations when users are experiencing high levels of stress."

[1860] This invention makes it possible to ensure a safe and comfortable environment for the baby, as well as reduce the mental burden on the user. The system uses various sensors and advanced analytical means to monitor the baby's and user's condition in real time and provide appropriate responses.

[1861] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1862] Step 1: Data acquisition and transmission (device)

[1863] Specific operation: Each sensor in the terminal (video acquisition means, vibration detection means, environmental sensor means, illuminance detection means) collects its own data. The video acquisition means captures video of the baby, and the vibration detection means monitors the baby's breathing. The environmental sensor means measures room temperature and humidity, and the illuminance detection means measures the brightness of the room.

[1864] Inputs: Baby's posture, facial expression, breathing, room temperature, humidity, and brightness.

[1865] Data processing and calculation: Each sensor collects data in real time and transmits it to the server via communication means.

[1866] Output: Various sensor data transferred to the server

[1867] Step 2: User emotion recognition (device)

[1868] Specific operation: The device's emotion recognition system captures the user's voice and video, acquiring voice tone and facial expression data. This data is sent to a server in real time for emotion analysis.

[1869] Input: User's voice data, video data

[1870] Data processing and calculation: The emotion recognition system collects data to analyze the user's emotions based on an emotion model and sends it to the server.

[1871] Output: Audio and video data sent to the server

[1872] Step 3: Data analysis and anomaly detection (server)

[1873] Specific operation: The server analyzes the received data and uses advanced algorithms to monitor the baby's posture, facial expression, breathing patterns, and room environment. Simultaneously, it analyzes the user's emotional state.

[1874] Input: Sensor data, audio data, and video data transmitted from the device.

[1875] Data processing and calculation: The server analyzes the data using analytical algorithms and detects anomalies. This includes determining whether the baby has stopped breathing or whether the user is in a high-stress state.

[1876] Output: Anomaly detection results, user emotional state analysis results

[1877] Step 4: Notification (Server)

[1878] Specific operation: If a server anomaly is detected, or if a user is experiencing stress, an emergency notification will be sent to the user using a notification system. The notification will arrive on a device such as a smartphone or tablet.

[1879] Input: Anomaly detection results, user emotional state analysis results

[1880] Data processing and calculation: When an anomaly is detected, immediately prepare a notification and send it to the user's device. Customize the message content according to the type of anomaly and the emotional state of the user.

[1881] Output: Notification message to the user

[1882] Step 5: Environment Control (Server)

[1883] Specific operation: Based on the analysis results, the server sends instructions to the terminal to control the settings of the air conditioner and lighting in order to maintain an optimal environment (temperature, humidity, brightness) in the bedroom.

[1884] Input: Baby's condition data, indoor environment data

[1885] Data processing and calculation: The server calculates the settings necessary to maintain a comfortable environment for the baby and sends instructions to the terminal.

[1886] Output: Control instructions for air conditioners and lighting.

[1887] Step 6: Learning and Advice (Server)

[1888] Specific operation: The server interacts with the childcare record application to learn the baby's daily routines (sleep, meals, elimination, etc.). Based on the learned patterns, it sends timely advice and notifications to the user.

[1889] Input: Childcare record data, past sensor data

[1890] Data Processing and Calculation: Analyze data from childcare record applications to analyze the baby's daily routines. Use the analysis results to generate appropriate advice.

[1891] Output: Advice messages, notification messages

[1892] The above outlines the processing flow of this system's program. By collecting and analyzing data in real time, this system provides comprehensive support to enhance the safety of babies and the comfort of users.

[1893] (Application Example 2)

[1894] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1895] Traditional systems for managing the safety and comfort of babies could monitor the baby's condition but failed to consider the user's emotions. Similarly, in physical stores, it was difficult to monitor customer emotions and behavior in real time and implement appropriate responses and environmental controls. This resulted in insufficient improvement in the quality of childcare and customer service, sometimes leading to delays in providing optimal support.

[1896] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an analysis means, a notification means, a control means, an automatic notification means, a customer emotion notification means, and a store environment control means. This enables monitoring of the baby's safety and comfort, as well as responses to customer emotions and automatic control of the store environment.

[1897] A "camera system" is a device used to detect the posture and facial condition of babies or customers in real time.

[1898] A "vibration sensor" is a device used to detect a baby's breathing.

[1899] A "temperature and humidity sensor means" is a sensor for measuring the temperature and humidity in a bedroom or store.

[1900] An "illuminance sensor means" is a sensor used to measure the brightness inside a bedroom or store.

[1901] "Communication means" refers to means for transmitting data from the camera means, vibration sensor means, temperature and humidity sensor means, and illuminance sensor means to the server.

[1902] "Analysis means" refers to means for monitoring the safety and comfort of babies and customers by analyzing the aforementioned data.

[1903] "Notification means" refers to a means for notifying a user or staff member when an abnormality is detected by the analysis means.

[1904] "Control means" refers to means for controlling the environment of a bedroom or store based on the analysis means.

[1905] An "emotion engine" is a means of detecting customer emotions in real time.

[1906] A "customer sentiment notification system" is a means of analyzing customer sentiment data and notifying staff of appropriate responses.

[1907] "Store environment control means" refers to means for automatically adjusting the store environment based on customer sentiment data.

[1908] "Learning methods" refer to techniques for learning individual patterns by linking them with the baby's care records.

[1909] "Advice means" refers to means for providing timely advice and notifications to the user based on the learning means.

[1910] This invention is a system for managing the safety and comfort of babies. Furthermore, the same technology can be used to monitor customer emotions and behavior in real time in physical stores, thereby improving customer service.

[1911] System Configuration

[1912] 1. Terminal

[1913] Camera System: This camera is used to detect the posture and facial condition of babies and customers in real time. The video data acquired from the camera is transmitted to the server described below.

[1914] Vibration sensor: This sensor is used to detect the baby's breathing. The vibration data is transmitted to a server.

[1915] Temperature and humidity sensor means: A sensor for measuring the temperature and humidity in a bedroom or store. This data is also transmitted to the server.

[1916] Illuminance sensor: This is a sensor used to measure the brightness in a bedroom or store. This data is also transmitted to the server.

[1917] Communication method: This is a communication device used to transmit data acquired from the above-mentioned sensors to the server. For example, Wi-Fi modules or Bluetooth are used.

[1918] Emotion Engine: A device that acquires customer voice and video data and detects customer emotions in real time.

[1919] 2. Server

[1920] Analysis method: This method involves analyzing data transmitted from terminals on a server to monitor the safety and comfort of babies and customers. Specifically, it utilizes image analysis and data mining techniques.

[1921] Notification method: This is a means of notifying users or store staff when an anomaly is detected by the analysis method. Notifications are made via devices such as smartphones and tablets.

[1922] Control means: A means for transmitting instructions to a terminal to control the environment of a bedroom or store based on analysis means. For example, it can be used to adjust the temperature of an air conditioner or change the brightness of lighting.

[1923] Customer sentiment notification method: This method analyzes customer sentiment data and notifies staff of appropriate responses. For example, if a customer expresses dissatisfaction, a notification is sent to staff prompting a quick response.

[1924] Store environment control means: This means automatically adjusting the store environment based on customer emotion data. For example, it may change the store temperature or background music.

[1925] 3. User

[1926] User Devices: Users or store staff can receive notifications using devices such as smartphones and tablets, and manage the baby's condition, environment, and customer's emotional state through the application. They can also receive timely advice and notifications provided by the system.

[1927] Specific example

[1928] Common usage examples

[1929] Terminal: The camera detects the baby's posture and sends the data to the server.

[1930] Terminal: The emotion engine analyzes the customer's facial expressions and sends them to the server.

[1931] Server: If the server analyzes the data and determines that everything is normal, it notifies the user or store staff with messages such as "The baby is safe" or "The customer is satisfied."

[1932] Terminal: The temperature and humidity sensor detects when the indoor temperature is too high and sends data to the server.

[1933] Server: Sends a cooling command to the air conditioner and adjusts it to the appropriate temperature.

[1934] Example of a prompt

[1935] Application: Design a smart store watch application to enhance the customer experience in physical stores. Using cameras and sensors, monitor customer emotions and behavior in real time, automating all responses and environmental controls. For example, if a customer appears distressed, quickly notify staff and adjust settings such as air conditioning or music to create a more relaxing atmosphere.

[1936] Examples of use in emergencies

[1937] Terminal: A vibration sensor detects when the baby stops breathing for 20 seconds and sends the data to the server.

[1938] Server: The system determines this is an emergency and sends an emergency notification to the parents stating, "There is an abnormality in the baby's breathing."

[1939] User: Receive notifications on your smartphone and respond quickly.

[1940] Server: Simultaneously analyzes the user's emotions and, if it determines that the user is experiencing stress, sends a message saying, "Please calm down. We will address this immediately."

[1941] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1942] Step 1:

[1943] The device uses a camera to detect the posture and facial features of babies and customers in real time and acquire video data. The input is camera footage, and the output is the detected face and posture data. Specifically, it uses OpenCV and the dlib library to capture video and execute a face detection algorithm.

[1944] Step 2:

[1945] The device uses a vibration sensor to detect the baby's breathing. The input is vibration data from the sensor, and the output is data indicating the presence of breathing. The sensor data is acquired at regular intervals and processed into data for analyzing the breathing pattern.

[1946] Step 3:

[1947] The terminal uses temperature and humidity sensors to measure and acquire data on the temperature and humidity in a bedroom or store. The input is the reading from the temperature and humidity sensors, and the output is accurate temperature and humidity data for the room. This data is used as foundational data for analysis and control described later.

[1948] Step 4:

[1949] The terminal uses an illuminance sensor to measure the brightness in a bedroom or store and acquire data. The input is the illuminance sensor reading, and the output is the room's illuminance data. The measured data is sent to the server.

[1950] Step 5:

[1951] The terminal uses communication methods to transmit data acquired from the camera, vibration sensor, temperature and humidity sensor, and illuminance sensor to the server. The input is the data acquired from each sensor, and the output is the data transferred to the server. Wi-Fi, Bluetooth, and other technologies are used for communication.

[1952] Step 6:

[1953] The server uses analysis tools to analyze the transmitted data and monitor the safety and comfort of babies and customers. The input is sensor data transmitted from the terminal, and the output is the analysis results. Specifically, it uses image analysis algorithms and machine learning models to detect various anomaly patterns.

[1954] Step 7:

[1955] The server uses a notification system to notify users or store staff if it detects an anomaly in the analysis process. The input is the analysis result, and the output is a notification message. This notification is sent via a push notification system to smartphones and tablets.

[1956] Step 8:

[1957] The server uses control means to send instructions to the terminal to control the environment of a bedroom or store based on analysis means. The input is the analysis result, and the output is the environment control instruction. For example, it may issue instructions to adjust the set temperature of an air conditioner or the brightness of lighting.

[1958] Step 9:

[1959] The terminal uses an emotion engine to acquire customer voice and video data and detect the customer's emotions. The input is voice and video data, and the output is analyzed emotion data. An emotion recognition model is used to detect changes in emotion, and this data is sent to the server.

[1960] Step 10:

[1961] The server uses a customer sentiment notification system to analyze customer sentiment data and notify staff of appropriate actions. The input is sentiment data sent from the terminal, and the output is a notification to staff. The notification content changes according to the customer's status, prompting a quick response for dissatisfied customers.

[1962] Step 11:

[1963] The server uses store environment control means to automatically adjust the store environment based on customer emotion data. The input is analyzed emotion data, and the output is environment control instructions. For example, it adjusts the store temperature and background music.

[1964] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1965] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1966] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1967] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1968] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1969] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1970] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1971] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1972] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1973] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1974] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1975] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1976] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1977] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1978] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1979] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by usi...

Claims

1. A camera system that detects the baby's posture and facial condition in real time, A vibration sensor means for detecting a baby's breathing, A temperature and humidity sensor means for measuring the temperature and humidity of a bedroom, An illuminance sensor means for measuring the brightness of a bedroom, A communication means for transmitting data from the camera means, vibration sensor means, temperature and humidity sensor means, and illuminance sensor means to a server, The server includes an analysis means that monitors the safety and comfort of the baby by analyzing the data, The aforementioned analysis means includes a notification means that notifies the user when an abnormality is detected, A control means for controlling the bedroom environment based on the analysis means, A system that includes this.

2. A learning method that links with the baby's care records and learns individual patterns, An advice means that provides timely advice and notifications to the user based on the learning means, The system according to claim 1, further comprising:

3. The system according to claim 1, characterized in that it issues an emergency notification when the analysis means detects that the baby is lying face down or has its face covered.

Images