System

Abstract

A system is provided.SOLUTION: A system comprising: means for using a terminal that is no longer in use as a baby monitor and transmitting video and audio data from the baby monitor to a server; means for analyzing the video and audio data and detecting a danger; and means for transmitting a push notification to a terminal of a user and automatically playing an audio message from the baby monitor when the danger is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Raising an infant is extremely difficult, as parents often have to juggle multiple tasks, including work, childcare, and housework, leading to sleep deprivation. This is especially true for babies, who require constant attention to ensure their safety and promote their proper development. Forming a baby's sleep habits and toilet training can also be a significant burden for parents. There is a need for an effective support system to resolve these issues and reduce the burden on parents. [Means for solving the problem]

[0005] The present invention provides a system that uses an unused device as a baby monitor, including means for transmitting video and audio data from the baby monitor to a server, means for analyzing the video and audio data to detect danger, and means for sending a push notification to the user's device and automatically playing an audio message from the baby monitor when the danger is detected. The system also includes means for playing music from the baby monitor to help the baby fall asleep and inputting the baby's sleep data into the user's device, means for recording the sleep data on the server, and means for sending a push notification to the user's device when the baby wakes up and cries. The system further includes means for inputting the baby's daily toilet use timings from the user's device, analyzing the toilet data on the server, and means for predicting the next toilet use timing and sending a push notification to the user's device. In this way, a system is realized that allows parents to remotely check the safety of their baby and take appropriate action at the optimal time.

[0006] "Baby monitor" refers to a device containing a camera and microphone that is used to remotely check on a baby.

[0007] "Terminal" refers to an electronic device used to perform a specific task.

[0008] "Server" refers to a computer system that processes requests from client devices over a network and serves or stores data.

[0009] "User" refers to the parent or guardian who uses the system to manage the safety and development of their baby.

[0010] "Video data" refers to data containing visual information captured by a camera.

[0011] "Audio data" refers to data containing sound information captured by a microphone.

[0012] "Analysis" refers to the process of processing received data to extract meaning and determine whether it meets certain criteria.

[0013] "Push notification" refers to a real-time notification message sent from a server to a user's device when a specific event occurs.

[0014] "Voice Message" means recorded sound data used as a warning or instruction for a particular situation.

[0015] "Sleep music" refers to specific music or sounds that are played to help babies fall asleep easily.

[0016] "Sleep data" refers to information that records what time the baby goes to sleep and what time they wake up.

[0017] "Toilet data" refers to information that records the time a baby uses the toilet.

[0018] "Forecasting" refers to the process of estimating future events based on past data. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention is a system that provides functions for monitoring infants, sleep training, and toilet training in a single application. The main components of this system are an unused device (baby monitor), the user's smartphone, and a cloud server. Details of each function and their implementation are described below.

[0041] 1. Monitoring function

[0042] Device (baby monitor):

[0043] Use an old smartphone as a baby monitor by installing an app and using the camera and microphone to capture real-time video and audio, which is then sent to a server.

[0044] server:

[0045] It receives video and audio data and analyzes it in real time. Specifically, it uses computer vision to analyze the video data and detect abnormalities in the baby's movements and posture. It also uses voice recognition AI to detect crying and other abnormal sounds. If danger is detected, the server sends a push notification to the user's smartphone and instructs the baby monitor to automatically play an audio warning.

[0046] User:

[0047] Users can check on their baby's condition in real time from their smartphone, and can respond immediately if necessary.

[0048] Examples:

[0049] While the baby is sleeping, the baby monitor uses a camera and microphone to send video and audio to a server. Computer vision and voice recognition AI detect when the baby kicks the covers or cries. In this case, the user's smartphone receives a push notification saying "The baby kicked the covers," and the baby monitor plays an audio warning saying "Danger! Stop!"

[0050] 2. Sleep training function

[0051] Device (baby monitor):

[0052] Before your baby falls asleep, play specific sleep music from the baby monitor, and even after your baby falls asleep, the sound sensor will detect any crying that wakes them up.

[0053] server:

[0054] The system receives and analyzes the baby's sleep data sent from the user's smartphone, and if it determines that the baby has woken up, it sends a push notification to the user's smartphone.

[0055] User:

[0056] Users can enter the time their baby goes to sleep and wakes up into the app, and this data is sent to a server. If the baby starts crying, the user will receive a push notification on their smartphone so they can take action.

[0057] Examples:

[0058] To get the baby to sleep alone, the user turns on the sleep training mode. The baby monitor starts playing selected sleep music. When the baby wakes up and cries in the middle of the night, the user's smartphone receives a notification saying "Your baby is crying."

[0059] 3. Toilet training function

[0060] Device (user's smartphone):

[0061] Users input the number of times their baby uses the toilet each day, which allows the server to analyze the intervals and timing of toilet visits.

[0062] server:

[0063] Based on the input data, the next time to go to the toilet is predicted. A push notification is periodically sent to the user's smartphone to let them know when it is time to go to the toilet next.

[0064] User:

[0065] Users who receive the notification can take their baby to the toilet at the appropriate time, which improves the success rate of potty training.

[0066] Examples:

[0067] Users input the time their baby spends using the toilet into the app each day. Based on this, the app predicts when their baby will next go to the toilet, and sends a push notification to the user's smartphone at the appropriate time.

[0068] This system significantly reduces the burden of childcare on parents and provides an environment where infants can live safely and healthily.The real-time monitoring, sleep training, and toilet training functions are realized by linking with a cloud server and making effective use of unused smartphones.

[0069] The processing flow will be explained below.

[0070] 1. Monitoring function

[0071] Device (baby monitor)

[0072] Step 1:

[0073] Install a dedicated app on the device that will serve as the baby monitor and set it up to use the camera and microphone.

[0074] Step 2:

[0075] The device captures video and audio data in real time and transmits it to a server.

[0076] server

[0077] Step 3:

[0078] The server analyzes the received video and audio data, and uses computer vision and voice recognition AI to detect the baby's movements and cries.

[0079] Step 4:

[0080] If a risk is detected as a result of data analysis, the server will send a push notification to the user's device.

[0081] Step 5:

[0082] At the same time, it sends an instruction to the baby monitor to automatically play an audio warning (e.g., "Danger! Stop!").

[0083] User

[0084] Step 6:

[0085] Users can check push notifications on their smartphones and monitor their baby's condition in real time.

[0086] 2. Sleep training function

[0087] Device (baby monitor)

[0088] Step 1:

[0089] When the user is getting ready to put the baby to sleep, the baby monitor is set to sleep training mode.

[0090] Step 2:

[0091] Depending on your settings, the baby monitor will play sleep music to help your baby fall asleep.

[0092] server

[0093] Step 3:

[0094] When the user enters their baby's sleep data (time they went to sleep, time they woke up) into a dedicated app, the server receives and records this data.

[0095] Step 4:

[0096] The server also periodically analyzes the audio data to detect when the baby wakes up crying.

[0097] Step 5:

[0098] If crying is detected, the server immediately sends a push notification to the user's smartphone.

[0099] User

[0100] Step 6:

[0101] The user sees the push notification and goes to check on the baby.

[0102] 3. Toilet training function

[0103] Device (user's smartphone)

[0104] Step 1:

[0105] The user enters the amount of time their baby uses the toilet each day into the app.

[0106] server

[0107] Step 2:

[0108] The server receives, records and analyzes the input toilet data.

[0109] Step 3:

[0110] Based on the analyzed data, the next time to use the toilet is predicted.

[0111] Step 4:

[0112] When the next toilet visit is approaching, the server will send a push notification to the user's smartphone.

[0113] User

[0114] Step 5:

[0115] The user checks the push notification and takes the baby to the toilet at the appropriate time.

[0116] ---

[0117] The above processing steps realize a system in which the baby monitor function, sleep training function, and toilet training function operate seamlessly, reducing the burden on parents.

[0118] Example 1

[0119] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0120] Conventional baby monitoring systems require expensive dedicated equipment and lack versatility. They also struggle to detect abnormalities in real time and respond immediately, making it difficult to reduce the burden of childcare. Furthermore, few systems offer consistent support for infant sleep and toilet training.

[0121] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0122] In this invention, the server uses an unused mobile device as a monitoring device and includes means for transmitting video and audio data from the monitoring device to a cloud computing environment, means for analyzing the video and audio data to detect abnormalities, and means for sending a push notification to the user's mobile communication device and automatically playing an audio message from the monitoring device when the abnormality is detected, thereby reducing the burden of childcare and enabling real-time detection and immediate response to abnormalities.

[0123] A "disused mobile device" is a portable electronic device that can continue to operate even after the user has finished normal use, and can be used for dedicated purposes by installing a specific application.

[0124] The "monitoring device" is a device that uses a camera and microphone to capture the baby's movements in real time and transmits the data to a cloud computing environment.

[0125] A "cloud computing environment" is a network of servers that provide computing resources and storage via the Internet, and analyzes and records the data received.

[0126] "Video and audio data" means visual and audio information captured from a monitoring device, including the baby's movements, crying, etc.

[0127] "Analysis" is the process of analyzing the transmitted video and audio data to detect anomalies or specific patterns.

[0128] "Detecting abnormalities" means determining whether there is any unusual behavior or risk in the baby's movements or voice.

[0129] "Push notifications" are a technology that allows smartphone apps to notify users of information in real time, and are used to issue warnings and alerts.

[0130] "Automatically play voice message" is a function that automatically plays a pre-set voice warning from the monitoring device when an abnormality is detected.

[0131] This invention is a system that provides infant monitoring, sleep training (hereafter referred to as "Nentore"), and toilet training (hereafter referred to as "Toi-train") through a single application. The main components of this system are an unused mobile device (monitoring device), the user's mobile communication device, and a cloud computing environment. Details of each function and an embodiment of the system are described below.

[0132] 1. Monitoring function

[0133] Terminal (monitoring device):

[0134] Abandoned mobile devices are used as surveillance devices, with applications installed that use the camera and microphone to capture real-time video and audio, and this data is transmitted to a cloud computing environment.

[0135] server:

[0136] The system receives video and audio data and analyzes it in real time. Specifically, it uses machine learning and computer vision technology to analyze the video data and detect abnormalities in the baby's movements and posture. It also uses voice recognition AI to detect crying and other abnormal sounds. If an abnormality is detected, the server sends a push notification to the user's mobile communication device and instructs the monitoring device to play an audio warning.

[0137] User:

[0138] Users can check the baby's condition in real time from their mobile communication device and take immediate action if necessary.

[0139] Examples:

[0140] While the baby is sleeping, the monitoring device uses a camera and microphone to transmit video and audio to a cloud computing environment. Computer vision technology and voice recognition AI detect when the baby kicks the covers or cries. In response, a push notification is sent to the user's mobile device stating, "The baby kicked the covers," and the monitoring device plays an audio warning saying, "Danger! Stop!"

[0141] 2. Sleep training function

[0142] Terminal (monitoring device):

[0143] Before your baby falls asleep, the monitor plays specific sleep music, and even after the baby falls asleep, the sound sensor detects any crying that may occur.

[0144] server:

[0145] The system receives and analyzes the baby's sleep data sent from the user's mobile communication device, and if it determines that the baby has woken up, it sends a push notification to the user's mobile communication device.

[0146] User:

[0147] Users enter the times their baby goes to sleep and wakes up into the application. This data is sent to a server for analysis. If the baby starts crying, the user receives a push notification on their mobile device, allowing them to take action.

[0148] Examples:

[0149] The user turns on sleep training mode in the application to help the baby fall asleep independently. The monitoring device starts playing selected sleep music. If the baby wakes up and cries during the night, the user's mobile device receives a notification that "Baby is crying."

[0150] 3. Toilet training function

[0151] Device (user's mobile communication device):

[0152] Users input the amount of time their baby spends using the toilet each day, which allows the cloud computing environment to analyze the intervals and timing of toilet visits.

[0153] server:

[0154] Based on the input data, the system predicts when the next toilet visit will be made, and periodically sends a push notification to the user's mobile device to let them know when it is time to go to the toilet next.

[0155] User:

[0156] Users will receive notifications so they can take their baby to the toilet at the right time, improving the success rate of potty training.

[0157] Examples:

[0158] The user inputs the time their baby uses the toilet each day into the application, and the server then predicts the next time the baby will use the toilet and sends a notification to the user's mobile device at the appropriate time.

[0159] Example prompt sentence:

[0160] "Please explain the specific processing steps of a system that uses computer vision and voice recognition to detect abnormalities in a baby monitoring function and send a push notification to the user's mobile communication device. Also, please provide a specific example of how the cloud computing environment and the monitoring device work together."

[0161] The system reduces the burden of childcare and provides a safe and healthy environment for infants. It includes real-time monitoring, sleep training, and toilet training functions, and is realized by linking with a cloud computing environment and making effective use of unused mobile devices.

[0162] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0163] Monitoring function

[0164] Step 1:

[0165] Data capture by terminal (monitoring device)

[0166] Input: An unused mobile device with the application installed, camera and microphone can act as input devices.

[0167] How it works: Uses a camera and microphone to capture video and audio of your baby in real time.

[0168] Output: Captured video and audio data.

[0169] Step 2:

[0170] Data transmission from the terminal (monitoring device) to the cloud computing environment

[0171] Input: Captured video and audio data.

[0172] How it works: Capture data is sent in real time to a cloud computing environment.

[0173] Output: Video and audio data received by the cloud computing environment.

[0174] Step 3:

[0175] Data analysis by server

[0176] Input: Video and audio data received by the cloud computing environment.

[0177] How it works: It uses computer vision technology and voice recognition AI to analyze incoming data and detect abnormalities such as baby movement or crying.

[0178] Output: Analysis result (normal / abnormal).

[0179] Step 4:

[0180] Server sends push notification and audio alert instructions

[0181] Input: When an anomaly is detected as a result of data analysis.

[0182] What it does: Sends a push notification to the user's mobile device and simultaneously instructs the monitoring device to play an audio alert.

[0183] Output: Push notification to user's mobile device and playback of audio message from monitoring device.

[0184] Step 5:

[0185] User notification and response

[0186] Input: A push notification sent to the user's mobile device.

[0187] Action: Check push notifications and rush to your baby if necessary.

[0188] Output: Baby safety check and problem solving.

[0189] Sleep training function

[0190] Step 1:

[0191] Playing sleep music on a terminal (monitoring device)

[0192] Input: Sleep music selection based on user preferences.

[0193] What it does: The monitoring device plays the selected sleep music.

[0194] Output: Sleep music played.

[0195] Step 2:

[0196] Detecting baby's wake-up sound using a terminal (monitoring device)

[0197] Input: Baby crying and movement sounds.

[0198] How it works: The sound sensor detects your baby's crying or any abnormal sounds.

[0199] Output: Detected sound data.

[0200] Step 3:

[0201] Entering sleep data from the user's mobile communication device

[0202] Input: Baby's sleep and wake times.

[0203] Action: A user enters data through an application.

[0204] Output: The input sleep data.

[0205] Step 4:

[0206] Sleep data analysis and notification by server

[0207] Input: Sleep data and sound sensor data sent from the user's mobile communication device.

[0208] What it does: Detects when the baby wakes up and sends a push notification to the user's mobile device.

[0209] Output: A push notification to the user's mobile device.

[0210] Toilet training function

[0211] Step 1:

[0212] Input of toilet usage time from user's mobile communication device

[0213] Enter: your baby's potty time.

[0214] Action: A user enters data through an application.

[0215] Output: The input toilet data.

[0216] Step 2:

[0217] Server-based analysis and prediction of toilet data

[0218] Input: Restroom data sent from the user's mobile communication device.

[0219] How it works: A cloud computing environment analyzes data and predicts when the next toilet visit will occur.

[0220] Output: Prediction of next toilet timing.

[0221] Step 3:

[0222] Server-driven push notifications

[0223] Input: Predicted next toilet time.

[0224] What it does: Sends a timely push notification to the user's mobile device saying "Next bathroom break."

[0225] Output: A push notification to the user's mobile device.

[0226] As a result, this system reduces the burden of childcare and provides a safe and healthy environment for babies. Each function works in conjunction with each other to provide real-time monitoring, effective sleep training, and proper toilet training.

[0227] (Application example 1)

[0228] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0229] The challenge is to provide a highly efficient, low-cost monitoring system that ensures safety in the home and responds quickly to abnormalities. In particular, there is a need for a system that can effectively utilize unused terminals and safely monitor the home even when the user is away.

[0230] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0231] In this invention, the server includes means for using an unused terminal as a monitoring device and transmitting video and audio data from the monitoring device to the server, means for analyzing the video and audio data to detect an abnormality, means for sending a push notification to the user's terminal and automatically playing an audio message from the monitoring device when the abnormality is detected, and means for monitoring the user's return home status and sending a confirmation notification if the user does not return home at the scheduled time. This makes it possible to maintain home safety even when the user is away and to respond quickly when an abnormality occurs.

[0232] A "terminal no longer in use" is an electronic device such as a smartphone or tablet that the user used in the past but is no longer using.

[0233] A "monitoring device" is a device that reuses unused terminals to collect video and audio data and detect abnormalities.

[0234] "Video and audio data" means real-time video and audio information captured by a surveillance device.

[0235] "Server" means a computer system or cloud-based system for receiving and analyzing video and audio data.

[0236] "Abnormality" refers to suspicious movements or sounds that are different from normal as a result of the surveillance device's analysis of video and audio data.

[0237] A "push notification" is a real-time notification message sent from a server to a user's device.

[0238] A "voice message" is a sound such as a warning sound that is played from a monitoring device when an abnormality is detected.

[0239] A "user's terminal" is a mobile device such as a smartphone or tablet that a user uses on a daily basis.

[0240] The "return home status" refers to whether the user has returned home at the scheduled time designated by the user.

[0241] A "confirmation notice" is a confirmation message sent to the user's terminal if the user does not return home at the scheduled time.

[0242] The "warning sound" is a sound that warns a suspicious person or a user when an abnormality is detected.

[0243] "Video data" refers to digital information of video captured by a monitoring device, which is recorded for abnormality detection and transmitted to a server.

[0244] The present invention provides a system for strengthening home security by utilizing unused terminals as monitoring devices. Specific embodiments for implementing this system will be described below.

[0245] composition

[0246] Terminal (monitoring device):

[0247] Abandoned smartphones and tablets are used as surveillance devices equipped with cameras and microphones to capture real-time video and audio.

[0248] server:

[0249] This is a computer system for receiving and analyzing video and audio data. The server analyzes the data using software libraries such as OpenCV and voice recognition AI.

[0250] On the user's device:

[0251] These are mobile devices that users use on a daily basis, such as smartphones and tablets. A dedicated application for this system is installed on the user's device, allowing them to receive notifications in real time.

[0252] Processing flow

[0253] 1. Video and audio data capture:

[0254] The monitoring device captures video and audio data in real time and transmits the data to a server.

[0255] 2. Data Analysis:

[0256] The server analyzes the video and audio data it receives. Specifically, it uses OpenCV to detect video anomalies, such as when a suspicious person is captured on camera. It also uses voice recognition AI to detect abnormal sounds.

[0257] 3. Push notifications and sound alerts:

[0258] If the server detects an abnormality, it sends a push notification to the user's device and simultaneously plays an alarm sound from the monitoring device.

[0259] 4. Monitoring when users return home:

[0260] The server monitors whether the user has returned home, and if the user has not returned home at the scheduled time, sends a confirmation notice to the user's terminal.

[0261] Specific examples

[0262] For example, a housewife places an unused smartphone at the front door while she goes out shopping. This monitoring device captures video and audio in real time and sends them to a server. The server analyzes the video data, and if it detects a suspicious person at the front door, it sends a push notification to the user's device saying, "A suspicious person has been detected at the front door!" At the same time, the monitoring device sounds an alarm to warn the suspicious person. Furthermore, the server monitors whether the user has returned home at the scheduled time, and if the user has not returned home at the scheduled time, it sends a confirmation notification to the user's device asking, "Did you return home as planned?"

[0263] Prompt Sentence Examples

[0264] "Functional description of this application: We want to create a smart home security system that captures video and audio in real time and detects suspicious activity. Specifically, we will use unused smartphones as security devices, and if an abnormality is detected, the system will upload the data to a cloud server, send a push notification to the user, and play an alarm."

[0265] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0266] Step 1:

[0267] The monitoring device captures video and audio data in real time and sends it to a server. The input includes real-time video and audio captured by the camera and microphone of the monitoring device. The output is sent to the server via the Internet. The specific operation of the monitoring device is to capture video with the camera and record audio with the microphone, and then transfer these as digital data to the server.

[0268] Step 2:

[0269] The server analyzes the video and audio data it receives. The input includes real-time video and audio data sent from the surveillance equipment. The server uses OpenCV to analyze the video data and performs facial recognition and motion detection to detect suspicious activity. It also uses voice recognition AI to detect abnormal sounds (for example, the sound of breaking glass or screaming). The output is alert information if suspicious activity or audio is detected. Specifically, the server analyzes the video data frame by frame and the audio data as a sound waveform.

[0270] Step 3:

[0271] If the server detects an anomaly, it sends a push notification to the user's device. The input includes the alert information generated by the server. The output is a push notification sent to the user's device stating "An anomaly has been detected." The push notification includes detailed information such as the type of anomaly and the time it was detected. Specifically, the server sends a notification to the user's smartphone through a push notification service.

[0272] Step 4:

[0273] If an abnormality is detected, the monitoring device automatically plays an alarm sound. The input includes the alert information sent from the server. As an output, the monitoring device plays an alarm sound to warn suspicious individuals on-site and those in the vicinity. Specifically, the monitoring device uses its built-in speaker to play a pre-set alarm sound file.

[0274] Step 5:

[0275] It monitors the user's return home status and sends a confirmation notification if the user does not return home at the scheduled time. The input includes the user's pre-set information and the server's timestamp information. The output is a confirmation notification sent to the user's smartphone asking, "Did you return home as scheduled?" Specifically, the server checks the user's return home information at the specified time and checks for any abnormalities. It then automatically sends a confirmation notification.

[0276] As described above, this system performs specific data processing and calculations at each step, and when an abnormality is detected, it notifies the user with an alert and even plays a warning sound to ensure the safety of the home.

[0277] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0278] This invention combines a system that provides functions for monitoring infants, sleep training, and toilet training with an emotion engine that recognizes the user's emotions. The main components of this system are an unused device (baby monitor), the user's smartphone, a cloud server, and the emotion engine. Below, we will explain an embodiment in which each function and emotion engine are combined.

[0279] 1. Monitoring function

[0280] Device (baby monitor)

[0281] A dedicated app is installed on an unused smartphone, and the camera and microphone are used to capture real-time video and audio, which is then sent to a server.

[0282] server

[0283] It receives video and audio data and analyzes it using computer vision and voice recognition AI. It detects baby movements and cries, and sends push notifications to the user's device if there is danger. It also instructs the baby monitor to automatically issue an audio alert.

[0284] Emotion Engine

[0285] It analyzes the user's emotions from voice data and input data to determine whether the user is tired, stressed, relaxed, etc. Based on this, it adjusts the content and timing of push notifications.

[0286] User

[0287] Users can check their baby's condition in real time from their smartphone and can take prompt action when notified.

[0288] Specific examples

[0289] While the baby is sleeping, the baby monitor captures video and audio and sends them to the server. The server detects when the baby kicks off the covers and determines whether there is any danger. At the same time, the emotion engine analyzes the user's situation. If it determines that the user is tired, it sends a push notification to reduce the burden on the parent by softening the tone of the notification or by eliminating redundant information to make it concise.

[0290] 2. Sleep training function

[0291] Device (baby monitor)

[0292] When you're getting your baby ready for bed, you can set the baby monitor to sleep mode and play specific sleep music. Even after your baby falls asleep, the sound sensor will still detect crying.

[0293] server

[0294] It receives the baby's sleep data sent from the user's smartphone and records it in a database. If crying is detected, it sends a push notification to the user's smartphone.

[0295] Emotion Engine

[0296] The system analyzes the user's emotions and adjusts the timing and content of sleep notifications. For example, if the user is feeling stressed, it can send a relaxing message.

[0297] User

[0298] Users can enter the times their baby goes to sleep and wakes up into the app, and if crying is detected, they will receive a push notification and be able to take action quickly.

[0299] Specific examples

[0300] To get the baby to sleep alone, the user turns on the sleep training mode. The baby monitor plays sleep music, and the baby falls asleep. When the baby starts crying in the middle of the night, the user's smartphone receives a notification saying "Your baby is crying." However, if the user is feeling stressed, the notification content is adjusted to say something like "It's okay, your baby is just crying a little."

[0301] 3. Toilet training function

[0302] Device (user's smartphone)

[0303] The user enters the amount of time their baby uses the toilet each day into the app.

[0304] server

[0305] The system receives and analyzes the entered toilet data, predicts the next toilet time, and sends a push notification to the user's smartphone.

[0306] Emotion Engine

[0307] The system analyzes the user's emotions and adjusts the toilet timing notification accordingly. For example, if the user is busy, the notification can be delayed a little.

[0308] User

[0309] Users who receive the notification can take their baby to the toilet at the appropriate time.

[0310] Specific examples

[0311] The user inputs the time their baby goes to the toilet into the app every day. The server predicts the next time the baby will go to the toilet, and if the emotion engine determines that the user is busy, it adjusts the notification timing. For example, the user might receive a notification saying, "The next time to go to the toilet is approaching, but you still have a little time."

[0312] This allows the system to take into account the parent's state and emotions, providing more effective and flexible childcare support. By combining real-time monitoring, sleep training, and toilet training functions with the emotion engine, the system further reduces the burden on parents and supports the safety and growth of infants.

[0313] The processing flow will be explained below.

[0314] 1. Monitoring function

[0315] Device (baby monitor)

[0316] Step 1:

[0317] Install the dedicated app on the device that will serve as the baby monitor and enable the camera and microphone.

[0318] Step 2:

[0319] The device captures video and audio data in real time and transmits the data to a server.

[0320] server

[0321] Step 3:

[0322] The server analyzes the received video and audio data, and uses computer vision to analyze the video data and detect any abnormal movements of the baby.

[0323] Step 4:

[0324] At the same time, voice recognition AI is used to analyze audio data and detect crying and abnormal sounds.

[0325] Step 5:

[0326] If a risk is detected, the server sends a push notification to the user's device.

[0327] Step 6:

[0328] The server sends instructions to the baby monitor to play an automatic audio warning (e.g., "Danger! Stop!").

[0329] User

[0330] Step 7:

[0331] Users can check push notifications and monitor their baby's condition in real time.

[0332] 2. Sleep training function

[0333] Device (baby monitor)

[0334] Step 1:

[0335] The user prepares the baby for sleep and sets the baby monitor to sleep mode.

[0336] Step 2:

[0337] The baby monitor plays selected sleep music.

[0338] server

[0339] Step 3:

[0340] The user enters the baby's sleep data (time they went to sleep, time they woke up) into a dedicated app.

[0341] Step 4:

[0342] The server records the entered sleep data in a database.

[0343] Step 5:

[0344] If the baby wakes up crying, the server analyzes the audio data and sends a push notification if crying is detected.

[0345] Emotion Engine

[0346] Step 6:

[0347] The emotion engine analyzes the user's emotional state and adjusts the content and timing of sleep notifications.

[0348] User

[0349] Step 7:

[0350] The user sees the push notification and goes to check on the baby.

[0351] 3. Toilet training function

[0352] Device (user's smartphone)

[0353] Step 1:

[0354] The user enters the amount of time their baby uses the toilet each day into the app.

[0355] server

[0356] Step 2:

[0357] The server receives and records the input toilet data.

[0358] Step 3:

[0359] The server analyzes the toilet data and predicts the next time to use the toilet.

[0360] Step 4:

[0361] A push notification will be sent to the user's smartphone when the next toilet visit is approaching.

[0362] Emotion Engine

[0363] Step 5:

[0364] The emotion engine analyzes the user's emotional state and adjusts the content and timing of toilet notification.

[0365] User

[0366] Step 6:

[0367] The user receives the notification and takes the baby to the toilet at the appropriate time.

[0368] ---

[0369] These processing steps realize a system in which the baby monitor function, sleep training function, and toilet training function work in conjunction with the emotion engine to provide appropriate support according to the parent's emotional state.

[0370] Example 2

[0371] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0372] While conventional baby monitor systems have functions for monitoring infants and training (sleep and toileting), they lack support that takes into account the emotions and state of the parents, which often leaves parents feeling tired and stressed, and the burden of childcare remains.

[0373] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0374] In this invention, the server includes means for using an unused device as a monitoring device and transmitting visual and audio data from the monitoring device to a central processing unit, means for analyzing the visual and audio data to detect danger, means for sending a push notification to the user's device and automatically playing an audio message from the monitoring device when the danger is detected, and means for evaluating the user's emotional state using an emotion analysis engine and adjusting the content and timing of the push notification based on the evaluation result. This enables notifications and support that take parents' emotions and states into consideration, thereby reducing the burden of child-rearing.

[0375] "Device" refers to a device or terminal held by a user, including devices used as monitor devices and smartphones used by users.

[0376] "Monitoring devices" refers to devices used to monitor and check on infants, including old smartphones.

[0377] "Visual Data" refers to real-time video information captured by the camera of a monitoring device.

[0378] "Acoustic Data" refers to real-time audio information collected by the microphone of a monitoring device.

[0379] "Central processing unit" refers to a device or system, such as a cloud or server, that receives and analyzes data sent from a monitor device.

[0380] "Danger" refers to abnormal behavior or a situation in an infant, and includes cases where the infant is deemed to be in danger.

[0381] "Push Notification" refers to alert messages or notifications sent in real time from a central processing unit to a user's device.

[0382] "Audible message" refers to a voice or sound alert that is automatically played from a monitoring device.

[0383] "Emotion analysis engine" refers to an algorithm or system that analyzes a user's voice or input data to assess the user's emotional state.

[0384] "Sleep Data" refers to information relating to an infant's sleep patterns and duration, including data recorded and analyzed by a central processing unit.

[0385] "Toilet data" refers to information about an infant's daily toilet use time and timing, including data that is analyzed by a central processing unit.

[0386] This invention is a system that combines a system that provides functions for monitoring infants, sleep training, and toilet training with an emotion analysis engine that recognizes user emotions. The main components are an unused device (monitoring device), the user's smartphone, a cloud server, and the emotion analysis engine. Below, we will explain in detail an embodiment in which each function is combined with the emotion analysis engine.

[0387] Monitoring function

[0388] Terminal (monitor device)

[0389] The monitoring device is an unused smartphone, which is then installed with a dedicated app. The app uses the smartphone's camera and microphone to capture real-time visual and acoustic data, which is then transmitted via the internet to a central processing unit located in the cloud.

[0390] Server (cloud server)

[0391] The cloud server receives visual and acoustic data sent from the monitoring device. This data is analyzed using computer vision algorithms (e.g., OpenCV) and speech recognition AI (e.g., speech recognition API). As a result of the analysis, the cloud server detects the infant's movements and cries and detects danger based on this. If danger is detected, a push notification is sent to the user's smartphone and an audio message is automatically played on the monitoring device.

[0392] Sentiment Analysis Engine

[0393] The emotion analysis engine analyzes the user's voice and input data to determine whether they are tired, stressed, or relaxed, and adjusts the content and timing of push notifications accordingly.

[0394] User

[0395] Users can check the condition of their baby in real time from their smartphone and can take prompt action when notified. For example, if a notification is received that a baby is in danger, they can rush to the scene immediately.

[0396] Specific examples

[0397] While the baby is sleeping, the monitor device captures video and audio and sends them to a cloud server. The cloud server detects when the baby kicks off the covers and determines whether there is any danger. At the same time, an emotion analysis engine analyzes the user's situation. If the user is tired, a push notification will be sent to soften the tone of the notification and simplify the message to reduce the burden on parents.

[0398] Prompt Sentence Examples

[0399] "Baby danger detection. Please suggest a way to soften the tone of the notification if the user is tired."

[0400] Sleep training function

[0401] Terminal (monitor device)

[0402] The monitoring device is set to sleep mode when preparing the baby for sleep, which plays specific sleep-inducing music and has sound sensors that detect crying even after the baby has fallen asleep.

[0403] Server (cloud server)

[0404] It receives the baby's sleep data sent from the user's smartphone and records it in a database. If the baby cries, it sends a push notification to the user's smartphone.

[0405] Sentiment Analysis Engine

[0406] The emotion analysis engine analyzes the user's emotions and adjusts the timing and content of sleep notifications accordingly. For example, if the user is feeling stressed, it will send a relaxing message.

[0407] User

[0408] Users enter the times their baby goes to sleep and wakes up into the app, and if crying is detected, they receive a push notification and can take action.

[0409] Specific examples

[0410] When the user turns on the sleep training mode, the monitor device plays sleep music to lull the baby to sleep. If crying is detected in the middle of the night, the cloud server sends a notification to the user's smartphone saying, "Your baby is crying." However, if the user is feeling stressed, the notification content will be adjusted to say, "It's okay, your baby is just crying a little."

[0411] Prompt Sentence Examples

[0412] "When detecting a baby's cry, create an appropriate relaxation message if the user is feeling stressed."

[0413] Toilet training function

[0414] Device (user's smartphone)

[0415] Users enter their baby's daily toilet use times into the app.

[0416] Server (cloud server)

[0417] The server receives and analyzes the input toilet data, and based on the analysis results, predicts the next toilet time and sends a push notification to the user's smartphone.

[0418] Sentiment Analysis Engine

[0419] The emotion analysis engine analyzes the user's emotions and adjusts the toilet timing notification accordingly. If the user is busy, the notification can be delayed.

[0420] User

[0421] When the user receives the notification, they can take the baby to the toilet at the appropriate time.

[0422] Specific examples

[0423] The user inputs the time their baby goes to the toilet into the app every day. The cloud server predicts when the next toilet visit will be, and if the emotion analysis engine determines that the user is busy, the notification timing will be adjusted. For example, the user might receive a notification saying, "The next toilet visit is approaching, but you still have a little time."

[0424] Prompt Sentence Examples

[0425] "Please suggest a way to predict when a baby needs to go to the toilet and adjust the notification timing based on the user's emotions."

[0426] With these functions, the system takes into account the emotions and state of the parent, providing more effective and flexible childcare support. By combining the monitoring, sleep training, and toilet training functions with the emotion analysis engine, the system further reduces the burden on parents and supports the safety and growth of infants.

[0427] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0428] Monitoring function processing steps

[0429] Step 1:

[0430] Terminal (monitor device)

[0431] Input: Video and audio from a monitor device with a dedicated app installed, camera and microphone.

[0432] Specific operation: Launch the dedicated app for the monitor device.

[0433] Processing: The app activates the monitor device's camera and microphone to capture video and audio in real time.

[0434] Output: Captured visual and acoustic data.

[0435] Step 2:

[0436] Terminal (monitor device)

[0437] Input: Captured visual and acoustic data.

[0438] Specific operation: Real-time video and audio data is sent to a cloud server via the Internet.

[0439] Processing: The dedicated app initiates a network connection to send the data, compresses and encodes the data, and sends it to the cloud server.

[0440] Output: Visual and acoustic data sent to cloud server.

[0441] Step 3:

[0442] Server (cloud server)

[0443] Input: Visual and acoustic data sent from the monitor device.

[0444] Specific operation: The cloud server receives the data and records it in a log.

[0445] Processing: Visual data is analyzed with computer vision algorithms (e.g., OpenCV), and acoustic data is analyzed with speech recognition AI (e.g., speech recognition API).

[0446] Output: The results of the analysis include the detection of infant movements and crying sounds.

[0447] Step 4:

[0448] Server (cloud server)

[0449] Input: Analysis results of visual and acoustic data.

[0450] Specific behavior: Generates a push notification if a danger is detected.

[0451] Processing: The system assesses the level of danger based on the detection of the infant's movements and cries, and if it determines that there is danger, it sends a push notification to the user's smartphone and instructs the monitoring device to send an acoustic message.

[0452] Output: Push notification sent to the user's smartphone.

[0453] Step 5:

[0454] Sentiment Analysis Engine

[0455] Input: User voice and typing data.

[0456] Specific operation: The sentiment analysis engine acquires the data and begins analysis.

[0457] Processing: A voice emotion analysis algorithm is used to determine the user's emotional state.

[0458] Output: User's emotional state as a result of emotion analysis (fatigue, stress, relaxed, etc.).

[0459] Step 6:

[0460] Server (cloud server)

[0461] Input: Sentiment analysis results from the sentiment analysis engine.

[0462] Specific operation: The cloud server adjusts the notification content and timing.

[0463] Processing: Based on the results of sentiment analysis, adjust the content (message tone, redundancy of information, etc.) and timing of push notifications.

[0464] Output: The adjusted push notification content.

[0465] Step 7:

[0466] User (smartphone)

[0467] Input: Push notification from cloud server.

[0468] Specific behavior: The user's smartphone displays the notification content.

[0469] Processing: The user confirms the push notification and sees and hears the baby in real time.

[0470] Output: User's response actions (checking and responding to the infant, etc.).

[0471] Processing steps for sleep training functions

[0472] Step 1:

[0473] Terminal (monitor device)

[0474] Input: Sleep training mode setting instruction from user.

[0475] Specific operation: The user sets the sleep training mode in the app.

[0476] Processing: Play specific sleep music from the monitor device depending on the sleep training mode.

[0477] Output: Sleep music is playing.

[0478] Step 2:

[0479] Terminal (monitor device)

[0480] Input: Audio capture by monitor device.

[0481] Specific function: The sound sensor captures sound and detects the baby's crying.

[0482] Processing: Send the crying data to the cloud server.

[0483] Output: Crying data sent to the cloud server.

[0484] Step 3:

[0485] Server (cloud server)

[0486] Input: Sleep data sent from the user's smartphone.

[0487] Specific operation: The server records it in the database.

[0488] Processing: The data is stored to analyze your baby's sleep patterns.

[0489] Output: Recorded sleep data.

[0490] Step 4:

[0491] Server (cloud server)

[0492] Input: Crying data analysis results.

[0493] Specific behavior: Generates a push notification.

[0494] Action: If crying is detected, a push notification is sent to the user's smartphone.

[0495] Output: Push notification sent to the user's smartphone.

[0496] Step 5:

[0497] Sentiment Analysis Engine

[0498] Input: User voice and typing data.

[0499] Specific operation: The sentiment analysis engine acquires the data and begins analysis.

[0500] Processing: A voice emotion analysis algorithm is used to determine the user's emotional state.

[0501] Output: User's emotional state (stressed, relaxed, etc.) as a result of emotion analysis.

[0502] Step 6:

[0503] Server (cloud server)

[0504] Input: Sentiment analysis results from the sentiment analysis engine.

[0505] Specific operation: The cloud server adjusts the notification content and timing.

[0506] Processing: Based on the results of sentiment analysis, adjust the content (message tone, redundancy of information, etc.) and timing of push notifications.

[0507] Output: The adjusted push notification content.

[0508] Step 7:

[0509] User (smartphone)

[0510] Input: Push notification from cloud server.

[0511] Specific operation: The user's smartphone displays the notification and checks the sleep data.

[0512] Action: The user checks the push notification and checks the baby's status in real time.

[0513] Output: User's response actions (responding to crying, recording sleep data, etc.).

[0514] Toilet training function processing steps

[0515] Step 1:

[0516] Device (user's smartphone)

[0517] Input: Your baby's daily toilet use timings.

[0518] Specific operation: The user inputs the timing of toilet use into the app.

[0519] Processing: The entered data is recorded by the app and sent to the cloud server.

[0520] Output: Toilet data sent to the cloud server.

[0521] Step 2:

[0522] Server (cloud server)

[0523] Input: Toilet data sent from the user's smartphone.

[0524] Specific operation: The server receives the data and begins analyzing it.

[0525] Processing: Predict the next toilet timing based on accumulated toilet data.

[0526] Output: Predicted next toilet time.

[0527] Step 3:

[0528] Server (cloud server)

[0529] Input: Predicted next toilet time.

[0530] Specific behavior: Generates a push notification to the user's smartphone.

[0531] Processing: Based on the prediction results, a message notifying the user of the next toilet visit is created and sent to the user's smartphone.

[0532] Output: Push notification sent to the user's smartphone.

[0533] Step 4:

[0534] Sentiment Analysis Engine

[0535] Input: User voice and typing data.

[0536] Specific operation: The sentiment analysis engine acquires the data and begins analysis.

[0537] Processing: A voice emotion analysis algorithm is used to determine the user's emotional state.

[0538] Output: User's emotional state (tired, stressed, busy, etc.) as a result of sentiment analysis.

[0539] Step 5:

[0540] Server (cloud server)

[0541] Input: Sentiment analysis results from the sentiment analysis engine.

[0542] Specific operation: The cloud server adjusts the notification content and timing.

[0543] Processing: Based on the results of sentiment analysis, adjust the content (message tone, redundancy of information, etc.) and timing of push notifications.

[0544] Output: The adjusted push notification content.

[0545] Step 6:

[0546] User (smartphone)

[0547] Input: Push notification from cloud server.

[0548] Specific behavior: The user's smartphone displays the notification content.

[0549] Action: The user checks the push notification and knows when to go to the bathroom in real time.

[0550] Output: Toilet guidance action by the user.

[0551] In this way, the system provides childcare support that takes into account the parent's emotions and state through each processing step, improving the effectiveness of supervision, sleep training, and toilet training.

[0552] (Application example 2)

[0553] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0554] Efficient and safe management of the status of workers and machinery in factories is important for improving the working environment and increasing productivity. However, existing monitoring systems are limited to simply detecting and reporting abnormalities and do not take into consideration the emotional state and workload of users. This can increase stress and burden on managers and delay appropriate responses. The present invention aims to solve these problems and provide a more convenient monitoring and management system.

[0555] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0556] In this invention, the server includes: a means for using an unused terminal as a monitoring device and transmitting video and audio data from the monitoring device to the server; a means for analyzing the video and audio data to detect an abnormality; a means for sending a push notification to the user's terminal and automatically playing an audio message from the monitoring device when the abnormality is detected; a means for analyzing the user's emotions and adjusting the content and timing of the notification; a means for analyzing the worker's status from the monitoring device and inputting the worker's status data into the user's terminal; a means for recording the status data in the server and sending a push notification to the user's terminal when the worker causes an abnormality; and a means for transmitting the adjusted notification to the user's terminal. This enables appropriate notification based on the user's emotional state at the same time as detecting an abnormality, thereby improving the safety and efficiency of the work environment.

[0557] An "obsolete device" is an electronic device that was previously in use but is no longer in use.

[0558] "Surveillance equipment" means equipment installed to monitor a specific area or object and capable of capturing video and audio data.

[0559] "Video and audio data" refers to digital data containing visual and audio information captured using a camera and microphone.

[0560] A "server" is a computer system that stores, processes, and manages data over a network.

[0561] An "anomaly" is an event or circumstance that deviates from normal operating or environmental conditions and requires immediate attention.

[0562] A "push notification" is a message that is automatically sent to a user's device when a specific event or state change occurs.

[0563] A "voice message" is an audio content that is recorded as audio data and played back.

[0564] "Emotion analysis" is the process of identifying a user's emotional state from speech or other input data.

[0565] "Adjusting notification content and timing" means changing the content of the message sent to the user and the timing at which the message is sent depending on the state or situation of the user.

[0566] "Worker status data" is digital data that indicates the worker's behavior, location, health condition, etc.

[0567] "Analysis" is the process of breaking down and evaluating data to extract useful information.

[0568] "Recording on the server" means storing the acquired data on the server so that it can be referenced or used later as needed.

[0569] "Adjusted notifications" are push notifications whose content or delivery timing has been changed based on the results of sentiment analysis.

[0570] MODE FOR CARRYING OUT THE INVENTION

[0571] A system for implementing the present invention includes the following configuration.

[0572] First, we use unused devices as surveillance devices. These devices have built-in cameras and microphones that capture video and audio data in real time, which is then transmitted to a cloud server via a network.

[0573] The cloud server uses computer vision technology and a voice recognition engine to analyze the received video and audio data, utilizing open source technologies such as OpenCV and TensorFlow, making it possible to detect abnormalities in workers and machinery.

[0574] If an abnormality is detected, the server sends a push notification to the user's device. This notification is displayed on the user's smartphone or PC. In addition, the monitoring device automatically plays a voice message regarding the abnormality. This voice message is pre-recorded and includes details of the abnormality and instructions on how to respond.

[0575] To analyze the user's emotional state, the cloud server is equipped with an emotion analysis engine. This engine analyzes voice data and other input data (e.g., text messages entered by the user) to determine the user's emotional state. For emotion analysis, voice analysis technologies such as Amazon Polly and Google Cloud Speech-to-Text can be used.

[0576] The content and timing of notifications are tailored based on the user's emotional state: if the user is feeling stressed, notifications can be sent in a softer tone or in a more concise manner without unnecessary information.

[0577] The system also has the function of monitoring the status of workers. Data acquired from the monitoring device is recorded on a server, and if an abnormality occurs, an appropriate notification is sent to the user's device. This improves worker safety and productivity.

[0578] Examples:

[0579] For example, if a monitoring device installed in a factory detects worker movements or a machine malfunction, the video and audio data is first sent to a cloud server. The cloud server analyzes the data and identifies the malfunction. At the same time, an emotion analysis engine analyzes the user's stress and fatigue levels and adjusts the notification content accordingly. The user's device may receive a notification such as, "An abnormality has occurred on machine A. Please check the details." However, if it is determined that the user is feeling stressed, a softer-toned notification may be sent, such as, "There is a minor malfunction on machine A. Please check."

[0580] Prompt for the generative AI model:

[0581] For example, the following might be a prompt to input to a generative AI model:

[0582] Please explain the functions of a system that supports putting a baby to sleep and toilet training. For example, a baby monitor captures video and audio while the baby is sleeping and sends them to a server. If the server detects danger and determines that the user is tired, it will send a push notification to reduce the burden on the parent by softening the tone of the notification or simplifying it by removing redundant information.

[0583] This allows the present invention to provide an efficient and safe monitoring and management system that takes into account the emotional state and workload of the user.

[0584] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0585] Processing Steps

[0586] Step 1: Booting the device and capturing data

[0587] The device starts up and uses the camera and microphone to capture video and audio data in real time. This data is acquired through the camera sensor and microphone. The input data is video frames and audio waveform data, which are converted and prepared in digital format.

[0588] Step 2: Send data

[0589] The device sends the captured video and audio data to the cloud server. The data is transmitted using a network communication protocol (e.g., HTTP or WebSocket). The input is the captured data, and the output is the status of the completion of data transmission to the server.

[0590] Step 3: Data analysis (server side)

[0591] The server analyzes the received data, using computer vision technology (e.g., OpenCV) for video data and a speech recognition engine (e.g., Google Cloud Speech-to-Text) for audio data. The input data are video frames and audio files sent from the device, and the output is anomaly detection information as the analysis result.

[0592] Step 4: Anomaly detection

[0593] The server detects anomalies based on the analysis results. If an anomaly is detected, it immediately starts a process to notify the user of that information. The input is the data analysis results, and the output is the anomaly determination result.

[0594] Step 5: Sentiment Analysis

[0595] The server analyzes the user's emotional state. To do this, it uses the voice data and text data from the user to identify the emotional state. Using an emotion analysis engine (e.g., Amazon Polly), the input data is the user's response and voice data, and the output is the user's emotional state.

[0596] Step 6: Adjust notification content and timing

[0597] The server adjusts the content and timing of notifications based on the emotion analysis results. If the user is feeling stressed, the notification content will be briefer and the tone will be softer. The input is the emotion analysis results, and the output is the adjusted content and timing of notifications.

[0598] Step 7: Send notification

[0599] The server sends the adjusted notification content to the user's device. This notification is sent as a push notification and displayed on the user's smartphone or PC. The input is the adjusted notification content, and the output is the notification sent to the user's device.

[0600] Step 8: Play a voice message when an error occurs

[0601] When an abnormality is detected, the monitoring device automatically plays back a voice message containing the details of the abnormality and instructions for how to respond. The input is the abnormality detection information, and the output is the playback of the voice message.

[0602] Examples:

[0603] For example, if a monitoring device installed in a factory detects worker movements or a machine malfunction, the video and audio data is first sent to a cloud server. The cloud server analyzes the data and identifies the malfunction. At the same time, an emotion analysis engine analyzes the user's stress and fatigue levels and adjusts the notification content accordingly. The user's device may receive a notification such as, "An abnormality has occurred on machine A. Please check the details." However, if it is determined that the user is feeling stressed, a softer-toned notification may be sent, such as, "There is a minor malfunction on machine A. Please check."

[0604] Example prompt for a generative AI model:

[0605] Please explain the functions of a system that supports putting a baby to sleep and toilet training. For example, a baby monitor captures video and audio while the baby is sleeping and sends them to a server. If the server detects danger and determines that the user is tired, it will send a push notification to reduce the burden on the parent by softening the tone of the notification or simplifying it by removing redundant information.

[0606] In this way, specific input data and output data are clearly defined in each processing step, and the system operates by processing data and performing calculations based on them.

[0607] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0608] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0609] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0610] [Second embodiment]

[0611] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0612] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0613] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0614] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0615] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0616] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0617] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0618] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0619] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0620] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0621] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0622] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0623] This invention is a system that provides functions for monitoring infants, sleep training, and toilet training in a single application. The main components of this system are an unused device (baby monitor), the user's smartphone, and a cloud server. Details of each function and their implementation are described below.

[0624] 1. Monitoring function

[0625] Device (baby monitor):

[0626] Use an old smartphone as a baby monitor by installing an app and using the camera and microphone to capture real-time video and audio, which is then sent to a server.

[0627] server:

[0628] It receives video and audio data and analyzes it in real time. Specifically, it uses computer vision to analyze the video data and detect abnormalities in the baby's movements and posture. It also uses voice recognition AI to detect crying and other abnormal sounds. If danger is detected, the server sends a push notification to the user's smartphone and instructs the baby monitor to automatically play an audio warning.

[0629] User:

[0630] Users can check on their baby's condition in real time from their smartphone, and can respond immediately if necessary.

[0631] Examples:

[0632] While the baby is sleeping, the baby monitor uses a camera and microphone to send video and audio to a server. Computer vision and voice recognition AI detect when the baby kicks the covers or cries. In this case, the user's smartphone receives a push notification saying "The baby kicked the covers," and the baby monitor plays an audio warning saying "Danger! Stop!"

[0633] 2. Sleep training function

[0634] Device (baby monitor):

[0635] Before your baby falls asleep, play specific sleep music from the baby monitor, and even after your baby falls asleep, the sound sensor will detect any crying that wakes them up.

[0636] server:

[0637] The system receives and analyzes the baby's sleep data sent from the user's smartphone, and if it determines that the baby has woken up, it sends a push notification to the user's smartphone.

[0638] User:

[0639] Users can enter the time their baby goes to sleep and wakes up into the app, and this data is sent to a server. If the baby starts crying, the user will receive a push notification on their smartphone so they can take action.

[0640] Examples:

[0641] To get the baby to sleep alone, the user turns on the sleep training mode. The baby monitor starts playing selected sleep music. When the baby wakes up and cries in the middle of the night, the user's smartphone receives a notification saying "Your baby is crying."

[0642] 3. Toilet training function

[0643] Device (user's smartphone):

[0644] Users input the number of times their baby uses the toilet each day, which allows the server to analyze the intervals and timing of toilet visits.

[0645] server:

[0646] Based on the input data, the next time to go to the toilet is predicted. A push notification is periodically sent to the user's smartphone to let them know when it is time to go to the toilet next.

[0647] User:

[0648] Users who receive the notification can take their baby to the toilet at the appropriate time, which improves the success rate of potty training.

[0649] Examples:

[0650] Users input the time their baby spends using the toilet into the app each day. Based on this, the app predicts when their baby will next go to the toilet, and sends a push notification to the user's smartphone at the appropriate time.

[0651] This system significantly reduces the burden of childcare on parents and provides an environment where infants can live safely and healthily.The real-time monitoring, sleep training, and toilet training functions are realized by linking with a cloud server and making effective use of unused smartphones.

[0652] The processing flow will be explained below.

[0653] 1. Monitoring function

[0654] Device (baby monitor)

[0655] Step 1:

[0656] Install a dedicated app on the device that will serve as the baby monitor and set it up to use the camera and microphone.

[0657] Step 2:

[0658] The device captures video and audio data in real time and transmits it to a server.

[0659] server

[0660] Step 3:

[0661] The server analyzes the received video and audio data, and uses computer vision and voice recognition AI to detect the baby's movements and cries.

[0662] Step 4:

[0663] If a risk is detected as a result of data analysis, the server will send a push notification to the user's device.

[0664] Step 5:

[0665] At the same time, it sends an instruction to the baby monitor to automatically play an audio warning (e.g., "Danger! Stop!").

[0666] User

[0667] Step 6:

[0668] Users can check push notifications on their smartphones and monitor their baby's condition in real time.

[0669] 2. Sleep training function

[0670] Device (baby monitor)

[0671] Step 1:

[0672] When the user is getting ready to put the baby to sleep, the baby monitor is set to sleep training mode.

[0673] Step 2:

[0674] Depending on your settings, the baby monitor will play sleep music to help your baby fall asleep.

[0675] server

[0676] Step 3:

[0677] When the user enters their baby's sleep data (time they went to sleep, time they woke up) into a dedicated app, the server receives and records this data.

[0678] Step 4:

[0679] The server also periodically analyzes the audio data to detect when the baby wakes up crying.

[0680] Step 5:

[0681] If crying is detected, the server immediately sends a push notification to the user's smartphone.

[0682] User

[0683] Step 6:

[0684] The user sees the push notification and goes to check on the baby.

[0685] 3. Toilet training function

[0686] Device (user's smartphone)

[0687] Step 1:

[0688] The user enters the amount of time their baby uses the toilet each day into the app.

[0689] server

[0690] Step 2:

[0691] The server receives, records and analyzes the input toilet data.

[0692] Step 3:

[0693] Based on the analyzed data, the next time to use the toilet is predicted.

[0694] Step 4:

[0695] When the next toilet visit is approaching, the server will send a push notification to the user's smartphone.

[0696] User

[0697] Step 5:

[0698] The user checks the push notification and takes the baby to the toilet at the appropriate time.

[0699] ---

[0700] The above processing steps realize a system in which the baby monitor function, sleep training function, and toilet training function operate seamlessly, reducing the burden on parents.

[0701] Example 1

[0702] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0703] Conventional baby monitoring systems require expensive dedicated equipment and lack versatility. They also struggle to detect abnormalities in real time and respond immediately, making it difficult to reduce the burden of childcare. Furthermore, few systems offer consistent support for infant sleep and toilet training.

[0704] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0705] In this invention, the server uses an unused mobile device as a monitoring device and includes means for transmitting video and audio data from the monitoring device to a cloud computing environment, means for analyzing the video and audio data to detect abnormalities, and means for sending a push notification to the user's mobile communication device and automatically playing an audio message from the monitoring device when the abnormality is detected, thereby reducing the burden of childcare and enabling real-time detection and immediate response to abnormalities.

[0706] A "disused mobile device" is a portable electronic device that can continue to operate even after the user has finished normal use, and can be used for dedicated purposes by installing a specific application.

[0707] The "monitoring device" is a device that uses a camera and microphone to capture the baby's movements in real time and transmits the data to a cloud computing environment.

[0708] A "cloud computing environment" is a network of servers that provide computing resources and storage via the Internet, and analyzes and records the data received.

[0709] "Video and audio data" means visual and audio information captured from a monitoring device, including the baby's movements, crying, etc.

[0710] "Analysis" is the process of analyzing the transmitted video and audio data to detect anomalies or specific patterns.

[0711] "Detecting abnormalities" means determining whether there is any unusual behavior or risk in the baby's movements or voice.

[0712] "Push notifications" are a technology that allows smartphone apps to notify users of information in real time, and are used to issue warnings and alerts.

[0713] "Automatically play voice message" is a function that automatically plays a pre-set voice warning from the monitoring device when an abnormality is detected.

[0714] This invention is a system that provides infant monitoring, sleep training (hereafter referred to as "Nentore"), and toilet training (hereafter referred to as "Toi-train") through a single application. The main components of this system are an unused mobile device (monitoring device), the user's mobile communication device, and a cloud computing environment. Details of each function and an embodiment of the system are described below.

[0715] 1. Monitoring function

[0716] Terminal (monitoring device):

[0717] Abandoned mobile devices are used as surveillance devices, with applications installed that use the camera and microphone to capture real-time video and audio, and this data is transmitted to a cloud computing environment.

[0718] server:

[0719] The system receives video and audio data and analyzes it in real time. Specifically, it uses machine learning and computer vision technology to analyze the video data and detect abnormalities in the baby's movements and posture. It also uses voice recognition AI to detect crying and other abnormal sounds. If an abnormality is detected, the server sends a push notification to the user's mobile communication device and instructs the monitoring device to play an audio warning.

[0720] User:

[0721] Users can check the baby's condition in real time from their mobile communication device and take immediate action if necessary.

[0722] Examples:

[0723] While the baby is sleeping, the monitoring device uses a camera and microphone to transmit video and audio to a cloud computing environment. Computer vision technology and voice recognition AI detect when the baby kicks the covers or cries. In response, a push notification is sent to the user's mobile device stating, "The baby kicked the covers," and the monitoring device plays an audio warning saying, "Danger! Stop!"

[0724] 2. Sleep training function

[0725] Terminal (monitoring device):

[0726] Before your baby falls asleep, the monitor plays specific sleep music, and even after the baby falls asleep, the sound sensor detects any crying that may occur.

[0727] server:

[0728] The system receives and analyzes the baby's sleep data sent from the user's mobile communication device, and if it determines that the baby has woken up, it sends a push notification to the user's mobile communication device.

[0729] User:

[0730] Users enter the times their baby goes to sleep and wakes up into the application. This data is sent to a server for analysis. If the baby starts crying, the user receives a push notification on their mobile device, allowing them to take action.

[0731] Examples:

[0732] The user turns on sleep training mode in the application to help the baby fall asleep independently. The monitoring device starts playing selected sleep music. If the baby wakes up and cries during the night, the user's mobile device receives a notification that "Baby is crying."

[0733] 3. Toilet training function

[0734] Device (user's mobile communication device):

[0735] Users input the amount of time their baby spends using the toilet each day, which allows the cloud computing environment to analyze the intervals and timing of toilet visits.

[0736] server:

[0737] Based on the input data, the system predicts when the next toilet visit will be made, and periodically sends a push notification to the user's mobile device to let them know when it is time to go to the toilet next.

[0738] User:

[0739] Users will receive notifications so they can take their baby to the toilet at the right time, improving the success rate of potty training.

[0740] Examples:

[0741] The user inputs the time their baby uses the toilet each day into the application, and the server then predicts the next time the baby will use the toilet and sends a notification to the user's mobile device at the appropriate time.

[0742] Example prompt sentence:

[0743] "Please explain the specific processing steps of a system that uses computer vision and voice recognition to detect abnormalities in a baby monitoring function and send a push notification to the user's mobile communication device. Also, please provide a specific example of how the cloud computing environment and the monitoring device work together."

[0744] The system reduces the burden of childcare and provides a safe and healthy environment for infants. It includes real-time monitoring, sleep training, and toilet training functions, and is realized by linking with a cloud computing environment and making effective use of unused mobile devices.

[0745] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0746] Monitoring function

[0747] Step 1:

[0748] Data capture by terminal (monitoring device)

[0749] Input: An unused mobile device with the application installed, camera and microphone can act as input devices.

[0750] How it works: Uses a camera and microphone to capture video and audio of your baby in real time.

[0751] Output: Captured video and audio data.

[0752] Step 2:

[0753] Data transmission from the terminal (monitoring device) to the cloud computing environment

[0754] Input: Captured video and audio data.

[0755] How it works: Capture data is sent in real time to a cloud computing environment.

[0756] Output: Video and audio data received by the cloud computing environment.

[0757] Step 3:

[0758] Data analysis by server

[0759] Input: Video and audio data received by the cloud computing environment.

[0760] How it works: It uses computer vision technology and voice recognition AI to analyze incoming data and detect abnormalities such as baby movement or crying.

[0761] Output: Analysis result (normal / abnormal).

[0762] Step 4:

[0763] Server sends push notification and audio alert instructions

[0764] Input: When an anomaly is detected as a result of data analysis.

[0765] What it does: Sends a push notification to the user's mobile device and simultaneously instructs the monitoring device to play an audio alert.

[0766] Output: Push notification to user's mobile device and playback of audio message from monitoring device.

[0767] Step 5:

[0768] User notification and response

[0769] Input: A push notification sent to the user's mobile device.

[0770] Action: Check push notifications and rush to your baby if necessary.

[0771] Output: Baby safety check and problem solving.

[0772] Sleep training function

[0773] Step 1:

[0774] Playing sleep music on a terminal (monitoring device)

[0775] Input: Sleep music selection based on user preferences.

[0776] What it does: The monitoring device plays the selected sleep music.

[0777] Output: Sleep music played.

[0778] Step 2:

[0779] Detecting baby's wake-up sound using a terminal (monitoring device)

[0780] Input: Baby crying and movement sounds.

[0781] How it works: The sound sensor detects your baby's crying or any abnormal sounds.

[0782] Output: Detected sound data.

[0783] Step 3:

[0784] Entering sleep data from the user's mobile communication device

[0785] Input: Baby's sleep and wake times.

[0786] Action: A user enters data through an application.

[0787] Output: The input sleep data.

[0788] Step 4:

[0789] Sleep data analysis and notification by server

[0790] Input: Sleep data and sound sensor data sent from the user's mobile communication device.

[0791] What it does: Detects when the baby wakes up and sends a push notification to the user's mobile device.

[0792] Output: A push notification to the user's mobile device.

[0793] Toilet training function

[0794] Step 1:

[0795] Input of toilet usage time from user's mobile communication device

[0796] Enter: your baby's potty time.

[0797] Action: A user enters data through an application.

[0798] Output: The input toilet data.

[0799] Step 2:

[0800] Server-based analysis and prediction of toilet data

[0801] Input: Restroom data sent from the user's mobile communication device.

[0802] How it works: A cloud computing environment analyzes data and predicts when the next toilet visit will occur.

[0803] Output: Prediction of next toilet timing.

[0804] Step 3:

[0805] Server-driven push notifications

[0806] Input: Predicted next toilet time.

[0807] What it does: Sends a timely push notification to the user's mobile device saying "Next bathroom break."

[0808] Output: A push notification to the user's mobile device.

[0809] As a result, this system reduces the burden of childcare and provides a safe and healthy environment for babies. Each function works in conjunction with each other to provide real-time monitoring, effective sleep training, and proper toilet training.

[0810] (Application example 1)

[0811] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0812] The challenge is to provide a highly efficient, low-cost monitoring system that ensures safety in the home and responds quickly to abnormalities. In particular, there is a need for a system that can effectively utilize unused terminals and safely monitor the home even when the user is away.

[0813] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0814] In this invention, the server includes means for using an unused terminal as a monitoring device and transmitting video and audio data from the monitoring device to the server, means for analyzing the video and audio data to detect an abnormality, means for sending a push notification to the user's terminal and automatically playing an audio message from the monitoring device when the abnormality is detected, and means for monitoring the user's return home status and sending a confirmation notification if the user does not return home at the scheduled time. This makes it possible to maintain home safety even when the user is away and to respond quickly when an abnormality occurs.

[0815] A "terminal no longer in use" is an electronic device such as a smartphone or tablet that the user used in the past but is no longer using.

[0816] A "monitoring device" is a device that reuses unused terminals to collect video and audio data and detect abnormalities.

[0817] "Video and audio data" means real-time video and audio information captured by a surveillance device.

[0818] "Server" means a computer system or cloud-based system for receiving and analyzing video and audio data.

[0819] "Abnormality" refers to suspicious movements or sounds that are different from normal as a result of the surveillance device's analysis of video and audio data.

[0820] A "push notification" is a real-time notification message sent from a server to a user's device.

[0821] A "voice message" is a sound such as a warning sound that is played from a monitoring device when an abnormality is detected.

[0822] A "user's terminal" is a mobile device such as a smartphone or tablet that a user uses on a daily basis.

[0823] The "return home status" refers to whether the user has returned home at the scheduled time designated by the user.

[0824] A "confirmation notice" is a confirmation message sent to the user's terminal if the user does not return home at the scheduled time.

[0825] The "warning sound" is a sound that warns a suspicious person or a user when an abnormality is detected.

[0826] "Video data" refers to digital information of video captured by a monitoring device, which is recorded for abnormality detection and transmitted to a server.

[0827] The present invention provides a system for strengthening home security by utilizing unused terminals as monitoring devices. Specific embodiments for implementing this system will be described below.

[0828] composition

[0829] Terminal (monitoring device):

[0830] Abandoned smartphones and tablets are used as surveillance devices equipped with cameras and microphones to capture real-time video and audio.

[0831] server:

[0832] This is a computer system for receiving and analyzing video and audio data. The server analyzes the data using software libraries such as OpenCV and voice recognition AI.

[0833] On the user's device:

[0834] These are mobile devices that users use on a daily basis, such as smartphones and tablets. A dedicated application for this system is installed on the user's device, allowing them to receive notifications in real time.

[0835] Processing flow

[0836] 1. Video and audio data capture:

[0837] The monitoring device captures video and audio data in real time and transmits the data to a server.

[0838] 2. Data Analysis:

[0839] The server analyzes the video and audio data it receives. Specifically, it uses OpenCV to detect video anomalies, such as when a suspicious person is captured on camera. It also uses voice recognition AI to detect abnormal sounds.

[0840] 3. Push notifications and sound alerts:

[0841] If the server detects an abnormality, it sends a push notification to the user's device and simultaneously plays an alarm sound from the monitoring device.

[0842] 4. Monitoring when users return home:

[0843] The server monitors whether the user has returned home, and if the user has not returned home at the scheduled time, sends a confirmation notice to the user's terminal.

[0844] Specific examples

[0845] For example, a housewife places an unused smartphone at the front door while she goes out shopping. This monitoring device captures video and audio in real time and sends them to a server. The server analyzes the video data, and if it detects a suspicious person at the front door, it sends a push notification to the user's device saying, "A suspicious person has been detected at the front door!" At the same time, the monitoring device sounds an alarm to warn the suspicious person. Furthermore, the server monitors whether the user has returned home at the scheduled time, and if the user has not returned home at the scheduled time, it sends a confirmation notification to the user's device asking, "Did you return home as planned?"

[0846] Prompt Sentence Examples

[0847] "Functional description of this application: We want to create a smart home security system that captures video and audio in real time and detects suspicious activity. Specifically, we will use unused smartphones as security devices, and if an abnormality is detected, the system will upload the data to a cloud server, send a push notification to the user, and play an alarm."

[0848] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0849] Step 1:

[0850] The monitoring device captures video and audio data in real time and sends it to a server. The input includes real-time video and audio captured by the camera and microphone of the monitoring device. The output is sent to the server via the Internet. The specific operation of the monitoring device is to capture video with the camera and record audio with the microphone, and then transfer these as digital data to the server.

[0851] Step 2:

[0852] The server analyzes the video and audio data it receives. The input includes real-time video and audio data sent from the surveillance equipment. The server uses OpenCV to analyze the video data and performs facial recognition and motion detection to detect suspicious activity. It also uses voice recognition AI to detect abnormal sounds (for example, the sound of breaking glass or screaming). The output is alert information if suspicious activity or audio is detected. Specifically, the server analyzes the video data frame by frame and the audio data as a sound waveform.

[0853] Step 3:

[0854] If the server detects an anomaly, it sends a push notification to the user's device. The input includes the alert information generated by the server. The output is a push notification sent to the user's device stating "An anomaly has been detected." The push notification includes detailed information such as the type of anomaly and the time it was detected. Specifically, the server sends a notification to the user's smartphone through a push notification service.

[0855] Step 4:

[0856] If an abnormality is detected, the monitoring device automatically plays an alarm sound. The input includes the alert information sent from the server. As an output, the monitoring device plays an alarm sound to warn suspicious individuals on-site and those in the vicinity. Specifically, the monitoring device uses its built-in speaker to play a pre-set alarm sound file.

[0857] Step 5:

[0858] It monitors the user's return home status and sends a confirmation notification if the user does not return home at the scheduled time. The input includes the user's pre-set information and the server's timestamp information. The output is a confirmation notification sent to the user's smartphone asking, "Did you return home as scheduled?" Specifically, the server checks the user's return home information at the specified time and checks for any abnormalities. It then automatically sends a confirmation notification.

[0859] As described above, this system performs specific data processing and calculations at each step, and when an abnormality is detected, it notifies the user with an alert and even plays a warning sound to ensure the safety of the home.

[0860] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0861] This invention combines a system that provides functions for monitoring infants, sleep training, and toilet training with an emotion engine that recognizes the user's emotions. The main components of this system are an unused device (baby monitor), the user's smartphone, a cloud server, and the emotion engine. Below, we will explain an embodiment in which each function and emotion engine are combined.

[0862] 1. Monitoring function

[0863] Device (baby monitor)

[0864] A dedicated app is installed on an unused smartphone, and the camera and microphone are used to capture real-time video and audio, which is then sent to a server.

[0865] server

[0866] It receives video and audio data and analyzes it using computer vision and voice recognition AI. It detects baby movements and cries, and sends push notifications to the user's device if there is danger. It also instructs the baby monitor to automatically issue an audio alert.

[0867] Emotion Engine

[0868] It analyzes the user's emotions from voice data and input data to determine whether the user is tired, stressed, relaxed, etc. Based on this, it adjusts the content and timing of push notifications.

[0869] User

[0870] Users can check their baby's condition in real time from their smartphone and can take prompt action when notified.

[0871] Specific examples

[0872] While the baby is sleeping, the baby monitor captures video and audio and sends them to the server. The server detects when the baby kicks off the covers and determines whether there is any danger. At the same time, the emotion engine analyzes the user's situation. If it determines that the user is tired, it sends a push notification to reduce the burden on the parent by softening the tone of the notification or by eliminating redundant information to make it concise.

[0873] 2. Sleep training function

[0874] Device (baby monitor)

[0875] When you're getting your baby ready for bed, you can set the baby monitor to sleep mode and play specific sleep music. Even after your baby falls asleep, the sound sensor will still detect crying.

[0876] server

[0877] It receives the baby's sleep data sent from the user's smartphone and records it in a database. If crying is detected, it sends a push notification to the user's smartphone.

[0878] Emotion Engine

[0879] The system analyzes the user's emotions and adjusts the timing and content of sleep notifications. For example, if the user is feeling stressed, it can send a relaxing message.

[0880] User

[0881] Users can enter the times their baby goes to sleep and wakes up into the app, and if crying is detected, they will receive a push notification and be able to take action quickly.

[0882] Specific examples

[0883] To get the baby to sleep alone, the user turns on the sleep training mode. The baby monitor plays sleep music, and the baby falls asleep. When the baby starts crying in the middle of the night, the user's smartphone receives a notification saying "Your baby is crying." However, if the user is feeling stressed, the notification content is adjusted to say something like "It's okay, your baby is just crying a little."

[0884] 3. Toilet training function

[0885] Device (user's smartphone)

[0886] The user enters the amount of time their baby uses the toilet each day into the app.

[0887] server

[0888] The system receives and analyzes the entered toilet data, predicts the next toilet time, and sends a push notification to the user's smartphone.

[0889] Emotion Engine

[0890] The system analyzes the user's emotions and adjusts the toilet timing notification accordingly. For example, if the user is busy, the notification can be delayed a little.

[0891] User

[0892] Users who receive the notification can take their baby to the toilet at the appropriate time.

[0893] Specific examples

[0894] The user inputs the time their baby goes to the toilet into the app every day. The server predicts the next time the baby will go to the toilet, and if the emotion engine determines that the user is busy, it adjusts the notification timing. For example, the user might receive a notification saying, "The next time to go to the toilet is approaching, but you still have a little time."

[0895] This allows the system to take into account the parent's state and emotions, providing more effective and flexible childcare support. By combining real-time monitoring, sleep training, and toilet training functions with the emotion engine, the system further reduces the burden on parents and supports the safety and growth of infants.

[0896] The processing flow will be explained below.

[0897] 1. Monitoring function

[0898] Device (baby monitor)

[0899] Step 1:

[0900] Install the dedicated app on the device that will serve as the baby monitor and enable the camera and microphone.

[0901] Step 2:

[0902] The device captures video and audio data in real time and transmits the data to a server.

[0903] server

[0904] Step 3:

[0905] The server analyzes the received video and audio data, and uses computer vision to analyze the video data and detect any abnormal movements of the baby.

[0906] Step 4:

[0907] At the same time, voice recognition AI is used to analyze audio data and detect crying and abnormal sounds.

[0908] Step 5:

[0909] If a risk is detected, the server sends a push notification to the user's device.

[0910] Step 6:

[0911] The server sends instructions to the baby monitor to play an automatic audio warning (e.g., "Danger! Stop!").

[0912] User

[0913] Step 7:

[0914] Users can check push notifications and monitor their baby's condition in real time.

[0915] 2. Sleep training function

[0916] Device (baby monitor)

[0917] Step 1:

[0918] The user prepares the baby for sleep and sets the baby monitor to sleep mode.

[0919] Step 2:

[0920] The baby monitor plays selected sleep music.

[0921] server

[0922] Step 3:

[0923] The user enters the baby's sleep data (time they went to sleep, time they woke up) into a dedicated app.

[0924] Step 4:

[0925] The server records the entered sleep data in a database.

[0926] Step 5:

[0927] If the baby wakes up crying, the server analyzes the audio data and sends a push notification if crying is detected.

[0928] Emotion Engine

[0929] Step 6:

[0930] The emotion engine analyzes the user's emotional state and adjusts the content and timing of sleep notifications.

[0931] User

[0932] Step 7:

[0933] The user sees the push notification and goes to check on the baby.

[0934] 3. Toilet training function

[0935] Device (user's smartphone)

[0936] Step 1:

[0937] The user enters the amount of time their baby uses the toilet each day into the app.

[0938] server

[0939] Step 2:

[0940] The server receives and records the input toilet data.

[0941] Step 3:

[0942] The server analyzes the toilet data and predicts the next time to use the toilet.

[0943] Step 4:

[0944] A push notification will be sent to the user's smartphone when the next toilet visit is approaching.

[0945] Emotion Engine

[0946] Step 5:

[0947] The emotion engine analyzes the user's emotional state and adjusts the content and timing of toilet notification.

[0948] User

[0949] Step 6:

[0950] The user receives the notification and takes the baby to the toilet at the appropriate time.

[0951] ---

[0952] These processing steps realize a system in which the baby monitor function, sleep training function, and toilet training function work in conjunction with the emotion engine to provide appropriate support according to the parent's emotional state.

[0953] Example 2

[0954] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0955] While conventional baby monitor systems have functions for monitoring infants and training (sleep and toileting), they lack support that takes into account the emotions and state of the parents, which often leaves parents feeling tired and stressed, and the burden of childcare remains.

[0956] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0957] In this invention, the server includes means for using an unused device as a monitoring device and transmitting visual and audio data from the monitoring device to a central processing unit, means for analyzing the visual and audio data to detect danger, means for sending a push notification to the user's device and automatically playing an audio message from the monitoring device when the danger is detected, and means for evaluating the user's emotional state using an emotion analysis engine and adjusting the content and timing of the push notification based on the evaluation result. This enables notifications and support that take parents' emotions and states into consideration, thereby reducing the burden of child-rearing.

[0958] "Device" refers to a device or terminal held by a user, including devices used as monitor devices and smartphones used by users.

[0959] "Monitoring devices" refers to devices used to monitor and check on infants, including old smartphones.

[0960] "Visual Data" refers to real-time video information captured by the camera of a monitoring device.

[0961] "Acoustic Data" refers to real-time audio information collected by the microphone of a monitoring device.

[0962] "Central processing unit" refers to a device or system, such as a cloud or server, that receives and analyzes data sent from a monitor device.

[0963] "Danger" refers to abnormal behavior or a situation in an infant, and includes cases where the infant is deemed to be in danger.

[0964] "Push Notification" refers to alert messages or notifications sent in real time from a central processing unit to a user's device.

[0965] "Audible message" refers to a voice or sound alert that is automatically played from a monitoring device.

[0966] "Emotion analysis engine" refers to an algorithm or system that analyzes a user's voice or input data to assess the user's emotional state.

[0967] "Sleep Data" refers to information relating to an infant's sleep patterns and duration, including data recorded and analyzed by a central processing unit.

[0968] "Toilet data" refers to information about an infant's daily toilet use time and timing, including data that is analyzed by a central processing unit.

[0969] This invention is a system that combines a system that provides functions for monitoring infants, sleep training, and toilet training with an emotion analysis engine that recognizes user emotions. The main components are an unused device (monitoring device), the user's smartphone, a cloud server, and the emotion analysis engine. Below, we will explain in detail an embodiment in which each function is combined with the emotion analysis engine.

[0970] Monitoring function

[0971] Terminal (monitor device)

[0972] The monitoring device is an unused smartphone, which is then installed with a dedicated app. The app uses the smartphone's camera and microphone to capture real-time visual and acoustic data, which is then transmitted via the internet to a central processing unit located in the cloud.

[0973] Server (cloud server)

[0974] The cloud server receives visual and acoustic data sent from the monitoring device. This data is analyzed using computer vision algorithms (e.g., OpenCV) and speech recognition AI (e.g., speech recognition API). As a result of the analysis, the cloud server detects the infant's movements and cries and detects danger based on this. If danger is detected, a push notification is sent to the user's smartphone and an audio message is automatically played on the monitoring device.

[0975] Sentiment Analysis Engine

[0976] The emotion analysis engine analyzes the user's voice and input data to determine whether they are tired, stressed, or relaxed, and adjusts the content and timing of push notifications accordingly.

[0977] User

[0978] Users can check the condition of their baby in real time from their smartphone and can take prompt action when notified. For example, if a notification is received that a baby is in danger, they can rush to the scene immediately.

[0979] Specific examples

[0980] While the baby is sleeping, the monitor device captures video and audio and sends them to a cloud server. The cloud server detects when the baby kicks off the covers and determines whether there is any danger. At the same time, an emotion analysis engine analyzes the user's situation. If the user is tired, a push notification will be sent to soften the tone of the notification and simplify the message to reduce the burden on parents.

[0981] Prompt Sentence Examples

[0982] "Baby danger detection. Please suggest a way to soften the tone of the notification if the user is tired."

[0983] Sleep training function

[0984] Terminal (monitor device)

[0985] The monitoring device is set to sleep mode when preparing the baby for sleep, which plays specific sleep-inducing music and has sound sensors that detect crying even after the baby has fallen asleep.

[0986] Server (cloud server)

[0987] It receives the baby's sleep data sent from the user's smartphone and records it in a database. If the baby cries, it sends a push notification to the user's smartphone.

[0988] Sentiment Analysis Engine

[0989] The emotion analysis engine analyzes the user's emotions and adjusts the timing and content of sleep notifications accordingly. For example, if the user is feeling stressed, it will send a relaxing message.

[0990] User

[0991] Users enter the times their baby goes to sleep and wakes up into the app, and if crying is detected, they receive a push notification and can take action.

[0992] Specific examples

[0993] When the user turns on the sleep training mode, the monitor device plays sleep music to lull the baby to sleep. If crying is detected in the middle of the night, the cloud server sends a notification to the user's smartphone saying, "Your baby is crying." However, if the user is feeling stressed, the notification content will be adjusted to say, "It's okay, your baby is just crying a little."

[0994] Prompt Sentence Examples

[0995] "When detecting a baby's cry, create an appropriate relaxation message if the user is feeling stressed."

[0996] Toilet training function

[0997] Device (user's smartphone)

[0998] Users enter their baby's daily toilet use times into the app.

[0999] Server (cloud server)

[1000] The server receives and analyzes the input toilet data, and based on the analysis results, predicts the next toilet time and sends a push notification to the user's smartphone.

[1001] Sentiment Analysis Engine

[1002] The emotion analysis engine analyzes the user's emotions and adjusts the toilet timing notification accordingly. If the user is busy, the notification can be delayed.

[1003] User

[1004] When the user receives the notification, they can take the baby to the toilet at the appropriate time.

[1005] Specific examples

[1006] The user inputs the time their baby goes to the toilet into the app every day. The cloud server predicts when the next toilet visit will be, and if the emotion analysis engine determines that the user is busy, the notification timing will be adjusted. For example, the user might receive a notification saying, "The next toilet visit is approaching, but you still have a little time."

[1007] Prompt Sentence Examples

[1008] "Please suggest a way to predict when a baby needs to go to the toilet and adjust the notification timing based on the user's emotions."

[1009] With these functions, the system takes into account the emotions and state of the parent, providing more effective and flexible childcare support. By combining the monitoring, sleep training, and toilet training functions with the emotion analysis engine, the system further reduces the burden on parents and supports the safety and growth of infants.

[1010] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1011] Monitoring function processing steps

[1012] Step 1:

[1013] Terminal (monitor device)

[1014] Input: Video and audio from a monitor device with a dedicated app installed, camera and microphone.

[1015] Specific operation: Launch the dedicated app for the monitor device.

[1016] Processing: The app activates the monitor device's camera and microphone to capture video and audio in real time.

[1017] Output: Captured visual and acoustic data.

[1018] Step 2:

[1019] Terminal (monitor device)

[1020] Input: Captured visual and acoustic data.

[1021] Specific operation: Real-time video and audio data is sent to a cloud server via the Internet.

[1022] Processing: The dedicated app initiates a network connection to send the data, compresses and encodes the data, and sends it to the cloud server.

[1023] Output: Visual and acoustic data sent to cloud server.

[1024] Step 3:

[1025] Server (cloud server)

[1026] Input: Visual and acoustic data sent from the monitor device.

[1027] Specific operation: The cloud server receives the data and records it in a log.

[1028] Processing: Visual data is analyzed with computer vision algorithms (e.g., OpenCV), and acoustic data is analyzed with speech recognition AI (e.g., speech recognition API).

[1029] Output: The results of the analysis include the detection of infant movements and crying sounds.

[1030] Step 4:

[1031] Server (cloud server)

[1032] Input: Analysis results of visual and acoustic data.

[1033] Specific behavior: Generates a push notification if a danger is detected.

[1034] Processing: The system assesses the level of danger based on the detection of the infant's movements and cries, and if it determines that there is danger, it sends a push notification to the user's smartphone and instructs the monitoring device to send an acoustic message.

[1035] Output: Push notification sent to the user's smartphone.

[1036] Step 5:

[1037] Sentiment Analysis Engine

[1038] Input: User voice and typing data.

[1039] Specific operation: The sentiment analysis engine acquires the data and begins analysis.

[1040] Processing: A voice emotion analysis algorithm is used to determine the user's emotional state.

[1041] Output: User's emotional state as a result of emotion analysis (fatigue, stress, relaxed, etc.).

[1042] Step 6:

[1043] Server (cloud server)

[1044] Input: Sentiment analysis results from the sentiment analysis engine.

[1045] Specific operation: The cloud server adjusts the notification content and timing.

[1046] Processing: Based on the results of sentiment analysis, adjust the content (message tone, redundancy of information, etc.) and timing of push notifications.

[1047] Output: The adjusted push notification content.

[1048] Step 7:

[1049] User (smartphone)

[1050] Input: Push notification from cloud server.

[1051] Specific behavior: The user's smartphone displays the notification content.

[1052] Processing: The user confirms the push notification and sees and hears the baby in real time.

[1053] Output: User's response actions (checking and responding to the infant, etc.).

[1054] Processing steps for sleep training functions

[1055] Step 1:

[1056] Terminal (monitor device)

[1057] Input: Sleep training mode setting instruction from user.

[1058] Specific operation: The user sets the sleep training mode in the app.

[1059] Processing: Play specific sleep music from the monitor device depending on the sleep training mode.

[1060] Output: Sleep music is playing.

[1061] Step 2:

[1062] Terminal (monitor device)

[1063] Input: Audio capture by monitor device.

[1064] Specific function: The sound sensor captures sound and detects the baby's crying.

[1065] Processing: Send the crying data to the cloud server.

[1066] Output: Crying data sent to the cloud server.

[1067] Step 3:

[1068] Server (cloud server)

[1069] Input: Sleep data sent from the user's smartphone.

[1070] Specific operation: The server records it in the database.

[1071] Processing: The data is stored to analyze your baby's sleep patterns.

[1072] Output: Recorded sleep data.

[1073] Step 4:

[1074] Server (cloud server)

[1075] Input: Crying data analysis results.

[1076] Specific behavior: Generates a push notification.

[1077] Action: If crying is detected, a push notification is sent to the user's smartphone.

[1078] Output: Push notification sent to the user's smartphone.

[1079] Step 5:

[1080] Sentiment Analysis Engine

[1081] Input: User voice and typing data.

[1082] Specific operation: The sentiment analysis engine acquires the data and begins analysis.

[1083] Processing: A voice emotion analysis algorithm is used to determine the user's emotional state.

[1084] Output: User's emotional state (stressed, relaxed, etc.) as a result of emotion analysis.

[1085] Step 6:

[1086] Server (cloud server)

[1087] Input: Sentiment analysis results from the sentiment analysis engine.

[1088] Specific operation: The cloud server adjusts the notification content and timing.

[1089] Processing: Based on the results of sentiment analysis, adjust the content (message tone, redundancy of information, etc.) and timing of push notifications.

[1090] Output: The adjusted push notification content.

[1091] Step 7:

[1092] User (smartphone)

[1093] Input: Push notification from cloud server.

[1094] Specific operation: The user's smartphone displays the notification and checks the sleep data.

[1095] Action: The user checks the push notification and checks the baby's status in real time.

[1096] Output: User's response actions (responding to crying, recording sleep data, etc.).

[1097] Toilet training function processing steps

[1098] Step 1:

[1099] Device (user's smartphone)

[1100] Input: Your baby's daily toilet use timings.

[1101] Specific operation: The user inputs the timing of toilet use into the app.

[1102] Processing: The entered data is recorded by the app and sent to the cloud server.

[1103] Output: Toilet data sent to the cloud server.

[1104] Step 2:

[1105] Server (cloud server)

[1106] Input: Toilet data sent from the user's smartphone.

[1107] Specific operation: The server receives the data and begins analyzing it.

[1108] Processing: Predict the next toilet timing based on accumulated toilet data.

[1109] Output: Predicted next toilet time.

[1110] Step 3:

[1111] Server (cloud server)

[1112] Input: Predicted next toilet time.

[1113] Specific behavior: Generates a push notification to the user's smartphone.

[1114] Processing: Based on the prediction results, a message notifying the user of the next toilet visit is created and sent to the user's smartphone.

[1115] Output: Push notification sent to the user's smartphone.

[1116] Step 4:

[1117] Sentiment Analysis Engine

[1118] Input: User voice and typing data.

[1119] Specific operation: The sentiment analysis engine acquires the data and begins analysis.

[1120] Processing: A voice emotion analysis algorithm is used to determine the user's emotional state.

[1121] Output: User's emotional state (tired, stressed, busy, etc.) as a result of sentiment analysis.

[1122] Step 5:

[1123] Server (cloud server)

[1124] Input: Sentiment analysis results from the sentiment analysis engine.

[1125] Specific operation: The cloud server adjusts the notification content and timing.

[1126] Processing: Based on the results of sentiment analysis, adjust the content (message tone, redundancy of information, etc.) and timing of push notifications.

[1127] Output: The adjusted push notification content.

[1128] Step 6:

[1129] User (smartphone)

[1130] Input: Push notification from cloud server.

[1131] Specific behavior: The user's smartphone displays the notification content.

[1132] Action: The user checks the push notification and knows when to go to the bathroom in real time.

[1133] Output: Toilet guidance action by the user.

[1134] In this way, the system provides childcare support that takes into account the parent's emotions and state through each processing step, improving the effectiveness of supervision, sleep training, and toilet training.

[1135] (Application example 2)

[1136] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1137] Efficient and safe management of the status of workers and machinery in factories is important for improving the working environment and increasing productivity. However, existing monitoring systems are limited to simply detecting and reporting abnormalities and do not take into consideration the emotional state and workload of users. This can increase stress and burden on managers and delay appropriate responses. The present invention aims to solve these problems and provide a more convenient monitoring and management system.

[1138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1139] In this invention, the server includes: a means for using an unused terminal as a monitoring device and transmitting video and audio data from the monitoring device to the server; a means for analyzing the video and audio data to detect an abnormality; a means for sending a push notification to the user's terminal and automatically playing an audio message from the monitoring device when the abnormality is detected; a means for analyzing the user's emotions and adjusting the content and timing of the notification; a means for analyzing the worker's status from the monitoring device and inputting the worker's status data into the user's terminal; a means for recording the status data in the server and sending a push notification to the user's terminal when the worker causes an abnormality; and a means for transmitting the adjusted notification to the user's terminal. This enables appropriate notification based on the user's emotional state at the same time as detecting an abnormality, thereby improving the safety and efficiency of the work environment.

[1140] An "obsolete device" is an electronic device that was previously in use but is no longer in use.

[1141] "Surveillance equipment" means equipment installed to monitor a specific area or object and capable of capturing video and audio data.

[1142] "Video and audio data" refers to digital data containing visual and audio information captured using a camera and microphone.

[1143] A "server" is a computer system that stores, processes, and manages data over a network.

[1144] An "anomaly" is an event or circumstance that deviates from normal operating or environmental conditions and requires immediate attention.

[1145] A "push notification" is a message that is automatically sent to a user's device when a specific event or state change occurs.

[1146] A "voice message" is an audio content that is recorded as audio data and played back.

[1147] "Emotion analysis" is the process of identifying a user's emotional state from speech or other input data.

[1148] "Adjusting notification content and timing" means changing the content of the message sent to the user and the timing at which the message is sent depending on the state or situation of the user.

[1149] "Worker status data" is digital data that indicates the worker's behavior, location, health condition, etc.

[1150] "Analysis" is the process of breaking down and evaluating data to extract useful information.

[1151] "Recording on the server" means storing the acquired data on the server so that it can be referenced or used later as needed.

[1152] "Adjusted notifications" are push notifications whose content or delivery timing has been changed based on the results of sentiment analysis.

[1153] MODE FOR CARRYING OUT THE INVENTION

[1154] A system for implementing the present invention includes the following configuration.

[1155] First, we use unused devices as surveillance devices. These devices have built-in cameras and microphones that capture video and audio data in real time, which is then transmitted to a cloud server via a network.

[1156] The cloud server uses computer vision technology and a voice recognition engine to analyze the received video and audio data, utilizing open source technologies such as OpenCV and TensorFlow, making it possible to detect abnormalities in workers and machinery.

[1157] If an abnormality is detected, the server sends a push notification to the user's device. This notification is displayed on the user's smartphone or PC. In addition, the monitoring device automatically plays a voice message regarding the abnormality. This voice message is pre-recorded and includes details of the abnormality and instructions on how to respond.

[1158] To analyze the user's emotional state, the cloud server is equipped with an emotion analysis engine. This engine analyzes voice data and other input data (e.g., text messages entered by the user) to determine the user's emotional state. For emotion analysis, voice analysis technologies such as Amazon Polly and Google Cloud Speech-to-Text can be used.

[1159] The content and timing of notifications are tailored based on the user's emotional state: if the user is feeling stressed, notifications can be sent in a softer tone or in a more concise manner without unnecessary information.

[1160] The system also has the function of monitoring the status of workers. Data acquired from the monitoring device is recorded on a server, and if an abnormality occurs, an appropriate notification is sent to the user's device. This improves worker safety and productivity.

[1161] Examples:

[1162] For example, if a monitoring device installed in a factory detects worker movements or a machine malfunction, the video and audio data is first sent to a cloud server. The cloud server analyzes the data and identifies the malfunction. At the same time, an emotion analysis engine analyzes the user's stress and fatigue levels and adjusts the notification content accordingly. The user's device may receive a notification such as, "An abnormality has occurred on machine A. Please check the details." However, if it is determined that the user is feeling stressed, a softer-toned notification may be sent, such as, "There is a minor malfunction on machine A. Please check."

[1163] Prompt for the generative AI model:

[1164] For example, the following might be a prompt to input to a generative AI model:

[1165] Please explain the functions of a system that supports putting a baby to sleep and toilet training. For example, a baby monitor captures video and audio while the baby is sleeping and sends them to a server. If the server detects danger and determines that the user is tired, it will send a push notification to reduce the burden on the parent by softening the tone of the notification or simplifying it by removing redundant information.

[1166] This allows the present invention to provide an efficient and safe monitoring and management system that takes into account the emotional state and workload of the user.

[1167] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1168] Processing Steps

[1169] Step 1: Booting the device and capturing data

[1170] The device starts up and uses the camera and microphone to capture video and audio data in real time. This data is acquired through the camera sensor and microphone. The input data is video frames and audio waveform data, which are converted and prepared in digital format.

[1171] Step 2: Send data

[1172] The device sends the captured video and audio data to the cloud server. The data is transmitted using a network communication protocol (e.g., HTTP or WebSocket). The input is the captured data, and the output is the status of the completion of data transmission to the server.

[1173] Step 3: Data analysis (server side)

[1174] The server analyzes the received data, using computer vision technology (e.g., OpenCV) for video data and a speech recognition engine (e.g., Google Cloud Speech-to-Text) for audio data. The input data are video frames and audio files sent from the device, and the output is anomaly detection information as the analysis result.

[1175] Step 4: Anomaly detection

[1176] The server detects anomalies based on the analysis results. If an anomaly is detected, it immediately starts a process to notify the user of that information. The input is the data analysis results, and the output is the anomaly determination result.

[1177] Step 5: Sentiment Analysis

[1178] The server analyzes the user's emotional state. To do this, it uses the voice data and text data from the user to identify the emotional state. Using an emotion analysis engine (e.g., Amazon Polly), the input data is the user's response and voice data, and the output is the user's emotional state.

[1179] Step 6: Adjust notification content and timing

[1180] The server adjusts the content and timing of notifications based on the emotion analysis results. If the user is feeling stressed, the notification content will be briefer and the tone will be softer. The input is the emotion analysis results, and the output is the adjusted content and timing of notifications.

[1181] Step 7: Send notification

[1182] The server sends the adjusted notification content to the user's device. This notification is sent as a push notification and displayed on the user's smartphone or PC. The input is the adjusted notification content, and the output is the notification sent to the user's device.

[1183] Step 8: Play a voice message when an error occurs

[1184] When an abnormality is detected, the monitoring device automatically plays back a voice message containing the details of the abnormality and instructions for how to respond. The input is the abnormality detection information, and the output is the playback of the voice message.

[1185] Examples:

[1186] For example, if a monitoring device installed in a factory detects worker movements or a machine malfunction, the video and audio data is first sent to a cloud server. The cloud server analyzes the data and identifies the malfunction. At the same time, an emotion analysis engine analyzes the user's stress and fatigue levels and adjusts the notification content accordingly. The user's device may receive a notification such as, "An abnormality has occurred on machine A. Please check the details." However, if it is determined that the user is feeling stressed, a softer-toned notification may be sent, such as, "There is a minor malfunction on machine A. Please check."

[1187] Example prompt for a generative AI model:

[1188] Please explain the functions of a system that supports putting a baby to sleep and toilet training. For example, a baby monitor captures video and audio while the baby is sleeping and sends them to a server. If the server detects danger and determines that the user is tired, it will send a push notification to reduce the burden on the parent by softening the tone of the notification or simplifying it by removing redundant information.

[1189] In this way, specific input data and output data are clearly defined in each processing step, and the system operates by processing data and performing calculations based on them.

[1190] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1191] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1192] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1193] [Third embodiment]

[1194] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1195] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1196] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1197] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1198] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1199] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1200] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1201] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1202] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1203] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1204] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1205] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1206] This invention is a system that provides functions for monitoring infants, sleep training, and toilet training in a single application. The main components of this system are an unused device (baby monitor), the user's smartphone, and a cloud server. Details of each function and their implementation are described below.

[1207] 1. Monitoring function

[1208] Device (baby monitor):

[1209] Use an old smartphone as a baby monitor by installing an app and using the camera and microphone to capture real-time video and audio, which is then sent to a server.

[1210] server:

[1211] It receives video and audio data and analyzes it in real time. Specifically, it uses computer vision to analyze the video data and detect abnormalities in the baby's movements and posture. It also uses voice recognition AI to detect crying and other abnormal sounds. If danger is detected, the server sends a push notification to the user's smartphone and instructs the baby monitor to automatically play an audio warning.

[1212] User:

[1213] Users can check on their baby's condition in real time from their smartphone, and can respond immediately if necessary.

[1214] Examples:

[1215] While the baby is sleeping, the baby monitor uses a camera and microphone to send video and audio to a server. Computer vision and voice recognition AI detect when the baby kicks the covers or cries. In this case, the user's smartphone receives a push notification saying "The baby kicked the covers," and the baby monitor plays an audio warning saying "Danger! Stop!"

[1216] 2. Sleep training function

[1217] Device (baby monitor):

[1218] Before your baby falls asleep, play specific sleep music from the baby monitor, and even after your baby falls asleep, the sound sensor will detect any crying that wakes them up.

[1219] server:

[1220] The system receives and analyzes the baby's sleep data sent from the user's smartphone, and if it determines that the baby has woken up, it sends a push notification to the user's smartphone.

[1221] User:

[1222] Users can enter the time their baby goes to sleep and wakes up into the app, and this data is sent to a server. If the baby starts crying, the user will receive a push notification on their smartphone so they can take action.

[1223] Examples:

[1224] To get the baby to sleep alone, the user turns on the sleep training mode. The baby monitor starts playing selected sleep music. When the baby wakes up and cries in the middle of the night, the user's smartphone receives a notification saying "Your baby is crying."

[1225] 3. Toilet training function

[1226] Device (user's smartphone):

[1227] Users input the number of times their baby uses the toilet each day, which allows the server to analyze the intervals and timing of toilet visits.

[1228] server:

[1229] Based on the input data, the next time to go to the toilet is predicted. A push notification is periodically sent to the user's smartphone to let them know when it is time to go to the toilet next.

[1230] User:

[1231] Users who receive the notification can take their baby to the toilet at the appropriate time, which improves the success rate of potty training.

[1232] Examples:

[1233] Users input the time their baby spends using the toilet into the app each day. Based on this, the app predicts when their baby will next go to the toilet, and sends a push notification to the user's smartphone at the appropriate time.

[1234] This system significantly reduces the burden of childcare on parents and provides an environment where infants can live safely and healthily.The real-time monitoring, sleep training, and toilet training functions are realized by linking with a cloud server and making effective use of unused smartphones.

[1235] The processing flow will be explained below.

[1236] 1. Monitoring function

[1237] Device (baby monitor)

[1238] Step 1:

[1239] Install a dedicated app on the device that will serve as the baby monitor and set it up to use the camera and microphone.

[1240] Step 2:

[1241] The device captures video and audio data in real time and transmits it to a server.

[1242] server

[1243] Step 3:

[1244] The server analyzes the received video and audio data, and uses computer vision and voice recognition AI to detect the baby's movements and cries.

[1245] Step 4:

[1246] If a risk is detected as a result of data analysis, the server will send a push notification to the user's device.

[1247] Step 5:

[1248] At the same time, it sends an instruction to the baby monitor to automatically play an audio warning (e.g., "Danger! Stop!").

[1249] User

[1250] Step 6:

[1251] Users can check push notifications on their smartphones and monitor their baby's condition in real time.

[1252] 2. Sleep training function

[1253] Device (baby monitor)

[1254] Step 1:

[1255] When the user is getting ready to put the baby to sleep, the baby monitor is set to sleep training mode.

[1256] Step 2:

[1257] Depending on your settings, the baby monitor will play sleep music to help your baby fall asleep.

[1258] server

[1259] Step 3:

[1260] When the user enters their baby's sleep data (time they went to sleep, time they woke up) into a dedicated app, the server receives and records this data.

[1261] Step 4:

[1262] The server also periodically analyzes the audio data to detect when the baby wakes up crying.

[1263] Step 5:

[1264] If crying is detected, the server immediately sends a push notification to the user's smartphone.

[1265] User

[1266] Step 6:

[1267] The user sees the push notification and goes to check on the baby.

[1268] 3. Toilet training function

[1269] Device (user's smartphone)

[1270] Step 1:

[1271] The user enters the amount of time their baby uses the toilet each day into the app.

[1272] server

[1273] Step 2:

[1274] The server receives, records and analyzes the input toilet data.

[1275] Step 3:

[1276] Based on the analyzed data, the next time to use the toilet is predicted.

[1277] Step 4:

[1278] When the next toilet visit is approaching, the server will send a push notification to the user's smartphone.

[1279] User

[1280] Step 5:

[1281] The user checks the push notification and takes the baby to the toilet at the appropriate time.

[1282] ---

[1283] The above processing steps realize a system in which the baby monitor function, sleep training function, and toilet training function operate seamlessly, reducing the burden on parents.

[1284] Example 1

[1285] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1286] Conventional baby monitoring systems require expensive dedicated equipment and lack versatility. They also struggle to detect abnormalities in real time and respond immediately, making it difficult to reduce the burden of childcare. Furthermore, few systems offer consistent support for infant sleep and toilet training.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1288] In this invention, the server uses an unused mobile device as a monitoring device and includes means for transmitting video and audio data from the monitoring device to a cloud computing environment, means for analyzing the video and audio data to detect abnormalities, and means for sending a push notification to the user's mobile communication device and automatically playing an audio message from the monitoring device when the abnormality is detected, thereby reducing the burden of childcare and enabling real-time detection and immediate response to abnormalities.

[1289] A "disused mobile device" is a portable electronic device that can continue to operate even after the user has finished normal use, and can be used for dedicated purposes by installing a specific application.

[1290] The "monitoring device" is a device that uses a camera and microphone to capture the baby's movements in real time and transmits the data to a cloud computing environment.

[1291] A "cloud computing environment" is a network of servers that provide computing resources and storage via the Internet, and analyzes and records the data received.

[1292] "Video and audio data" means visual and audio information captured from a monitoring device, including the baby's movements, crying, etc.

[1293] "Analysis" is the process of analyzing the transmitted video and audio data to detect anomalies or specific patterns.

[1294] "Detecting abnormalities" means determining whether there is any unusual behavior or risk in the baby's movements or voice.

[1295] "Push notifications" are a technology that allows smartphone apps to notify users of information in real time, and are used to issue warnings and alerts.

[1296] "Automatically play voice message" is a function that automatically plays a pre-set voice warning from the monitoring device when an abnormality is detected.

[1297] This invention is a system that provides infant monitoring, sleep training (hereafter referred to as "Nentore"), and toilet training (hereafter referred to as "Toi-train") through a single application. The main components of this system are an unused mobile device (monitoring device), the user's mobile communication device, and a cloud computing environment. Details of each function and an embodiment of the system are described below.

[1298] 1. Monitoring function

[1299] Terminal (monitoring device):

[1300] Abandoned mobile devices are used as surveillance devices, with applications installed that use the camera and microphone to capture real-time video and audio, and this data is transmitted to a cloud computing environment.

[1301] server:

[1302] The system receives video and audio data and analyzes it in real time. Specifically, it uses machine learning and computer vision technology to analyze the video data and detect abnormalities in the baby's movements and posture. It also uses voice recognition AI to detect crying and other abnormal sounds. If an abnormality is detected, the server sends a push notification to the user's mobile communication device and instructs the monitoring device to play an audio warning.

[1303] User:

[1304] Users can check the baby's condition in real time from their mobile communication device and take immediate action if necessary.

[1305] Examples:

[1306] While the baby is sleeping, the monitoring device uses a camera and microphone to transmit video and audio to a cloud computing environment. Computer vision technology and voice recognition AI detect when the baby kicks the covers or cries. In response, a push notification is sent to the user's mobile device stating, "The baby kicked the covers," and the monitoring device plays an audio warning saying, "Danger! Stop!"

[1307] 2. Sleep training function

[1308] Terminal (monitoring device):

[1309] Before your baby falls asleep, the monitor plays specific sleep music, and even after the baby falls asleep, the sound sensor detects any crying that may occur.

[1310] server:

[1311] The system receives and analyzes the baby's sleep data sent from the user's mobile communication device, and if it determines that the baby has woken up, it sends a push notification to the user's mobile communication device.

[1312] User:

[1313] Users enter the times their baby goes to sleep and wakes up into the application. This data is sent to a server for analysis. If the baby starts crying, the user receives a push notification on their mobile device, allowing them to take action.

[1314] Examples:

[1315] The user turns on sleep training mode in the application to help the baby fall asleep independently. The monitoring device starts playing selected sleep music. If the baby wakes up and cries during the night, the user's mobile device receives a notification that "Baby is crying."

[1316] 3. Toilet training function

[1317] Device (user's mobile communication device):

[1318] Users input the amount of time their baby spends using the toilet each day, which allows the cloud computing environment to analyze the intervals and timing of toilet visits.

[1319] server:

[1320] Based on the input data, the system predicts when the next toilet visit will be made, and periodically sends a push notification to the user's mobile device to let them know when it is time to go to the toilet next.

[1321] User:

[1322] Users will receive notifications so they can take their baby to the toilet at the right time, improving the success rate of potty training.

[1323] Examples:

[1324] The user inputs the time their baby uses the toilet each day into the application, and the server then predicts the next time the baby will use the toilet and sends a notification to the user's mobile device at the appropriate time.

[1325] Example prompt sentence:

[1326] "Please explain the specific processing steps of a system that uses computer vision and voice recognition to detect abnormalities in a baby monitoring function and send a push notification to the user's mobile communication device. Also, please provide a specific example of how the cloud computing environment and the monitoring device work together."

[1327] The system reduces the burden of childcare and provides a safe and healthy environment for infants. It includes real-time monitoring, sleep training, and toilet training functions, and is realized by linking with a cloud computing environment and making effective use of unused mobile devices.

[1328] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1329] Monitoring function

[1330] Step 1:

[1331] Data capture by terminal (monitoring device)

[1332] Input: An unused mobile device with the application installed, camera and microphone can act as input devices.

[1333] How it works: Uses a camera and microphone to capture video and audio of your baby in real time.

[1334] Output: Captured video and audio data.

[1335] Step 2:

[1336] Data transmission from the terminal (monitoring device) to the cloud computing environment

[1337] Input: Captured video and audio data.

[1338] How it works: Capture data is sent in real time to a cloud computing environment.

[1339] Output: Video and audio data received by the cloud computing environment.

[1340] Step 3:

[1341] Data analysis by server

[1342] Input: Video and audio data received by the cloud computing environment.

[1343] How it works: It uses computer vision technology and voice recognition AI to analyze incoming data and detect abnormalities such as baby movement or crying.

[1344] Output: Analysis result (normal / abnormal).

[1345] Step 4:

[1346] Server sends push notification and audio alert instructions

[1347] Input: When an anomaly is detected as a result of data analysis.

[1348] What it does: Sends a push notification to the user's mobile device and simultaneously instructs the monitoring device to play an audio alert.

[1349] Output: Push notification to user's mobile device and playback of audio message from monitoring device.

[1350] Step 5:

[1351] User notification and response

[1352] Input: A push notification sent to the user's mobile device.

[1353] Action: Check push notifications and rush to your baby if necessary.

[1354] Output: Baby safety check and problem solving.

[1355] Sleep training function

[1356] Step 1:

[1357] Playing sleep music on a terminal (monitoring device)

[1358] Input: Sleep music selection based on user preferences.

[1359] What it does: The monitoring device plays the selected sleep music.

[1360] Output: Sleep music played.

[1361] Step 2:

[1362] Detecting baby's wake-up sound using a terminal (monitoring device)

[1363] Input: Baby crying and movement sounds.

[1364] How it works: The sound sensor detects your baby's crying or any abnormal sounds.

[1365] Output: Detected sound data.

[1366] Step 3:

[1367] Entering sleep data from the user's mobile communication device

[1368] Input: Baby's sleep and wake times.

[1369] Action: A user enters data through an application.

[1370] Output: The input sleep data.

[1371] Step 4:

[1372] Sleep data analysis and notification by server

[1373] Input: Sleep data and sound sensor data sent from the user's mobile communication device.

[1374] What it does: Detects when the baby wakes up and sends a push notification to the user's mobile device.

[1375] Output: A push notification to the user's mobile device.

[1376] Toilet training function

[1377] Step 1:

[1378] Input of toilet usage time from user's mobile communication device

[1379] Enter: your baby's potty time.

[1380] Action: A user enters data through an application.

[1381] Output: The input toilet data.

[1382] Step 2:

[1383] Server-based analysis and prediction of toilet data

[1384] Input: Restroom data sent from the user's mobile communication device.

[1385] How it works: A cloud computing environment analyzes data and predicts when the next toilet visit will occur.

[1386] Output: Prediction of next toilet timing.

[1387] Step 3:

[1388] Server-driven push notifications

[1389] Input: Predicted next toilet time.

[1390] What it does: Sends a timely push notification to the user's mobile device saying "Next bathroom break."

[1391] Output: A push notification to the user's mobile device.

[1392] As a result, this system reduces the burden of childcare and provides a safe and healthy environment for babies. Each function works in conjunction with each other to provide real-time monitoring, effective sleep training, and proper toilet training.

[1393] (Application example 1)

[1394] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1395] The challenge is to provide a highly efficient, low-cost monitoring system that ensures safety in the home and responds quickly to abnormalities. In particular, there is a need for a system that can effectively utilize unused terminals and safely monitor the home even when the user is away.

[1396] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1397] In this invention, the server includes means for using an unused terminal as a monitoring device and transmitting video and audio data from the monitoring device to the server, means for analyzing the video and audio data to detect an abnormality, means for sending a push notification to the user's terminal and automatically playing an audio message from the monitoring device when the abnormality is detected, and means for monitoring the user's return home status and sending a confirmation notification if the user does not return home at the scheduled time. This makes it possible to maintain home safety even when the user is away and to respond quickly when an abnormality occurs.

[1398] A "terminal no longer in use" is an electronic device such as a smartphone or tablet that the user used in the past but is no longer using.

[1399] A "monitoring device" is a device that reuses unused terminals to collect video and audio data and detect abnormalities.

[1400] "Video and audio data" means real-time video and audio information captured by a surveillance device.

[1401] "Server" means a computer system or cloud-based system for receiving and analyzing video and audio data.

[1402] "Abnormality" refers to suspicious movements or sounds that are different from normal as a result of the surveillance device's analysis of video and audio data.

[1403] A "push notification" is a real-time notification message sent from a server to a user's device.

[1404] A "voice message" is a sound such as a warning sound that is played from a monitoring device when an abnormality is detected.

[1405] A "user's terminal" is a mobile device such as a smartphone or tablet that a user uses on a daily basis.

[1406] The "return home status" refers to whether the user has returned home at the scheduled time designated by the user.

[1407] A "confirmation notice" is a confirmation message sent to the user's terminal if the user does not return home at the scheduled time.

[1408] The "warning sound" is a sound that warns a suspicious person or a user when an abnormality is detected.

[1409] "Video data" refers to digital information of video captured by a monitoring device, which is recorded for abnormality detection and transmitted to a server.

[1410] The present invention provides a system for strengthening home security by utilizing unused terminals as monitoring devices. Specific embodiments for implementing this system will be described below.

[1411] composition

[1412] Terminal (monitoring device):

[1413] Abandoned smartphones and tablets are used as surveillance devices equipped with cameras and microphones to capture real-time video and audio.

[1414] server:

[1415] This is a computer system for receiving and analyzing video and audio data. The server analyzes the data using software libraries such as OpenCV and voice recognition AI.

[1416] On the user's device:

[1417] These are mobile devices that users use on a daily basis, such as smartphones and tablets. A dedicated application for this system is installed on the user's device, allowing them to receive notifications in real time.

[1418] Processing flow

[1419] 1. Video and audio data capture:

[1420] The monitoring device captures video and audio data in real time and transmits the data to a server.

[1421] 2. Data Analysis:

[1422] The server analyzes the video and audio data it receives. Specifically, it uses OpenCV to detect video anomalies, such as when a suspicious person is captured on camera. It also uses voice recognition AI to detect abnormal sounds.

[1423] 3. Push notifications and sound alerts:

[1424] If the server detects an abnormality, it sends a push notification to the user's device and simultaneously plays an alarm sound from the monitoring device.

[1425] 4. Monitoring when users return home:

[1426] The server monitors whether the user has returned home, and if the user has not returned home at the scheduled time, sends a confirmation notice to the user's terminal.

[1427] Specific examples

[1428] For example, a housewife places an unused smartphone at the front door while she goes out shopping. This monitoring device captures video and audio in real time and sends them to a server. The server analyzes the video data, and if it detects a suspicious person at the front door, it sends a push notification to the user's device saying, "A suspicious person has been detected at the front door!" At the same time, the monitoring device sounds an alarm to warn the suspicious person. Furthermore, the server monitors whether the user has returned home at the scheduled time, and if the user has not returned home at the scheduled time, it sends a confirmation notification to the user's device asking, "Did you return home as planned?"

[1429] Prompt Sentence Examples

[1430] "Functional description of this application: We want to create a smart home security system that captures video and audio in real time and detects suspicious activity. Specifically, we will use unused smartphones as security devices, and if an abnormality is detected, the system will upload the data to a cloud server, send a push notification to the user, and play an alarm."

[1431] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1432] Step 1:

[1433] The monitoring device captures video and audio data in real time and sends it to a server. The input includes real-time video and audio captured by the camera and microphone of the monitoring device. The output is sent to the server via the Internet. The specific operation of the monitoring device is to capture video with the camera and record audio with the microphone, and then transfer these as digital data to the server.

[1434] Step 2:

[1435] The server analyzes the video and audio data it receives. The input includes real-time video and audio data sent from the surveillance equipment. The server uses OpenCV to analyze the video data and performs facial recognition and motion detection to detect suspicious activity. It also uses voice recognition AI to detect abnormal sounds (for example, the sound of breaking glass or screaming). The output is alert information if suspicious activity or audio is detected. Specifically, the server analyzes the video data frame by frame and the audio data as a sound waveform.

[1436] Step 3:

[1437] If the server detects an anomaly, it sends a push notification to the user's device. The input includes the alert information generated by the server. The output is a push notification sent to the user's device stating "An anomaly has been detected." The push notification includes detailed information such as the type of anomaly and the time it was detected. Specifically, the server sends a notification to the user's smartphone through a push notification service.

[1438] Step 4:

[1439] If an abnormality is detected, the monitoring device automatically plays an alarm sound. The input includes the alert information sent from the server. As an output, the monitoring device plays an alarm sound to warn suspicious individuals on-site and those in the vicinity. Specifically, the monitoring device uses its built-in speaker to play a pre-set alarm sound file.

[1440] Step 5:

[1441] It monitors the user's return home status and sends a confirmation notification if the user does not return home at the scheduled time. The input includes the user's pre-set information and the server's timestamp information. The output is a confirmation notification sent to the user's smartphone asking, "Did you return home as scheduled?" Specifically, the server checks the user's return home information at the specified time and checks for any abnormalities. It then automatically sends a confirmation notification.

[1442] As described above, this system performs specific data processing and calculations at each step, and when an abnormality is detected, it notifies the user with an alert and even plays a warning sound to ensure the safety of the home.

[1443] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1444] This invention combines a system that provides functions for monitoring infants, sleep training, and toilet training with an emotion engine that recognizes the user's emotions. The main components of this system are an unused device (baby monitor), the user's smartphone, a cloud server, and the emotion engine. Below, we will explain an embodiment in which each function and emotion engine are combined.

[1445] 1. Monitoring function

[1446] Device (baby monitor)

[1447] A dedicated app is installed on an unused smartphone, and the camera and microphone are used to capture real-time video and audio, which is then sent to a server.

[1448] server

[1449] It receives video and audio data and analyzes it using computer vision and voice recognition AI. It detects baby movements and cries, and sends push notifications to the user's device if there is danger. It also instructs the baby monitor to automatically issue an audio alert.

[1450] Emotion Engine

[1451] It analyzes the user's emotions from voice data and input data to determine whether the user is tired, stressed, relaxed, etc. Based on this, it adjusts the content and timing of push notifications.

[1452] User

[1453] Users can check their baby's condition in real time from their smartphone and can take prompt action when notified.

[1454] Specific examples

[1455] While the baby is sleeping, the baby monitor captures video and audio and sends them to the server. The server detects when the baby kicks off the covers and determines whether there is any danger. At the same time, the emotion engine analyzes the user's situation. If it determines that the user is tired, it sends a push notification to reduce the burden on the parent by softening the tone of the notification or by eliminating redundant information to make it concise.

[1456] 2. Sleep training function

[1457] Device (baby monitor)

[1458] When you're getting your baby ready for bed, you can set the baby monitor to sleep mode and play specific sleep music. Even after your baby falls asleep, the sound sensor will still detect crying.

[1459] server

[1460] It receives the baby's sleep data sent from the user's smartphone and records it in a database. If crying is detected, it sends a push notification to the user's smartphone.

[1461] Emotion Engine

[1462] The system analyzes the user's emotions and adjusts the timing and content of sleep notifications. For example, if the user is feeling stressed, it can send a relaxing message.

[1463] User

[1464] Users can enter the times their baby goes to sleep and wakes up into the app, and if crying is detected, they will receive a push notification and be able to take action quickly.

[1465] Specific examples

[1466] To get the baby to sleep alone, the user turns on the sleep training mode. The baby monitor plays sleep music, and the baby falls asleep. When the baby starts crying in the middle of the night, the user's smartphone receives a notification saying "Your baby is crying." However, if the user is feeling stressed, the notification content is adjusted to say something like "It's okay, your baby is just crying a little."

[1467] 3. Toilet training function

[1468] Device (user's smartphone)

[1469] The user enters the amount of time their baby uses the toilet each day into the app.

[1470] server

[1471] The system receives and analyzes the entered toilet data, predicts the next toilet time, and sends a push notification to the user's smartphone.

[1472] Emotion Engine

[1473] The system analyzes the user's emotions and adjusts the toilet timing notification accordingly. For example, if the user is busy, the notification can be delayed a little.

[1474] User

[1475] Users who receive the notification can take their baby to the toilet at the appropriate time.

[1476] Specific examples

[1477] The user inputs the time their baby goes to the toilet into the app every day. The server predicts the next time the baby will go to the toilet, and if the emotion engine determines that the user is busy, it adjusts the notification timing. For example, the user might receive a notification saying, "The next time to go to the toilet is approaching, but you still have a little time."

[1478] This allows the system to take into account the parent's state and emotions, providing more effective and flexible childcare support. By combining real-time monitoring, sleep training, and toilet training functions with the emotion engine, the system further reduces the burden on parents and supports the safety and growth of infants.

[1479] The processing flow will be explained below.

[1480] 1. Monitoring function

[1481] Device (baby monitor)

[1482] Step 1:

[1483] Install the dedicated app on the device that will serve as the baby monitor and enable the camera and microphone.

[1484] Step 2:

[1485] The device captures video and audio data in real time and transmits the data to a server.

[1486] server

[1487] Step 3:

[1488] The server analyzes the received video and audio data, and uses computer vision to analyze the video data and detect any abnormal movements of the baby.

[1489] Step 4:

[1490] At the same time, voice recognition AI is used to analyze audio data and detect crying and abnormal sounds.

[1491] Step 5:

[1492] If a risk is detected, the server sends a push notification to the user's device.

[1493] Step 6:

[1494] The server sends instructions to the baby monitor to play an automatic audio warning (e.g., "Danger! Stop!").

[1495] User

[1496] Step 7:

[1497] Users can check push notifications and monitor their baby's condition in real time.

[1498] 2. Sleep training function

[1499] Device (baby monitor)

[1500] Step 1:

[1501] The user prepares the baby for sleep and sets the baby monitor to sleep mode.

[1502] Step 2:

[1503] The baby monitor plays selected sleep music.

[1504] server

[1505] Step 3:

[1506] The user enters the baby's sleep data (time they went to sleep, time they woke up) into a dedicated app.

[1507] Step 4:

[1508] The server records the entered sleep data in a database.

[1509] Step 5:

[1510] If the baby wakes up crying, the server analyzes the audio data and sends a push notification if crying is detected.

[1511] Emotion Engine

[1512] Step 6:

[1513] The emotion engine analyzes the user's emotional state and adjusts the content and timing of sleep notifications.

[1514] User

[1515] Step 7:

[1516] The user sees the push notification and goes to check on the baby.

[1517] 3. Toilet training function

[1518] Device (user's smartphone)

[1519] Step 1:

[1520] The user enters the amount of time their baby uses the toilet each day into the app.

[1521] server

[1522] Step 2:

[1523] The server receives and records the input toilet data.

[1524] Step 3:

[1525] The server analyzes the toilet data and predicts the next time to use the toilet.

[1526] Step 4:

[1527] A push notification will be sent to the user's smartphone when the next toilet visit is approaching.

[1528] Emotion Engine

[1529] Step 5:

[1530] The emotion engine analyzes the user's emotional state and adjusts the content and timing of toilet notification.

[1531] User

[1532] Step 6:

[1533] The user receives the notification and takes the baby to the toilet at the appropriate time.

[1534] ---

[1535] These processing steps realize a system in which the baby monitor function, sleep training function, and toilet training function work in conjunction with the emotion engine to provide appropriate support according to the parent's emotional state.

[1536] Example 2

[1537] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1538] While conventional baby monitor systems have functions for monitoring infants and training (sleep and toileting), they lack support that takes into account the emotions and state of the parents, which often leaves parents feeling tired and stressed, and the burden of childcare remains.

[1539] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1540] In this invention, the server includes means for using an unused device as a monitoring device and transmitting visual and audio data from the monitoring device to a central processing unit, means for analyzing the visual and audio data to detect danger, means for sending a push notification to the user's device and automatically playing an audio message from the monitoring device when the danger is detected, and means for evaluating the user's emotional state using an emotion analysis engine and adjusting the content and timing of the push notification based on the evaluation result. This enables notifications and support that take parents' emotions and states into consideration, thereby reducing the burden of child-rearing.

[1541] "Device" refers to a device or terminal held by a user, including devices used as monitor devices and smartphones used by users.

[1542] "Monitoring devices" refers to devices used to monitor and check on infants, including old smartphones.

[1543] "Visual Data" refers to real-time video information captured by the camera of a monitoring device.

[1544] "Acoustic Data" refers to real-time audio information collected by the microphone of a monitoring device.

[1545] "Central processing unit" refers to a device or system, such as a cloud or server, that receives and analyzes data sent from a monitor device.

[1546] "Danger" refers to abnormal behavior or a situation in an infant, and includes cases where the infant is deemed to be in danger.

[1547] "Push Notification" refers to alert messages or notifications sent in real time from a central processing unit to a user's device.

[1548] "Audible message" refers to a voice or sound alert that is automatically played from a monitoring device.

[1549] "Emotion analysis engine" refers to an algorithm or system that analyzes a user's voice or input data to assess the user's emotional state.

[1550] "Sleep Data" refers to information relating to an infant's sleep patterns and duration, including data recorded and analyzed by a central processing unit.

[1551] "Toilet data" refers to information about an infant's daily toilet use time and timing, including data that is analyzed by a central processing unit.

[1552] This invention is a system that combines a system that provides functions for monitoring infants, sleep training, and toilet training with an emotion analysis engine that recognizes user emotions. The main components are an unused device (monitoring device), the user's smartphone, a cloud server, and the emotion analysis engine. Below, we will explain in detail an embodiment in which each function is combined with the emotion analysis engine.

[1553] Monitoring function

[1554] Terminal (monitor device)

[1555] The monitoring device is an unused smartphone, which is then installed with a dedicated app. The app uses the smartphone's camera and microphone to capture real-time visual and acoustic data, which is then transmitted via the internet to a central processing unit located in the cloud.

[1556] Server (cloud server)

[1557] The cloud server receives visual and acoustic data sent from the monitoring device. This data is analyzed using computer vision algorithms (e.g., OpenCV) and speech recognition AI (e.g., speech recognition API). As a result of the analysis, the cloud server detects the infant's movements and cries and detects danger based on this. If danger is detected, a push notification is sent to the user's smartphone and an audio message is automatically played on the monitoring device.

[1558] Sentiment Analysis Engine

[1559] The emotion analysis engine analyzes the user's voice and input data to determine whether they are tired, stressed, or relaxed, and adjusts the content and timing of push notifications accordingly.

[1560] User

[1561] Users can check the condition of their baby in real time from their smartphone and can take prompt action when notified. For example, if a notification is received that a baby is in danger, they can rush to the scene immediately.

[1562] Specific examples

[1563] While the baby is sleeping, the monitor device captures video and audio and sends them to a cloud server. The cloud server detects when the baby kicks off the covers and determines whether there is any danger. At the same time, an emotion analysis engine analyzes the user's situation. If the user is tired, a push notification will be sent to soften the tone of the notification and simplify the message to reduce the burden on parents.

[1564] Prompt Sentence Examples

[1565] "Baby danger detection. Please suggest a way to soften the tone of the notification if the user is tired."

[1566] Sleep training function

[1567] Terminal (monitor device)

[1568] The monitoring device is set to sleep mode when preparing the baby for sleep, which plays specific sleep-inducing music and has sound sensors that detect crying even after the baby has fallen asleep.

[1569] Server (cloud server)

[1570] It receives the baby's sleep data sent from the user's smartphone and records it in a database. If the baby cries, it sends a push notification to the user's smartphone.

[1571] Sentiment Analysis Engine

[1572] The emotion analysis engine analyzes the user's emotions and adjusts the timing and content of sleep notifications accordingly. For example, if the user is feeling stressed, it will send a relaxing message.

[1573] User

[1574] Users enter the times their baby goes to sleep and wakes up into the app, and if crying is detected, they receive a push notification and can take action.

[1575] Specific examples

[1576] When the user turns on the sleep training mode, the monitor device plays sleep music to lull the baby to sleep. If crying is detected in the middle of the night, the cloud server sends a notification to the user's smartphone saying, "Your baby is crying." However, if the user is feeling stressed, the notification content will be adjusted to say, "It's okay, your baby is just crying a little."

[1577] Prompt Sentence Examples

[1578] "When detecting a baby's cry, create an appropriate relaxation message if the user is feeling stressed."

[1579] Toilet training function

[1580] Device (user's smartphone)

[1581] Users enter their baby's daily toilet use times into the app.

[1582] Server (cloud server)

[1583] The server receives and analyzes the input toilet data, and based on the analysis results, predicts the next toilet time and sends a push notification to the user's smartphone.

[1584] Sentiment Analysis Engine

[1585] The emotion analysis engine analyzes the user's emotions and adjusts the toilet timing notification accordingly. If the user is busy, the notification can be delayed.

[1586] User

[1587] When the user receives the notification, they can take the baby to the toilet at the appropriate time.

[1588] Specific examples

[1589] The user inputs the time their baby goes to the toilet into the app every day. The cloud server predicts when the next toilet visit will be, and if the emotion analysis engine determines that the user is busy, the notification timing will be adjusted. For example, the user might receive a notification saying, "The next toilet visit is approaching, but you still have a little time."

[1590] Prompt Sentence Examples

[1591] "Please suggest a way to predict when a baby needs to go to the toilet and adjust the notification timing based on the user's emotions."

[1592] With these functions, the system takes into account the emotions and state of the parent, providing more effective and flexible childcare support. By combining the monitoring, sleep training, and toilet training functions with the emotion analysis engine, the system further reduces the burden on parents and supports the safety and growth of infants.

[1593] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1594] Monitoring function processing steps

[1595] Step 1:

[1596] Terminal (monitor device)

[1597] Input: Video and audio from a monitor device with a dedicated app installed, camera and microphone.

[1598] Specific operation: Launch the dedicated app for the monitor device.

[1599] Processing: The app activates the monitor device's camera and microphone to capture video and audio in real time.

[1600] Output: Captured visual and acoustic data.

[1601] Step 2:

[1602] Terminal (monitor device)

[1603] Input: Captured visual and acoustic data.

[1604] Specific operation: Real-time video and audio data is sent to a cloud server via the Internet.

[1605] Processing: The dedicated app initiates a network connection to send the data, compresses and encodes the data, and sends it to the cloud server.

[1606] Output: Visual and acoustic data sent to cloud server.

[1607] Step 3:

[1608] Server (cloud server)

[1609] Input: Visual and acoustic data sent from the monitor device.

[1610] Specific operation: The cloud server receives the data and records it in a log.

[1611] Processing: Visual data is analyzed with computer vision algorithms (e.g., OpenCV), and acoustic data is analyzed with speech recognition AI (e.g., speech recognition API).

[1612] Output: The results of the analysis include the detection of infant movements and crying sounds.

[1613] Step 4:

[1614] Server (cloud server)

[1615] Input: Analysis results of visual and acoustic data.

[1616] Specific behavior: Generates a push notification if a danger is detected.

[1617] Processing: The system assesses the level of danger based on the detection of the infant's movements and cries, and if it determines that there is danger, it sends a push notification to the user's smartphone and instructs the monitoring device to send an acoustic message.

[1618] Output: Push notification sent to the user's smartphone.

[1619] Step 5:

[1620] Sentiment Analysis Engine

[1621] Input: User voice and typing data.

[1622] Specific operation: The sentiment analysis engine acquires the data and begins analysis.

[1623] Processing: A voice emotion analysis algorithm is used to determine the user's emotional state.

[1624] Output: User's emotional state as a result of emotion analysis (fatigue, stress, relaxed, etc.).

[1625] Step 6:

[1626] Server (cloud server)

[1627] Input: Sentiment analysis results from the sentiment analysis engine.

[1628] Specific operation: The cloud server adjusts the notification content and timing.

[1629] Processing: Based on the results of sentiment analysis, adjust the content (message tone, redundancy of information, etc.) and timing of push notifications.

[1630] Output: The adjusted push notification content.

[1631] Step 7:

[1632] User (smartphone)

[1633] Input: Push notification from cloud server.

[1634] Specific behavior: The user's smartphone displays the notification content.

[1635] Processing: The user confirms the push notification and sees and hears the baby in real time.

[1636] Output: User's response actions (checking and responding to the infant, etc.).

[1637] Processing steps for sleep training functions

[1638] Step 1:

[1639] Terminal (monitor device)

[1640] Input: Sleep training mode setting instruction from user.

[1641] Specific operation: The user sets the sleep training mode in the app.

[1642] Processing: Play specific sleep music from the monitor device depending on the sleep training mode.

[1643] Output: Sleep music is playing.

[1644] Step 2:

[1645] Terminal (monitor device)

[1646] Input: Audio capture by monitor device.

[1647] Specific function: The sound sensor captures sound and detects the baby's crying.

[1648] Processing: Send the crying data to the cloud server.

[1649] Output: Crying data sent to the cloud server.

[1650] Step 3:

[1651] Server (cloud server)

[1652] Input: Sleep data sent from the user's smartphone.

[1653] Specific operation: The server records it in the database.

[1654] Processing: The data is stored to analyze your baby's sleep patterns.

[1655] Output: Recorded sleep data.

[1656] Step 4:

[1657] Server (cloud server)

[1658] Input: Crying data analysis results.

[1659] Specific behavior: Generates a push notification.

[1660] Action: If crying is detected, a push notification is sent to the user's smartphone.

[1661] Output: Push notification sent to the user's smartphone.

[1662] Step 5:

[1663] Sentiment Analysis Engine

[1664] Input: User voice and typing data.

[1665] Specific operation: The sentiment analysis engine acquires the data and begins analysis.

[1666] Processing: A voice emotion analysis algorithm is used to determine the user's emotional state.

[1667] Output: User's emotional state (stressed, relaxed, etc.) as a result of emotion analysis.

[1668] Step 6:

[1669] Server (cloud server)

[1670] Input: Sentiment analysis results from the sentiment analysis engine.

[1671] Specific operation: The cloud server adjusts the notification content and timing.

[1672] Processing: Based on the results of sentiment analysis, adjust the content (message tone, redundancy of information, etc.) and timing of push notifications.

[1673] Output: The adjusted push notification content.

[1674] Step 7:

[1675] User (smartphone)

[1676] Input: Push notification from cloud server.

[1677] Specific operation: The user's smartphone displays the notification and checks the sleep data.

[1678] Action: The user checks the push notification and checks the baby's status in real time.

[1679] Output: User's response actions (responding to crying, recording sleep data, etc.).

[1680] Toilet training function processing steps

[1681] Step 1:

[1682] Device (user's smartphone)

[1683] Input: Your baby's daily toilet use timings.

[1684] Specific operation: The user inputs the timing of toilet use into the app.

[1685] Processing: The entered data is recorded by the app and sent to the cloud server.

[1686] Output: Toilet data sent to the cloud server.

[1687] Step 2:

[1688] Server (cloud server)

[1689] Input: Toilet data sent from the user's smartphone.

[1690] Specific operation: The server receives the data and begins analyzing it.

[1691] Processing: Predict the next toilet timing based on accumulated toilet data.

[1692] Output: Predicted next toilet time.

[1693] Step 3:

[1694] Server (cloud server)

[1695] Input: Predicted next toilet time.

[1696] Specific behavior: Generates a push notification to the user's smartphone.

[1697] Processing: Based on the prediction results, a message notifying the user of the next toilet visit is created and sent to the user's smartphone.

[1698] Output: Push notification sent to the user's smartphone.

[1699] Step 4:

[1700] Sentiment Analysis Engine

[1701] Input: User voice and typing data.

[1702] Specific operation: The sentiment analysis engine acquires the data and begins analysis.

[1703] Processing: A voice emotion analysis algorithm is used to determine the user's emotional state.

[1704] Output: User's emotional state (tired, stressed, busy, etc.) as a result of sentiment analysis.

[1705] Step 5:

[1706] Server (cloud server)

[1707] Input: Sentiment analysis results from the sentiment analysis engine.

[1708] Specific operation: The cloud server adjusts the notification content and timing.

[1709] Processing: Based on the results of sentiment analysis, adjust the content (message tone, redundancy of information, etc.) and timing of push notifications.

[1710] Output: The adjusted push notification content.

[1711] Step 6:

[1712] User (smartphone)

[1713] Input: Push notification from cloud server.

[1714] Specific behavior: The user's smartphone displays the notification content.

[1715] Action: The user checks the push notification and knows when to go to the bathroom in real time.

[1716] Output: Toilet guidance action by the user.

[1717] In this way, the system provides childcare support that takes into account the parent's emotions and state through each processing step, improving the effectiveness of supervision, sleep training, and toilet training.

[1718] (Application example 2)

[1719] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1720] Efficient and safe management of the status of workers and machinery in factories is important for improving the working environment and increasing productivity. However, existing monitoring systems are limited to simply detecting and reporting abnormalities and do not take into consideration the emotional state and workload of users. This can increase stress and burden on managers and delay appropriate responses. The present invention aims to solve these problems and provide a more convenient monitoring and management system.

[1721] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1722] In this invention, the server includes: a means for using an unused terminal as a monitoring device and transmitting video and audio data from the monitoring device to the server; a means for analyzing the video and audio data to detect an abnormality; a means for sending a push notification to the user's terminal and automatically playing an audio message from the monitoring device when the abnormality is detected; a means for analyzing the user's emotions and adjusting the content and timing of the notification; a means for analyzing the worker's status from the monitoring device and inputting the worker's status data into the user's terminal; a means for recording the status data in the server and sending a push notification to the user's terminal when the worker causes an abnormality; and a means for transmitting the adjusted notification to the user's terminal. This enables appropriate notification based on the user's emotional state at the same time as detecting an abnormality, thereby improving the safety and efficiency of the work environment.

[1723] An "obsolete device" is an electronic device that was previously in use but is no longer in use.

[1724] "Surveillance equipment" means equipment installed to monitor a specific area or object and capable of capturing video and audio data.

[1725] "Video and audio data" refers to digital data containing visual and audio information captured using a camera and microphone.

[1726] A "server" is a computer system that stores, processes, and manages data over a network.

[1727] An "anomaly" is an event or circumstance that deviates from normal operating or environmental conditions and requires immediate attention.

[1728] A "push notification" is a message that is automatically sent to a user's device when a specific event or state change occurs.

[1729] A "voice message" is an audio content that is recorded as audio data and played back.

[1730] "Emotion analysis" is the process of identifying a user's emotional state from speech or other input data.

[1731] "Adjusting notification content and timing" means changing the content of the message sent to the user and the timing at which the message is sent depending on the state or situation of the user.

[1732] "Worker status data" is digital data that indicates the worker's behavior, location, health condition, etc.

[1733] "Analysis" is the process of breaking down and evaluating data to extract useful information.

[1734] "Recording on the server" means storing the acquired data on the server so that it can be referenced or used later as needed.

[1735] "Adjusted notifications" are push notifications whose content or delivery timing has been changed based on the results of sentiment analysis.

[1736] MODE FOR CARRYING OUT THE INVENTION

[1737] A system for implementing the present invention includes the following configuration.

[1738] First, we use unused devices as surveillance devices. These devices have built-in cameras and microphones that capture video and audio data in real time, which is then transmitted to a cloud server via a network.

[1739] The cloud server uses computer vision technology and a voice recognition engine to analyze the received video and audio data, utilizing open source technologies such as OpenCV and TensorFlow, making it possible to detect abnormalities in workers and machinery.

[1740] If an abnormality is detected, the server sends a push notification to the user's device. This notification is displayed on the user's smartphone or PC. In addition, the monitoring device automatically plays a voice message regarding the abnormality. This voice message is pre-recorded and includes details of the abnormality and instructions on how to respond.

[1741] To analyze the user's emotional state, the cloud server is equipped with an emotion analysis engine. This engine analyzes voice data and other input data (e.g., text messages entered by the user) to determine the user's emotional state. For emotion analysis, voice analysis technologies such as Amazon Polly and Google Cloud Speech-to-Text can be used.

[1742] The content and timing of notifications are tailored based on the user's emotional state: if the user is feeling stressed, notifications can be sent in a softer tone or in a more concise manner without unnecessary information.

[1743] The system also has the function of monitoring the status of workers. Data acquired from the monitoring device is recorded on a server, and if an abnormality occurs, an appropriate notification is sent to the user's device. This improves worker safety and productivity.

[1744] Examples:

[1745] For example, if a monitoring device installed in a factory detects worker movements or a machine malfunction, the video and audio data is first sent to a cloud server. The cloud server analyzes the data and identifies the malfunction. At the same time, an emotion analysis engine analyzes the user's stress and fatigue levels and adjusts the notification content accordingly. The user's device may receive a notification such as, "An abnormality has occurred on machine A. Please check the details." However, if it is determined that the user is feeling stressed, a softer-toned notification may be sent, such as, "There is a minor malfunction on machine A. Please check."

[1746] Prompt for the generative AI model:

[1747] For example, the following might be a prompt to input to a generative AI model:

[1748] Please explain the functions of a system that supports putting a baby to sleep and toilet training. For example, a baby monitor captures video and audio while the baby is sleeping and sends them to a server. If the server detects danger and determines that the user is tired, it will send a push notification to reduce the burden on the parent by softening the tone of the notification or simplifying it by removing redundant information.

[1749] This allows the present invention to provide an efficient and safe monitoring and management system that takes into account the emotional state and workload of the user.

[1750] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1751] Processing Steps

[1752] Step 1: Booting the device and capturing data

[1753] The device starts up and uses the camera and microphone to capture video and audio data in real time. This data is acquired through the camera sensor and microphone. The input data is video frames and audio waveform data, which are converted and prepared in digital format.

[1754] Step 2: Send data

[1755] The device sends the captured video and audio data to the cloud server. The data is transmitted using a network communication protocol (e.g., HTTP or WebSocket). The input is the captured data, and the output is the status of the completion of data transmission to the server.

[1756] Step 3: Data analysis (server side)

[1757] The server analyzes the received data, using computer vision technology (e.g., OpenCV) for video data and a speech recognition engine (e.g., Google Cloud Speech-to-Text) for audio data. The input data are video frames and audio files sent from the device, and the output is anomaly detection information as the analysis result.

[1758] Step 4: Anomaly detection

[1759] The server detects anomalies based on the analysis results. If an anomaly is detected, it immediately starts a process to notify the user of that information. The input is the data analysis results, and the output is the anomaly determination result.

[1760] Step 5: Sentiment Analysis

[1761] The server analyzes the user's emotional state. To do this, it uses the voice data and text data from the user to identify the emotional state. Using an emotion analysis engine (e.g., Amazon Polly), the input data is the user's response and voice data, and the output is the user's emotional state.

[1762] Step 6: Adjust notification content and timing

[1763] The server adjusts the content and timing of notifications based on the emotion analysis results. If the user is feeling stressed, the notification content will be briefer and the tone will be softer. The input is the emotion analysis results, and the output is the adjusted content and timing of notifications.

[1764] Step 7: Send notification

[1765] The server sends the adjusted notification content to the user's device. This notification is sent as a push notification and displayed on the user's smartphone or PC. The input is the adjusted notification content, and the output is the notification sent to the user's device.

[1766] Step 8: Play a voice message when an error occurs

[1767] When an abnormality is detected, the monitoring device automatically plays back a voice message containing the details of the abnormality and instructions for how to respond. The input is the abnormality detection information, and the output is the playback of the voice message.

[1768] Examples:

[1769] For example, if a monitoring device installed in a factory detects worker movements or a machine malfunction, the video and audio data is first sent to a cloud server. The cloud server analyzes the data and identifies the malfunction. At the same time, an emotion analysis engine analyzes the user's stress and fatigue levels and adjusts the notification content accordingly. The user's device may receive a notification such as, "An abnormality has occurred on machine A. Please check the details." However, if it is determined that the user is feeling stressed, a softer-toned notification may be sent, such as, "There is a minor malfunction on machine A. Please check."

[1770] Example prompt for a generative AI model:

[1771] Please explain the functions of a system that supports putting a baby to sleep and toilet training. For example, a baby monitor captures video and audio while the baby is sleeping and sends them to a server. If the server detects danger and determines that the user is tired, it will send a push notification to reduce the burden on the parent by softening the tone of the notification or simplifying it by removing redundant information.

[1772] In this way, specific input data and output data are clearly defined in each processing step, and the system operates by processing data and performing calculations based on them.

[1773] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1774] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1775] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1776] [Fourth embodiment]

[1777] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1778] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1779] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1780] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1781] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1782] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1783] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1784] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1785] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1786] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1787] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1788] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1789] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1790] This invention is a system that provides functions for monitoring infants, sleep training, and toilet training in a single application. The main components of this system are an unused device (baby monitor), the user's smartphone, and a cloud server. Details of each function and their implementation are described below.

[1791] 1. Monitoring function

[1792] Device (baby monitor):

[1793] Use an old smartphone as a baby monitor by installing an app and using the camera and microphone to capture real-time video and audio, which is then sent to a server.

[1794] server:

[1795] It receives video and audio data and analyzes it in real time. Specifically, it uses computer vision to analyze the video data and detect abnormalities in the baby's movements and posture. It also uses voice recognition AI to detect crying and other abnormal sounds. If danger is detected, the server sends a push notification to the user's smartphone and instructs the baby monitor to automatically play an audio warning.

[1796] User:

[1797] Users can check on their baby's condition in real time from their smartphone, and can respond immediately if necessary.

[1798] Examples:

[1799] While the baby is sleeping, the baby monitor uses a camera and microphone to send video and audio to a server. Computer vision and voice recognition AI detect when the baby kicks the covers or cries. In this case, the user's smartphone receives a push notification saying "The baby kicked the covers," and the baby monitor plays an audio warning saying "Danger! Stop!"

[1800] 2. Sleep training function

[1801] Device (baby monitor):

[1802] Before your baby falls asleep, play specific sleep music from the baby monitor, and even after your baby falls asleep, the sound sensor will detect any crying that wakes them up.

[1803] server:

[1804] The system receives and analyzes the baby's sleep data sent from the user's smartphone, and if it determines that the baby has woken up, it sends a push notification to the user's smartphone.

[1805] User:

[1806] Users can enter the time their baby goes to sleep and wakes up into the app, and this data is sent to a server. If the baby starts crying, the user will receive a push notification on their smartphone so they can take action.

[1807] Examples:

[1808] To get the baby to sleep alone, the user turns on the sleep training mode. The baby monitor starts playing selected sleep music. When the baby wakes up and cries in the middle of the night, the user's smartphone receives a notification saying "Your baby is crying."

[1809] 3. Toilet training function

[1810] Device (user's smartphone):

[1811] Users input the number of times their baby uses the toilet each day, which allows the server to analyze the intervals and timing of toilet visits.

[1812] server:

[1813] Based on the input data, the next time to go to the toilet is predicted. A push notification is periodically sent to the user's smartphone to let them know when it is time to go to the toilet next.

[1814] User:

[1815] Users who receive the notification can take their baby to the toilet at the appropriate time, which improves the success rate of potty training.

[1816] Examples:

[1817] Users input the time their baby spends using the toilet into the app each day. Based on this, the app predicts when their baby will next go to the toilet, and sends a push notification to the user's smartphone at the appropriate time.

[1818] This system significantly reduces the burden of childcare on parents and provides an environment where infants can live safely and healthily.The real-time monitoring, sleep training, and toilet training functions are realized by linking with a cloud server and making effective use of unused smartphones.

[1819] The processing flow will be explained below.

[1820] 1. Monitoring function

[1821] Device (baby monitor)

[1822] Step 1:

[1823] Install a dedicated app on the device that will serve as the baby monitor and set it up to use the camera and microphone.

[1824] Step 2:

[1825] The device captures video and audio data in real time and transmits it to a server.

[1826] server

[1827] Step 3:

[1828] The server analyzes the received video and audio data, and uses computer vision and voice recognition AI to detect the baby's movements and cries.

[1829] Step 4:

[1830] If a risk is detected as a result of data analysis, the server will send a push notification to the user's device.

[1831] Step 5:

[1832] At the same time, it sends an instruction to the baby monitor to automatically play an audio warning (e.g., "Danger! Stop!").

[1833] User

[1834] Step 6:

[1835] Users can check push notifications on their smartphones and monitor their baby's condition in real time.

[1836] 2. Sleep training function

[1837] Device (baby monitor)

[1838] Step 1:

[1839] When the user is getting ready to put the baby to sleep, the baby monitor is set to sleep training mode.

[1840] Step 2:

[1841] Depending on your settings, the baby monitor will play sleep music to help your baby fall asleep.

[1842] server

[1843] Step 3:

[1844] When the user enters their baby's sleep data (time they went to sleep, time they woke up) into a dedicated app, the server receives and records this data.

[1845] Step 4:

[1846] The server also periodically analyzes the audio data to detect when the baby wakes up crying.

[1847] Step 5:

[1848] If crying is detected, the server immediately sends a push notification to the user's smartphone.

[1849] User

[1850] Step 6:

[1851] The user sees the push notification and goes to check on the baby.

[1852] 3. Toilet training function

[1853] Device (user's smartphone)

[1854] Step 1:

[1855] The user enters the amount of time their baby uses the toilet each day into the app.

[1856] server

[1857] Step 2:

[1858] The server receives, records and analyzes the input toilet data.

[1859] Step 3:

[1860] Based on the analyzed data, the next time to use the toilet is predicted.

[1861] Step 4:

[1862] When the next toilet visit is approaching, the server will send a push notification to the user's smartphone.

[1863] User

[1864] Step 5:

[1865] The user checks the push notification and takes the baby to the toilet at the appropriate time.

[1866] ---

[1867] The above processing steps realize a system in which the baby monitor function, sleep training function, and toilet training function operate seamlessly, reducing the burden on parents.

[1868] Example 1

[1869] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1870] Conventional baby monitoring systems require expensive dedicated equipment and lack versatility. They also struggle to detect abnormalities in real time and respond immediately, making it difficult to reduce the burden of childcare. Furthermore, few systems offer consistent support for infant sleep and toilet training.

[1871] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1872] In this invention, the server uses an unused mobile device as a monitoring device and includes means for transmitting video and audio data from the monitoring device to a cloud computing environment, means for analyzing the video and audio data to detect abnormalities, and means for sending a push notification to the user's mobile communication device and automatically playing an audio message from the monitoring device when the abnormality is detected, thereby reducing the burden of childcare and enabling real-time detection and immediate response to abnormalities.

[1873] A "disused mobile device" is a portable electronic device that can continue to operate even after the user has finished normal use, and can be used for dedicated purposes by installing a specific application.

[1874] The "monitoring device" is a device that uses a camera and microphone to capture the baby's movements in real time and transmits the data to a cloud computing environment.

[1875] A "cloud computing environment" is a network of servers that provide computing resources and storage via the Internet, and analyzes and records the data received.

[1876] "Video and audio data" means visual and audio information captured from a monitoring device, including the baby's movements, crying, etc.

[1877] "Analysis" is the process of analyzing the transmitted video and audio data to detect anomalies or specific patterns.

[1878] "Detecting abnormalities" means determining whether there is any unusual behavior or risk in the baby's movements or voice.

[1879] "Push notifications" are a technology that allows smartphone apps to notify users of information in real time, and are used to issue warnings and alerts.

[1880] "Automatically play voice message" is a function that automatically plays a pre-set voice warning from the monitoring device when an abnormality is detected.

[1881] This invention is a system that provides infant monitoring, sleep training (hereafter referred to as "Nentore"), and toilet training (hereafter referred to as "Toi-train") through a single application. The main components of this system are an unused mobile device (monitoring device), the user's mobile communication device, and a cloud computing environment. Details of each function and an embodiment of the system are described below.

[1882] 1. Monitoring function

[1883] Terminal (monitoring device):

[1884] Abandoned mobile devices are used as surveillance devices, with applications installed that use the camera and microphone to capture real-time video and audio, and this data is transmitted to a cloud computing environment.

[1885] server:

[1886] The system receives video and audio data and analyzes it in real time. Specifically, it uses machine learning and computer vision technology to analyze the video data and detect abnormalities in the baby's movements and posture. It also uses voice recognition AI to detect crying and other abnormal sounds. If an abnormality is detected, the server sends a push notification to the user's mobile communication device and instructs the monitoring device to play an audio warning.

[1887] User:

[1888] Users can check the baby's condition in real time from their mobile communication device and take immediate action if necessary.

[1889] Examples:

[1890] While the baby is sleeping, the monitoring device uses a camera and microphone to transmit video and audio to a cloud computing environment. Computer vision technology and voice recognition AI detect when the baby kicks the covers or cries. In response, a push notification is sent to the user's mobile device stating, "The baby kicked the covers," and the monitoring device plays an audio warning saying, "Danger! Stop!"

[1891] 2. Sleep training function

[1892] Terminal (monitoring device):

[1893] Before your baby falls asleep, the monitor plays specific sleep music, and even after the baby falls asleep, the sound sensor detects any crying that may occur.

[1894] server:

[1895] The system receives and analyzes the baby's sleep data sent from the user's mobile communication device, and if it determines that the baby has woken up, it sends a push notification to the user's mobile communication device.

[1896] User:

[1897] Users enter the times their baby goes to sleep and wakes up into the application. This data is sent to a server for analysis. If the baby starts crying, the user receives a push notification on their mobile device, allowing them to take action.

[1898] Examples:

[1899] The user turns on sleep training mode in the application to help the baby fall asleep independently. The monitoring device starts playing selected sleep music. If the baby wakes up and cries during the night, the user's mobile device receives a notification that "Baby is crying."

[1900] 3. Toilet training function

[1901] Device (user's mobile communication device):

[1902] Users input the amount of time their baby spends using the toilet each day, which allows the cloud computing environment to analyze the intervals and timing of toilet visits.

[1903] server:

[1904] Based on the input data, the system predicts when the next toilet visit will be made, and periodically sends a push notification to the user's mobile device to let them know when it is time to go to the toilet next.

[1905] User:

[1906] Users will receive notifications so they can take their baby to the toilet at the right time, improving the success rate of potty training.

[1907] Examples:

[1908] The user inputs the time their baby uses the toilet each day into the application, and the server then predicts the next time the baby will use the toilet and sends a notification to the user's mobile device at the appropriate time.

[1909] Example prompt sentence:

[1910] "Please explain the specific processing steps of a system that uses computer vision and voice recognition to detect abnormalities in a baby monitoring function and send a push notification to the user's mobile communication device. Also, please provide a specific example of how the cloud computing environment and the monitoring device work together."

[1911] The system reduces the burden of childcare and provides a safe and healthy environment for infants. It includes real-time monitoring, sleep training, and toilet training functions, and is realized by linking with a cloud computing environment and making effective use of unused mobile devices.

[1912] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1913] Monitoring function

[1914] Step 1:

[1915] Data capture by terminal (monitoring device)

[1916] Input: An unused mobile device with the application installed, camera and microphone can act as input devices.

[1917] How it works: Uses a camera and microphone to capture video and audio of your baby in real time.

[1918] Output: Captured video and audio data.

[1919] Step 2:

[1920] Data transmission from the terminal (monitoring device) to the cloud computing environment

[1921] Input: Captured video and audio data.

[1922] How it works: Capture data is sent in real time to a cloud computing environment.

[1923] Output: Video and audio data received by the cloud computing environment.

[1924] Step 3:

[1925] Data analysis by server

[1926] Input: Video and audio data received by the cloud computing environment.

[1927] How it works: It uses computer vision technology and voice recognition AI to analyze incoming data and detect abnormalities such as baby movement or crying.

[1928] Output: Analysis result (normal / abnormal).

[1929] Step 4:

[1930] Server sends push notification and audio alert instructions

[1931] Input: When an anomaly is detected as a result of data analysis.

[1932] What it does: Sends a push notification to the user's mobile device and simultaneously instructs the monitoring device to play an audio alert.

[1933] Output: Push notification to user's mobile device and playback of audio message from monitoring device.

[1934] Step 5:

[1935] User notification and response

[1936] Input: A push notification sent to the user's mobile device.

[1937] Action: Check push notifications and rush to your baby if necessary.

[1938] Output: Baby safety check and problem solving.

[1939] Sleep training function

[1940] Step 1:

[1941] Playing sleep music on a terminal (monitoring device)

[1942] Input: Sleep music selection based on user preferences.

[1943] What it does: The monitoring device plays the selected sleep music.

[1944] Output: Sleep music played.

[1945] Step 2:

[1946] Detecting baby's wake-up sound using a terminal (monitoring device)

[1947] Input: Baby crying and movement sounds.

[1948] How it works: The sound sensor detects your baby's crying or any abnormal sounds.

[1949] Output: Detected sound data.

[1950] Step 3:

[1951] Entering sleep data from the user's mobile communication device

[1952] Input: Baby's sleep and wake times.

[1953] Action: A user enters data through an application.

[1954] Output: The input sleep data.

[1955] Step 4:

[1956] Sleep data analysis and notification by server

[1957] Input: Sleep data and sound sensor data sent from the user's mobile communication device.

[1958] What it does: Detects when the baby wakes up and sends a push notification to the user's mobile device.

[1959] Output: A push notification to the user's mobile device.

[1960] Toilet training function

[1961] Step 1:

[1962] Input of toilet usage time from user's mobile communication device

[1963] Enter: your baby's potty time.

[1964] Action: A user enters data through an application.

[1965] Output: The input toilet data.

[1966] Step 2:

[1967] Server-based analysis and prediction of toilet data

[1968] Input: Restroom data sent from the user's mobile communication device.

[1969] How it works: A cloud computing environment analyzes data and predicts when the next toilet visit will occur.

[1970] Output: Prediction of next toilet timing.

[1971] Step 3:

[1972] Server-driven push notifications

[1973] Input: Predicted next toilet time.

[1974] What it does: Sends a timely push notification to the user's mobile device saying "Next bathroom break."

[1975] Output: A push notification to the user's mobile device.

[1976] As a result, this system reduces the burden of childcare and provides a safe and healthy environment for babies. Each function works in conjunction with each other to provide real-time monitoring, effective sleep training, and proper toilet training.

[1977] (Application example 1)

[1978] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1979] The challenge is to provide a highly efficient, low-cost monitoring system that ensures safety in the home and responds quickly to abnormalities. In particular, there is a need for a system that can effectively utilize unused terminals and safely monitor the home even when the user is away.

[1980] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1981] In this invention, the server includes means for using an unused terminal as a monitoring device and transmitting video and audio data from the monitoring device to the server, means for analyzing the video and audio data to detect an abnormality, means for sending a push notification to the user's terminal and automatically playing an audio message from the monitoring device when the abnormality is detected, and means for monitoring the user's return home status and sending a confirmation notification if the user does not return home at the scheduled time. This makes it possible to maintain home safety even when the user is away and to respond quickly when an abnormality occurs.

[1982] A "terminal no longer in use" is an electronic device such as a smartphone or tablet that the user used in the past but is no longer using.

[1983] A "monitoring device" is a device that reuses unused terminals to collect video and audio data and detect abnormalities.

[1984] "Video and audio data" means real-time video and audio information captured by a surveillance device.

[1985] "Server" means a computer system or cloud-based system for receiving and analyzing video and audio data.

[1986] "Abnormality" refers to suspicious movements or sounds that are different from normal as a result of the surveillance device's analysis of video and audio data.

[1987] A "push notification" is a real-time notification message sent from a server to a user's device.

[1988] A "voice message" is a sound such as a warning sound that is played from a monitoring device when an abnormality is detected.

[1989] A "user's terminal" is a mobile device such as a smartphone or tablet that a user uses on a daily basis.

[1990] The "return home status" refers to whether the user has returned home at the scheduled time designated by the user.

[1991] A "confirmation notice" is a confirmation message sent to the user's terminal if the user does not return home at the scheduled time.

[1992] The "warning sound" is a sound that warns a suspicious person or a user when an abnormality is detected.

[1993] "Video data" refers to digital information of video captured by a monitoring device, which is recorded for abnormality detection and transmitted to a server.

[1994] The present invention provides a system for strengthening home security by utilizing unused terminals as monitoring devices. Specific embodiments for implementing this system will be described below.

[1995] composition

[1996] Terminal (monitoring device):

[1997] Abandoned smartphones and tablets are used as surveillance devices equipped with cameras and microphones to capture real-time video and audio.

[1998] server:

[1999] This is a computer system for receiving and analyzing video and audio data. The server analyzes the data using software libraries such as OpenCV and voice recognition AI.

[2000] On the user's device:

[2001] These are mobile devices that users use on a daily basis, such as smartphones and tablets. A dedicated application for this system is installed on the user's device, allowing them to receive notifications in real time.

[2002] Processing flow

[2003] 1. Video and audio data capture:

[2004] The monitoring device captures video and audio data in real time and transmits the data to a server.

[2005] 2. Data Analysis:

[2006] The server analyzes the video and audio data it receives. Specifically, it uses OpenCV to detect video anomalies, such as when a suspicious person is captured on camera. It also uses voice recognition AI to detect abnormal sounds.

[2007] 3. Push notifications and sound alerts:

[2008] If the server detects an abnormality, it sends a push notification to the user's device and simultaneously plays an alarm sound from the monitoring device.

[2009] 4. Monitoring when users return home:

[2010] The server monitors whether the user has returned home, and if the user has not returned home at the scheduled time, sends a confirmation notice to the user's terminal.

[2011] Specific examples

[2012] For example, a housewife places an unused smartphone at the front door while she goes out shopping. This monitoring device captures video and audio in real time and sends them to a server. The server analyzes the video data, and if it detects a suspicious person at the front door, it sends a push notification to the user's device saying, "A suspicious person has been detected at the front door!" At the same time, the monitoring device sounds an alarm to warn the suspicious person. Furthermore, the server monitors whether the user has returned home at the scheduled time, and if the user has not returned home at the scheduled time, it sends a confirmation notification to the user's device asking, "Did you return home as planned?"

[2013] Prompt Sentence Examples

[2014] "Functional description of this application: We want to create a smart home security system that captures video and audio in real time and detects suspicious activity. Specifically, we will use unused smartphones as security devices, and if an abnormality is detected, the system will upload the data to a cloud server, send a push notification to the user, and play an alarm."

[2015] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2016] Step 1:

[2017] The monitoring device captures video and audio data in real time and sends it to a server. The input includes real-time video and audio captured by the camera and microphone of the monitoring device. The output is sent to the server via the Internet. The specific operation of the monitoring device is to capture video with the camera and record audio with the microphone, and then transfer these as digital data to the server.

[2018] Step 2:

[2019] The server analyzes the video and audio data it receives. The input includes real-time video and audio data sent from the surveillance equipment. The server uses OpenCV to analyze the video data and performs facial recognition and motion detection to detect suspicious activity. It also uses voice recognition AI to detect abnormal sounds (for example, the sound of breaking glass or screaming). The output is alert information if suspicious activity or audio is detected. Specifically, the server analyzes the video data frame by frame and the audio data as a sound waveform.

[2020] Step 3:

[2021] If the server detects an anomaly, it sends a push notification to the user's device. The input includes the alert information generated by the server. The output is a push notification sent to the user's device stating "An anomaly has been detected." The push notification includes detailed information such as the type of anomaly and the time it was detected. Specifically, the server sends a notification to the user's smartphone through a push notification service.

[2022] Step 4:

[2023] If an abnormality is detected, the monitoring device automatically plays an alarm sound. The input includes the alert information sent from the server. As an output, the monitoring device plays an alarm sound to warn suspicious individuals on-site and those in the vicinity. Specifically, the monitoring device uses its built-in speaker to play a pre-set alarm sound file.

[2024] Step 5:

[2025] It monitors the user's return home status and sends a confirmation notification if the user does not return home at the scheduled time. The input includes the user's pre-set information and the server's timestamp information. The output is a confirmation notification sent to the user's smartphone asking, "Did you return home as scheduled?" Specifically, the server checks the user's return home information at the specified time and checks for any abnormalities. It then automatically sends a confirmation notification.

[2026] As described above, this system performs specific data processing and calculations at each step, and when an abnormality is detected, it notifies the user with an alert and even plays a warning sound to ensure the safety of the home.

[2027] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2028] This invention combines a system that provides functions for monitoring infants, sleep training, and toilet training with an emotion engine that recognizes the user's emotions. The main components of this system are an unused device (baby monitor), the user's smartphone, a cloud server, and the emotion engine. Below, we will explain an embodiment in which each function and emotion engine are combined.

[2029] 1. Monitoring function

[2030] Device (baby monitor)

[2031] A dedicated app is installed on an unused smartphone, and the camera and microphone are used to capture real-time video and audio, which is then sent to a server.

[2032] server

[2033] It receives video and audio data and analyzes it using computer vision and voice recognition AI. It detects baby movements and cries, and sends push notifications to the user's device if there is danger. It also instructs the baby monitor to automatically issue an audio alert.

[2034] Emotion Engine

[2035] It analyzes the user's emotions from voice data and input data to determine whether the user is tired, stressed, relaxed, etc. Based on this, it adjusts the content and timing of push notifications.

[2036] User

[2037] Users can check their baby's condition in real time from their smartphone and can take prompt action when notified.

[2038] Specific examples

[2039] While the baby is sleeping, the baby monitor captures video and audio and sends them to the server. The server detects when the baby kicks off the covers and determines whether there is any danger. At the same time, the emotion engine analyzes the user's situation. If it determines that the user is tired, it sends a push notification to reduce the burden on the parent by softening the tone of the notification or by eliminating redundant information to make it concise.

[2040] 2. Sleep training function

[2041] Device (baby monitor)

[2042] When you're getting your baby ready for bed, you can set the baby monitor to sleep mode and play specific sleep music. Even after your baby falls asleep, the sound sensor will still detect crying.

[2043] server

[2044] It receives the baby's sleep data sent from the user's smartphone and records it in a database. If crying is detected, it sends a push notification to the user's smartphone.

[2045] Emotion Engine

[2046] The system analyzes the user's emotions and adjusts the timing and content of sleep notifications. For example, if the user is feeling stressed, it can send a relaxing message.

[2047] User

[2048] Users can enter the times their baby goes to sleep and wakes up into the app, and if crying is detected, they will receive a push notification and be able to take action quickly.

[2049] Specific examples

[2050] To get the baby to sleep alone, the user turns on the sleep training mode. The baby monitor plays sleep music, and the baby falls asleep. When the baby starts crying in the middle of the night, the user's smartphone receives a notification saying "Your baby is crying." However, if the user is feeling stressed, the notification content is adjusted to say something like "It's okay, your baby is just crying a little."

[2051] 3. Toilet training function

[2052] Device (user's smartphone)

[2053] The user enters the amount of time their baby uses the toilet each day into the app.

[2054] server

[2055] The system receives and analyzes the entered toilet data, predicts the next toilet time, and sends a push notification to the user's smartphone.

[2056] Emotion Engine

[2057] The system analyzes the user's emotions and adjusts the toilet timing notification accordingly. For example, if the user is busy, the notification can be delayed a little.

[2058] User

[2059] Users who receive the notification can take their baby to the toilet at the appropriate time.

[2060] Specific examples

[2061] The user inputs the time their baby goes to the toilet into the app every day. The server predicts the next time the baby will go to the toilet, and if the emotion engine determines that the user is busy, it adjusts the notification timing. For example, the user might receive a notification saying, "The next time to go to the toilet is approaching, but you still have a little time."

[2062] This allows the system to take into account the parent's state and emotions, providing more effective and flexible childcare support. By combining real-time monitoring, sleep training, and toilet training functions with the emotion engine, the system further reduces the burden on parents and supports the safety and growth of infants.

[2063] The processing flow will be explained below.

[2064] 1. Monitoring function

[2065] Device (baby monitor)

[2066] Step 1:

[2067] Install the dedicated app on the device that will serve as the baby monitor and enable the camera and microphone.

[2068] Step 2:

[2069] The device captures video and audio data in real time and transmits the data to a server.

[2070] server

[2071] Step 3:

[2072] The server analyzes the received video and audio data, and uses computer vision to analyze the video data and detect any abnormal movements of the baby.

[2073] Step 4:

[2074] At the same time, voice recognition AI is used to analyze audio data and detect crying and abnormal sounds.

[2075] Step 5:

[2076] If a risk is detected, the server sends a push notification to the user's device.

[2077] Step 6:

[2078] The server sends instructions to the baby monitor to play an automatic audio warning (e.g., "Danger! Stop!").

[2079] User

[2080] Step 7:

[2081] Users can check push notifications and monitor their baby's condition in real time.

[2082] 2. Sleep training function

[2083] Device (baby monitor)

[2084] Step 1:

[2085] The user prepares the baby for sleep and sets the baby monitor to sleep mode.

[2086] Step 2:

[2087] The baby monitor plays selected sleep music.

[2088] server

[2089] Step 3:

[2090] The user enters the baby's sleep data (time they went to sleep, time they woke up) into a dedicated app.

[2091] Step 4:

[2092] The server records the entered sleep data in a database.

[2093] Step 5:

[2094] If the baby wakes up crying, the server analyzes the audio data and sends a push notification if crying is detected.

[2095] Emotion Engine

[2096] Step 6:

[2097] The emotion engine analyzes the user's emotional state and adjusts the content and timing of sleep notifications.

[2098] User

[2099] Step 7:

[2100] The user sees the push notification and goes to check on the baby.

[2101] 3. Toilet training function

[2102] Device (user's smartphone)

[2103] Step 1:

[2104] The user enters the amount of time their baby uses the toilet each day into the app.

[2105] server

[2106] Step 2:

[2107] The server receives and records the input toilet data.

[2108] Step 3:

[2109] The server analyzes the toilet data and predicts the next time to use the toilet.

[2110] Step 4:

[2111] A push notification will be sent to the user's smartphone when the next toilet visit is approaching.

[2112] Emotion Engine

[2113] Step 5:

[2114] The emotion engine analyzes the user's emotional state and adjusts the content and timing of toilet notification.

[2115] User

[2116] Step 6:

[2117] The user receives the notification and takes the baby to the toilet at the appropriate time.

[2118] ---

[2119] These processing steps realize a system in which the baby monitor function, sleep training function, and toilet training function work in conjunction with the emotion engine to provide appropriate support according to the parent's emotional state.

[2120] Example 2

[2121] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2122] While conventional baby monitor systems have functions for monitoring infants and training (sleep and toileting), they lack support that takes into account the emotions and state of the parents, which often leaves parents feeling tired and stressed, and the burden of childcare remains.

[2123] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2124] In this invention, the server includes means for using an unused device as a monitoring device and transmitting visual and audio data from the monitoring device to a central processing unit, means for analyzing the visual and audio data to detect danger, means for sending a push notification to the user's device and automatically playing an audio message from the monitoring device when the danger is detected, and means for evaluating the user's emotional state using an emotion analysis engine and adjusting the content and timing of the push notification based on the evaluation result. This enables notifications and support that take parents' emotions and states into consideration, thereby reducing the burden of child-rearing.

[2125] "Device" refers to a device or terminal held by a user, including devices used as monitor devices and smartphones used by users.

[2126] "Monitoring devices" refers to devices used to monitor and check on infants, including old smartphones.

[2127] "Visual Data" refers to real-time video information captured by the camera of a monitoring device.

[2128] "Acoustic Data" refers to real-time audio information collected by the microphone of a monitoring device.

[2129] "Central processing unit" refers to a device or system, such as a cloud or server, that receives and analyzes data sent from a monitor device.

[2130] "Danger" refers to abnormal behavior or a situation in an infant, and includes cases where the infant is deemed to be in danger.

[2131] "Push Notification" refers to alert messages or notifications sent in real time from a central processing unit to a user's device.

[2132] "Audible message" refers to a voice or sound alert that is automatically played from a monitoring device.

[2133] "Emotion analysis engine" refers to an algorithm or system that analyzes a user's voice or input data to assess the user's emotional state.

[2134] "Sleep Data" refers to information relating to an infant's sleep patterns and duration, including data recorded and analyzed by a central processing unit.

[2135] "Toilet data" refers to information about an infant's daily toilet use time and timing, including data that is analyzed by a central processing unit.

[2136] This invention is a system that combines a system that provides functions for monitoring infants, sleep training, and toilet training with an emotion analysis engine that recognizes user emotions. The main components are an unused device (monitoring device), the user's smartphone, a cloud server, and the emotion analysis engine. Below, we will explain in detail an embodiment in which each function is combined with the emotion analysis engine.

[2137] Monitoring function

[2138] Terminal (monitor device)

[2139] The monitoring device is an unused smartphone, which is then installed with a dedicated app. The app uses the smartphone's camera and microphone to capture real-time visual and acoustic data, which is then transmitted via the internet to a central processing unit located in the cloud.

[2140] Server (cloud server)

[2141] The cloud server receives visual and acoustic data sent from the monitoring device. This data is analyzed using computer vision algorithms (e.g., OpenCV) and speech recognition AI (e.g., speech recognition API). As a result of the analysis, the cloud server detects the infant's movements and cries and detects danger based on this. If danger is detected, a push notification is sent to the user's smartphone and an audio message is automatically played on the monitoring device.

[2142] Sentiment Analysis Engine

[2143] The emotion analysis engine analyzes the user's voice and input data to determine whether they are tired, stressed, or relaxed, and adjusts the content and timing of push notifications accordingly.

[2144] User

[2145] Users can check the condition of their baby in real time from their smartphone and can take prompt action when notified. For example, if a notification is received that a baby is in danger, they can rush to the scene immediately.

[2146] Specific examples

[2147] While the baby is sleeping, the monitor device captures video and audio and sends them to a cloud server. The cloud server detects when the baby kicks off the covers and determines whether there is any danger. At the same time, an emotion analysis engine analyzes the user's situation. If the user is tired, a push notification will be sent to soften the tone of the notification and simplify the message to reduce the burden on parents.

[2148] Prompt Sentence Examples

[2149] "Baby danger detection. Please suggest a way to soften the tone of the notification if the user is tired."

[2150] Sleep training function

[2151] Terminal (monitor device)

[2152] The monitoring device is set to sleep mode when preparing the baby for sleep, which plays specific sleep-inducing music and has sound sensors that detect crying even after the baby has fallen asleep.

[2153] Server (cloud server)

[2154] It receives the baby's sleep data sent from the user's smartphone and records it in a database. If the baby cries, it sends a push notification to the user's smartphone.

[2155] Sentiment Analysis Engine

[2156] The emotion analysis engine analyzes the user's emotions and adjusts the timing and content of sleep notifications accordingly. For example, if the user is feeling stressed, it will send a relaxing message.

[2157] User

[2158] Users enter the times their baby goes to sleep and wakes up into the app, and if crying is detected, they receive a push notification and can take action.

[2159] Specific examples

[2160] When the user turns on the sleep training mode, the monitor device plays sleep music to lull the baby to sleep. If crying is detected in the middle of the night, the cloud server sends a notification to the user's smartphone saying, "Your baby is crying." However, if the user is feeling stressed, the notification content will be adjusted to say, "It's okay, your baby is just crying a little."

[2161] Prompt Sentence Examples

[2162] "When detecting a baby's cry, create an appropriate relaxation message if the user is feeling stressed."

[2163] Toilet training function

[2164] Device (user's smartphone)

[2165] Users enter their baby's daily toilet use times into the app.

[2166] Server (cloud server)

[2167] The server receives and analyzes the input toilet data, and based on the analysis results, predicts the next toilet time and sends a push notification to the user's smartphone.

[2168] Sentiment Analysis Engine

[2169] The emotion analysis engine analyzes the user's emotions and adjusts the toilet timing notification accordingly. If the user is busy, the notification can be delayed.

[2170] User

[2171] When the user receives the notification, they can take the baby to the toilet at the appropriate time.

[2172] Specific examples

[2173] The user inputs the time their baby goes to the toilet into the app every day. The cloud server predicts when the next toilet visit will be, and if the emotion analysis engine determines that the user is busy, the notification timing will be adjusted. For example, the user might receive a notification saying, "The next toilet visit is approaching, but you still have a little time."

[2174] Prompt Sentence Examples

[2175] "Please suggest a way to predict when a baby needs to go to the toilet and adjust the notification timing based on the user's emotions."

[2176] With these functions, the system takes into account the emotions and state of the parent, providing more effective and flexible childcare support. By combining the monitoring, sleep training, and toilet training functions with the emotion analysis engine, the system further reduces the burden on parents and supports the safety and growth of infants.

[2177] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2178] Monitoring function processing steps

[2179] Step 1:

[2180] Terminal (monitor device)

[2181] Input: Video and audio from a monitor device with a dedicated app installed, camera and microphone.

[2182] Specific operation: Launch the dedicated app for the monitor device.

[2183] Processing: The app activates the monitor device's camera and microphone to capture video and audio in real time.

[2184] Output: Captured visual and acoustic data.

[2185] Step 2:

[2186] Terminal (monitor device)

[2187] Input: Captured visual and acoustic data.

[2188] Specific operation: Real-time video and audio data is sent to a cloud server via the Internet.

[2189] Processing: The dedicated app initiates a network connection to send the data, compresses and encodes the data, and sends it to the cloud server.

[2190] Output: Visual and acoustic data sent to cloud server.

[2191] Step 3:

[2192] Server (cloud server)

[2193] Input: Visual and acoustic data sent from the monitor device.

[2194] Specific operation: The cloud server receives the data and records it in a log.

[2195] Processing: Visual data is analyzed with computer vision algorithms (e.g., OpenCV), and acoustic data is analyzed with speech recognition AI (e.g., speech recognition API).

[2196] Output: The results of the analysis include the detection of infant movements and crying sounds.

[2197] Step 4:

[2198] Server (cloud server)

[2199] Input: Analysis results of visual and acoustic data.

[2200] Specific behavior: Generates a push notification if a danger is detected.

[2201] Processing: The system assesses the level of danger based on the detection of the infant's movements and cries, and if it determines that there is danger, it sends a push notification to the user's smartphone and instructs the monitoring device to send an acoustic message.

[2202] Output: Push notification sent to the user's smartphone.

[2203] Step 5:

[2204] Sentiment Analysis Engine

[2205] Input: User voice and typing data.

[2206] Specific operation: The sentiment analysis engine acquires the data and begins analysis.

[2207] Processing: A voice emotion analysis algorithm is used to determine the user's emotional state.

[2208] Output: User's emotional state as a result of emotion analysis (fatigue, stress, relaxed, etc.).

[2209] Step 6:

[2210] Server (cloud server)

[2211] Input: Sentiment analysis results from the sentiment analysis engine.

[2212] Specific operation: The cloud server adjusts the notification content and timing.

[2213] Processing: Based on the results of sentiment analysis, adjust the content (message tone, redundancy of information, etc.) and timing of push notifications.

[2214] Output: The adjusted push notification content.

[2215] Step 7:

[2216] User (smartphone)

[2217] Input: Push notification from cloud server.

[2218] Specific behavior: The user's smartphone displays the notification content.

[2219] Processing: The user confirms the push notification and sees and hears the baby in real time.

[2220] Output: User's response actions (checking and responding to the infant, etc.).

[2221] Processing steps for sleep training functions

[2222] Step 1:

[2223] Terminal (monitor device)

[2224] Input: Sleep training mode setting instruction from user.

[2225] Specific operation: The user sets the sleep training mode in the app.

[2226] Processing: Play specific sleep music from the monitor device depending on the sleep training mode.

[2227] Output: Sleep music is playing.

[2228] Step 2:

[2229] Terminal (monitor device)

[2230] Input: Audio capture by monitor device.

[2231] Specific function: The sound sensor captures sound and detects the baby's crying.

[2232] Processing: Send the crying data to the cloud server.

[2233] Output: Crying data sent to the cloud server.

[2234] Step 3:

[2235] Server (cloud server)

[2236] Input: Sleep data sent from the user's smartphone.

[2237] Specific operation: The server records it in the database.

[2238] Processing: The data is stored to analyze your baby's sleep patterns.

[2239] Output: Recorded sleep data.

[2240] Step 4:

[2241] Server (cloud server)

[2242] Input: Crying data analysis results.

[2243] Specific behavior: Generates a push notification.

[2244] Action: If crying is detected, a push notification is sent to the user's smartphone.

[2245] Output: Push notification sent to the user's smartphone.

[2246] Step 5:

[2247] Sentiment Analysis Engine

[2248] Input: User voice and typing data.

[2249] Specific operation: The sentiment analysis engine acquires the data and begins analysis.

[2250] Processing: A voice emotion analysis algorithm is used to determine the user's emotional state.

[2251] Output: User's emotional state (stressed, relaxed, etc.) as a result of emotion analysis.

[2252] Step 6:

[2253] Server (cloud server)

[2254] Input: Sentiment analysis results from the sentiment analysis engine.

[2255] Specific operation: The cloud server adjusts the notification content and timing.

[2256] Processing: Based on the results of sentiment analysis, adjust the content (message tone, redundancy of information, etc.) and timing of push notifications.

[2257] Output: The adjusted push notification content.

[2258] Step 7:

[2259] User (smartphone)

[2260] Input: Push notification from cloud server.

[2261] Specific operation: The user's smartphone displays the notification and checks the sleep data.

[2262] Action: The user checks the push notification and checks the baby's status in real time.

[2263] Output: User's response actions (responding to crying, recording sleep data, etc.).

[2264] Toilet training function processing steps

[2265] Step 1:

[2266] Device (user's smartphone)

[2267] Input: Your baby's daily toilet use timings.

[2268] Specific operation: The user inputs the timing of toilet use into the app.

[2269] Processing: The entered data is recorded by the app and sent to the cloud server.

[2270] Output: Toilet data sent to the cloud server.

[2271] Step 2:

[2272] Server (cloud server)

[2273] Input: Toilet data sent from the user's smartphone.

[2274] Specific operation: The server receives the data and begins analyzing it.

[2275] Processing: Predict the next toilet timing based on accumulated toilet data.

[2276] Output: Predicted next toilet time.

[2277] Step 3:

[2278] Server (cloud server)

[2279] Input: Predicted next toilet time.

[2280] Specific behavior: Generates a push notification to the user's smartphone.

[2281] Processing: Based on the prediction results, a message notifying the user of the next toilet visit is created and sent to the user's smartphone.

[2282] Output: Push notification sent to the user's smartphone.

[2283] Step 4:

[2284] Sentiment Analysis Engine

[2285] Input: User voice and typing data.

[2286] Specific operation: The sentiment analysis engine acquires the data and begins analysis.

[2287] Processing: A voice emotion analysis algorithm is used to determine the user's emotional state.

[2288] Output: User's emotional state (tired, stressed, busy, etc.) as a result of sentiment analysis.

[2289] Step 5:

[2290] Server (cloud server)

[2291] Input: Sentiment analysis results from the sentiment analysis engine.

[2292] Specific operation: The cloud server adjusts the notification content and timing.

[2293] Processing: Based on the results of sentiment analysis, adjust the content (message tone, redundancy of information, etc.) and timing of push notifications.

[2294] Output: The adjusted push notification content.

[2295] Step 6:

[2296] User (smartphone)

[2297] Input: Push notification from cloud server.

[2298] Specific behavior: The user's smartphone displays the notification content.

[2299] Action: The user checks the push notification and knows when to go to the bathroom in real time.

[2300] Output: Toilet guidance action by the user.

[2301] In this way, the system provides childcare support that takes into account the parent's emotions and state through each processing step, improving the effectiveness of supervision, sleep training, and toilet training.

[2302] (Application example 2)

[2303] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2304] Efficient and safe management of the status of workers and machinery in factories is important for improving the working environment and increasing productivity. However, existing monitoring systems are limited to simply detecting and reporting abnormalities and do not take into consideration the emotional state and workload of users. This can increase stress and burden on managers and delay appropriate responses. The present invention aims to solve these problems and provide a more convenient monitoring and management system.

[2305] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2306] In this invention, the server includes: a means for using an unused terminal as a monitoring device and transmitting video and audio data from the monitoring device to the server; a means for analyzing the video and audio data to detect an abnormality; a means for sending a push notification to the user's terminal and automatically playing an audio message from the monitoring device when the abnormality is detected; a means for analyzing the user's emotions and adjusting the content and timing of the notification; a means for analyzing the worker's status from the monitoring device and inputting the worker's status data into the user's terminal; a means for recording the status data in the server and sending a push notification to the user's terminal when the worker causes an abnormality; and a means for transmitting the adjusted notification to the user's terminal. This enables appropriate notification based on the user's emotional state at the same time as detecting an abnormality, thereby improving the safety and efficiency of the work environment.

[2307] An "obsolete device" is an electronic device that was previously in use but is no longer in use.

[2308] "Surveillance equipment" means equipment installed to monitor a specific area or object and capable of capturing video and audio data.

[2309] "Video and audio data" refers to digital data containing visual and audio information captured using a camera and microphone.

[2310] A "server" is a computer system that stores, processes, and manages data over a network.

[2311] An "anomaly" is an event or circumstance that deviates from normal operating or environmental conditions and requires immediate attention.

[2312] A "push notification" is a message that is automatically sent to a user's device when a specific event or state change occurs.

[2313] A "voice message" is an audio content that is recorded as audio data and played back.

[2314] "Emotion analysis" is the process of identifying a user's emotional state from speech or other input data.

[2315] "Adjusting notification content and timing" means changing the content of the message sent to the user and the timing at which the message is sent depending on the state or situation of the user.

[2316] "Worker status data" is digital data that indicates the worker's behavior, location, health condition, etc.

[2317] "Analysis" is the process of breaking down and evaluating data to extract useful information.

[2318] "Recording on the server" means storing the acquired data on the server so that it can be referenced or used later as needed.

[2319] "Adjusted notifications" are push notifications whose content or delivery timing has been changed based on the results of sentiment analysis.

[2320] MODE FOR CARRYING OUT THE INVENTION

[2321] A system for implementing the present invention includes the following configuration.

[2322] First, we use unused devices as surveillance devices. These devices have built-in cameras and microphones that capture video and audio data in real time, which is then transmitted to a cloud server via a network.

[2323] The cloud server uses computer vision technology and a voice recognition engine to analyze the received video and audio data, utilizing open source technologies such as OpenCV and TensorFlow, making it possible to detect abnormalities in workers and machinery.

[2324] If an abnormality is detected, the server sends a push notification to the user's device. This notification is displayed on the user's smartphone or PC. In addition, the monitoring device automatically plays a voice message regarding the abnormality. This voice message is pre-recorded and includes details of the abnormality and instructions on how to respond.

[2325] To analyze the user's emotional state, the cloud server is equipped with an emotion analysis engine. This engine analyzes voice data and other input data (e.g., text messages entered by the user) to determine the user's emotional state. For emotion analysis, voice analysis technologies such as Amazon Polly and Google Cloud Speech-to-Text can be used.

[2326] The content and timing of notifications are tailored based on the user's emotional state: if the user is feeling stressed, notifications can be sent in a softer tone or in a more concise manner without unnecessary information.

[2327] The system also has the function of monitoring the status of workers. Data acquired from the monitoring device is recorded on a server, and if an abnormality occurs, an appropriate notification is sent to the user's device. This improves worker safety and productivity.

[2328] Examples:

[2329] For example, if a monitoring device installed in a factory detects worker movements or a machine malfunction, the video and audio data is first sent to a cloud server. The cloud server analyzes the data and identifies the malfunction. At the same time, an emotion analysis engine analyzes the user's stress and fatigue levels and adjusts the notification content accordingly. The user's device may receive a notification such as, "An abnormality has occurred on machine A. Please check the details." However, if it is determined that the user is feeling stressed, a softer-toned notification may be sent, such as, "There is a minor malfunction on machine A. Please check."

[2330] Prompt for the generative AI model:

[2331] For example, the following might be a prompt to input to a generative AI model:

[2332] Please explain the functions of a system that supports putting a baby to sleep and toilet training. For example, a baby monitor captures video and audio while the baby is sleeping and sends them to a server. If the server detects danger and determines that the user is tired, it will send a push notification to reduce the burden on the parent by softening the tone of the notification or simplifying it by removing redundant information.

[2333] This allows the present invention to provide an efficient and safe monitoring and management system that takes into account the emotional state and workload of the user.

[2334] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2335] Processing Steps

[2336] Step 1: Booting the device and capturing data

[2337] The device starts up and uses the camera and microphone to capture video and audio data in real time. This data is acquired through the camera sensor and microphone. The input data is video frames and audio waveform data, which are converted and prepared in digital format.

[2338] Step 2: Send data

[2339] The device sends the captured video and audio data to the cloud server. The data is transmitted using a network communication protocol (e.g., HTTP or WebSocket). The input is the captured data, and the output is the status of the completion of data transmission to the server.

[2340] Step 3: Data analysis (server side)

[2341] The server analyzes the received data, using computer vision technology (e.g., OpenCV) for video data and a speech recognition engine (e.g., Google Cloud Speech-to-Text) for audio data. The input data are video frames and audio files sent from the device, and the output is anomaly detection information as the analysis result.

[2342] Step 4: Anomaly detection

[2343] The server detects anomalies based on the analysis results. If an anomaly is detected, it immediately starts a process to notify the user of that information. The input is the data analysis results, and the output is the anomaly determination result.

[2344] Step 5: Sentiment Analysis

[2345] The server analyzes the user's emotional state. To do this, it uses the voice data and text data from the user to identify the emotional state. Using an emotion analysis engine (e.g., Amazon Polly), the input data is the user's response and voice data, and the output is the user's emotional state.

[2346] Step 6: Adjust notification content and timing

[2347] The server adjusts the content and timing of notifications based on the emotion analysis results. If the user is feeling stressed, the notification content will be briefer and the tone will be softer. The input is the emotion analysis results, and the output is the adjusted content and timing of notifications.

[2348] Step 7: Send notification

[2349] The server sends the adjusted notification content to the user's device. This notification is sent as a push notification and displayed on the user's smartphone or PC. The input is the adjusted notification content, and the output is the notification sent to the user's device.

[2350] Step 8: Play a voice message when an error occurs

[2351] When an abnormality is detected, the monitoring device automatically plays back a voice message containing the details of the abnormality and instructions for how to respond. The input is the abnormality detection information, and the output is the playback of the voice message.

[2352] Examples:

[2353] For example, if a monitoring device installed in a factory detects worker movements or a machine malfunction, the video and audio data is first sent to a cloud server. The cloud server analyzes the data and identifies the malfunction. At the same time, an emotion analysis engine analyzes the user's stress and fatigue levels and adjusts the notification content accordingly. The user's device may receive a notification such as, "An abnormality has occurred on machine A. Please check the details." However, if it is determined that the user is feeling stressed, a softer-toned notification may be sent, such as, "There is a minor malfunction on machine A. Please check."

[2354] Example prompt for a generative AI model:

[2355] Please explain the functions of a system that supports putting a baby to sleep and toilet training. For example, a baby monitor captures video and audio while the baby is sleeping and sends them to a server. If the server detects danger and determines that the user is tired, it will send a push notification to reduce the burden on the parent by softening the tone of the notification or simplifying it by removing redundant information.

[2356] In this way, specific input data and output data are clearly defined in each processing step, and the system operates by processing data and performing calculations based on them.

[2357] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2358] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2359] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2360] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2361] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2362] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2363] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2364] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2365] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2366] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural ...

Claims

1. A means for using an unused terminal as a baby monitor and transmitting video and audio data from the baby monitor to a server; means for analyzing the video and audio data to detect danger; The system includes a means for sending a push notification to the user's terminal when the danger is detected and automatically playing an audio message from the baby monitor.

2. A means for playing music from the baby monitor to help the baby fall asleep and inputting the baby's sleep data into the user's device; The system according to claim 1 , further comprising means for recording the sleep data in a server and sending a push notification to the user's terminal when the baby wakes up and cries.

3. A means for inputting the timing of the baby's daily toilet use from a user's terminal and analyzing the toilet data in a server; The system according to claim 1 , further comprising means for predicting the next toilet timing and sending a push notification to the user's terminal.

Images