system

Abstract

Provide a system. The system includes: a means for capturing image data using a camera and transmitting the image data to a server in real time; A means for receiving video data and performing image recognition of people and animals in the video data; A method for analyzing behavior based on image recognition results and detecting dangerous behavior; means for issuing an audio alert to notify the user of the detected unsafe behavior; and means for notifying a user terminal of the detected risky behavior.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern homes, monitoring and protecting pets, babies, young children, and other household members is important. However, it is difficult for people to constantly monitor these households, and dangerous situations can occur when people are not paying attention. It is also difficult to immediately recognize and report the intrusion of suspicious individuals. There is a need for a system that can solve these problems and provide a safe and secure living environment. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the present invention provides the following means: A system including means for capturing image data using a camera and transmitting it to a server in real time, means for receiving the video data and performing image recognition on people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting dangerous behavior, means for issuing an audio alert to notify the user of the detected dangerous behavior, and means for notifying the user of the detected dangerous behavior to a user terminal. The system also includes means for notifying the user of an intrusion by using facial recognition, and means for recording the detected video data and analysis results in a database for use in later analysis.

[0006] A "camera" is a device for capturing images and video.

[0007] "Image data" is a digital representation of visual information captured by a camera.

[0008] A "server" is a computer system that provides data processing and storage functions over a network.

[0009] "Real-time" refers to a state in which processing and communication are carried out immediately without delay.

[0010] "Video data" is dynamic visual information that represents multiple images in succession.

[0011] "Receiving" refers to the act of receiving transmitted data.

[0012] "Image recognition" is a technology that allows a computer to identify people and objects in an image and analyze their features.

[0013] "Behavioral analysis" is the process of evaluating and analyzing a target's behavioral patterns based on information detected by image recognition.

[0014] "Dangerous behavior" refers to behavior that could potentially cause an accident or trouble if performed by the subject.

[0015] "Voice alert" is a means of drawing attention by issuing a voice message or warning sound when a specific situation occurs.

[0016] A "user terminal" is an electronic device used by a user, such as a smartphone, tablet, or PC.

[0017] "Notification" refers to the act of transmitting specific information or warnings to a receiving device.

[0018] "Facial recognition" is a technology that uses image recognition technology to identify facial features and identify a person.

[0019] "Trespass" refers to the act of entering a particular area without permission.

[0020] A "database" is a system for efficiently storing, searching, and updating large amounts of data.

[0021] "Analysis results" refers to the results of analysis conducted based on the data and the contents of the report. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] MODE FOR CARRYING OUT THE INVENTION

[0044] To achieve the functions described in the claims, the present invention provides a system that includes the following elements: Specifically, a camera, a server, a terminal, and a software module are combined to create an environment in which users can live their daily lives with peace of mind.

[0045] Program processing overview

[0046] The present invention is a system that processes data between devices as follows, and ultimately provides the user with the information they need.

[0047] 1. Camera Interface

[0048] A camera connected to the device captures images of the inside of the house in real time, for example, periodically taking pictures of the living room or children's room.

[0049] The video data captured by the terminal is transmitted to a server via the Internet.

[0050] 2. Receiving and processing video data

[0051] The server temporarily stores the video data received from the terminal in a buffer.

[0052] The server divides the video data into frames and inputs them into an image recognition module.

[0053] 3. Image Recognition

[0054] The image recognition module detects people and pets in the frame and recognizes their faces and shapes, for example, the face of a baby or a pet dog.

[0055] The image recognition module compares the image against a database to determine whether it is a known person or pet.

[0056] 4. Behavior analysis

[0057] The server performs behavioral analysis based on the information obtained from the image recognition module, for example, detecting a baby trying to crawl out of its crib.

[0058] The behavior analysis module develops defined risky behaviors and determines whether the detected behavior matches them.

[0059] 5. Alert System

[0060] The server generates an audio alert message if risky behavior is detected.

[0061] The device will emit a warning sound or message through the speaker to the user, for example, "Your baby is about to fall out of his crib!"

[0062] 6. Notification System

[0063] The server sends push notifications to users' smartphones or PCs, instantly informing them of any dangerous behavior or intruders.

[0064] For example, if a pet starts chewing on an electrical cord, the user will receive a notification on their smartphone saying, "Your pet is chewing on the electrical cord!"

[0065] 7. Database Management

[0066] The server stores the detected video data and analysis results in a database.

[0067] The database management module organizes the data for later access and creates backups as needed.

[0068] Specific examples

[0069] 1. Pet monitoring

[0070] A camera installed on the device periodically captures footage of the pet dog.

[0071] The server receives the video data and analyzes the dog's behavior using image recognition and behavior analysis modules.

[0072] If a dog starts chewing on an electrical cord, the server will play an audio message saying "That's dangerous!" and simultaneously send a notification to the user's smartphone.

[0073] 2. Baby watching

[0074] The device's camera captures images of the baby in real time.

[0075] The server analyzes the baby's behavior and detects dangerous movements.

[0076] When the baby tries to crawl out of the crib, the device will emit an audio alert saying "Baby is moving!" and send a push notification to the user's smartphone.

[0077] This allows users to ensure the safety of pets, babies, and small children anytime, anywhere. This system prevents danger in daily life and provides users with a sense of security.

[0078] The processing flow will be explained below.

[0079] Step 1:

[0080] The device's camera captures video of the home in real time at a set frame rate, and the video data is divided into frames at regular time intervals.

[0081] Step 2:

[0082] The captured video frames are sent directly to the server using a dedicated communication protocol, ensuring data security.

[0083] Step 3:

[0084] The server stores the video frames received from the terminal in a buffer in order, allowing subsequent processing to proceed smoothly.

[0085] Step 4:

[0086] The server sends the buffered video frames to an image recognition module that detects people and pets in each frame, using specific algorithms to identify faces and shapes.

[0087] Step 5:

[0088] The image recognition module compares the detection results with a database to determine whether the target is registered, for example, by comparing the faces of family members with facial images stored in the database.

[0089] Step 6:

[0090] The server passes the image recognition results to the behavior analysis module, which then analyzes multiple behavioral patterns to determine whether the target's behavior is dangerous.

[0091] Step 7:

[0092] The behavior analysis module analyzes behavioral patterns and detects dangerous behavior (for example, a baby about to fall or a pet chewing on an electrical cord).

[0093] Step 8:

[0094] If the server detects any risky behavior, it will instantly generate an audio alert message, which is generated based on pre-defined phrases.

[0095] Step 9:

[0096] The device receives the audio alert message and issues a voice warning through the speaker, for example, "That's dangerous!"

[0097] Step 10:

[0098] The server then sends a push notification to the user's smartphone or PC in parallel, containing details of the risky behavior detected.

[0099] Step 11:

[0100] Users can check notifications on their smartphones or PCs and rush to the scene if necessary. For example, they can tap the pop-up notification to check details.

[0101] Step 12:

[0102] The server stores the detected video data and analysis results in a database, which can be used for later review and analysis.

[0103] Step 13:

[0104] The database management module periodically creates backups of data and makes them available for system recovery, thereby preventing data loss.

[0105] Example 1

[0106] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0107] Ensuring the safety of children and pets is a major concern in modern homes. Conventional surveillance systems simply record video footage, but are unable to detect danger or send alerts in real time, making it difficult to respond to situations that require a rapid response. Furthermore, there is a lack of means to manage and analyze massive amounts of video data, making efficient data storage and use a challenge.

[0108] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0109] In this invention, the server includes means for capturing image data using a camera and transmitting it to the server in real time, means for receiving the video data and performing image recognition of people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting dangerous behavior, means for issuing an audio alert to notify the user of the detected dangerous behavior, means for notifying the user of the detected dangerous behavior, means for temporarily storing the video data in a buffer, means for storing the detected video data and analysis results in a database for later access, means for analyzing each frame using an image recognition module, means for tracking movements in the video using a behavior analysis module and detecting specific patterns, and means for generating an audio alert message. This enables real-time monitoring and analysis of video data inside the home, rapid detection of dangerous behavior and issuing an alert, and efficient storage and management of video data.

[0110] A "camera" is an optical device for capturing image data.

[0111] A "server" is a computer system that processes and manages data in response to requests from clients.

[0112] A "terminal" is a computer system or device that has input and output devices and can be directly operated by a user.

[0113] "Image data" refers to visual information captured by a camera and stored in digital form.

[0114] "Real-time" means near-instant processing with minimal delay.

[0115] A "buffer" is a memory area that functions as a temporary storage area for data and ensures a smooth flow of data processing.

[0116] "Image recognition" is the process of using computer vision techniques to detect specific objects or patterns in video data.

[0117] "Behavioral analysis" is the process of analyzing patterns of movement and behavior to predict intentions and next actions.

[0118] A "voice alert" is a system that uses voice to warn or notify.

[0119] A "user terminal" is a device operated by a user who uses the system, and typically refers to a smartphone or personal computer.

[0120] A "database" is a structured data storage system for efficiently managing, retrieving, and storing data.

[0121] An "image recognition module" is a software component that analyzes image data to recognize specific objects or people.

[0122] The "behavioral analysis module" is a software component that analyzes behavior from recognized data and detects specific patterns and anomalies.

[0123] A "voice alert message" is a message containing a warning or notification generated using voice synthesis technology.

[0124] "People and animals" refers to living beings such as humans and pets that appear in image data.

[0125] This invention combines cameras, servers, terminals, and software modules to provide an environment in which users can live their daily lives with peace of mind. Specifically, it is a system that monitors images from inside the home in real time and issues alerts and notifications as needed.

[0126] Hardware and Software Configuration

[0127] 1. Camera

[0128] Device: A camera is used to capture video from inside the home. For example, cameras installed in the living room or children's room capture video data in real time.

[0129] The video data is transmitted to a server via the Internet.

[0130] 2. Server

[0131] Server: Receives video data sent from the device and temporarily stores it in a buffer. This buffer can hold several seconds of video.

[0132] The received video data is divided into frames and input into the image recognition module.

[0133] Image Recognition Module: Detects people and pets from video frames and recognizes their faces and shapes. For example, using Tensorflow (registered trademark) or OpenCV.

[0134] A software module (behavior analysis module) runs to perform behavior analysis based on the recognized information.

[0135] The module detects risky behavior and handles the results appropriately.

[0136] 3. Alert System

[0137] Server: Uses a text-to-speech (TTS) engine to generate audio alert messages if risky behavior is identified as a result of behavioral analysis.

[0138] Device: Play an audio alert through the speaker, for example, "Baby is about to fall out of his crib!"

[0139] 4. Notification System

[0140] Server: Sends push notifications to users' smartphones or PCs. To do this, you can use a notification service such as Firebase Cloud Messaging (FCM).

[0141] Users: They can check notifications on their smartphones and take action quickly.

[0142] 5. Database Management

[0143] Server: Stores the detected video data and analysis results in a database, such as MySQL (registered trademark) or PostgreSQL.

[0144] The stored data is organized and regularly backed up for later access.

[0145] Specific examples

[0146] 1. Pet monitoring

[0147] Device: A camera installed in the living room captures footage of a pet dog.

[0148] Server: Receives video data and detects the dog using an image recognition module. It then analyzes its behavior and, for example, if the dog starts biting an electrical cord, determines that this behavior is dangerous.

[0149] Server: Generates and broadcasts the audio alert "That's dangerous!". At the same time, a notification is sent to the user's smartphone saying "Your pet is chewing on the electrical cord!"

[0150] 2. Baby watching

[0151] Device: A camera installed in the child's room captures real-time footage of the baby.

[0152] Server: Receives the video data and detects the baby's position using the image recognition module.Then, the behavior analysis module detects the baby's movements to crawl out of the crib.

[0153] Server: Generates and sends an audio alert in response to this dangerous behavior: "The baby is about to fall out of the crib!" At the same time, a notification is sent to the user's smartphone saying, "The baby is moving!"

[0154] This allows users to ensure the safety of pets, babies, and small children anytime, anywhere. This system prevents danger in daily life and provides users with a sense of security.

[0155] Prompt Sentence Examples

[0156] Text-to-speech prompt for when your baby is about to fall out of his crib: "Your baby is about to fall out of his crib!"

[0157] If your pet starts chewing on an electrical cord, a push notification prompt will appear: "Your pet is chewing on the electrical cord!"

[0158] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0159] Program processing steps

[0160] Explanation divided into processing steps

[0161] Step 1: Video capture and transmission

[0162] Step 2: Receiving and buffering video data

[0163] Step 3: Image Recognition

[0164] Step 4: Behavioral analysis

[0165] Step 5: Alert Generation

[0166] Step 6: Send notifications

[0167] Step 7: Data Management and Storage

[0168] ---

[0169] Specific explanation of each processing step

[0170] Step 1: Video capture and transmission

[0171] Device: A camera connected to the device captures video from inside the home in real time, for example capturing video of the living room at 30 frames per second.

[0172] Input: Image captured by the camera.

[0173] Data processing: Encode the captured video into a compressed format such as H.264.

[0174] Output: Encoded video data.

[0175] Specific operation: The encoded video data is sent to a server via the Internet, for example, at a rate of 1MB per second.

[0176] Step 2: Receiving and buffering video data

[0177] Server: Receives video data sent from the device. For example, it waits for streaming data on a specific port on the server.

[0178] Input: Video data sent from the device.

[0179] Data calculation: The received video data is temporarily stored in a buffer. The buffer size is set so that it can hold several seconds of video data.

[0180] Output: Buffered video data.

[0181] What it does: The server receives 1MB of data per second over an inbound TCP connection and stores it in a buffer.

[0182] Step 3: Image Recognition

[0183] Server: The video data stored in the buffer is divided into frames and input into the image recognition module.

[0184] Input: Buffered frame data.

[0185] Data computation: Each frame is analyzed by an image recognition module to detect people and pets, for example using TensorFlow or OpenCV.

[0186] Output: Information about detected people and pets.

[0187] What it does: It analyzes video at 30 frames per second and runs an algorithm to detect the baby's face in each frame.

[0188] Step 4: Behavioral analysis

[0189] Server: Performs behavioral analysis based on information obtained from the image recognition module. Tracks the movements of people and pets in the video and detects specific patterns (e.g., crawling out of a crib).

[0190] Input: Person and pet information obtained from the image recognition module.

[0191] Data calculation: The behavioral analysis module tracks movements and determines whether they match defined risky behaviors.

[0192] Output: Risky behavior detection results.

[0193] Specific operation: If the baby moves to a dangerous location, the location information and movement pattern are analyzed to determine whether it constitutes dangerous behavior.

[0194] Step 5: Alert Generation

[0195] Server: If risky behavior is detected as a result of behavioral analysis, it generates an audio alert message and converts the alert message into audio using a Text-to-Speech (TTS) engine.

[0196] Input: The behavioral analysis results and the corresponding text alert message.

[0197] Data Calculation: Converts text into a voice alert message using a TTS engine.

[0198] Output: Audio alert message.

[0199] Specific behavior: Generates the message "Your baby is about to fall out of his crib!" and plays it through speakers in the home.

[0200] Step 6: Send notifications

[0201] Server: If dangerous behavior or anomalies are detected, a push notification is sent to the user's smartphone or PC. Services such as Firebase Cloud Messaging (FCM) may be used.

[0202] Input: The behavior analysis results and the corresponding notification message.

[0203] Data Computing: Deliver messages using a notification service such as FCM.

[0204] Output: Notifications sent to the user's smartphone or PC.

[0205] Specific operation: When the baby engages in dangerous behavior, the user will receive a notification on their smartphone saying, "Your baby is moving!"

[0206] Step 7: Data Management and Storage

[0207] Server: Stores the detected video data and analysis results in a database. RDBMS such as MySQL or PostgreSQL is often used.

[0208] Input: Video data and analysis results.

[0209] Data Computation: Stores data in a database, creates indexes, and creates regular backups.

[0210] Output: Data stored in a database.

[0211] Specific operation: Video data along with analysis results are stored in a database with a timestamp, allowing users to verify past data.

[0212] (Application example 1)

[0213] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0214] The challenge is to enhance home security by detecting suspicious individuals or dangerous behavior in real time and notifying users promptly. It is also necessary to send push notifications to users if suspicious behavior persists and save video frames as a record to facilitate later analysis and evidence collection.

[0215] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0216] In this invention, the server includes means for capturing image data using a camera and transmitting it to the server in real time, means for receiving the video data and performing image recognition of people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting suspicious behavior, means for issuing an audio alert to notify the user of the detected suspicious behavior, means for notifying the user of the detected suspicious behavior to a user terminal, means for notifying the user via push notification if the suspicious behavior continues, and means for saving the detected video frames as a log file, thereby enabling early detection of suspicious people and dangerous behavior and rapid response.

[0217] A "camera" is a device that captures video data and transmits the video to a server.

[0218] "Real-time" refers to processing the target images and data instantly and without delay.

[0219] A "server" is a computer system that receives video data sent from a camera and performs various data processing.

[0220] "Video data" refers to digital data of moving or still images captured by a camera.

[0221] "Image recognition" is a technology that detects and identifies specific objects such as people and animals within video data.

[0222] "Suspicious behavior" refers to behavior that is unusual or dangerous based on the results of image recognition.

[0223] "Audio alert" is a means of issuing a warning sound or voice message when suspicious activity is detected.

[0224] "User terminal" refers to a communication device such as a smartphone or PC owned by a user.

[0225] A "push notification" is an alert or notification that is sent instantly from a server to a user device.

[0226] A "log file" is a file that records system operation status and event information, and can be used for later analysis and verification.

[0227] The present invention describes an embodiment of a system for strengthening home security and detecting suspicious individuals and dangerous behavior in real time. This system is configured by combining a camera, a server, a user terminal, and various software modules.

[0228] Components

[0229] 1. Camera

[0230] Cameras installed in homes capture real-time video of what is happening in and around the home, and the video data is sent to a server via Wi-Fi or a wired connection.

[0231] 2. Server

[0232] The server receives the video data and has a buffer for temporary storage.

[0233] Video data is processed using various software modules (image recognition module, behavior analysis module, alert generation module, notification module).

[0234] 3. User Device

[0235] User devices such as smartphones and PCs communicate with the server and receive push notifications.

[0236] Software Module

[0237] 1. Image Recognition Module

[0238] The received video data is divided into frames and specific objects (people, animals) are detected.

[0239] It utilizes existing deep learning technologies such as OpenCV's DNN module.

[0240] 2. Behavioral Analysis Module

[0241] Based on the information obtained from the image recognition module, behavioral analysis is performed to detect suspicious behavior.

[0242] If suspicious behavior continues for a certain period of time, an alert is generated.

[0243] 3. Alert Generation Module

[0244] When suspicious behavior is detected, audio alerts and messages are generated and played over speakers in the home.

[0245] 4. Notification Module

[0246] The server will then send a push notification to the user's device regarding any suspicious behavior or dangerous situations that it detects.

[0247] For example, a notification such as "There's a suspicious person at the front door!" is immediately forwarded to the user.

[0248] 5. Data storage module

[0249] The detected video frames and analysis results are saved as a log file, making them available for later analysis.

[0250] Processing Description

[0251] The server receives video data from the camera in real time and temporarily stores it in a buffer. The video data is analyzed through an image recognition module to detect specific objects (people or animals). The behavior analysis module then analyzes the movements of these objects to detect any suspicious behavior.

[0252] If the detected suspicious behavior continues for a certain period of time, the alert generation module will generate an audio warning or message and send it to a speaker in the home, while the notification module will send a push notification to the user's device to immediately notify them of the suspicious activity.

[0253] In addition, the data storage module stores the detected video frames and analysis results as a log file for later analysis and evidence collection.

[0254] Examples and prompts

[0255] For example, if a suspicious person is detected at the entrance, the image recognition module will identify the person, and the behavior analysis module will analyze the person's movements. If the suspicious behavior continues, the alert generation module will play an audio message saying "There's a suspicious person at the entrance!", and at the same time, the notification module will send a push notification to the user's device.

[0256] Examples of prompts include, "What happens if an intruder is detected in your home?", "How does your system detect an intruder?", and "What is the threshold for generating an alert?"

[0257] As a result, the security system based on the present invention can significantly improve safety within the home and encourage users to take prompt and appropriate action.

[0258] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0259] Step 1:

[0260] The server receives video data from cameras installed in homes in real time. The input data is real-time video data captured by the cameras, and the server temporarily stores it in a buffer.

[0261] Step 2:

[0262] The server divides the temporarily stored video data into frames and sends them to the image recognition module. The input data here are video frames, which the image recognition module receives and detects people and animals using OpenCV's DNN module, etc. The output data is position information and probability information for objects (people and animals) present in the video frames.

[0263] Step 3:

[0264] The server sends the object information obtained from the image recognition module to the behavior analysis module. The input data is the object information, and the behavior analysis module uses this information to analyze suspicious behavior. The output data is the type of analyzed behavior and its risk level.

[0265] Step 4:

[0266] The server uses the alert generation module to create a voice alert based on the suspicious behavior information obtained from the behavior analysis module. The input data is the suspicious behavior information, and the alert generation module generates a warning message based on this. The output data is a warning message or a voice file. The device plays this warning sound or message.

[0267] Step 5:

[0268] The server notifies the user device of the suspicious behavior information. The input data is the suspicious behavior information and the generated warning message. The notification module generates a push notification based on this and sends it to the user device. The output data is the push notification message. The push notification is displayed on the device and a warning is sent to the user.

[0269] Step 6:

[0270] The server uses a data storage module to store the detected video frames and analysis results as a log file. The input data are the video frames and their analysis results, and the data storage module creates a log file based on them. The output data is the stored log file, which can be used for later analysis or as evidence.

[0271] For example, if a user asks, "What happens if a suspicious person is detected in the house?", the suspicious person will be detected through the above steps, a warning will be sent to the user, and necessary measures will be taken. In this way, the present invention improves home safety and enables early detection of suspicious people and rapid response.

[0272] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0273] MODE FOR CARRYING OUT THE INVENTION

[0274] To achieve the functions recited in the claims, the present invention provides a system including the following elements, which monitors and protects pets, babies, and young children, recognizes the user's emotions as needed, and provides corresponding alerts, allowing the user to live their daily lives with peace of mind.

[0275] Program processing overview

[0276] The present invention is a system that operates by combining a camera, a server, a terminal, and a software module, and specifically performs data processing in the following manner.

[0277] 1. Camera Interface

[0278] A camera attached to the device captures visual data from inside the home in real time.

[0279] The terminal transmits the captured video data to a server via the Internet.

[0280] 2. Receiving and processing video data

[0281] The server temporarily stores the video data received from the terminal in a buffer and divides it into frames.

[0282] The server inputs the video frames into an image recognition module.

[0283] 3. Image Recognition

[0284] An image recognition module identifies people and pets in each frame.

[0285] Identified faces and shapes are matched against a known database to determine if the subject is registered.

[0286] 4. Behavior analysis

[0287] The server analyzes behavior based on the results of image recognition and detects risky behavior.

[0288] Specifically, it detects actions such as a baby falling or a pet chewing on an electrical cord.

[0289] 5. Alert System

[0290] The server generates an audio alert for detected risky behavior.

[0291] The device will issue an audio warning through the speaker.

[0292] 6. Notification System

[0293] The server sends push notifications to users' smartphones or PCs about dangerous behavior or intrusions.

[0294] The user receives a notification and checks for more information.

[0295] 7. Emotion Engine

[0296] The emotion engine installed on the server recognizes the user's emotions, for example, by analyzing the user's voice and facial expressions to determine their emotional state.

[0297] The content of alerts and notifications is adjusted based on the analysis results of the emotion engine. For example, if the user is feeling stressed, notifications with a high level of urgency will be sent first.

[0298] 8. Database Management

[0299] The server stores the video data, image recognition results, behavioral analysis results, and emotion data in a database.

[0300] The management module organizes this data and makes it available for later reanalysis or backup.

[0301] Specific examples

[0302] 1. Pet monitoring

[0303] The device's camera captures video of your pet dog in real time.

[0304] The server receives the video data and analyzes the dog's behavior using image recognition and behavior analysis modules.

[0305] If a dog starts biting an electrical cord, the server generates a voice message saying, "That's dangerous!" and the device issues a voice warning. The emotion engine checks the user's stress level and sends an emergency notification to the smartphone if necessary.

[0306] 2. Baby watching

[0307] The device's camera captures images of the baby and sends them to a server in real time.

[0308] The server analyzes the baby's behavior and detects dangerous movements (such as an imminent fall).

[0309] If the baby tries to crawl out of the crib, the device will issue an audio alert saying "Baby moving!" The server will use an emotion engine to determine the parent's level of concern and adjust the urgency of the notification accordingly. For example, if the user is busy, they can choose not to receive detailed notifications.

[0310] This provides a system that not only monitors and protects pets, babies, and small children, but also takes into account the user's emotions, reducing stress for the user and creating a safer and more comfortable living environment.

[0311] The processing flow will be explained below.

[0312] MODE FOR CARRYING OUT THE INVENTION

[0313] In order to achieve the functions described in the claims, the present invention provides a system including the following elements: In particular, processing steps for monitoring and protecting pets, babies, and young children are specifically described.

[0314] Step 1:

[0315] The camera captures video of the inside of the house in real time at a set frame rate, with the video divided into frames at regular time intervals.

[0316] Step 2:

[0317] The device transmits the captured video frames to a server over the Internet, where the transmission is secure using an encrypted communication protocol.

[0318] Step 3:

[0319] The server stores the video frames received from the terminal in a buffer in order, allowing subsequent processing to proceed smoothly.

[0320] Step 4:

[0321] The server inputs the buffered video frames into an image recognition module to detect people or pets in each frame, for example, identifying the faces and shapes of babies or pets.

[0322] Step 5:

[0323] The image recognition module checks the detected person or pet data against a database to determine whether the target is already registered, for example, by comparing it with facial images of family members stored in the database.

[0324] Step 6:

[0325] The server passes the image recognition results to the behavioral analysis module, which then begins analyzing the behavior, identifying risky behaviors, such as a baby trying to crawl out of its crib.

[0326] Step 7:

[0327] Based on the behavioral patterns analyzed by the behavior analysis module, dangerous behavior (for example, a pet chewing on an electrical cord or a baby about to fall) is detected.

[0328] Step 8:

[0329] If the server detects risky behavior, it generates an audio alert message based on a predefined phrase.

[0330] Step 9:

[0331] The device receives the audio alert message and issues a warning through the speaker, such as "That's dangerous!"

[0332] Step 10:

[0333] The server sends a push notification to the user's smartphone or PC, which contains details of the risky behavior.

[0334] Step 11:

[0335] The user receives a push notification on their smartphone or PC and checks the details, for example by tapping on the pop-up notification to check the details of the risky behavior.

[0336] Step 12:

[0337] An emotion engine installed on the server analyzes the user's emotions in real time, detecting, for example, stress or anxiety from the user's voice and facial expressions.

[0338] Step 13:

[0339] Based on the analysis results of the emotion engine, the server adjusts the content of audio alerts and push notifications. For example, if the user is in a high stress state, a notification with more specific information will be sent.

[0340] Step 14:

[0341] The server stores the detected video data, image recognition results, behavioral analysis results, and emotion data in a database.

[0342] Step 15:

[0343] The database management module periodically backs up data and stores it for future analysis and restoration, thus preventing data loss.

[0344] This allows users to better monitor and protect their pets, babies and young children, and the alert system also takes into account the user's emotional state to provide more relevant and reassuring notifications.

[0345] Example 2

[0346] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0347] In modern homes, it is important to safely monitor pets, babies, and young children and respond quickly when necessary. However, conventional surveillance systems are limited to simple video capture and notification, and do not support specific behavioral analysis of people and animals or adjust the urgency of notifications based on the user's emotions. This makes it difficult to respond quickly and appropriately to dangerous situations, resulting in a lack of user peace of mind and convenience.

[0348] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing image data using a camera and transmitting it to the server in real time, a means for receiving video data and performing image recognition of people and animals in the video data, a means for analyzing behavior based on the image recognition results and detecting dangerous behavior, a means for issuing an audio alert to notify the user of the detected dangerous behavior, a means for notifying the user of the detected dangerous behavior to a user terminal, a means for recording analysis data and emotion data in a database and using it for later analysis, and a means for analyzing the user's emotion and adjusting the urgency of the notification. This makes it possible to monitor and protect pets, babies, and young children, and to provide appropriate alerts and notifications taking the user's emotion into consideration.

[0349] "Camera" means a device that captures visual data. Commonly used terms include digital cameras and webcams.

[0350] A "server" is a computer system that receives, processes, and manages data over a network.

[0351] A "terminal" is a device that is directly operated by a user, and is a computer to which peripheral devices such as a camera and speakers are connected.

[0352] "Image data" refers to digital data containing visual information captured by a camera.

[0353] "Real-time" refers to near-instant processing and response without delay.

[0354] "Video data" refers to a moving image made up of a series of multiple image data.

[0355] "Image recognition" is the process of automatically identifying people, animals, and objects in video data.

[0356] "Behavioral analysis" is the process of evaluating a subject's behavior based on the results of image recognition to detect specific patterns or anomalies.

[0357] "Riskful behavior" refers to behavior that puts a subject in a potentially dangerous situation and may threaten their safety.

[0358] A "voice alert" is a notification that issues a warning via voice when risky behavior is detected.

[0359] A "user terminal" refers to a computer device such as a smartphone or PC used by a user.

[0360] A "notification" is a message that notifies the user of important information or warnings.

[0361] A "database" is a system for efficiently storing, managing, and searching structured data.

[0362] "Emotional data" refers to digital data that represents a user's emotional state.

[0363] The "urgency of a notification" is an indicator of the degree to which a notification requires an immediate response.

[0364] The present invention is a system for monitoring and protecting pets, babies, and small children, and provides a safe living environment by recognizing the user's emotions and providing notifications accordingly. Specifically, it is realized using a camera, a server, a terminal, and multiple software modules.

[0365] A camera attached to the device captures video data from inside the home in real time. The hardware used is a typical surveillance camera, such as a digital camera or webcam. The captured video data is sent from the device to a server via the Internet. The software used for this is a library that streams and transmits video data, such as OpenCV or FFmpeg.

[0366] The server stores the video data received from the device in a buffer and then divides it into frames. The cloud platform used is a "cloud storage service," examples of which include "AWS (registered trademark) S3" and "Google (registered trademark) Cloud Storage." The server passes the divided frames to an image recognition module, which identifies people and pets contained in each frame. This image recognition uses an "image recognition library" or "deep learning model," specifically "YOLOv5" or "TensorFlow."

[0367] The identified target's face and shape are compared with a known database on the server to determine whether the target is registered. The behavioral analysis module then analyzes the target's movements based on the image recognition results and detects risky behavior. This behavioral analysis is performed using a "motion analysis library," such as "OpenPose" or "Scikit-learn."

[0368] If a dangerous behavior is detected, the server generates an audio alert. This is generated using a speech synthesis service, such as Amazon Polly or Google Text-to-Speech. The generated audio alert is then played through the device's speaker. The device is equipped with a standard speaker, such as a Bluetooth speaker or a smart speaker.

[0369] The server then notifies the user of any detected risky behavior via their smartphone or PC. Notifications are sent using a push notification service, which uses Firebase Cloud Messaging. Users can receive notifications and check detailed information on their smartphone or PC.

[0370] Furthermore, the server is equipped with an emotion engine that analyzes the user's voice and facial expressions to determine their emotional state. This emotion analysis uses an "emotion recognition service," which uses emotion recognition APIs from IBM Watson (registered trademark) and Microsoft (registered trademark) Azure (registered trademark). The alert content and urgency of the notification are adjusted based on the analysis results of the emotion engine. If the user is feeling stressed, notifications with a higher urgency level are sent first.

[0371] Finally, the server stores the video data, image recognition results, behavioral analysis results, and emotion data in a database. The database used is a database management system, such as MongoDB or PostgreSQL, which allows for data organization, reanalysis, and backup.

[0372] Specific examples

[0373] 1. Pet monitoring

[0374] The device's camera captures video of your pet dog in real time.

[0375] The server receives the video data and analyzes the dog's behavior using image recognition and behavior analysis modules.

[0376] If a dog starts biting an electrical cord, the server generates a voice message saying, "That's dangerous!" and the device issues a voice warning. The emotion engine checks the user's stress level and sends an emergency notification to the smartphone if necessary.

[0377] 2. Baby watching

[0378] The device's camera captures images of the baby and sends them to a server in real time.

[0379] The server analyzes the baby's behavior and detects dangerous movements (such as an imminent fall).

[0380] If the baby tries to crawl out of the crib, the device will issue an audio alert saying "Baby moving!" The server will use an emotion engine to determine the parent's level of concern and adjust the urgency of the notification accordingly. For example, if the user is busy, they can choose not to receive detailed notifications.

[0381] As described above, the present invention provides a system for monitoring and protecting pets, babies, and small children, and is flexible and considers the user's emotions, reducing stress for the user and creating a safer and more comfortable living environment.

[0382] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0383] Step 1:

[0384] Video Capture

[0385] A camera attached to the device captures visual data from inside the home in real time. Specifically, the camera continuously captures video data at about 30 frames per second. The hardware used is a digital camera or webcam. The input is the visual data captured by the camera, and the output is video data captured in real time.

[0386] Step 2:

[0387] Data transmission

[0388] The device sends the captured video data to a server via the Internet. The software used is "OpenCV" and a network library. The input is real-time video data, and the output is the video data sent to the server. Specifically, the device divides the video data into packets at regular intervals (for example, every second) and sends them to the server via the network.

[0389] Step 3:

[0390] Data reception

[0391] The server temporarily stores the video data received from the device in a buffer. The cloud platform used is a "cloud storage service" that includes AWS S3 and Google Cloud Storage. The input is the video data sent from the device, and the output is the video data stored in the buffer in the server. Specifically, the server reconstructs the received packets and stores them in the buffer as video data.

[0392] Step 4:

[0393] Frame Split

[0394] The server divides the video data into frames. This process uses the OpenCV library. The input is the video data stored in the buffer, and the output is each frame. Specifically, the server divides the video data into 30 frames per second and passes each frame to subsequent processing.

[0395] Step 5:

[0396] Image Recognition

[0397] The server inputs video frames into an image recognition module to identify people and pets in each frame. The image recognition libraries used are "YOLOv5" and "TensorFlow." The input is video data divided into frames, and the output is information about the identified objects (position, type, etc.). Specifically, the object detection algorithm is applied to each frame to obtain the object's coordinates and class label.

[0398] Step 6:

[0399] Database collation

[0400] The server compares the identified face and shape with a known database. The database includes "face recognition software" and "shape recognition modules." The input is the information of the identified subject, and the output is whether the subject is registered in the database. Specifically, the server extracts the subject's features and compares them with entries in the database to determine whether they match.

[0401] Step 7:

[0402] Behavioral analysis

[0403] The server analyzes behavior based on the results of image recognition and detects risky behavior. The software libraries used are "OpenPose" and "Scikit-learn." The input is the movement and location information of the identified object, and the output is the behavior analysis results and the detection of risky behavior. Specifically, the server analyzes time-series data and detects specific patterns (e.g., falls, biting electrical cords, etc.).

[0404] Step 8:

[0405] Audio alert generation

[0406] If the server detects risky behavior, it immediately generates a corresponding audio alert. This audio alert is generated using a "speech synthesis service," specifically "Amazon Polly" or "Google Text-to-Speech." The input is the detection result of risky behavior, and the output is the generated audio file. Specifically, the server sends a text message to the speech synthesis service, which generates audio data.

[0407] Step 9:

[0408] Voice alert

[0409] The device issues the audio alert sent from the server through a speaker. A typical speaker, such as a Bluetooth speaker or a smart speaker, is attached to the device. The input is the generated audio file, and the output is the transmission of the audio alert. Specifically, the device plays the audio file and issues the audio alert from the speaker.

[0410] Step 10:

[0411] Send notifications

[0412] The server sends push notifications of detected risky behavior or intrusions to the user's smartphone or PC. The push notification service used is "Firebase Cloud Messaging." The input is the detection result of risky behavior and the notification content, and the output is the notification sent to the user's device. Specifically, the server generates a push notification message and sends it to the user's device via the cloud messaging service.

[0413] Step 11:

[0414] emotion recognition

[0415] The emotion engine installed on the server recognizes the user's emotions. Emotion analysis uses "emotion recognition services," including "IBM Watson" and "Microsoft Azure." The input is data based on the user's voice and facial expressions, and the output is the user's emotional state. Specifically, the server analyzes the user's voice data and facial expression data and evaluates the state using the emotion engine.

[0416] Step 12:

[0417] Emotion-based notification adjustment

[0418] The server adjusts the alert content and notification urgency based on the analysis results of the emotion engine. The input is the user's emotional state and the detection results of risky behavior, and the output is the adjusted alert and notification. Specifically, the server determines whether the user is feeling stressed and sets the system to send notifications with a high level of urgency first.

[0419] Step 13:

[0420] Data storage

[0421] The server stores video data, image recognition results, behavioral analysis results, and emotion data in a database. The database management systems used are "MongoDB" and "PostgreSQL." The input is the various analysis result data, and the output is the data stored in the database. Specifically, the server formats the various data appropriately and inserts it into the database.

[0422] (Application example 2)

[0423] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0424] There is a need for monitoring and protecting pets, babies, and young children in the home, and for providing prompt and appropriate alerts through user emotion recognition when necessary. Conventional systems focus on detecting dangerous behavior, but are unable to consider the user's emotional state, making it difficult to reduce stress and anxiety. Furthermore, there are challenges in detecting dangerous behavior and providing notifications accurately and immediately.

[0425] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0426] In this invention, the server includes means for capturing image data using a camera and transmitting it to the server in real time, means for receiving the video data and performing image recognition of people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting dangerous behavior, means for issuing an audio alert to notify the user of the detected dangerous behavior, means for notifying the user of the detected dangerous behavior to a user terminal, and means for adjusting the content of the notification or alert based on the emotion recognition results. This makes it possible to monitor and protect pets, babies, and young children in the home, and allows for flexible responses according to the user's emotional state, providing a safer and more secure living environment.

[0427] A "camera" is a device for capturing image data.

[0428] A "server" is a central computer for receiving, storing, processing, and distributing data over a network.

[0429] "Image data" refers to visual information captured using a camera.

[0430] "Real-time" refers to processing occurring immediately without delay.

[0431] "Video data" refers to moving image information that is made up of a series of multiple image frames.

[0432] A "person" is an object that shows a human shape or face.

[0433] "Animals" refers to non-human living things such as pets and livestock.

[0434] "Image recognition" is the technology of identifying and analyzing objects and patterns in digital images.

[0435] "Behavioral analysis" is the process of evaluating and analyzing the behavior of a subject based on the results of image recognition.

[0436] "Dangerous behavior" is behavior that has the potential to cause accidents or trouble.

[0437] A "voice alert" is a warning method that calls attention by issuing a voice message.

[0438] A "user terminal" is a device (such as a smartphone or PC) that is directly operated by a user.

[0439] A "notification" is a means of informing a user of specific information.

[0440] "Emotion recognition" is a technology that analyzes a user's voice, facial expressions, etc. to determine their emotional state.

[0441] "Notification content" is the content of the message sent to the user.

[0442] "Alert content" refers to the content of a message to warn or alert.

[0443] A "database" is a system for managing an organized collection of data.

[0444] The present invention is a system for monitoring and protecting pets, babies, and small children in the home, recognizing the user's emotions and providing corresponding alerts as needed. The system operates using cameras, a server, a terminal, and multiple software modules.

[0445] Hardware and Software Configuration

[0446] Cameras: Use the built-in camera on your smartphone or a home security camera that captures video data in real time and sends it to a server.

[0447] Server: Acts as the central computer that receives, stores, processes, and distributes data. The server contains the following main modules:

[0448] Image Recognition Module: Uses TensorFlow and Keras to identify people and animals in video frames.

[0449] Behavioral analysis module: Detects risky behavior based on image recognition results.

[0450] Emotion Recognition Module: Uses TensorFlow and Keras to analyze the user's voice and facial expressions to determine their emotional state.

[0451] Database management module: Stores video data and analysis results for later reanalysis and backup.

[0452] Device: A smartphone or PC used by a user. These devices receive notifications and alerts from the server and transmit them to the user.

[0453] Communication: Data communication between the server, terminal, and camera is carried out via the Internet.

[0454] Program processing

[0455] The server receives video data captured by the camera in real time and temporarily stores it in a buffer. Each frame is input into an image recognition module to identify people and animals. Based on the identification results, a behavior analysis module detects dangerous behavior and generates an audio alert. It also sends a notification of the dangerous behavior to the user's device.

[0456] To take the user's emotions into account, the emotion recognition module recognizes the user's emotional state and adjusts the content of notifications and alerts based on the analysis results. For example, if the user is feeling stressed, notifications with a high level of urgency can be sent first.

[0457] Usage example

[0458] Pet Monitoring:

[0459] If the system detects that a pet dog is biting an electrical cord, it will generate an audio alert saying "That's dangerous!" and send a notification to a smartphone. It will also prioritize emergency notifications if it detects that the user is in a stressful state.

[0460] Baby monitoring:

[0461] When the baby is about to crawl out of the crib, the system will issue an audio alert saying "Baby moving!", and an emotion recognition module will analyze the parent's emotions to adjust the urgency of the notification - for example, it can reduce detailed notifications if the parent is busy.

[0462] Prompt Sentence Examples

[0463] If you want to generate a pet watching scenario, the prompt is as follows:

[0464] "Generate a pet monitoring scenario"

[0465] If you want to generate a baby watching scenario, the prompt is as follows:

[0466] "Generate a baby watching scenario"

[0467] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0468] Step 1:

[0469] Camera Interface:

[0470] The device uses a camera to capture video data from within the home in real time. The input is the video data captured by the camera, and the output is frames of this video data. The device encodes these frames and transmits them to a server via the Internet.

[0471] Step 2:

[0472] Video data reception and processing:

[0473] The server receives the video data sent from the device and temporarily stores it in a buffer. The input is the encoded video data, and the output is each divided frame. The server then decodes these frames sequentially and inputs them into the image recognition module.

[0474] Step 3:

[0475] Image Recognition:

[0476] The server uses TensorFlow and Keras to input video frames into an image recognition module. The input is image data for each frame, and the output is information about identified people and animals. The server analyzes each frame to identify people and animals and verify their identity by matching them with a known database.

[0477] Step 4:

[0478] Behavior analysis:

[0479] The server analyzes behavior based on the image recognition results and detects dangerous behavior. The input is the identification information obtained from the image recognition module, and the output is the type and urgency of the detected dangerous behavior. For example, it can detect when a baby is about to fall or when a pet is biting an electrical cord.

[0480] Step 5:

[0481] Alert System:

[0482] The server generates an audio alert for the detected risky behavior and issues it to the device. The input is the behavior analysis results, and the output is the audio alert data. The device receives this data and plays the audio alert through its built-in speaker.

[0483] Step 6:

[0484] Notification system:

[0485] The server sends push notifications of risky behavior and intrusions to the user's smartphone or PC. The input is the behavior analysis results and emotion recognition results, and the output is a notification message sent to the user's device. The user receives the notification and checks the detailed information.

[0486] Step 7:

[0487] Emotion Engine:

[0488] The server uses an emotion engine to analyze the user's voice and facial expressions to determine their emotional state. The input is the user's audio and video data, and the output is an estimated emotional state. Based on this result, the server adjusts the content and urgency of alerts and notifications.

[0489] Step 8:

[0490] Database Management:

[0491] The server stores video data, image recognition results, behavioral analysis results, and emotion data in a database. The input is all processed data, and the output is historical data stored in the database. This allows for future reanalysis and backup.

[0492] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0493] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0494] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0495] [Second embodiment]

[0496] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0497] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0498] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0499] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0500] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0501] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0502] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0503] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0504] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0505] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0506] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0507] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0508] MODE FOR CARRYING OUT THE INVENTION

[0509] To achieve the functions described in the claims, the present invention provides a system that includes the following elements: Specifically, a camera, a server, a terminal, and a software module are combined to create an environment in which users can live their daily lives with peace of mind.

[0510] Program processing overview

[0511] The present invention is a system that processes data between devices as follows, and ultimately provides the user with the information they need.

[0512] 1. Camera Interface

[0513] A camera connected to the device captures images of the inside of the house in real time, for example, periodically taking pictures of the living room or children's room.

[0514] The video data captured by the terminal is transmitted to a server via the Internet.

[0515] 2. Receiving and processing video data

[0516] The server temporarily stores the video data received from the terminal in a buffer.

[0517] The server divides the video data into frames and inputs them into an image recognition module.

[0518] 3. Image Recognition

[0519] The image recognition module detects people and pets in the frame and recognizes their faces and shapes, for example, the face of a baby or a pet dog.

[0520] The image recognition module compares the image against a database to determine whether it is a known person or pet.

[0521] 4. Behavior analysis

[0522] The server performs behavioral analysis based on the information obtained from the image recognition module, for example, detecting a baby trying to crawl out of its crib.

[0523] The behavior analysis module develops defined risky behaviors and determines whether the detected behavior matches them.

[0524] 5. Alert System

[0525] The server generates an audio alert message if risky behavior is detected.

[0526] The device will emit a warning sound or message through the speaker to the user, for example, "Your baby is about to fall out of his crib!"

[0527] 6. Notification System

[0528] The server sends push notifications to users' smartphones or PCs, instantly informing them of any dangerous behavior or intruders.

[0529] For example, if a pet starts chewing on an electrical cord, the user will receive a notification on their smartphone saying, "Your pet is chewing on the electrical cord!"

[0530] 7. Database Management

[0531] The server stores the detected video data and analysis results in a database.

[0532] The database management module organizes the data for later access and creates backups as needed.

[0533] Specific examples

[0534] 1. Pet monitoring

[0535] A camera installed on the device periodically captures footage of the pet dog.

[0536] The server receives the video data and analyzes the dog's behavior using image recognition and behavior analysis modules.

[0537] If a dog starts chewing on an electrical cord, the server will play an audio message saying "That's dangerous!" and simultaneously send a notification to the user's smartphone.

[0538] 2. Baby watching

[0539] The device's camera captures images of the baby in real time.

[0540] The server analyzes the baby's behavior and detects dangerous movements.

[0541] When the baby tries to crawl out of the crib, the device will emit an audio alert saying "Baby is moving!" and send a push notification to the user's smartphone.

[0542] This allows users to ensure the safety of pets, babies, and small children anytime, anywhere. This system prevents danger in daily life and provides users with a sense of security.

[0543] The processing flow will be explained below.

[0544] Step 1:

[0545] The device's camera captures video of the home in real time at a set frame rate, and the video data is divided into frames at regular time intervals.

[0546] Step 2:

[0547] The captured video frames are sent directly to the server using a dedicated communication protocol, ensuring data security.

[0548] Step 3:

[0549] The server stores the video frames received from the terminal in a buffer in order, allowing subsequent processing to proceed smoothly.

[0550] Step 4:

[0551] The server sends the buffered video frames to an image recognition module that detects people and pets in each frame, using specific algorithms to identify faces and shapes.

[0552] Step 5:

[0553] The image recognition module compares the detection results with a database to determine whether the target is registered, for example, by comparing the faces of family members with facial images stored in the database.

[0554] Step 6:

[0555] The server passes the image recognition results to the behavior analysis module, which then analyzes multiple behavioral patterns to determine whether the target's behavior is dangerous.

[0556] Step 7:

[0557] The behavior analysis module analyzes behavioral patterns and detects dangerous behavior (for example, a baby about to fall or a pet chewing on an electrical cord).

[0558] Step 8:

[0559] If the server detects any risky behavior, it will instantly generate an audio alert message, which is generated based on pre-defined phrases.

[0560] Step 9:

[0561] The device receives the audio alert message and issues a voice warning through the speaker, for example, "That's dangerous!"

[0562] Step 10:

[0563] The server then sends a push notification to the user's smartphone or PC in parallel, containing details of the risky behavior detected.

[0564] Step 11:

[0565] Users can check notifications on their smartphones or PCs and rush to the scene if necessary. For example, they can tap the pop-up notification to check details.

[0566] Step 12:

[0567] The server stores the detected video data and analysis results in a database, which can be used for later review and analysis.

[0568] Step 13:

[0569] The database management module periodically creates backups of data and makes them available for system recovery, thereby preventing data loss.

[0570] Example 1

[0571] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0572] Ensuring the safety of children and pets is a major concern in modern homes. Conventional surveillance systems simply record video footage, but are unable to detect danger or send alerts in real time, making it difficult to respond to situations that require a rapid response. Furthermore, there is a lack of means to manage and analyze massive amounts of video data, making efficient data storage and use a challenge.

[0573] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0574] In this invention, the server includes means for capturing image data using a camera and transmitting it to the server in real time, means for receiving the video data and performing image recognition of people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting dangerous behavior, means for issuing an audio alert to notify the user of the detected dangerous behavior, means for notifying the user of the detected dangerous behavior, means for temporarily storing the video data in a buffer, means for storing the detected video data and analysis results in a database for later access, means for analyzing each frame using an image recognition module, means for tracking movements in the video using a behavior analysis module and detecting specific patterns, and means for generating an audio alert message. This enables real-time monitoring and analysis of video data inside the home, rapid detection of dangerous behavior and issuing an alert, and efficient storage and management of video data.

[0575] A "camera" is an optical device for capturing image data.

[0576] A "server" is a computer system that processes and manages data in response to requests from clients.

[0577] A "terminal" is a computer system or device that has input and output devices and can be directly operated by a user.

[0578] "Image data" refers to visual information captured by a camera and stored in digital form.

[0579] "Real-time" means near-instant processing with minimal delay.

[0580] A "buffer" is a memory area that functions as a temporary storage area for data and ensures a smooth flow of data processing.

[0581] "Image recognition" is the process of using computer vision techniques to detect specific objects or patterns in video data.

[0582] "Behavioral analysis" is the process of analyzing patterns of movement and behavior to predict intentions and next actions.

[0583] A "voice alert" is a system that uses voice to warn or notify.

[0584] A "user terminal" is a device operated by a user who uses the system, and typically refers to a smartphone or personal computer.

[0585] A "database" is a structured data storage system for efficiently managing, retrieving, and storing data.

[0586] An "image recognition module" is a software component that analyzes image data to recognize specific objects or people.

[0587] The "behavioral analysis module" is a software component that analyzes behavior from recognized data and detects specific patterns and anomalies.

[0588] A "voice alert message" is a message containing a warning or notification generated using voice synthesis technology.

[0589] "People and animals" refers to living beings such as humans and pets that appear in image data.

[0590] This invention combines cameras, servers, terminals, and software modules to provide an environment in which users can live their daily lives with peace of mind. Specifically, it is a system that monitors images from inside the home in real time and issues alerts and notifications as needed.

[0591] Hardware and Software Configuration

[0592] 1. Camera

[0593] Device: A camera is used to capture video from inside the home. For example, cameras installed in the living room or children's room capture video data in real time.

[0594] The video data is transmitted to a server via the Internet.

[0595] 2. Server

[0596] Server: Receives video data sent from the device and temporarily stores it in a buffer. This buffer can hold several seconds of video.

[0597] The received video data is divided into frames and input into the image recognition module.

[0598] Image Recognition Module: Detects people and pets from video frames and recognizes their faces and shapes, for example using TensorFlow or OpenCV.

[0599] A software module (behavior analysis module) runs to perform behavior analysis based on the recognized information.

[0600] The module detects risky behavior and handles the results appropriately.

[0601] 3. Alert System

[0602] Server: Uses a text-to-speech (TTS) engine to generate audio alert messages if risky behavior is identified as a result of behavioral analysis.

[0603] Device: Play an audio alert through the speaker, for example, "Baby is about to fall out of his crib!"

[0604] 4. Notification System

[0605] Server: Sends push notifications to users' smartphones or PCs. To do this, you can use a notification service such as Firebase Cloud Messaging (FCM).

[0606] Users: They can check notifications on their smartphones and take action quickly.

[0607] 5. Database Management

[0608] Server: Stores the detected video data and analysis results in a database, such as MySQL or PostgreSQL.

[0609] The stored data is organized and regularly backed up for later access.

[0610] Specific examples

[0611] 1. Pet monitoring

[0612] Device: A camera installed in the living room captures footage of a pet dog.

[0613] Server: Receives video data and detects the dog using an image recognition module. It then analyzes its behavior and, for example, if the dog starts biting an electrical cord, determines that this behavior is dangerous.

[0614] Server: Generates and broadcasts the audio alert "That's dangerous!". At the same time, a notification is sent to the user's smartphone saying "Your pet is chewing on the electrical cord!"

[0615] 2. Baby watching

[0616] Device: A camera installed in the child's room captures real-time footage of the baby.

[0617] Server: Receives the video data and detects the baby's position using the image recognition module.Then, the behavior analysis module detects the baby's movements to crawl out of the crib.

[0618] Server: Generates and sends an audio alert in response to this dangerous behavior: "The baby is about to fall out of the crib!" At the same time, a notification is sent to the user's smartphone saying, "The baby is moving!"

[0619] This allows users to ensure the safety of pets, babies, and small children anytime, anywhere. This system prevents danger in daily life and provides users with a sense of security.

[0620] Prompt Sentence Examples

[0621] Text-to-speech prompt for when your baby is about to fall out of his crib: "Your baby is about to fall out of his crib!"

[0622] If your pet starts chewing on an electrical cord, a push notification prompt will appear: "Your pet is chewing on the electrical cord!"

[0623] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0624] Program processing steps

[0625] Explanation divided into processing steps

[0626] Step 1: Video capture and transmission

[0627] Step 2: Receiving and buffering video data

[0628] Step 3: Image Recognition

[0629] Step 4: Behavioral analysis

[0630] Step 5: Alert Generation

[0631] Step 6: Send notifications

[0632] Step 7: Data Management and Storage

[0633] ---

[0634] Specific explanation of each processing step

[0635] Step 1: Video capture and transmission

[0636] Device: A camera connected to the device captures video from inside the home in real time, for example capturing video of the living room at 30 frames per second.

[0637] Input: Image captured by the camera.

[0638] Data processing: Encode the captured video into a compressed format such as H.264.

[0639] Output: Encoded video data.

[0640] Specific operation: The encoded video data is sent to a server via the Internet, for example, at a rate of 1MB per second.

[0641] Step 2: Receiving and buffering video data

[0642] Server: Receives video data sent from the device. For example, it waits for streaming data on a specific port on the server.

[0643] Input: Video data sent from the device.

[0644] Data calculation: The received video data is temporarily stored in a buffer. The buffer size is set so that it can hold several seconds of video data.

[0645] Output: Buffered video data.

[0646] What it does: The server receives 1MB of data per second over an inbound TCP connection and stores it in a buffer.

[0647] Step 3: Image Recognition

[0648] Server: The video data stored in the buffer is divided into frames and input into the image recognition module.

[0649] Input: Buffered frame data.

[0650] Data computation: Each frame is analyzed by an image recognition module to detect people and pets, for example using TensorFlow or OpenCV.

[0651] Output: Information about detected people and pets.

[0652] What it does: It analyzes video at 30 frames per second and runs an algorithm to detect the baby's face in each frame.

[0653] Step 4: Behavioral analysis

[0654] Server: Performs behavioral analysis based on information obtained from the image recognition module. Tracks the movements of people and pets in the video and detects specific patterns (e.g., crawling out of a crib).

[0655] Input: Person and pet information obtained from the image recognition module.

[0656] Data calculation: The behavioral analysis module tracks movements and determines whether they match defined risky behaviors.

[0657] Output: Risky behavior detection results.

[0658] Specific operation: If the baby moves to a dangerous location, the location information and movement pattern are analyzed to determine whether it constitutes dangerous behavior.

[0659] Step 5: Alert Generation

[0660] Server: If risky behavior is detected as a result of behavioral analysis, it generates an audio alert message and converts the alert message into audio using a Text-to-Speech (TTS) engine.

[0661] Input: The behavioral analysis results and the corresponding text alert message.

[0662] Data Calculation: Converts text into a voice alert message using a TTS engine.

[0663] Output: Audio alert message.

[0664] Specific behavior: Generates the message "Your baby is about to fall out of his crib!" and plays it through speakers in the home.

[0665] Step 6: Send notifications

[0666] Server: If dangerous behavior or anomalies are detected, a push notification is sent to the user's smartphone or PC. Services such as Firebase Cloud Messaging (FCM) may be used.

[0667] Input: The behavior analysis results and the corresponding notification message.

[0668] Data Computing: Deliver messages using a notification service such as FCM.

[0669] Output: Notifications sent to the user's smartphone or PC.

[0670] Specific operation: When the baby engages in dangerous behavior, the user will receive a notification on their smartphone saying, "Your baby is moving!"

[0671] Step 7: Data Management and Storage

[0672] Server: Stores the detected video data and analysis results in a database. RDBMS such as MySQL or PostgreSQL is often used.

[0673] Input: Video data and analysis results.

[0674] Data Computation: Stores data in a database, creates indexes, and creates regular backups.

[0675] Output: Data stored in a database.

[0676] Specific operation: Video data along with analysis results are stored in a database with a timestamp, allowing users to verify past data.

[0677] (Application example 1)

[0678] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0679] The challenge is to enhance home security by detecting suspicious individuals or dangerous behavior in real time and notifying users promptly. It is also necessary to send push notifications to users if suspicious behavior persists and save video frames as a record to facilitate later analysis and evidence collection.

[0680] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0681] In this invention, the server includes means for capturing image data using a camera and transmitting it to the server in real time, means for receiving the video data and performing image recognition of people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting suspicious behavior, means for issuing an audio alert to notify the user of the detected suspicious behavior, means for notifying the user of the detected suspicious behavior to a user terminal, means for notifying the user via push notification if the suspicious behavior continues, and means for saving the detected video frames as a log file, thereby enabling early detection of suspicious people and dangerous behavior and rapid response.

[0682] A "camera" is a device that captures video data and transmits the video to a server.

[0683] "Real-time" refers to processing the target images and data instantly and without delay.

[0684] A "server" is a computer system that receives video data sent from a camera and performs various data processing.

[0685] "Video data" refers to digital data of moving or still images captured by a camera.

[0686] "Image recognition" is a technology that detects and identifies specific objects such as people and animals within video data.

[0687] "Suspicious behavior" refers to behavior that is unusual or dangerous based on the results of image recognition.

[0688] "Audio alert" is a means of issuing a warning sound or voice message when suspicious activity is detected.

[0689] "User terminal" refers to a communication device such as a smartphone or PC owned by a user.

[0690] A "push notification" is an alert or notification that is sent instantly from a server to a user device.

[0691] A "log file" is a file that records system operation status and event information, and can be used for later analysis and verification.

[0692] The present invention describes an embodiment of a system for strengthening home security and detecting suspicious individuals and dangerous behavior in real time. This system is configured by combining a camera, a server, a user terminal, and various software modules.

[0693] Components

[0694] 1. Camera

[0695] Cameras installed in homes capture real-time video of what is happening in and around the home, and the video data is sent to a server via Wi-Fi or a wired connection.

[0696] 2. Server

[0697] The server receives the video data and has a buffer for temporary storage.

[0698] Video data is processed using various software modules (image recognition module, behavior analysis module, alert generation module, notification module).

[0699] 3. User Device

[0700] User devices such as smartphones and PCs communicate with the server and receive push notifications.

[0701] Software Module

[0702] 1. Image Recognition Module

[0703] The received video data is divided into frames and specific objects (people, animals) are detected.

[0704] It utilizes existing deep learning technologies such as OpenCV's DNN module.

[0705] 2. Behavioral Analysis Module

[0706] Based on the information obtained from the image recognition module, behavioral analysis is performed to detect suspicious behavior.

[0707] If suspicious behavior continues for a certain period of time, an alert is generated.

[0708] 3. Alert Generation Module

[0709] When suspicious behavior is detected, audio alerts and messages are generated and played over speakers in the home.

[0710] 4. Notification Module

[0711] The server will then send a push notification to the user's device regarding any suspicious behavior or dangerous situations that it detects.

[0712] For example, a notification such as "There's a suspicious person at the front door!" is immediately forwarded to the user.

[0713] 5. Data storage module

[0714] The detected video frames and analysis results are saved as a log file, making them available for later analysis.

[0715] Processing Description

[0716] The server receives video data from the camera in real time and temporarily stores it in a buffer. The video data is analyzed through an image recognition module to detect specific objects (people or animals). The behavior analysis module then analyzes the movements of these objects to detect any suspicious behavior.

[0717] If the detected suspicious behavior continues for a certain period of time, the alert generation module will generate an audio warning or message and send it to a speaker in the home, while the notification module will send a push notification to the user's device to immediately notify them of the suspicious activity.

[0718] In addition, the data storage module stores the detected video frames and analysis results as a log file for later analysis and evidence collection.

[0719] Examples and prompts

[0720] For example, if a suspicious person is detected at the entrance, the image recognition module will identify the person, and the behavior analysis module will analyze the person's movements. If the suspicious behavior continues, the alert generation module will play an audio message saying "There's a suspicious person at the entrance!", and at the same time, the notification module will send a push notification to the user's device.

[0721] Examples of prompts include, "What happens if an intruder is detected in your home?", "How does your system detect an intruder?", and "What is the threshold for generating an alert?"

[0722] As a result, the security system based on the present invention can significantly improve safety within the home and encourage users to take prompt and appropriate action.

[0723] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0724] Step 1:

[0725] The server receives video data from cameras installed in homes in real time. The input data is real-time video data captured by the cameras, and the server temporarily stores it in a buffer.

[0726] Step 2:

[0727] The server divides the temporarily stored video data into frames and sends them to the image recognition module. The input data here are video frames, which the image recognition module receives and detects people and animals using OpenCV's DNN module, etc. The output data is position information and probability information for objects (people and animals) present in the video frames.

[0728] Step 3:

[0729] The server sends the object information obtained from the image recognition module to the behavior analysis module. The input data is the object information, and the behavior analysis module uses this information to analyze suspicious behavior. The output data is the type of analyzed behavior and its risk level.

[0730] Step 4:

[0731] The server uses the alert generation module to create a voice alert based on the suspicious behavior information obtained from the behavior analysis module. The input data is the suspicious behavior information, and the alert generation module generates a warning message based on this. The output data is a warning message or a voice file. The device plays this warning sound or message.

[0732] Step 5:

[0733] The server notifies the user device of the suspicious behavior information. The input data is the suspicious behavior information and the generated warning message. The notification module generates a push notification based on this and sends it to the user device. The output data is the push notification message. The push notification is displayed on the device and a warning is sent to the user.

[0734] Step 6:

[0735] The server uses a data storage module to store the detected video frames and analysis results as a log file. The input data are the video frames and their analysis results, and the data storage module creates a log file based on them. The output data is the stored log file, which can be used for later analysis or as evidence.

[0736] For example, if a user asks, "What happens if a suspicious person is detected in the house?", the suspicious person will be detected through the above steps, a warning will be sent to the user, and necessary measures will be taken. In this way, the present invention improves home safety and enables early detection of suspicious people and rapid response.

[0737] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0738] MODE FOR CARRYING OUT THE INVENTION

[0739] To achieve the functions recited in the claims, the present invention provides a system including the following elements, which monitors and protects pets, babies, and young children, recognizes the user's emotions as needed, and provides corresponding alerts, allowing the user to live their daily lives with peace of mind.

[0740] Program processing overview

[0741] The present invention is a system that operates by combining a camera, a server, a terminal, and a software module, and specifically performs data processing in the following manner.

[0742] 1. Camera Interface

[0743] A camera attached to the device captures visual data from inside the home in real time.

[0744] The terminal transmits the captured video data to a server via the Internet.

[0745] 2. Receiving and processing video data

[0746] The server temporarily stores the video data received from the terminal in a buffer and divides it into frames.

[0747] The server inputs the video frames into an image recognition module.

[0748] 3. Image Recognition

[0749] An image recognition module identifies people and pets in each frame.

[0750] Identified faces and shapes are matched against a known database to determine if the subject is registered.

[0751] 4. Behavior analysis

[0752] The server analyzes behavior based on the results of image recognition and detects risky behavior.

[0753] Specifically, it detects actions such as a baby falling or a pet chewing on an electrical cord.

[0754] 5. Alert System

[0755] The server generates an audio alert for detected risky behavior.

[0756] The device will issue an audio warning through the speaker.

[0757] 6. Notification System

[0758] The server sends push notifications to users' smartphones or PCs about dangerous behavior or intrusions.

[0759] The user receives a notification and checks for more information.

[0760] 7. Emotion Engine

[0761] The emotion engine installed on the server recognizes the user's emotions, for example, by analyzing the user's voice and facial expressions to determine their emotional state.

[0762] The content of alerts and notifications is adjusted based on the analysis results of the emotion engine. For example, if the user is feeling stressed, notifications with a high level of urgency will be sent first.

[0763] 8. Database Management

[0764] The server stores the video data, image recognition results, behavioral analysis results, and emotion data in a database.

[0765] The management module organizes this data and makes it available for later reanalysis or backup.

[0766] Specific examples

[0767] 1. Pet monitoring

[0768] The device's camera captures video of your pet dog in real time.

[0769] The server receives the video data and analyzes the dog's behavior using image recognition and behavior analysis modules.

[0770] If a dog starts biting an electrical cord, the server generates a voice message saying, "That's dangerous!" and the device issues a voice warning. The emotion engine checks the user's stress level and sends an emergency notification to the smartphone if necessary.

[0771] 2. Baby watching

[0772] The device's camera captures images of the baby and sends them to a server in real time.

[0773] The server analyzes the baby's behavior and detects dangerous movements (such as an imminent fall).

[0774] If the baby tries to crawl out of the crib, the device will issue an audio alert saying "Baby moving!" The server will use an emotion engine to determine the parent's level of concern and adjust the urgency of the notification accordingly. For example, if the user is busy, they can choose not to receive detailed notifications.

[0775] This provides a system that not only monitors and protects pets, babies, and small children, but also takes into account the user's emotions, reducing stress for the user and creating a safer and more comfortable living environment.

[0776] The processing flow will be explained below.

[0777] MODE FOR CARRYING OUT THE INVENTION

[0778] In order to achieve the functions described in the claims, the present invention provides a system including the following elements: In particular, processing steps for monitoring and protecting pets, babies, and young children are specifically described.

[0779] Step 1:

[0780] The camera captures video of the inside of the house in real time at a set frame rate, with the video divided into frames at regular time intervals.

[0781] Step 2:

[0782] The device transmits the captured video frames to a server over the Internet, where the transmission is secure using an encrypted communication protocol.

[0783] Step 3:

[0784] The server stores the video frames received from the terminal in a buffer in order, allowing subsequent processing to proceed smoothly.

[0785] Step 4:

[0786] The server inputs the buffered video frames into an image recognition module to detect people or pets in each frame, for example, identifying the faces and shapes of babies or pets.

[0787] Step 5:

[0788] The image recognition module checks the detected person or pet data against a database to determine whether the target is already registered, for example, by comparing it with facial images of family members stored in the database.

[0789] Step 6:

[0790] The server passes the image recognition results to the behavioral analysis module, which then begins analyzing the behavior, identifying risky behaviors, such as a baby trying to crawl out of its crib.

[0791] Step 7:

[0792] Based on the behavioral patterns analyzed by the behavior analysis module, dangerous behavior (for example, a pet chewing on an electrical cord or a baby about to fall) is detected.

[0793] Step 8:

[0794] If the server detects risky behavior, it generates an audio alert message based on a predefined phrase.

[0795] Step 9:

[0796] The device receives the audio alert message and issues a warning through the speaker, such as "That's dangerous!"

[0797] Step 10:

[0798] The server sends a push notification to the user's smartphone or PC, which contains details of the risky behavior.

[0799] Step 11:

[0800] The user receives a push notification on their smartphone or PC and checks the details, for example by tapping on the pop-up notification to check the details of the risky behavior.

[0801] Step 12:

[0802] An emotion engine installed on the server analyzes the user's emotions in real time, detecting, for example, stress or anxiety from the user's voice and facial expressions.

[0803] Step 13:

[0804] Based on the analysis results of the emotion engine, the server adjusts the content of audio alerts and push notifications. For example, if the user is in a high stress state, a notification with more specific information will be sent.

[0805] Step 14:

[0806] The server stores the detected video data, image recognition results, behavioral analysis results, and emotion data in a database.

[0807] Step 15:

[0808] The database management module periodically backs up data and stores it for future analysis and restoration, thus preventing data loss.

[0809] This allows users to better monitor and protect their pets, babies and young children, and the alert system also takes into account the user's emotional state to provide more relevant and reassuring notifications.

[0810] Example 2

[0811] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0812] In modern homes, it is important to safely monitor pets, babies, and young children and respond quickly when necessary. However, conventional surveillance systems are limited to simple video capture and notification, and do not support specific behavioral analysis of people and animals or adjust the urgency of notifications based on the user's emotions. This makes it difficult to respond quickly and appropriately to dangerous situations, resulting in a lack of user peace of mind and convenience.

[0813] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing image data using a camera and transmitting it to the server in real time, a means for receiving video data and performing image recognition of people and animals in the video data, a means for analyzing behavior based on the image recognition results and detecting dangerous behavior, a means for issuing an audio alert to notify the user of the detected dangerous behavior, a means for notifying the user of the detected dangerous behavior to a user terminal, a means for recording analysis data and emotion data in a database and using it for later analysis, and a means for analyzing the user's emotion and adjusting the urgency of the notification. This makes it possible to monitor and protect pets, babies, and young children, and to provide appropriate alerts and notifications taking the user's emotion into consideration.

[0814] "Camera" means a device that captures visual data. Commonly used terms include digital cameras and webcams.

[0815] A "server" is a computer system that receives, processes, and manages data over a network.

[0816] A "terminal" is a device that is directly operated by a user, and is a computer to which peripheral devices such as a camera and speakers are connected.

[0817] "Image data" refers to digital data containing visual information captured by a camera.

[0818] "Real-time" refers to near-instant processing and response without delay.

[0819] "Video data" refers to a moving image made up of a series of multiple image data.

[0820] "Image recognition" is the process of automatically identifying people, animals, and objects in video data.

[0821] "Behavioral analysis" is the process of evaluating a subject's behavior based on the results of image recognition to detect specific patterns or anomalies.

[0822] "Riskful behavior" refers to behavior that puts a subject in a potentially dangerous situation and may threaten their safety.

[0823] A "voice alert" is a notification that issues a warning via voice when risky behavior is detected.

[0824] A "user terminal" refers to a computer device such as a smartphone or PC used by a user.

[0825] A "notification" is a message that notifies the user of important information or warnings.

[0826] A "database" is a system for efficiently storing, managing, and searching structured data.

[0827] "Emotional data" refers to digital data that represents a user's emotional state.

[0828] The "urgency of a notification" is an indicator of the degree to which a notification requires an immediate response.

[0829] The present invention is a system for monitoring and protecting pets, babies, and small children, and provides a safe living environment by recognizing the user's emotions and providing notifications accordingly. Specifically, it is realized using a camera, a server, a terminal, and multiple software modules.

[0830] A camera attached to the device captures video data from inside the home in real time. The hardware used is a typical surveillance camera, such as a digital camera or webcam. The captured video data is sent from the device to a server via the Internet. The software used for this is a library that streams and transmits video data, such as OpenCV or FFmpeg.

[0831] The server stores the video data received from the device in a buffer and then divides it into frames. The cloud platform used is a "cloud storage service," such as "AWS S3" or "Google Cloud Storage." The server then passes the divided frames to an image recognition module, which identifies people or pets in each frame. This image recognition uses an "image recognition library" or "deep learning model," such as "YOLOv5" or "TensorFlow."

[0832] The identified target's face and shape are compared with a known database on the server to determine whether the target is registered. The behavioral analysis module then analyzes the target's movements based on the image recognition results and detects risky behavior. This behavioral analysis is performed using a "motion analysis library," such as "OpenPose" or "Scikit-learn."

[0833] If a dangerous behavior is detected, the server generates an audio alert. This is generated using a speech synthesis service, such as Amazon Polly or Google Text-to-Speech. The generated audio alert is then played through the device's speaker. The device is equipped with a standard speaker, such as a Bluetooth speaker or a smart speaker.

[0834] The server then notifies the user of any detected risky behavior via their smartphone or PC. Notifications are sent using a push notification service, which uses Firebase Cloud Messaging. Users can receive notifications and check detailed information on their smartphone or PC.

[0835] Additionally, the server is equipped with an emotion engine that analyzes the user's voice and facial expressions to determine their emotional state. This emotion analysis uses an "emotion recognition service," which includes emotion recognition APIs from IBM Watson and Microsoft Azure. The alert content and urgency of the notification are adjusted based on the analysis results of the emotion engine. If the user is feeling stressed, notifications with a higher urgency level are sent first.

[0836] Finally, the server stores the video data, image recognition results, behavioral analysis results, and emotion data in a database. The database used is a database management system, such as MongoDB or PostgreSQL, which allows for data organization, reanalysis, and backup.

[0837] Specific examples

[0838] 1. Pet monitoring

[0839] The device's camera captures video of your pet dog in real time.

[0840] The server receives the video data and analyzes the dog's behavior using image recognition and behavior analysis modules.

[0841] If a dog starts biting an electrical cord, the server generates a voice message saying, "That's dangerous!" and the device issues a voice warning. The emotion engine checks the user's stress level and sends an emergency notification to the smartphone if necessary.

[0842] 2. Baby watching

[0843] The device's camera captures images of the baby and sends them to a server in real time.

[0844] The server analyzes the baby's behavior and detects dangerous movements (such as an imminent fall).

[0845] If the baby tries to crawl out of the crib, the device will issue an audio alert saying "Baby moving!" The server will use an emotion engine to determine the parent's level of concern and adjust the urgency of the notification accordingly. For example, if the user is busy, they can choose not to receive detailed notifications.

[0846] As described above, the present invention provides a system for monitoring and protecting pets, babies, and small children, and is flexible and considers the user's emotions, reducing stress for the user and creating a safer and more comfortable living environment.

[0847] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0848] Step 1:

[0849] Video Capture

[0850] A camera attached to the device captures visual data from inside the home in real time. Specifically, the camera continuously captures video data at about 30 frames per second. The hardware used is a digital camera or webcam. The input is the visual data captured by the camera, and the output is video data captured in real time.

[0851] Step 2:

[0852] Data transmission

[0853] The device sends the captured video data to a server via the Internet. The software used is "OpenCV" and a network library. The input is real-time video data, and the output is the video data sent to the server. Specifically, the device divides the video data into packets at regular intervals (for example, every second) and sends them to the server via the network.

[0854] Step 3:

[0855] Data reception

[0856] The server temporarily stores the video data received from the device in a buffer. The cloud platform used is a "cloud storage service" that includes AWS S3 and Google Cloud Storage. The input is the video data sent from the device, and the output is the video data stored in the buffer in the server. Specifically, the server reconstructs the received packets and stores them in the buffer as video data.

[0857] Step 4:

[0858] Frame Split

[0859] The server divides the video data into frames. This process uses the OpenCV library. The input is the video data stored in the buffer, and the output is each frame. Specifically, the server divides the video data into 30 frames per second and passes each frame to subsequent processing.

[0860] Step 5:

[0861] Image Recognition

[0862] The server inputs video frames into an image recognition module to identify people and pets in each frame. The image recognition libraries used are "YOLOv5" and "TensorFlow." The input is video data divided into frames, and the output is information about the identified objects (position, type, etc.). Specifically, the object detection algorithm is applied to each frame to obtain the object's coordinates and class label.

[0863] Step 6:

[0864] Database collation

[0865] The server compares the identified face and shape with a known database. The database includes "face recognition software" and "shape recognition modules." The input is the information of the identified subject, and the output is whether the subject is registered in the database. Specifically, the server extracts the subject's features and compares them with entries in the database to determine whether they match.

[0866] Step 7:

[0867] Behavioral analysis

[0868] The server analyzes behavior based on the results of image recognition and detects risky behavior. The software libraries used are "OpenPose" and "Scikit-learn." The input is the movement and location information of the identified object, and the output is the behavior analysis results and the detection of risky behavior. Specifically, the server analyzes time-series data and detects specific patterns (e.g., falls, biting electrical cords, etc.).

[0869] Step 8:

[0870] Audio alert generation

[0871] If the server detects risky behavior, it immediately generates a corresponding audio alert. This audio alert is generated using a "speech synthesis service," specifically "Amazon Polly" or "Google Text-to-Speech." The input is the detection result of risky behavior, and the output is the generated audio file. Specifically, the server sends a text message to the speech synthesis service, which generates audio data.

[0872] Step 9:

[0873] Voice alert

[0874] The device issues the audio alert sent from the server through a speaker. A typical speaker, such as a Bluetooth speaker or a smart speaker, is attached to the device. The input is the generated audio file, and the output is the transmission of the audio alert. Specifically, the device plays the audio file and issues the audio alert from the speaker.

[0875] Step 10:

[0876] Send notifications

[0877] The server sends push notifications of detected risky behavior or intrusions to the user's smartphone or PC. The push notification service used is "Firebase Cloud Messaging." The input is the detection result of risky behavior and the notification content, and the output is the notification sent to the user's device. Specifically, the server generates a push notification message and sends it to the user's device via the cloud messaging service.

[0878] Step 11:

[0879] emotion recognition

[0880] The emotion engine installed on the server recognizes the user's emotions. Emotion analysis uses "emotion recognition services," including "IBM Watson" and "Microsoft Azure." The input is data based on the user's voice and facial expressions, and the output is the user's emotional state. Specifically, the server analyzes the user's voice data and facial expression data and evaluates the state using the emotion engine.

[0881] Step 12:

[0882] Emotion-based notification adjustment

[0883] The server adjusts the alert content and notification urgency based on the analysis results of the emotion engine. The input is the user's emotional state and the detection results of risky behavior, and the output is the adjusted alert and notification. Specifically, the server determines whether the user is feeling stressed and sets the system to send notifications with a high level of urgency first.

[0884] Step 13:

[0885] Data storage

[0886] The server stores video data, image recognition results, behavioral analysis results, and emotion data in a database. The database management systems used are "MongoDB" and "PostgreSQL." The input is the various analysis result data, and the output is the data stored in the database. Specifically, the server formats the various data appropriately and inserts it into the database.

[0887] (Application example 2)

[0888] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0889] There is a need for monitoring and protecting pets, babies, and young children in the home, and for providing prompt and appropriate alerts through user emotion recognition when necessary. Conventional systems focus on detecting dangerous behavior, but are unable to consider the user's emotional state, making it difficult to reduce stress and anxiety. Furthermore, there are challenges in detecting dangerous behavior and providing notifications accurately and immediately.

[0890] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0891] In this invention, the server includes means for capturing image data using a camera and transmitting it to the server in real time, means for receiving the video data and performing image recognition of people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting dangerous behavior, means for issuing an audio alert to notify the user of the detected dangerous behavior, means for notifying the user of the detected dangerous behavior to a user terminal, and means for adjusting the content of the notification or alert based on the emotion recognition results. This makes it possible to monitor and protect pets, babies, and young children in the home, and allows for flexible responses according to the user's emotional state, providing a safer and more secure living environment.

[0892] A "camera" is a device for capturing image data.

[0893] A "server" is a central computer for receiving, storing, processing, and distributing data over a network.

[0894] "Image data" refers to visual information captured using a camera.

[0895] "Real-time" refers to processing occurring immediately without delay.

[0896] "Video data" refers to moving image information that is made up of a series of multiple image frames.

[0897] A "person" is an object that shows a human shape or face.

[0898] "Animals" refers to non-human living things such as pets and livestock.

[0899] "Image recognition" is the technology of identifying and analyzing objects and patterns in digital images.

[0900] "Behavioral analysis" is the process of evaluating and analyzing the behavior of a subject based on the results of image recognition.

[0901] "Dangerous behavior" is behavior that has the potential to cause accidents or trouble.

[0902] A "voice alert" is a warning method that calls attention by issuing a voice message.

[0903] A "user terminal" is a device (such as a smartphone or PC) that is directly operated by a user.

[0904] A "notification" is a means of informing a user of specific information.

[0905] "Emotion recognition" is a technology that analyzes a user's voice, facial expressions, etc. to determine their emotional state.

[0906] "Notification content" is the content of the message sent to the user.

[0907] "Alert content" refers to the content of a message to warn or alert.

[0908] A "database" is a system for managing an organized collection of data.

[0909] The present invention is a system for monitoring and protecting pets, babies, and small children in the home, recognizing the user's emotions and providing corresponding alerts as needed. The system operates using cameras, a server, a terminal, and multiple software modules.

[0910] Hardware and Software Configuration

[0911] Cameras: Use the built-in camera on your smartphone or a home security camera that captures video data in real time and sends it to a server.

[0912] Server: Acts as the central computer that receives, stores, processes, and distributes data. The server contains the following main modules:

[0913] Image Recognition Module: Uses TensorFlow and Keras to identify people and animals in video frames.

[0914] Behavioral analysis module: Detects risky behavior based on image recognition results.

[0915] Emotion Recognition Module: Uses TensorFlow and Keras to analyze the user's voice and facial expressions to determine their emotional state.

[0916] Database management module: Stores video data and analysis results for later reanalysis and backup.

[0917] Device: A smartphone or PC used by a user. These devices receive notifications and alerts from the server and transmit them to the user.

[0918] Communication: Data communication between the server, terminal, and camera is carried out via the Internet.

[0919] Program processing

[0920] The server receives video data captured by the camera in real time and temporarily stores it in a buffer. Each frame is input into an image recognition module to identify people and animals. Based on the identification results, a behavior analysis module detects dangerous behavior and generates an audio alert. It also sends a notification of the dangerous behavior to the user's device.

[0921] To take the user's emotions into account, the emotion recognition module recognizes the user's emotional state and adjusts the content of notifications and alerts based on the analysis results. For example, if the user is feeling stressed, notifications with a high level of urgency can be sent first.

[0922] Usage example

[0923] Pet Monitoring:

[0924] If the system detects that a pet dog is biting an electrical cord, it will generate an audio alert saying "That's dangerous!" and send a notification to a smartphone. It will also prioritize emergency notifications if it detects that the user is in a stressful state.

[0925] Baby monitoring:

[0926] When the baby is about to crawl out of the crib, the system will issue an audio alert saying "Baby moving!", and an emotion recognition module will analyze the parent's emotions to adjust the urgency of the notification - for example, it can reduce detailed notifications if the parent is busy.

[0927] Prompt Sentence Examples

[0928] If you want to generate a pet watching scenario, the prompt is as follows:

[0929] "Generate a pet monitoring scenario"

[0930] If you want to generate a baby watching scenario, the prompt is as follows:

[0931] "Generate a baby watching scenario"

[0932] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0933] Step 1:

[0934] Camera Interface:

[0935] The device uses a camera to capture video data from within the home in real time. The input is the video data captured by the camera, and the output is frames of this video data. The device encodes these frames and transmits them to a server via the Internet.

[0936] Step 2:

[0937] Video data reception and processing:

[0938] The server receives the video data sent from the device and temporarily stores it in a buffer. The input is the encoded video data, and the output is each divided frame. The server then decodes these frames sequentially and inputs them into the image recognition module.

[0939] Step 3:

[0940] Image Recognition:

[0941] The server uses TensorFlow and Keras to input video frames into an image recognition module. The input is image data for each frame, and the output is information about identified people and animals. The server analyzes each frame to identify people and animals and verify their identity by matching them with a known database.

[0942] Step 4:

[0943] Behavior analysis:

[0944] The server analyzes behavior based on the image recognition results and detects dangerous behavior. The input is the identification information obtained from the image recognition module, and the output is the type and urgency of the detected dangerous behavior. For example, it can detect when a baby is about to fall or when a pet is biting an electrical cord.

[0945] Step 5:

[0946] Alert System:

[0947] The server generates an audio alert for the detected risky behavior and issues it to the device. The input is the behavior analysis results, and the output is the audio alert data. The device receives this data and plays the audio alert through its built-in speaker.

[0948] Step 6:

[0949] Notification system:

[0950] The server sends push notifications of risky behavior and intrusions to the user's smartphone or PC. The input is the behavior analysis results and emotion recognition results, and the output is a notification message sent to the user's device. The user receives the notification and checks the detailed information.

[0951] Step 7:

[0952] Emotion Engine:

[0953] The server uses an emotion engine to analyze the user's voice and facial expressions to determine their emotional state. The input is the user's audio and video data, and the output is an estimated emotional state. Based on this result, the server adjusts the content and urgency of alerts and notifications.

[0954] Step 8:

[0955] Database Management:

[0956] The server stores video data, image recognition results, behavioral analysis results, and emotion data in a database. The input is all processed data, and the output is historical data stored in the database. This allows for future reanalysis and backup.

[0957] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0958] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0959] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0960] [Third embodiment]

[0961] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0962] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0963] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0964] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0965] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0966] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0967] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0968] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0969] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0970] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0971] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0972] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0973] MODE FOR CARRYING OUT THE INVENTION

[0974] To achieve the functions described in the claims, the present invention provides a system that includes the following elements: Specifically, a camera, a server, a terminal, and a software module are combined to create an environment in which users can live their daily lives with peace of mind.

[0975] Program processing overview

[0976] The present invention is a system that processes data between devices as follows, and ultimately provides the user with the information they need.

[0977] 1. Camera Interface

[0978] A camera connected to the device captures images of the inside of the house in real time, for example, periodically taking pictures of the living room or children's room.

[0979] The video data captured by the terminal is transmitted to a server via the Internet.

[0980] 2. Receiving and processing video data

[0981] The server temporarily stores the video data received from the terminal in a buffer.

[0982] The server divides the video data into frames and inputs them into an image recognition module.

[0983] 3. Image Recognition

[0984] The image recognition module detects people and pets in the frame and recognizes their faces and shapes, for example, the face of a baby or a pet dog.

[0985] The image recognition module compares the image against a database to determine whether it is a known person or pet.

[0986] 4. Behavior analysis

[0987] The server performs behavioral analysis based on the information obtained from the image recognition module, for example, detecting a baby trying to crawl out of its crib.

[0988] The behavior analysis module develops defined risky behaviors and determines whether the detected behavior matches them.

[0989] 5. Alert System

[0990] The server generates an audio alert message if risky behavior is detected.

[0991] The device will emit a warning sound or message through the speaker to the user, for example, "Your baby is about to fall out of his crib!"

[0992] 6. Notification System

[0993] The server sends push notifications to users' smartphones or PCs, instantly informing them of any dangerous behavior or intruders.

[0994] For example, if a pet starts chewing on an electrical cord, the user will receive a notification on their smartphone saying, "Your pet is chewing on the electrical cord!"

[0995] 7. Database Management

[0996] The server stores the detected video data and analysis results in a database.

[0997] The database management module organizes the data for later access and creates backups as needed.

[0998] Specific examples

[0999] 1. Pet monitoring

[1000] A camera installed on the device periodically captures footage of the pet dog.

[1001] The server receives the video data and analyzes the dog's behavior using image recognition and behavior analysis modules.

[1002] If a dog starts chewing on an electrical cord, the server will play an audio message saying "That's dangerous!" and simultaneously send a notification to the user's smartphone.

[1003] 2. Baby watching

[1004] The device's camera captures images of the baby in real time.

[1005] The server analyzes the baby's behavior and detects dangerous movements.

[1006] When the baby tries to crawl out of the crib, the device will emit an audio alert saying "Baby is moving!" and send a push notification to the user's smartphone.

[1007] This allows users to ensure the safety of pets, babies, and small children anytime, anywhere. This system prevents danger in daily life and provides users with a sense of security.

[1008] The processing flow will be explained below.

[1009] Step 1:

[1010] The device's camera captures video of the home in real time at a set frame rate, and the video data is divided into frames at regular time intervals.

[1011] Step 2:

[1012] The captured video frames are sent directly to the server using a dedicated communication protocol, ensuring data security.

[1013] Step 3:

[1014] The server stores the video frames received from the terminal in a buffer in order, allowing subsequent processing to proceed smoothly.

[1015] Step 4:

[1016] The server sends the buffered video frames to an image recognition module that detects people and pets in each frame, using specific algorithms to identify faces and shapes.

[1017] Step 5:

[1018] The image recognition module compares the detection results with a database to determine whether the target is registered, for example, by comparing the faces of family members with facial images stored in the database.

[1019] Step 6:

[1020] The server passes the image recognition results to the behavior analysis module, which then analyzes multiple behavioral patterns to determine whether the target's behavior is dangerous.

[1021] Step 7:

[1022] The behavior analysis module analyzes behavioral patterns and detects dangerous behavior (for example, a baby about to fall or a pet chewing on an electrical cord).

[1023] Step 8:

[1024] If the server detects any risky behavior, it will instantly generate an audio alert message, which is generated based on pre-defined phrases.

[1025] Step 9:

[1026] The device receives the audio alert message and issues a voice warning through the speaker, for example, "That's dangerous!"

[1027] Step 10:

[1028] The server then sends a push notification to the user's smartphone or PC in parallel, containing details of the risky behavior detected.

[1029] Step 11:

[1030] Users can check notifications on their smartphones or PCs and rush to the scene if necessary. For example, they can tap the pop-up notification to check details.

[1031] Step 12:

[1032] The server stores the detected video data and analysis results in a database, which can be used for later review and analysis.

[1033] Step 13:

[1034] The database management module periodically creates backups of data and makes them available for system recovery, thereby preventing data loss.

[1035] Example 1

[1036] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1037] Ensuring the safety of children and pets is a major concern in modern homes. Conventional surveillance systems simply record video footage, but are unable to detect danger or send alerts in real time, making it difficult to respond to situations that require a rapid response. Furthermore, there is a lack of means to manage and analyze massive amounts of video data, making efficient data storage and use a challenge.

[1038] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1039] In this invention, the server includes means for capturing image data using a camera and transmitting it to the server in real time, means for receiving the video data and performing image recognition of people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting dangerous behavior, means for issuing an audio alert to notify the user of the detected dangerous behavior, means for notifying the user of the detected dangerous behavior, means for temporarily storing the video data in a buffer, means for storing the detected video data and analysis results in a database for later access, means for analyzing each frame using an image recognition module, means for tracking movements in the video using a behavior analysis module and detecting specific patterns, and means for generating an audio alert message. This enables real-time monitoring and analysis of video data inside the home, rapid detection of dangerous behavior and issuing an alert, and efficient storage and management of video data.

[1040] A "camera" is an optical device for capturing image data.

[1041] A "server" is a computer system that processes and manages data in response to requests from clients.

[1042] A "terminal" is a computer system or device that has input and output devices and can be directly operated by a user.

[1043] "Image data" refers to visual information captured by a camera and stored in digital form.

[1044] "Real-time" means near-instant processing with minimal delay.

[1045] A "buffer" is a memory area that functions as a temporary storage area for data and ensures a smooth flow of data processing.

[1046] "Image recognition" is the process of using computer vision techniques to detect specific objects or patterns in video data.

[1047] "Behavioral analysis" is the process of analyzing patterns of movement and behavior to predict intentions and next actions.

[1048] A "voice alert" is a system that uses voice to warn or notify.

[1049] A "user terminal" is a device operated by a user who uses the system, and typically refers to a smartphone or personal computer.

[1050] A "database" is a structured data storage system for efficiently managing, retrieving, and storing data.

[1051] An "image recognition module" is a software component that analyzes image data to recognize specific objects or people.

[1052] The "behavioral analysis module" is a software component that analyzes behavior from recognized data and detects specific patterns and anomalies.

[1053] A "voice alert message" is a message containing a warning or notification generated using voice synthesis technology.

[1054] "People and animals" refers to living beings such as humans and pets that appear in image data.

[1055] This invention combines cameras, servers, terminals, and software modules to provide an environment in which users can live their daily lives with peace of mind. Specifically, it is a system that monitors images from inside the home in real time and issues alerts and notifications as needed.

[1056] Hardware and Software Configuration

[1057] 1. Camera

[1058] Device: A camera is used to capture video from inside the home. For example, cameras installed in the living room or children's room capture video data in real time.

[1059] The video data is transmitted to a server via the Internet.

[1060] 2. Server

[1061] Server: Receives video data sent from the device and temporarily stores it in a buffer. This buffer can hold several seconds of video.

[1062] The received video data is divided into frames and input into the image recognition module.

[1063] Image Recognition Module: Detects people and pets from video frames and recognizes their faces and shapes, for example using TensorFlow or OpenCV.

[1064] A software module (behavior analysis module) runs to perform behavior analysis based on the recognized information.

[1065] The module detects risky behavior and handles the results appropriately.

[1066] 3. Alert System

[1067] Server: Uses a text-to-speech (TTS) engine to generate audio alert messages if risky behavior is identified as a result of behavioral analysis.

[1068] Device: Play an audio alert through the speaker, for example, "Baby is about to fall out of his crib!"

[1069] 4. Notification System

[1070] Server: Sends push notifications to users' smartphones or PCs. To do this, you can use a notification service such as Firebase Cloud Messaging (FCM).

[1071] Users: They can check notifications on their smartphones and take action quickly.

[1072] 5. Database Management

[1073] Server: Stores the detected video data and analysis results in a database, such as MySQL or PostgreSQL.

[1074] The stored data is organized and regularly backed up for later access.

[1075] Specific examples

[1076] 1. Pet monitoring

[1077] Device: A camera installed in the living room captures footage of a pet dog.

[1078] Server: Receives video data and detects the dog using an image recognition module. It then analyzes its behavior and, for example, if the dog starts biting an electrical cord, determines that this behavior is dangerous.

[1079] Server: Generates and broadcasts the audio alert "That's dangerous!". At the same time, a notification is sent to the user's smartphone saying "Your pet is chewing on the electrical cord!"

[1080] 2. Baby watching

[1081] Device: A camera installed in the child's room captures real-time footage of the baby.

[1082] Server: Receives the video data and detects the baby's position using the image recognition module.Then, the behavior analysis module detects the baby's movements to crawl out of the crib.

[1083] Server: Generates and sends an audio alert in response to this dangerous behavior: "The baby is about to fall out of the crib!" At the same time, a notification is sent to the user's smartphone saying, "The baby is moving!"

[1084] This allows users to ensure the safety of pets, babies, and small children anytime, anywhere. This system prevents danger in daily life and provides users with a sense of security.

[1085] Prompt Sentence Examples

[1086] Text-to-speech prompt for when your baby is about to fall out of his crib: "Your baby is about to fall out of his crib!"

[1087] If your pet starts chewing on an electrical cord, a push notification prompt will appear: "Your pet is chewing on the electrical cord!"

[1088] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1089] Program processing steps

[1090] Explanation divided into processing steps

[1091] Step 1: Video capture and transmission

[1092] Step 2: Receiving and buffering video data

[1093] Step 3: Image Recognition

[1094] Step 4: Behavioral analysis

[1095] Step 5: Alert Generation

[1096] Step 6: Send notifications

[1097] Step 7: Data Management and Storage

[1098] ---

[1099] Specific explanation of each processing step

[1100] Step 1: Video capture and transmission

[1101] Device: A camera connected to the device captures video from inside the home in real time, for example capturing video of the living room at 30 frames per second.

[1102] Input: Image captured by the camera.

[1103] Data processing: Encode the captured video into a compressed format such as H.264.

[1104] Output: Encoded video data.

[1105] Specific operation: The encoded video data is sent to a server via the Internet, for example, at a rate of 1MB per second.

[1106] Step 2: Receiving and buffering video data

[1107] Server: Receives video data sent from the device. For example, it waits for streaming data on a specific port on the server.

[1108] Input: Video data sent from the device.

[1109] Data calculation: The received video data is temporarily stored in a buffer. The buffer size is set so that it can hold several seconds of video data.

[1110] Output: Buffered video data.

[1111] What it does: The server receives 1MB of data per second over an inbound TCP connection and stores it in a buffer.

[1112] Step 3: Image Recognition

[1113] Server: The video data stored in the buffer is divided into frames and input into the image recognition module.

[1114] Input: Buffered frame data.

[1115] Data computation: Each frame is analyzed by an image recognition module to detect people and pets, for example using TensorFlow or OpenCV.

[1116] Output: Information about detected people and pets.

[1117] What it does: It analyzes video at 30 frames per second and runs an algorithm to detect the baby's face in each frame.

[1118] Step 4: Behavioral analysis

[1119] Server: Performs behavioral analysis based on information obtained from the image recognition module. Tracks the movements of people and pets in the video and detects specific patterns (e.g., crawling out of a crib).

[1120] Input: Person and pet information obtained from the image recognition module.

[1121] Data calculation: The behavioral analysis module tracks movements and determines whether they match defined risky behaviors.

[1122] Output: Risky behavior detection results.

[1123] Specific operation: If the baby moves to a dangerous location, the location information and movement pattern are analyzed to determine whether it constitutes dangerous behavior.

[1124] Step 5: Alert Generation

[1125] Server: If risky behavior is detected as a result of behavioral analysis, it generates an audio alert message and converts the alert message into audio using a Text-to-Speech (TTS) engine.

[1126] Input: The behavioral analysis results and the corresponding text alert message.

[1127] Data Calculation: Converts text into a voice alert message using a TTS engine.

[1128] Output: Audio alert message.

[1129] Specific behavior: Generates the message "Your baby is about to fall out of his crib!" and plays it through speakers in the home.

[1130] Step 6: Send notifications

[1131] Server: If dangerous behavior or anomalies are detected, a push notification is sent to the user's smartphone or PC. Services such as Firebase Cloud Messaging (FCM) may be used.

[1132] Input: The behavior analysis results and the corresponding notification message.

[1133] Data Computing: Deliver messages using a notification service such as FCM.

[1134] Output: Notifications sent to the user's smartphone or PC.

[1135] Specific operation: When the baby engages in dangerous behavior, the user will receive a notification on their smartphone saying, "Your baby is moving!"

[1136] Step 7: Data Management and Storage

[1137] Server: Stores the detected video data and analysis results in a database. RDBMS such as MySQL or PostgreSQL is often used.

[1138] Input: Video data and analysis results.

[1139] Data Computation: Stores data in a database, creates indexes, and creates regular backups.

[1140] Output: Data stored in a database.

[1141] Specific operation: Video data along with analysis results are stored in a database with a timestamp, allowing users to verify past data.

[1142] (Application example 1)

[1143] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1144] The challenge is to enhance home security by detecting suspicious individuals or dangerous behavior in real time and notifying users promptly. It is also necessary to send push notifications to users if suspicious behavior persists and save video frames as a record to facilitate later analysis and evidence collection.

[1145] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1146] In this invention, the server includes means for capturing image data using a camera and transmitting it to the server in real time, means for receiving the video data and performing image recognition of people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting suspicious behavior, means for issuing an audio alert to notify the user of the detected suspicious behavior, means for notifying the user of the detected suspicious behavior to a user terminal, means for notifying the user via push notification if the suspicious behavior continues, and means for saving the detected video frames as a log file, thereby enabling early detection of suspicious people and dangerous behavior and rapid response.

[1147] A "camera" is a device that captures video data and transmits the video to a server.

[1148] "Real-time" refers to processing the target images and data instantly and without delay.

[1149] A "server" is a computer system that receives video data sent from a camera and performs various data processing.

[1150] "Video data" refers to digital data of moving or still images captured by a camera.

[1151] "Image recognition" is a technology that detects and identifies specific objects such as people and animals within video data.

[1152] "Suspicious behavior" refers to behavior that is unusual or dangerous based on the results of image recognition.

[1153] "Audio alert" is a means of issuing a warning sound or voice message when suspicious activity is detected.

[1154] "User terminal" refers to a communication device such as a smartphone or PC owned by a user.

[1155] A "push notification" is an alert or notification that is sent instantly from a server to a user device.

[1156] A "log file" is a file that records system operation status and event information, and can be used for later analysis and verification.

[1157] The present invention describes an embodiment of a system for strengthening home security and detecting suspicious individuals and dangerous behavior in real time. This system is configured by combining a camera, a server, a user terminal, and various software modules.

[1158] Components

[1159] 1. Camera

[1160] Cameras installed in homes capture real-time video of what is happening in and around the home, and the video data is sent to a server via Wi-Fi or a wired connection.

[1161] 2. Server

[1162] The server receives the video data and has a buffer for temporary storage.

[1163] Video data is processed using various software modules (image recognition module, behavior analysis module, alert generation module, notification module).

[1164] 3. User Device

[1165] User devices such as smartphones and PCs communicate with the server and receive push notifications.

[1166] Software Module

[1167] 1. Image Recognition Module

[1168] The received video data is divided into frames and specific objects (people, animals) are detected.

[1169] It utilizes existing deep learning technologies such as OpenCV's DNN module.

[1170] 2. Behavioral Analysis Module

[1171] Based on the information obtained from the image recognition module, behavioral analysis is performed to detect suspicious behavior.

[1172] If suspicious behavior continues for a certain period of time, an alert is generated.

[1173] 3. Alert Generation Module

[1174] When suspicious behavior is detected, audio alerts and messages are generated and played over speakers in the home.

[1175] 4. Notification Module

[1176] The server will then send a push notification to the user's device regarding any suspicious behavior or dangerous situations that it detects.

[1177] For example, a notification such as "There's a suspicious person at the front door!" is immediately forwarded to the user.

[1178] 5. Data storage module

[1179] The detected video frames and analysis results are saved as a log file, making them available for later analysis.

[1180] Processing Description

[1181] The server receives video data from the camera in real time and temporarily stores it in a buffer. The video data is analyzed through an image recognition module to detect specific objects (people or animals). The behavior analysis module then analyzes the movements of these objects to detect any suspicious behavior.

[1182] If the detected suspicious behavior continues for a certain period of time, the alert generation module will generate an audio warning or message and send it to a speaker in the home, while the notification module will send a push notification to the user's device to immediately notify them of the suspicious activity.

[1183] In addition, the data storage module stores the detected video frames and analysis results as a log file for later analysis and evidence collection.

[1184] Examples and prompts

[1185] For example, if a suspicious person is detected at the entrance, the image recognition module will identify the person, and the behavior analysis module will analyze the person's movements. If the suspicious behavior continues, the alert generation module will play an audio message saying "There's a suspicious person at the entrance!", and at the same time, the notification module will send a push notification to the user's device.

[1186] Examples of prompts include, "What happens if an intruder is detected in your home?", "How does your system detect an intruder?", and "What is the threshold for generating an alert?"

[1187] As a result, the security system based on the present invention can significantly improve safety within the home and encourage users to take prompt and appropriate action.

[1188] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1189] Step 1:

[1190] The server receives video data from cameras installed in homes in real time. The input data is real-time video data captured by the cameras, and the server temporarily stores it in a buffer.

[1191] Step 2:

[1192] The server divides the temporarily stored video data into frames and sends them to the image recognition module. The input data here are video frames, which the image recognition module receives and detects people and animals using OpenCV's DNN module, etc. The output data is position information and probability information for objects (people and animals) present in the video frames.

[1193] Step 3:

[1194] The server sends the object information obtained from the image recognition module to the behavior analysis module. The input data is the object information, and the behavior analysis module uses this information to analyze suspicious behavior. The output data is the type of analyzed behavior and its risk level.

[1195] Step 4:

[1196] The server uses the alert generation module to create a voice alert based on the suspicious behavior information obtained from the behavior analysis module. The input data is the suspicious behavior information, and the alert generation module generates a warning message based on this. The output data is a warning message or a voice file. The device plays this warning sound or message.

[1197] Step 5:

[1198] The server notifies the user device of the suspicious behavior information. The input data is the suspicious behavior information and the generated warning message. The notification module generates a push notification based on this and sends it to the user device. The output data is the push notification message. The push notification is displayed on the device and a warning is sent to the user.

[1199] Step 6:

[1200] The server uses a data storage module to store the detected video frames and analysis results as a log file. The input data are the video frames and their analysis results, and the data storage module creates a log file based on them. The output data is the stored log file, which can be used for later analysis or as evidence.

[1201] For example, if a user asks, "What happens if a suspicious person is detected in the house?", the suspicious person will be detected through the above steps, a warning will be sent to the user, and necessary measures will be taken. In this way, the present invention improves home safety and enables early detection of suspicious people and rapid response.

[1202] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1203] MODE FOR CARRYING OUT THE INVENTION

[1204] To achieve the functions recited in the claims, the present invention provides a system including the following elements, which monitors and protects pets, babies, and young children, recognizes the user's emotions as needed, and provides corresponding alerts, allowing the user to live their daily lives with peace of mind.

[1205] Program processing overview

[1206] The present invention is a system that operates by combining a camera, a server, a terminal, and a software module, and specifically performs data processing in the following manner.

[1207] 1. Camera Interface

[1208] A camera attached to the device captures visual data from inside the home in real time.

[1209] The terminal transmits the captured video data to a server via the Internet.

[1210] 2. Receiving and processing video data

[1211] The server temporarily stores the video data received from the terminal in a buffer and divides it into frames.

[1212] The server inputs the video frames into an image recognition module.

[1213] 3. Image Recognition

[1214] An image recognition module identifies people and pets in each frame.

[1215] Identified faces and shapes are matched against a known database to determine if the subject is registered.

[1216] 4. Behavior analysis

[1217] The server analyzes behavior based on the results of image recognition and detects risky behavior.

[1218] Specifically, it detects actions such as a baby falling or a pet chewing on an electrical cord.

[1219] 5. Alert System

[1220] The server generates an audio alert for detected risky behavior.

[1221] The device will issue an audio warning through the speaker.

[1222] 6. Notification System

[1223] The server sends push notifications to users' smartphones or PCs about dangerous behavior or intrusions.

[1224] The user receives a notification and checks for more information.

[1225] 7. Emotion Engine

[1226] The emotion engine installed on the server recognizes the user's emotions, for example, by analyzing the user's voice and facial expressions to determine their emotional state.

[1227] The content of alerts and notifications is adjusted based on the analysis results of the emotion engine. For example, if the user is feeling stressed, notifications with a high level of urgency will be sent first.

[1228] 8. Database Management

[1229] The server stores the video data, image recognition results, behavioral analysis results, and emotion data in a database.

[1230] The management module organizes this data and makes it available for later reanalysis or backup.

[1231] Specific examples

[1232] 1. Pet monitoring

[1233] The device's camera captures video of your pet dog in real time.

[1234] The server receives the video data and analyzes the dog's behavior using image recognition and behavior analysis modules.

[1235] If a dog starts biting an electrical cord, the server generates a voice message saying, "That's dangerous!" and the device issues a voice warning. The emotion engine checks the user's stress level and sends an emergency notification to the smartphone if necessary.

[1236] 2. Baby watching

[1237] The device's camera captures images of the baby and sends them to a server in real time.

[1238] The server analyzes the baby's behavior and detects dangerous movements (such as an imminent fall).

[1239] If the baby tries to crawl out of the crib, the device will issue an audio alert saying "Baby moving!" The server will use an emotion engine to determine the parent's level of concern and adjust the urgency of the notification accordingly. For example, if the user is busy, they can choose not to receive detailed notifications.

[1240] This provides a system that not only monitors and protects pets, babies, and small children, but also takes into account the user's emotions, reducing stress for the user and creating a safer and more comfortable living environment.

[1241] The processing flow will be explained below.

[1242] MODE FOR CARRYING OUT THE INVENTION

[1243] In order to achieve the functions described in the claims, the present invention provides a system including the following elements: In particular, processing steps for monitoring and protecting pets, babies, and young children are specifically described.

[1244] Step 1:

[1245] The camera captures video of the inside of the house in real time at a set frame rate, with the video divided into frames at regular time intervals.

[1246] Step 2:

[1247] The device transmits the captured video frames to a server over the Internet, where the transmission is secure using an encrypted communication protocol.

[1248] Step 3:

[1249] The server stores the video frames received from the terminal in a buffer in order, allowing subsequent processing to proceed smoothly.

[1250] Step 4:

[1251] The server inputs the buffered video frames into an image recognition module to detect people or pets in each frame, for example, identifying the faces and shapes of babies or pets.

[1252] Step 5:

[1253] The image recognition module checks the detected person or pet data against a database to determine whether the target is already registered, for example, by comparing it with facial images of family members stored in the database.

[1254] Step 6:

[1255] The server passes the image recognition results to the behavioral analysis module, which then begins analyzing the behavior, identifying risky behaviors, such as a baby trying to crawl out of its crib.

[1256] Step 7:

[1257] Based on the behavioral patterns analyzed by the behavior analysis module, dangerous behavior (for example, a pet chewing on an electrical cord or a baby about to fall) is detected.

[1258] Step 8:

[1259] If the server detects risky behavior, it generates an audio alert message based on a predefined phrase.

[1260] Step 9:

[1261] The device receives the audio alert message and issues a warning through the speaker, such as "That's dangerous!"

[1262] Step 10:

[1263] The server sends a push notification to the user's smartphone or PC, which contains details of the risky behavior.

[1264] Step 11:

[1265] The user receives a push notification on their smartphone or PC and checks the details, for example by tapping on the pop-up notification to check the details of the risky behavior.

[1266] Step 12:

[1267] An emotion engine installed on the server analyzes the user's emotions in real time, detecting, for example, stress or anxiety from the user's voice and facial expressions.

[1268] Step 13:

[1269] Based on the analysis results of the emotion engine, the server adjusts the content of audio alerts and push notifications. For example, if the user is in a high stress state, a notification with more specific information will be sent.

[1270] Step 14:

[1271] The server stores the detected video data, image recognition results, behavioral analysis results, and emotion data in a database.

[1272] Step 15:

[1273] The database management module periodically backs up data and stores it for future analysis and restoration, thus preventing data loss.

[1274] This allows users to better monitor and protect their pets, babies and young children, and the alert system also takes into account the user's emotional state to provide more relevant and reassuring notifications.

[1275] Example 2

[1276] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1277] In modern homes, it is important to safely monitor pets, babies, and young children and respond quickly when necessary. However, conventional surveillance systems are limited to simple video capture and notification, and do not support specific behavioral analysis of people and animals or adjust the urgency of notifications based on the user's emotions. This makes it difficult to respond quickly and appropriately to dangerous situations, resulting in a lack of user peace of mind and convenience.

[1278] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing image data using a camera and transmitting it to the server in real time, a means for receiving video data and performing image recognition of people and animals in the video data, a means for analyzing behavior based on the image recognition results and detecting dangerous behavior, a means for issuing an audio alert to notify the user of the detected dangerous behavior, a means for notifying the user of the detected dangerous behavior to a user terminal, a means for recording analysis data and emotion data in a database and using it for later analysis, and a means for analyzing the user's emotion and adjusting the urgency of the notification. This makes it possible to monitor and protect pets, babies, and young children, and to provide appropriate alerts and notifications taking the user's emotion into consideration.

[1279] "Camera" means a device that captures visual data. Commonly used terms include digital cameras and webcams.

[1280] A "server" is a computer system that receives, processes, and manages data over a network.

[1281] A "terminal" is a device that is directly operated by a user, and is a computer to which peripheral devices such as a camera and speakers are connected.

[1282] "Image data" refers to digital data containing visual information captured by a camera.

[1283] "Real-time" refers to near-instant processing and response without delay.

[1284] "Video data" refers to a moving image made up of a series of multiple image data.

[1285] "Image recognition" is the process of automatically identifying people, animals, and objects in video data.

[1286] "Behavioral analysis" is the process of evaluating a subject's behavior based on the results of image recognition to detect specific patterns or anomalies.

[1287] "Riskful behavior" refers to behavior that puts a subject in a potentially dangerous situation and may threaten their safety.

[1288] A "voice alert" is a notification that issues a warning via voice when risky behavior is detected.

[1289] A "user terminal" refers to a computer device such as a smartphone or PC used by a user.

[1290] A "notification" is a message that notifies the user of important information or warnings.

[1291] A "database" is a system for efficiently storing, managing, and searching structured data.

[1292] "Emotional data" refers to digital data that represents a user's emotional state.

[1293] The "urgency of a notification" is an indicator of the degree to which a notification requires an immediate response.

[1294] The present invention is a system for monitoring and protecting pets, babies, and small children, and provides a safe living environment by recognizing the user's emotions and providing notifications accordingly. Specifically, it is realized using a camera, a server, a terminal, and multiple software modules.

[1295] A camera attached to the device captures video data from inside the home in real time. The hardware used is a typical surveillance camera, such as a digital camera or webcam. The captured video data is sent from the device to a server via the Internet. The software used for this is a library that streams and transmits video data, such as OpenCV or FFmpeg.

[1296] The server stores the video data received from the device in a buffer and then divides it into frames. The cloud platform used is a "cloud storage service," such as "AWS S3" or "Google Cloud Storage." The server then passes the divided frames to an image recognition module, which identifies people or pets in each frame. This image recognition uses an "image recognition library" or "deep learning model," such as "YOLOv5" or "TensorFlow."

[1297] The identified target's face and shape are compared with a known database on the server to determine whether the target is registered. The behavioral analysis module then analyzes the target's movements based on the image recognition results and detects risky behavior. This behavioral analysis is performed using a "motion analysis library," such as "OpenPose" or "Scikit-learn."

[1298] If a dangerous behavior is detected, the server generates an audio alert. This is generated using a speech synthesis service, such as Amazon Polly or Google Text-to-Speech. The generated audio alert is then played through the device's speaker. The device is equipped with a standard speaker, such as a Bluetooth speaker or a smart speaker.

[1299] The server then notifies the user of any detected risky behavior via their smartphone or PC. Notifications are sent using a push notification service, which uses Firebase Cloud Messaging. Users can receive notifications and check detailed information on their smartphone or PC.

[1300] Additionally, the server is equipped with an emotion engine that analyzes the user's voice and facial expressions to determine their emotional state. This emotion analysis uses an "emotion recognition service," which includes emotion recognition APIs from IBM Watson and Microsoft Azure. The alert content and urgency of the notification are adjusted based on the analysis results of the emotion engine. If the user is feeling stressed, notifications with a higher urgency level are sent first.

[1301] Finally, the server stores the video data, image recognition results, behavioral analysis results, and emotion data in a database. The database used is a database management system, such as MongoDB or PostgreSQL, which allows for data organization, reanalysis, and backup.

[1302] Specific examples

[1303] 1. Pet monitoring

[1304] The device's camera captures video of your pet dog in real time.

[1305] The server receives the video data and analyzes the dog's behavior using image recognition and behavior analysis modules.

[1306] If a dog starts biting an electrical cord, the server generates a voice message saying, "That's dangerous!" and the device issues a voice warning. The emotion engine checks the user's stress level and sends an emergency notification to the smartphone if necessary.

[1307] 2. Baby watching

[1308] The device's camera captures images of the baby and sends them to a server in real time.

[1309] The server analyzes the baby's behavior and detects dangerous movements (such as an imminent fall).

[1310] If the baby tries to crawl out of the crib, the device will issue an audio alert saying "Baby moving!" The server will use an emotion engine to determine the parent's level of concern and adjust the urgency of the notification accordingly. For example, if the user is busy, they can choose not to receive detailed notifications.

[1311] As described above, the present invention provides a system for monitoring and protecting pets, babies, and small children, and is flexible and considers the user's emotions, reducing stress for the user and creating a safer and more comfortable living environment.

[1312] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1313] Step 1:

[1314] Video Capture

[1315] A camera attached to the device captures visual data from inside the home in real time. Specifically, the camera continuously captures video data at about 30 frames per second. The hardware used is a digital camera or webcam. The input is the visual data captured by the camera, and the output is video data captured in real time.

[1316] Step 2:

[1317] Data transmission

[1318] The device sends the captured video data to a server via the Internet. The software used is "OpenCV" and a network library. The input is real-time video data, and the output is the video data sent to the server. Specifically, the device divides the video data into packets at regular intervals (for example, every second) and sends them to the server via the network.

[1319] Step 3:

[1320] Data reception

[1321] The server temporarily stores the video data received from the device in a buffer. The cloud platform used is a "cloud storage service" that includes AWS S3 and Google Cloud Storage. The input is the video data sent from the device, and the output is the video data stored in the buffer in the server. Specifically, the server reconstructs the received packets and stores them in the buffer as video data.

[1322] Step 4:

[1323] Frame Split

[1324] The server divides the video data into frames. This process uses the OpenCV library. The input is the video data stored in the buffer, and the output is each frame. Specifically, the server divides the video data into 30 frames per second and passes each frame to subsequent processing.

[1325] Step 5:

[1326] Image Recognition

[1327] The server inputs video frames into an image recognition module to identify people and pets in each frame. The image recognition libraries used are "YOLOv5" and "TensorFlow." The input is video data divided into frames, and the output is information about the identified objects (position, type, etc.). Specifically, the object detection algorithm is applied to each frame to obtain the object's coordinates and class label.

[1328] Step 6:

[1329] Database collation

[1330] The server compares the identified face and shape with a known database. The database includes "face recognition software" and "shape recognition modules." The input is the information of the identified subject, and the output is whether the subject is registered in the database. Specifically, the server extracts the subject's features and compares them with entries in the database to determine whether they match.

[1331] Step 7:

[1332] Behavioral analysis

[1333] The server analyzes behavior based on the results of image recognition and detects risky behavior. The software libraries used are "OpenPose" and "Scikit-learn." The input is the movement and location information of the identified object, and the output is the behavior analysis results and the detection of risky behavior. Specifically, the server analyzes time-series data and detects specific patterns (e.g., falls, biting electrical cords, etc.).

[1334] Step 8:

[1335] Audio alert generation

[1336] If the server detects risky behavior, it immediately generates a corresponding audio alert. This audio alert is generated using a "speech synthesis service," specifically "Amazon Polly" or "Google Text-to-Speech." The input is the detection result of risky behavior, and the output is the generated audio file. Specifically, the server sends a text message to the speech synthesis service, which generates audio data.

[1337] Step 9:

[1338] Voice alert

[1339] The device issues the audio alert sent from the server through a speaker. A typical speaker, such as a Bluetooth speaker or a smart speaker, is attached to the device. The input is the generated audio file, and the output is the transmission of the audio alert. Specifically, the device plays the audio file and issues the audio alert from the speaker.

[1340] Step 10:

[1341] Send notifications

[1342] The server sends push notifications of detected risky behavior or intrusions to the user's smartphone or PC. The push notification service used is "Firebase Cloud Messaging." The input is the detection result of risky behavior and the notification content, and the output is the notification sent to the user's device. Specifically, the server generates a push notification message and sends it to the user's device via the cloud messaging service.

[1343] Step 11:

[1344] emotion recognition

[1345] The emotion engine installed on the server recognizes the user's emotions. Emotion analysis uses "emotion recognition services," including "IBM Watson" and "Microsoft Azure." The input is data based on the user's voice and facial expressions, and the output is the user's emotional state. Specifically, the server analyzes the user's voice data and facial expression data and evaluates the state using the emotion engine.

[1346] Step 12:

[1347] Emotion-based notification adjustment

[1348] The server adjusts the alert content and notification urgency based on the analysis results of the emotion engine. The input is the user's emotional state and the detection results of risky behavior, and the output is the adjusted alert and notification. Specifically, the server determines whether the user is feeling stressed and sets the system to send notifications with a high level of urgency first.

[1349] Step 13:

[1350] Data storage

[1351] The server stores video data, image recognition results, behavioral analysis results, and emotion data in a database. The database management systems used are "MongoDB" and "PostgreSQL." The input is the various analysis result data, and the output is the data stored in the database. Specifically, the server formats the various data appropriately and inserts it into the database.

[1352] (Application example 2)

[1353] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1354] There is a need for monitoring and protecting pets, babies, and young children in the home, and for providing prompt and appropriate alerts through user emotion recognition when necessary. Conventional systems focus on detecting dangerous behavior, but are unable to consider the user's emotional state, making it difficult to reduce stress and anxiety. Furthermore, there are challenges in detecting dangerous behavior and providing notifications accurately and immediately.

[1355] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1356] In this invention, the server includes means for capturing image data using a camera and transmitting it to the server in real time, means for receiving the video data and performing image recognition of people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting dangerous behavior, means for issuing an audio alert to notify the user of the detected dangerous behavior, means for notifying the user of the detected dangerous behavior to a user terminal, and means for adjusting the content of the notification or alert based on the emotion recognition results. This makes it possible to monitor and protect pets, babies, and young children in the home, and allows for flexible responses according to the user's emotional state, providing a safer and more secure living environment.

[1357] A "camera" is a device for capturing image data.

[1358] A "server" is a central computer for receiving, storing, processing, and distributing data over a network.

[1359] "Image data" refers to visual information captured using a camera.

[1360] "Real-time" refers to processing occurring immediately without delay.

[1361] "Video data" refers to moving image information that is made up of a series of multiple image frames.

[1362] A "person" is an object that shows a human shape or face.

[1363] "Animals" refers to non-human living things such as pets and livestock.

[1364] "Image recognition" is the technology of identifying and analyzing objects and patterns in digital images.

[1365] "Behavioral analysis" is the process of evaluating and analyzing the behavior of a subject based on the results of image recognition.

[1366] "Dangerous behavior" is behavior that has the potential to cause accidents or trouble.

[1367] A "voice alert" is a warning method that calls attention by issuing a voice message.

[1368] A "user terminal" is a device (such as a smartphone or PC) that is directly operated by a user.

[1369] A "notification" is a means of informing a user of specific information.

[1370] "Emotion recognition" is a technology that analyzes a user's voice, facial expressions, etc. to determine their emotional state.

[1371] "Notification content" is the content of the message sent to the user.

[1372] "Alert content" refers to the content of a message to warn or alert.

[1373] A "database" is a system for managing an organized collection of data.

[1374] The present invention is a system for monitoring and protecting pets, babies, and small children in the home, recognizing the user's emotions and providing corresponding alerts as needed. The system operates using cameras, a server, a terminal, and multiple software modules.

[1375] Hardware and Software Configuration

[1376] Cameras: Use the built-in camera on your smartphone or a home security camera that captures video data in real time and sends it to a server.

[1377] Server: Acts as the central computer that receives, stores, processes, and distributes data. The server contains the following main modules:

[1378] Image Recognition Module: Uses TensorFlow and Keras to identify people and animals in video frames.

[1379] Behavioral analysis module: Detects risky behavior based on image recognition results.

[1380] Emotion Recognition Module: Uses TensorFlow and Keras to analyze the user's voice and facial expressions to determine their emotional state.

[1381] Database management module: Stores video data and analysis results for later reanalysis and backup.

[1382] Device: A smartphone or PC used by a user. These devices receive notifications and alerts from the server and transmit them to the user.

[1383] Communication: Data communication between the server, terminal, and camera is carried out via the Internet.

[1384] Program processing

[1385] The server receives video data captured by the camera in real time and temporarily stores it in a buffer. Each frame is input into an image recognition module to identify people and animals. Based on the identification results, a behavior analysis module detects dangerous behavior and generates an audio alert. It also sends a notification of the dangerous behavior to the user's device.

[1386] To take the user's emotions into account, the emotion recognition module recognizes the user's emotional state and adjusts the content of notifications and alerts based on the analysis results. For example, if the user is feeling stressed, notifications with a high level of urgency can be sent first.

[1387] Usage example

[1388] Pet Monitoring:

[1389] If the system detects that a pet dog is biting an electrical cord, it will generate an audio alert saying "That's dangerous!" and send a notification to a smartphone. It will also prioritize emergency notifications if it detects that the user is in a stressful state.

[1390] Baby monitoring:

[1391] When the baby is about to crawl out of the crib, the system will issue an audio alert saying "Baby moving!", and an emotion recognition module will analyze the parent's emotions to adjust the urgency of the notification - for example, it can reduce detailed notifications if the parent is busy.

[1392] Prompt Sentence Examples

[1393] If you want to generate a pet watching scenario, the prompt is as follows:

[1394] "Generate a pet monitoring scenario"

[1395] If you want to generate a baby watching scenario, the prompt is as follows:

[1396] "Generate a baby watching scenario"

[1397] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1398] Step 1:

[1399] Camera Interface:

[1400] The device uses a camera to capture video data from within the home in real time. The input is the video data captured by the camera, and the output is frames of this video data. The device encodes these frames and transmits them to a server via the Internet.

[1401] Step 2:

[1402] Video data reception and processing:

[1403] The server receives the video data sent from the device and temporarily stores it in a buffer. The input is the encoded video data, and the output is each divided frame. The server then decodes these frames sequentially and inputs them into the image recognition module.

[1404] Step 3:

[1405] Image Recognition:

[1406] The server uses TensorFlow and Keras to input video frames into an image recognition module. The input is image data for each frame, and the output is information about identified people and animals. The server analyzes each frame to identify people and animals and verify their identity by matching them with a known database.

[1407] Step 4:

[1408] Behavior analysis:

[1409] The server analyzes behavior based on the image recognition results and detects dangerous behavior. The input is the identification information obtained from the image recognition module, and the output is the type and urgency of the detected dangerous behavior. For example, it can detect when a baby is about to fall or when a pet is biting an electrical cord.

[1410] Step 5:

[1411] Alert System:

[1412] The server generates an audio alert for the detected risky behavior and issues it to the device. The input is the behavior analysis results, and the output is the audio alert data. The device receives this data and plays the audio alert through its built-in speaker.

[1413] Step 6:

[1414] Notification system:

[1415] The server sends push notifications of risky behavior and intrusions to the user's smartphone or PC. The input is the behavior analysis results and emotion recognition results, and the output is a notification message sent to the user's device. The user receives the notification and checks the detailed information.

[1416] Step 7:

[1417] Emotion Engine:

[1418] The server uses an emotion engine to analyze the user's voice and facial expressions to determine their emotional state. The input is the user's audio and video data, and the output is an estimated emotional state. Based on this result, the server adjusts the content and urgency of alerts and notifications.

[1419] Step 8:

[1420] Database Management:

[1421] The server stores video data, image recognition results, behavioral analysis results, and emotion data in a database. The input is all processed data, and the output is historical data stored in the database. This allows for future reanalysis and backup.

[1422] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1423] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1424] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1425] [Fourth embodiment]

[1426] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1427] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1428] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1429] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1430] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1431] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1432] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1433] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1434] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1435] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1436] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1437] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1438] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1439] MODE FOR CARRYING OUT THE INVENTION

[1440] To achieve the functions described in the claims, the present invention provides a system that includes the following elements: Specifically, a camera, a server, a terminal, and a software module are combined to create an environment in which users can live their daily lives with peace of mind.

[1441] Program processing overview

[1442] The present invention is a system that processes data between devices as follows, and ultimately provides the user with the information they need.

[1443] 1. Camera Interface

[1444] A camera connected to the device captures images of the inside of the house in real time, for example, periodically taking pictures of the living room or children's room.

[1445] The video data captured by the terminal is transmitted to a server via the Internet.

[1446] 2. Receiving and processing video data

[1447] The server temporarily stores the video data received from the terminal in a buffer.

[1448] The server divides the video data into frames and inputs them into an image recognition module.

[1449] 3. Image Recognition

[1450] The image recognition module detects people and pets in the frame and recognizes their faces and shapes, for example, the face of a baby or a pet dog.

[1451] The image recognition module compares the image against a database to determine whether it is a known person or pet.

[1452] 4. Behavior analysis

[1453] The server performs behavioral analysis based on the information obtained from the image recognition module, for example, detecting a baby trying to crawl out of its crib.

[1454] The behavior analysis module develops defined risky behaviors and determines whether the detected behavior matches them.

[1455] 5. Alert System

[1456] The server generates an audio alert message if risky behavior is detected.

[1457] The device will emit a warning sound or message through the speaker to the user, for example, "Your baby is about to fall out of his crib!"

[1458] 6. Notification System

[1459] The server sends push notifications to users' smartphones or PCs, instantly informing them of any dangerous behavior or intruders.

[1460] For example, if a pet starts chewing on an electrical cord, the user will receive a notification on their smartphone saying, "Your pet is chewing on the electrical cord!"

[1461] 7. Database Management

[1462] The server stores the detected video data and analysis results in a database.

[1463] The database management module organizes the data for later access and creates backups as needed.

[1464] Specific examples

[1465] 1. Pet monitoring

[1466] A camera installed on the device periodically captures footage of the pet dog.

[1467] The server receives the video data and analyzes the dog's behavior using image recognition and behavior analysis modules.

[1468] If a dog starts chewing on an electrical cord, the server will play an audio message saying "That's dangerous!" and simultaneously send a notification to the user's smartphone.

[1469] 2. Baby watching

[1470] The device's camera captures images of the baby in real time.

[1471] The server analyzes the baby's behavior and detects dangerous movements.

[1472] When the baby tries to crawl out of the crib, the device will emit an audio alert saying "Baby is moving!" and send a push notification to the user's smartphone.

[1473] This allows users to ensure the safety of pets, babies, and small children anytime, anywhere. This system prevents danger in daily life and provides users with a sense of security.

[1474] The processing flow will be explained below.

[1475] Step 1:

[1476] The device's camera captures video of the home in real time at a set frame rate, and the video data is divided into frames at regular time intervals.

[1477] Step 2:

[1478] The captured video frames are sent directly to the server using a dedicated communication protocol, ensuring data security.

[1479] Step 3:

[1480] The server stores the video frames received from the terminal in a buffer in order, allowing subsequent processing to proceed smoothly.

[1481] Step 4:

[1482] The server sends the buffered video frames to an image recognition module that detects people and pets in each frame, using specific algorithms to identify faces and shapes.

[1483] Step 5:

[1484] The image recognition module compares the detection results with a database to determine whether the target is registered, for example, by comparing the faces of family members with facial images stored in the database.

[1485] Step 6:

[1486] The server passes the image recognition results to the behavior analysis module, which then analyzes multiple behavioral patterns to determine whether the target's behavior is dangerous.

[1487] Step 7:

[1488] The behavior analysis module analyzes behavioral patterns and detects dangerous behavior (for example, a baby about to fall or a pet chewing on an electrical cord).

[1489] Step 8:

[1490] If the server detects any risky behavior, it will instantly generate an audio alert message, which is generated based on pre-defined phrases.

[1491] Step 9:

[1492] The device receives the audio alert message and issues a voice warning through the speaker, for example, "That's dangerous!"

[1493] Step 10:

[1494] The server then sends a push notification to the user's smartphone or PC in parallel, containing details of the risky behavior detected.

[1495] Step 11:

[1496] Users can check notifications on their smartphones or PCs and rush to the scene if necessary. For example, they can tap the pop-up notification to check details.

[1497] Step 12:

[1498] The server stores the detected video data and analysis results in a database, which can be used for later review and analysis.

[1499] Step 13:

[1500] The database management module periodically creates backups of data and makes them available for system recovery, thereby preventing data loss.

[1501] Example 1

[1502] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1503] Ensuring the safety of children and pets is a major concern in modern homes. Conventional surveillance systems simply record video footage, but are unable to detect danger or send alerts in real time, making it difficult to respond to situations that require a rapid response. Furthermore, there is a lack of means to manage and analyze massive amounts of video data, making efficient data storage and use a challenge.

[1504] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1505] In this invention, the server includes means for capturing image data using a camera and transmitting it to the server in real time, means for receiving the video data and performing image recognition of people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting dangerous behavior, means for issuing an audio alert to notify the user of the detected dangerous behavior, means for notifying the user of the detected dangerous behavior, means for temporarily storing the video data in a buffer, means for storing the detected video data and analysis results in a database for later access, means for analyzing each frame using an image recognition module, means for tracking movements in the video using a behavior analysis module and detecting specific patterns, and means for generating an audio alert message. This enables real-time monitoring and analysis of video data inside the home, rapid detection of dangerous behavior and issuing an alert, and efficient storage and management of video data.

[1506] A "camera" is an optical device for capturing image data.

[1507] A "server" is a computer system that processes and manages data in response to requests from clients.

[1508] A "terminal" is a computer system or device that has input and output devices and can be directly operated by a user.

[1509] "Image data" refers to visual information captured by a camera and stored in digital form.

[1510] "Real-time" means near-instant processing with minimal delay.

[1511] A "buffer" is a memory area that functions as a temporary storage area for data and ensures a smooth flow of data processing.

[1512] "Image recognition" is the process of using computer vision techniques to detect specific objects or patterns in video data.

[1513] "Behavioral analysis" is the process of analyzing patterns of movement and behavior to predict intentions and next actions.

[1514] A "voice alert" is a system that uses voice to warn or notify.

[1515] A "user terminal" is a device operated by a user who uses the system, and typically refers to a smartphone or personal computer.

[1516] A "database" is a structured data storage system for efficiently managing, retrieving, and storing data.

[1517] An "image recognition module" is a software component that analyzes image data to recognize specific objects or people.

[1518] The "behavioral analysis module" is a software component that analyzes behavior from recognized data and detects specific patterns and anomalies.

[1519] A "voice alert message" is a message containing a warning or notification generated using voice synthesis technology.

[1520] "People and animals" refers to living beings such as humans and pets that appear in image data.

[1521] This invention combines cameras, servers, terminals, and software modules to provide an environment in which users can live their daily lives with peace of mind. Specifically, it is a system that monitors images from inside the home in real time and issues alerts and notifications as needed.

[1522] Hardware and Software Configuration

[1523] 1. Camera

[1524] Device: A camera is used to capture video from inside the home. For example, cameras installed in the living room or children's room capture video data in real time.

[1525] The video data is transmitted to a server via the Internet.

[1526] 2. Server

[1527] Server: Receives video data sent from the device and temporarily stores it in a buffer. This buffer can hold several seconds of video.

[1528] The received video data is divided into frames and input into the image recognition module.

[1529] Image Recognition Module: Detects people and pets from video frames and recognizes their faces and shapes, for example using TensorFlow or OpenCV.

[1530] A software module (behavior analysis module) runs to perform behavior analysis based on the recognized information.

[1531] The module detects risky behavior and handles the results appropriately.

[1532] 3. Alert System

[1533] Server: Uses a text-to-speech (TTS) engine to generate audio alert messages if risky behavior is identified as a result of behavioral analysis.

[1534] Device: Play an audio alert through the speaker, for example, "Baby is about to fall out of his crib!"

[1535] 4. Notification System

[1536] Server: Sends push notifications to users' smartphones or PCs. To do this, you can use a notification service such as Firebase Cloud Messaging (FCM).

[1537] Users: They can check notifications on their smartphones and take action quickly.

[1538] 5. Database Management

[1539] Server: Stores the detected video data and analysis results in a database, such as MySQL or PostgreSQL.

[1540] The stored data is organized and regularly backed up for later access.

[1541] Specific examples

[1542] 1. Pet monitoring

[1543] Device: A camera installed in the living room captures footage of a pet dog.

[1544] Server: Receives video data and detects the dog using an image recognition module. It then analyzes its behavior and, for example, if the dog starts biting an electrical cord, determines that this behavior is dangerous.

[1545] Server: Generates and broadcasts the audio alert "That's dangerous!". At the same time, a notification is sent to the user's smartphone saying "Your pet is chewing on the electrical cord!"

[1546] 2. Baby watching

[1547] Device: A camera installed in the child's room captures real-time footage of the baby.

[1548] Server: Receives the video data and detects the baby's position using the image recognition module.Then, the behavior analysis module detects the baby's movements to crawl out of the crib.

[1549] Server: Generates and sends an audio alert in response to this dangerous behavior: "The baby is about to fall out of the crib!" At the same time, a notification is sent to the user's smartphone saying, "The baby is moving!"

[1550] This allows users to ensure the safety of pets, babies, and small children anytime, anywhere. This system prevents danger in daily life and provides users with a sense of security.

[1551] Prompt Sentence Examples

[1552] Text-to-speech prompt for when your baby is about to fall out of his crib: "Your baby is about to fall out of his crib!"

[1553] If your pet starts chewing on an electrical cord, a push notification prompt will appear: "Your pet is chewing on the electrical cord!"

[1554] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1555] Program processing steps

[1556] Explanation divided into processing steps

[1557] Step 1: Video capture and transmission

[1558] Step 2: Receiving and buffering video data

[1559] Step 3: Image Recognition

[1560] Step 4: Behavioral analysis

[1561] Step 5: Alert Generation

[1562] Step 6: Send notifications

[1563] Step 7: Data Management and Storage

[1564] ---

[1565] Specific explanation of each processing step

[1566] Step 1: Video capture and transmission

[1567] Device: A camera connected to the device captures video from inside the home in real time, for example capturing video of the living room at 30 frames per second.

[1568] Input: Image captured by the camera.

[1569] Data processing: Encode the captured video into a compressed format such as H.264.

[1570] Output: Encoded video data.

[1571] Specific operation: The encoded video data is sent to a server via the Internet, for example, at a rate of 1MB per second.

[1572] Step 2: Receiving and buffering video data

[1573] Server: Receives video data sent from the device. For example, it waits for streaming data on a specific port on the server.

[1574] Input: Video data sent from the device.

[1575] Data calculation: The received video data is temporarily stored in a buffer. The buffer size is set so that it can hold several seconds of video data.

[1576] Output: Buffered video data.

[1577] What it does: The server receives 1MB of data per second over an inbound TCP connection and stores it in a buffer.

[1578] Step 3: Image Recognition

[1579] Server: The video data stored in the buffer is divided into frames and input into the image recognition module.

[1580] Input: Buffered frame data.

[1581] Data computation: Each frame is analyzed by an image recognition module to detect people and pets, for example using TensorFlow or OpenCV.

[1582] Output: Information about detected people and pets.

[1583] What it does: It analyzes video at 30 frames per second and runs an algorithm to detect the baby's face in each frame.

[1584] Step 4: Behavioral analysis

[1585] Server: Performs behavioral analysis based on information obtained from the image recognition module. Tracks the movements of people and pets in the video and detects specific patterns (e.g., crawling out of a crib).

[1586] Input: Person and pet information obtained from the image recognition module.

[1587] Data calculation: The behavioral analysis module tracks movements and determines whether they match defined risky behaviors.

[1588] Output: Risky behavior detection results.

[1589] Specific operation: If the baby moves to a dangerous location, the location information and movement pattern are analyzed to determine whether it constitutes dangerous behavior.

[1590] Step 5: Alert Generation

[1591] Server: If risky behavior is detected as a result of behavioral analysis, it generates an audio alert message and converts the alert message into audio using a Text-to-Speech (TTS) engine.

[1592] Input: The behavioral analysis results and the corresponding text alert message.

[1593] Data Calculation: Converts text into a voice alert message using a TTS engine.

[1594] Output: Audio alert message.

[1595] Specific behavior: Generates the message "Your baby is about to fall out of his crib!" and plays it through speakers in the home.

[1596] Step 6: Send notifications

[1597] Server: If dangerous behavior or anomalies are detected, a push notification is sent to the user's smartphone or PC. Services such as Firebase Cloud Messaging (FCM) may be used.

[1598] Input: The behavior analysis results and the corresponding notification message.

[1599] Data Computing: Deliver messages using a notification service such as FCM.

[1600] Output: Notifications sent to the user's smartphone or PC.

[1601] Specific operation: When the baby engages in dangerous behavior, the user will receive a notification on their smartphone saying, "Your baby is moving!"

[1602] Step 7: Data Management and Storage

[1603] Server: Stores the detected video data and analysis results in a database. RDBMS such as MySQL or PostgreSQL is often used.

[1604] Input: Video data and analysis results.

[1605] Data Computation: Stores data in a database, creates indexes, and creates regular backups.

[1606] Output: Data stored in a database.

[1607] Specific operation: Video data along with analysis results are stored in a database with a timestamp, allowing users to verify past data.

[1608] (Application example 1)

[1609] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1610] The challenge is to enhance home security by detecting suspicious individuals or dangerous behavior in real time and notifying users promptly. It is also necessary to send push notifications to users if suspicious behavior persists and save video frames as a record to facilitate later analysis and evidence collection.

[1611] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1612] In this invention, the server includes means for capturing image data using a camera and transmitting it to the server in real time, means for receiving the video data and performing image recognition of people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting suspicious behavior, means for issuing an audio alert to notify the user of the detected suspicious behavior, means for notifying the user of the detected suspicious behavior to a user terminal, means for notifying the user via push notification if the suspicious behavior continues, and means for saving the detected video frames as a log file, thereby enabling early detection of suspicious people and dangerous behavior and rapid response.

[1613] A "camera" is a device that captures video data and transmits the video to a server.

[1614] "Real-time" refers to processing the target images and data instantly and without delay.

[1615] A "server" is a computer system that receives video data sent from a camera and performs various data processing.

[1616] "Video data" refers to digital data of moving or still images captured by a camera.

[1617] "Image recognition" is a technology that detects and identifies specific objects such as people and animals within video data.

[1618] "Suspicious behavior" refers to behavior that is unusual or dangerous based on the results of image recognition.

[1619] "Audio alert" is a means of issuing a warning sound or voice message when suspicious activity is detected.

[1620] "User terminal" refers to a communication device such as a smartphone or PC owned by a user.

[1621] A "push notification" is an alert or notification that is sent instantly from a server to a user device.

[1622] A "log file" is a file that records system operation status and event information, and can be used for later analysis and verification.

[1623] The present invention describes an embodiment of a system for strengthening home security and detecting suspicious individuals and dangerous behavior in real time. This system is configured by combining a camera, a server, a user terminal, and various software modules.

[1624] Components

[1625] 1. Camera

[1626] Cameras installed in homes capture real-time video of what is happening in and around the home, and the video data is sent to a server via Wi-Fi or a wired connection.

[1627] 2. Server

[1628] The server receives the video data and has a buffer for temporary storage.

[1629] Video data is processed using various software modules (image recognition module, behavior analysis module, alert generation module, notification module).

[1630] 3. User Device

[1631] User devices such as smartphones and PCs communicate with the server and receive push notifications.

[1632] Software Module

[1633] 1. Image Recognition Module

[1634] The received video data is divided into frames and specific objects (people, animals) are detected.

[1635] It utilizes existing deep learning technologies such as OpenCV's DNN module.

[1636] 2. Behavioral Analysis Module

[1637] Based on the information obtained from the image recognition module, behavioral analysis is performed to detect suspicious behavior.

[1638] If suspicious behavior continues for a certain period of time, an alert is generated.

[1639] 3. Alert Generation Module

[1640] When suspicious behavior is detected, audio alerts and messages are generated and played over speakers in the home.

[1641] 4. Notification Module

[1642] The server will then send a push notification to the user's device regarding any suspicious behavior or dangerous situations that it detects.

[1643] For example, a notification such as "There's a suspicious person at the front door!" is immediately forwarded to the user.

[1644] 5. Data storage module

[1645] The detected video frames and analysis results are saved as a log file, making them available for later analysis.

[1646] Processing Description

[1647] The server receives video data from the camera in real time and temporarily stores it in a buffer. The video data is analyzed through an image recognition module to detect specific objects (people or animals). The behavior analysis module then analyzes the movements of these objects to detect any suspicious behavior.

[1648] If the detected suspicious behavior continues for a certain period of time, the alert generation module will generate an audio warning or message and send it to a speaker in the home, while the notification module will send a push notification to the user's device to immediately notify them of the suspicious activity.

[1649] In addition, the data storage module stores the detected video frames and analysis results as a log file for later analysis and evidence collection.

[1650] Examples and prompts

[1651] For example, if a suspicious person is detected at the entrance, the image recognition module will identify the person, and the behavior analysis module will analyze the person's movements. If the suspicious behavior continues, the alert generation module will play an audio message saying "There's a suspicious person at the entrance!", and at the same time, the notification module will send a push notification to the user's device.

[1652] Examples of prompts include, "What happens if an intruder is detected in your home?", "How does your system detect an intruder?", and "What is the threshold for generating an alert?"

[1653] As a result, the security system based on the present invention can significantly improve safety within the home and encourage users to take prompt and appropriate action.

[1654] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1655] Step 1:

[1656] The server receives video data from cameras installed in homes in real time. The input data is real-time video data captured by the cameras, and the server temporarily stores it in a buffer.

[1657] Step 2:

[1658] The server divides the temporarily stored video data into frames and sends them to the image recognition module. The input data here are video frames, which the image recognition module receives and detects people and animals using OpenCV's DNN module, etc. The output data is position information and probability information for objects (people and animals) present in the video frames.

[1659] Step 3:

[1660] The server sends the object information obtained from the image recognition module to the behavior analysis module. The input data is the object information, and the behavior analysis module uses this information to analyze suspicious behavior. The output data is the type of analyzed behavior and its risk level.

[1661] Step 4:

[1662] The server uses the alert generation module to create a voice alert based on the suspicious behavior information obtained from the behavior analysis module. The input data is the suspicious behavior information, and the alert generation module generates a warning message based on this. The output data is a warning message or a voice file. The device plays this warning sound or message.

[1663] Step 5:

[1664] The server notifies the user device of the suspicious behavior information. The input data is the suspicious behavior information and the generated warning message. The notification module generates a push notification based on this and sends it to the user device. The output data is the push notification message. The push notification is displayed on the device and a warning is sent to the user.

[1665] Step 6:

[1666] The server uses a data storage module to store the detected video frames and analysis results as a log file. The input data are the video frames and their analysis results, and the data storage module creates a log file based on them. The output data is the stored log file, which can be used for later analysis or as evidence.

[1667] For example, if a user asks, "What happens if a suspicious person is detected in the house?", the suspicious person will be detected through the above steps, a warning will be sent to the user, and necessary measures will be taken. In this way, the present invention improves home safety and enables early detection of suspicious people and rapid response.

[1668] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1669] MODE FOR CARRYING OUT THE INVENTION

[1670] To achieve the functions recited in the claims, the present invention provides a system including the following elements, which monitors and protects pets, babies, and young children, recognizes the user's emotions as needed, and provides corresponding alerts, allowing the user to live their daily lives with peace of mind.

[1671] Program processing overview

[1672] The present invention is a system that operates by combining a camera, a server, a terminal, and a software module, and specifically performs data processing in the following manner.

[1673] 1. Camera Interface

[1674] A camera attached to the device captures visual data from inside the home in real time.

[1675] The terminal transmits the captured video data to a server via the Internet.

[1676] 2. Receiving and processing video data

[1677] The server temporarily stores the video data received from the terminal in a buffer and divides it into frames.

[1678] The server inputs the video frames into an image recognition module.

[1679] 3. Image Recognition

[1680] An image recognition module identifies people and pets in each frame.

[1681] Identified faces and shapes are matched against a known database to determine if the subject is registered.

[1682] 4. Behavior analysis

[1683] The server analyzes behavior based on the results of image recognition and detects risky behavior.

[1684] Specifically, it detects actions such as a baby falling or a pet chewing on an electrical cord.

[1685] 5. Alert System

[1686] The server generates an audio alert for detected risky behavior.

[1687] The device will issue an audio warning through the speaker.

[1688] 6. Notification System

[1689] The server sends push notifications to users' smartphones or PCs about dangerous behavior or intrusions.

[1690] The user receives a notification and checks for more information.

[1691] 7. Emotion Engine

[1692] The emotion engine installed on the server recognizes the user's emotions, for example, by analyzing the user's voice and facial expressions to determine their emotional state.

[1693] The content of alerts and notifications is adjusted based on the analysis results of the emotion engine. For example, if the user is feeling stressed, notifications with a high level of urgency will be sent first.

[1694] 8. Database Management

[1695] The server stores the video data, image recognition results, behavioral analysis results, and emotion data in a database.

[1696] The management module organizes this data and makes it available for later reanalysis or backup.

[1697] Specific examples

[1698] 1. Pet monitoring

[1699] The device's camera captures video of your pet dog in real time.

[1700] The server receives the video data and analyzes the dog's behavior using image recognition and behavior analysis modules.

[1701] If a dog starts biting an electrical cord, the server generates a voice message saying, "That's dangerous!" and the device issues a voice warning. The emotion engine checks the user's stress level and sends an emergency notification to the smartphone if necessary.

[1702] 2. Baby watching

[1703] The device's camera captures images of the baby and sends them to a server in real time.

[1704] The server analyzes the baby's behavior and detects dangerous movements (such as an imminent fall).

[1705] If the baby tries to crawl out of the crib, the device will issue an audio alert saying "Baby moving!" The server will use an emotion engine to determine the parent's level of concern and adjust the urgency of the notification accordingly. For example, if the user is busy, they can choose not to receive detailed notifications.

[1706] This provides a system that not only monitors and protects pets, babies, and small children, but also takes into account the user's emotions, reducing stress for the user and creating a safer and more comfortable living environment.

[1707] The processing flow will be explained below.

[1708] MODE FOR CARRYING OUT THE INVENTION

[1709] In order to achieve the functions described in the claims, the present invention provides a system including the following elements: In particular, processing steps for monitoring and protecting pets, babies, and young children are specifically described.

[1710] Step 1:

[1711] The camera captures video of the inside of the house in real time at a set frame rate, with the video divided into frames at regular time intervals.

[1712] Step 2:

[1713] The device transmits the captured video frames to a server over the Internet, where the transmission is secure using an encrypted communication protocol.

[1714] Step 3:

[1715] The server stores the video frames received from the terminal in a buffer in order, allowing subsequent processing to proceed smoothly.

[1716] Step 4:

[1717] The server inputs the buffered video frames into an image recognition module to detect people or pets in each frame, for example, identifying the faces and shapes of babies or pets.

[1718] Step 5:

[1719] The image recognition module checks the detected person or pet data against a database to determine whether the target is already registered, for example, by comparing it with facial images of family members stored in the database.

[1720] Step 6:

[1721] The server passes the image recognition results to the behavioral analysis module, which then begins analyzing the behavior, identifying risky behaviors, such as a baby trying to crawl out of its crib.

[1722] Step 7:

[1723] Based on the behavioral patterns analyzed by the behavior analysis module, dangerous behavior (for example, a pet chewing on an electrical cord or a baby about to fall) is detected.

[1724] Step 8:

[1725] If the server detects risky behavior, it generates an audio alert message based on a predefined phrase.

[1726] Step 9:

[1727] The device receives the audio alert message and issues a warning through the speaker, such as "That's dangerous!"

[1728] Step 10:

[1729] The server sends a push notification to the user's smartphone or PC, which contains details of the risky behavior.

[1730] Step 11:

[1731] The user receives a push notification on their smartphone or PC and checks the details, for example by tapping on the pop-up notification to check the details of the risky behavior.

[1732] Step 12:

[1733] An emotion engine installed on the server analyzes the user's emotions in real time, detecting, for example, stress or anxiety from the user's voice and facial expressions.

[1734] Step 13:

[1735] Based on the analysis results of the emotion engine, the server adjusts the content of audio alerts and push notifications. For example, if the user is in a high stress state, a notification with more specific information will be sent.

[1736] Step 14:

[1737] The server stores the detected video data, image recognition results, behavioral analysis results, and emotion data in a database.

[1738] Step 15:

[1739] The database management module periodically backs up data and stores it for future analysis and restoration, thus preventing data loss.

[1740] This allows users to better monitor and protect their pets, babies and young children, and the alert system also takes into account the user's emotional state to provide more relevant and reassuring notifications.

[1741] Example 2

[1742] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1743] In modern homes, it is important to safely monitor pets, babies, and young children and respond quickly when necessary. However, conventional surveillance systems are limited to simple video capture and notification, and do not support specific behavioral analysis of people and animals or adjust the urgency of notifications based on the user's emotions. This makes it difficult to respond quickly and appropriately to dangerous situations, resulting in a lack of user peace of mind and convenience.

[1744] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing image data using a camera and transmitting it to the server in real time, a means for receiving video data and performing image recognition of people and animals in the video data, a means for analyzing behavior based on the image recognition results and detecting dangerous behavior, a means for issuing an audio alert to notify the user of the detected dangerous behavior, a means for notifying the user of the detected dangerous behavior to a user terminal, a means for recording analysis data and emotion data in a database and using it for later analysis, and a means for analyzing the user's emotion and adjusting the urgency of the notification. This makes it possible to monitor and protect pets, babies, and young children, and to provide appropriate alerts and notifications taking the user's emotion into consideration.

[1745] "Camera" means a device that captures visual data. Commonly used terms include digital cameras and webcams.

[1746] A "server" is a computer system that receives, processes, and manages data over a network.

[1747] A "terminal" is a device that is directly operated by a user, and is a computer to which peripheral devices such as a camera and speakers are connected.

[1748] "Image data" refers to digital data containing visual information captured by a camera.

[1749] "Real-time" refers to near-instant processing and response without delay.

[1750] "Video data" refers to a moving image made up of a series of multiple image data.

[1751] "Image recognition" is the process of automatically identifying people, animals, and objects in video data.

[1752] "Behavioral analysis" is the process of evaluating a subject's behavior based on the results of image recognition to detect specific patterns or anomalies.

[1753] "Riskful behavior" refers to behavior that puts a subject in a potentially dangerous situation and may threaten their safety.

[1754] A "voice alert" is a notification that issues a warning via voice when risky behavior is detected.

[1755] A "user terminal" refers to a computer device such as a smartphone or PC used by a user.

[1756] A "notification" is a message that notifies the user of important information or warnings.

[1757] A "database" is a system for efficiently storing, managing, and searching structured data.

[1758] "Emotional data" refers to digital data that represents a user's emotional state.

[1759] The "urgency of a notification" is an indicator of the degree to which a notification requires an immediate response.

[1760] The present invention is a system for monitoring and protecting pets, babies, and small children, and provides a safe living environment by recognizing the user's emotions and providing notifications accordingly. Specifically, it is realized using a camera, a server, a terminal, and multiple software modules.

[1761] A camera attached to the device captures video data from inside the home in real time. The hardware used is a typical surveillance camera, such as a digital camera or webcam. The captured video data is sent from the device to a server via the Internet. The software used for this is a library that streams and transmits video data, such as OpenCV or FFmpeg.

[1762] The server stores the video data received from the device in a buffer and then divides it into frames. The cloud platform used is a "cloud storage service," such as "AWS S3" or "Google Cloud Storage." The server then passes the divided frames to an image recognition module, which identifies people or pets in each frame. This image recognition uses an "image recognition library" or "deep learning model," such as "YOLOv5" or "TensorFlow."

[1763] The identified target's face and shape are compared with a known database on the server to determine whether the target is registered. The behavioral analysis module then analyzes the target's movements based on the image recognition results and detects risky behavior. This behavioral analysis is performed using a "motion analysis library," such as "OpenPose" or "Scikit-learn."

[1764] If a dangerous behavior is detected, the server generates an audio alert. This is generated using a speech synthesis service, such as Amazon Polly or Google Text-to-Speech. The generated audio alert is then played through the device's speaker. The device is equipped with a standard speaker, such as a Bluetooth speaker or a smart speaker.

[1765] The server then notifies the user of any detected risky behavior via their smartphone or PC. Notifications are sent using a push notification service, which uses Firebase Cloud Messaging. Users can receive notifications and check detailed information on their smartphone or PC.

[1766] Additionally, the server is equipped with an emotion engine that analyzes the user's voice and facial expressions to determine their emotional state. This emotion analysis uses an "emotion recognition service," which includes emotion recognition APIs from IBM Watson and Microsoft Azure. The alert content and urgency of the notification are adjusted based on the analysis results of the emotion engine. If the user is feeling stressed, notifications with a higher urgency level are sent first.

[1767] Finally, the server stores the video data, image recognition results, behavioral analysis results, and emotion data in a database. The database used is a database management system, such as MongoDB or PostgreSQL, which allows for data organization, reanalysis, and backup.

[1768] Specific examples

[1769] 1. Pet monitoring

[1770] The device's camera captures video of your pet dog in real time.

[1771] The server receives the video data and analyzes the dog's behavior using image recognition and behavior analysis modules.

[1772] If a dog starts biting an electrical cord, the server generates a voice message saying, "That's dangerous!" and the device issues a voice warning. The emotion engine checks the user's stress level and sends an emergency notification to the smartphone if necessary.

[1773] 2. Baby watching

[1774] The device's camera captures images of the baby and sends them to a server in real time.

[1775] The server analyzes the baby's behavior and detects dangerous movements (such as an imminent fall).

[1776] If the baby tries to crawl out of the crib, the device will issue an audio alert saying "Baby moving!" The server will use an emotion engine to determine the parent's level of concern and adjust the urgency of the notification accordingly. For example, if the user is busy, they can choose not to receive detailed notifications.

[1777] As described above, the present invention provides a system for monitoring and protecting pets, babies, and small children, and is flexible and considers the user's emotions, reducing stress for the user and creating a safer and more comfortable living environment.

[1778] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1779] Step 1:

[1780] Video Capture

[1781] A camera attached to the device captures visual data from inside the home in real time. Specifically, the camera continuously captures video data at about 30 frames per second. The hardware used is a digital camera or webcam. The input is the visual data captured by the camera, and the output is video data captured in real time.

[1782] Step 2:

[1783] Data transmission

[1784] The device sends the captured video data to a server via the Internet. The software used is "OpenCV" and a network library. The input is real-time video data, and the output is the video data sent to the server. Specifically, the device divides the video data into packets at regular intervals (for example, every second) and sends them to the server via the network.

[1785] Step 3:

[1786] Data reception

[1787] The server temporarily stores the video data received from the device in a buffer. The cloud platform used is a "cloud storage service" that includes AWS S3 and Google Cloud Storage. The input is the video data sent from the device, and the output is the video data stored in the buffer in the server. Specifically, the server reconstructs the received packets and stores them in the buffer as video data.

[1788] Step 4:

[1789] Frame Split

[1790] The server divides the video data into frames. This process uses the OpenCV library. The input is the video data stored in the buffer, and the output is each frame. Specifically, the server divides the video data into 30 frames per second and passes each frame to subsequent processing.

[1791] Step 5:

[1792] Image Recognition

[1793] The server inputs video frames into an image recognition module to identify people and pets in each frame. The image recognition libraries used are "YOLOv5" and "TensorFlow." The input is video data divided into frames, and the output is information about the identified objects (position, type, etc.). Specifically, the object detection algorithm is applied to each frame to obtain the object's coordinates and class label.

[1794] Step 6:

[1795] Database collation

[1796] The server compares the identified face and shape with a known database. The database includes "face recognition software" and "shape recognition modules." The input is the information of the identified subject, and the output is whether the subject is registered in the database. Specifically, the server extracts the subject's features and compares them with entries in the database to determine whether they match.

[1797] Step 7:

[1798] Behavioral analysis

[1799] The server analyzes behavior based on the results of image recognition and detects risky behavior. The software libraries used are "OpenPose" and "Scikit-learn." The input is the movement and location information of the identified object, and the output is the behavior analysis results and the detection of risky behavior. Specifically, the server analyzes time-series data and detects specific patterns (e.g., falls, biting electrical cords, etc.).

[1800] Step 8:

[1801] Audio alert generation

[1802] If the server detects risky behavior, it immediately generates a corresponding audio alert. This audio alert is generated using a "speech synthesis service," specifically "Amazon Polly" or "Google Text-to-Speech." The input is the detection result of risky behavior, and the output is the generated audio file. Specifically, the server sends a text message to the speech synthesis service, which generates audio data.

[1803] Step 9:

[1804] Voice alert

[1805] The device issues the audio alert sent from the server through a speaker. A typical speaker, such as a Bluetooth speaker or a smart speaker, is attached to the device. The input is the generated audio file, and the output is the transmission of the audio alert. Specifically, the device plays the audio file and issues the audio alert from the speaker.

[1806] Step 10:

[1807] Send notifications

[1808] The server sends push notifications of detected risky behavior or intrusions to the user's smartphone or PC. The push notification service used is "Firebase Cloud Messaging." The input is the detection result of risky behavior and the notification content, and the output is the notification sent to the user's device. Specifically, the server generates a push notification message and sends it to the user's device via the cloud messaging service.

[1809] Step 11:

[1810] emotion recognition

[1811] The emotion engine installed on the server recognizes the user's emotions. Emotion analysis uses "emotion recognition services," including "IBM Watson" and "Microsoft Azure." The input is data based on the user's voice and facial expressions, and the output is the user's emotional state. Specifically, the server analyzes the user's voice data and facial expression data and evaluates the state using the emotion engine.

[1812] Step 12:

[1813] Emotion-based notification adjustment

[1814] The server adjusts the alert content and notification urgency based on the analysis results of the emotion engine. The input is the user's emotional state and the detection results of risky behavior, and the output is the adjusted alert and notification. Specifically, the server determines whether the user is feeling stressed and sets the system to send notifications with a high level of urgency first.

[1815] Step 13:

[1816] Data storage

[1817] The server stores video data, image recognition results, behavioral analysis results, and emotion data in a database. The database management systems used are "MongoDB" and "PostgreSQL." The input is the various analysis result data, and the output is the data stored in the database. Specifically, the server formats the various data appropriately and inserts it into the database.

[1818] (Application example 2)

[1819] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1820] There is a need for monitoring and protecting pets, babies, and young children in the home, and for providing prompt and appropriate alerts through user emotion recognition when necessary. Conventional systems focus on detecting dangerous behavior, but are unable to consider the user's emotional state, making it difficult to reduce stress and anxiety. Furthermore, there are challenges in detecting dangerous behavior and providing notifications accurately and immediately.

[1821] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1822] In this invention, the server includes means for capturing image data using a camera and transmitting it to the server in real time, means for receiving the video data and performing image recognition of people and animals in the video data, means for analyzing behavior based on the image recognition results and detecting dangerous behavior, means for issuing an audio alert to notify the user of the detected dangerous behavior, means for notifying the user of the detected dangerous behavior to a user terminal, and means for adjusting the content of the notification or alert based on the emotion recognition results. This makes it possible to monitor and protect pets, babies, and young children in the home, and allows for flexible responses according to the user's emotional state, providing a safer and more secure living environment.

[1823] A "camera" is a device for capturing image data.

[1824] A "server" is a central computer for receiving, storing, processing, and distributing data over a network.

[1825] "Image data" refers to visual information captured using a camera.

[1826] "Real-time" refers to processing occurring immediately without delay.

[1827] "Video data" refers to moving image information that is made up of a series of multiple image frames.

[1828] A "person" is an object that shows a human shape or face.

[1829] "Animals" refers to non-human living things such as pets and livestock.

[1830] "Image recognition" is the technology of identifying and analyzing objects and patterns in digital images.

[1831] "Behavioral analysis" is the process of evaluating and analyzing the behavior of a subject based on the results of image recognition.

[1832] "Dangerous behavior" is behavior that has the potential to cause accidents or trouble.

[1833] A "voice alert" is a warning method that calls attention by issuing a voice message.

[1834] A "user terminal" is a device (such as a smartphone or PC) that is directly operated by a user.

[1835] A "notification" is a means of informing a user of specific information.

[1836] "Emotion recognition" is a technology that analyzes a user's voice, facial expressions, etc. to determine their emotional state.

[1837] "Notification content" is the content of the message sent to the user.

[1838] "Alert content" refers to the content of a message to warn or alert.

[1839] A "database" is a system for managing an organized collection of data.

[1840] The present invention is a system for monitoring and protecting pets, babies, and small children in the home, recognizing the user's emotions and providing corresponding alerts as needed. The system operates using cameras, a server, a terminal, and multiple software modules.

[1841] Hardware and Software Configuration

[1842] Cameras: Use the built-in camera on your smartphone or a home security camera that captures video data in real time and sends it to a server.

[1843] Server: Acts as the central computer that receives, stores, processes, and distributes data. The server contains the following main modules:

[1844] Image Recognition Module: Uses TensorFlow and Keras to identify people and animals in video frames.

[1845] Behavioral analysis module: Detects risky behavior based on image recognition results.

[1846] Emotion Recognition Module: Uses TensorFlow and Keras to analyze the user's voice and facial expressions to determine their emotional state.

[1847] Database management module: Stores video data and analysis results for later reanalysis and backup.

[1848] Device: A smartphone or PC used by a user. These devices receive notifications and alerts from the server and transmit them to the user.

[1849] Communication: Data communication between the server, terminal, and camera is carried out via the Internet.

[1850] Program processing

[1851] The server receives video data captured by the camera in real time and temporarily stores it in a buffer. Each frame is input into an image recognition module to identify people and animals. Based on the identification results, a behavior analysis module detects dangerous behavior and generates an audio alert. It also sends a notification of the dangerous behavior to the user's device.

[1852] To take the user's emotions into account, the emotion recognition module recognizes the user's emotional state and adjusts the content of notifications and alerts based on the analysis results. For example, if the user is feeling stressed, notifications with a high level of urgency can be sent first.

[1853] Usage example

[1854] Pet Monitoring:

[1855] If the system detects that a pet dog is biting an electrical cord, it will generate an audio alert saying "That's dangerous!" and send a notification to a smartphone. It will also prioritize emergency notifications if it detects that the user is in a stressful state.

[1856] Baby monitoring:

[1857] When the baby is about to crawl out of the crib, the system will issue an audio alert saying "Baby moving!", and an emotion recognition module will analyze the parent's emotions to adjust the urgency of the notification - for example, it can reduce detailed notifications if the parent is busy.

[1858] Prompt Sentence Examples

[1859] If you want to generate a pet watching scenario, the prompt is as follows:

[1860] "Generate a pet monitoring scenario"

[1861] If you want to generate a baby watching scenario, the prompt is as follows:

[1862] "Generate a baby watching scenario"

[1863] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1864] Step 1:

[1865] Camera Interface:

[1866] The device uses a camera to capture video data from within the home in real time. The input is the video data captured by the camera, and the output is frames of this video data. The device encodes these frames and transmits them to a server via the Internet.

[1867] Step 2:

[1868] Video data reception and processing:

[1869] The server receives the video data sent from the device and temporarily stores it in a buffer. The input is the encoded video data, and the output is each divided frame. The server then decodes these frames sequentially and inputs them into the image recognition module.

[1870] Step 3:

[1871] Image Recognition:

[1872] The server uses TensorFlow and Keras to input video frames into an image recognition module. The input is image data for each frame, and the output is information about identified people and animals. The server analyzes each frame to identify people and animals and verify their identity by matching them with a known database.

[1873] Step 4:

[1874] Behavior analysis:

[1875] The server analyzes behavior based on the image recognition results and detects dangerous behavior. The input is the identification information obtained from the image recognition module, and the output is the type and urgency of the detected dangerous behavior. For example, it can detect when a baby is about to fall or when a pet is biting an electrical cord.

[1876] Step 5:

[1877] Alert System:

[1878] The server generates an audio alert for the detected risky behavior and issues it to the device. The input is the behavior analysis results, and the output is the audio alert data. The device receives this data and plays the audio alert through its built-in speaker.

[1879] Step 6:

[1880] Notification system:

[1881] The server sends push notifications of risky behavior and intrusions to the user's smartphone or PC. The input is the behavior analysis results and emotion recognition results, and the output is a notification message sent to the user's device. The user receives the notification and checks the detailed information.

[1882] Step 7:

[1883] Emotion Engine:

[1884] The server uses an emotion engine to analyze the user's voice and facial expressions to determine their emotional state. The input is the user's audio and video data, and the output is an estimated emotional state. Based on this result, the server adjusts the content and urgency of alerts and notifications.

[1885] Step 8:

[1886] Database Management:

[1887] The server stores video data, image recognition results, behavioral analysis results, and emotion data in a database. The input is all processed data, and the output is historical data stored in the database. This allows for future reanalysis and backup.

[1888] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1889] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1890] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1891] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1892] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1893] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1894] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1895] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1896] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1897] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1898] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1899] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1900] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1901] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1902] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1903] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1904] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1905] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1906] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1907] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1908] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1909] The following is further disclosed regarding the above embodiment.

[1910] (Claim 1)

[1911] means for capturing image data using a camera and transmitting the image data to a server in real time;

[1912] A means for receiving video data and performing image recognition of people and animals in the video data;

[1913] A method for analyzing behavior based on image recognition results and detecting dangerous behavior;

[1914] means for issuing an audio alert to notify the user of the detected unsafe behavior;

[1915] A system including a means for notifying a user terminal of detected risky behavior.

[1916] (Claim 2)

[1917] 10. The system of claim 1, further comprising means for notifying a user when a stranger has entered the system using facial recognition.

[1918] (Claim 3)

[1919] 10. The system according to claim 1, further comprising means for recording the detected image data and the analysis results in a database for use in subsequent analysis.

[1920] "Example 1"

[1921] (Claim 1)

[1922] means for capturing image data using a camera and transmitting the image data to a server in real time;

[1923] A means for receiving video data and performing image recognition of people and animals in the video data;

[1924] A method for analyzing behavior based on image recognition results and detecting dangerous behavior;

[1925] means for issuing an audio alert to notify the user of the detected unsafe behavior;

[1926] means for notifying a user terminal of the detected risky behavior;

[1927] means for temporarily storing video data in a buffer;

[1928] A means to store the detected video data and analysis results in a database for later access;

[1929] means for analyzing each frame using an image recognition module;

[1930] a means for tracking movements in the video using a behavioral analysis module to detect certain patterns;

[1931] A system including means for generating an audio alert message.

[1932] (Claim 2)

[1933] 10. The system of claim 1, further comprising means for notifying a user when a stranger has entered the system using facial recognition.

[1934] (Claim 3)

[1935] 10. The system according to claim 1, further comprising means for recording the detected image data and the analysis results in a database for use in subsequent analysis.

[1936] "Application Example 1"

[1937] (Claim 1)

[1938] means for capturing image data using a camera and transmitting the image data to a server in real time;

[1939] A means for receiving video data and performing image recognition of people and animals in the video data;

[1940] A method for analyzing behavior based on image recognition results and detecting suspicious behavior;

[1941] a means for issuing an audio alert to notify the user of detected suspicious activity;

[1942] means for notifying a user terminal of detected suspicious behavior;

[1943] a means of notifying the user via push notification if suspicious behavior continues;

[1944] The system includes means for saving the detected video frames as a log file.

[1945] (Claim 2)

[1946] 10. The system of claim 1, further comprising means for notifying a user when a stranger has entered the system using facial recognition.

[1947] (Claim 3)

[1948] 10. The system according to claim 1, further comprising means for recording the detected image data and the analysis results in a database for use in subsequent analysis.

[1949] "Example 2: Combining Emotion Engines"

[1950] (Claim 1)

[1951] means for capturing image data using a camera and transmitting the image data to a server in real time;

[1952] A means for receiving video data and performing image recognition of people and animals in the video data;

[1953] A method for analyzing behavior based on image recognition results and detecting dangerous behavior;

[1954] means for issuing an audio alert to notify the user of the detected unsafe behavior;

[1955] means for notifying a user terminal of the detected risky behavior;

[1956] The system includes a means for recording the analytical data and emotional data in a database for later analysis.

[1957] (Claim 2)

[1958] 10. The system of claim 1, further comprising means for notifying a user when a stranger has entered the system using facial recognition.

[1959] (Claim 3)

[1960] 10. The system of claim 1, further comprising means for analyzing a user's sentiment and adjusting the urgency of the notification.

[1961] "Application example 2 when combining emotion engines"

[1962] (Claim 1)

[1963] means for capturing image data using a camera and transmitting the image data to a server in real time;

[1964] A means for receiving video data and performing image recognition of people and animals in the video data;

[1965] A method for analyzing behavior based on image recognition results and detecting dangerous behavior;

[1966] means for issuing an audio alert to notify the user of the detected unsafe behavior;

[1967] means for notifying a user terminal of the detected risky behavior;

[1968] and means for adjusting notification or alert content based on emotion recognition results.

[1969] (Claim 2)

[1970] 10. The system of claim 1, further comprising means for notifying a user when a stranger has entered the system using facial recognition.

[1971] (Claim 3)

[1972] 10. The system according to claim 1, further comprising means for recording the detected image data and the analysis results in a database for use in subsequent analysis. [Explanation of symbols]

[1973] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for capturing image data using a camera and transmitting the image data to a server in real time; A means for receiving video data and performing image recognition of people and animals in the video data; A method for analyzing behavior based on image recognition results and detecting dangerous behavior; means for issuing an audio alert to notify the user of the detected unsafe behavior; and means for notifying a user terminal of the detected risky behavior.

2. The system of claim 1 further comprising means for notifying a user when a stranger has entered the system using facial recognition.

3. 2. The system according to claim 1, further comprising means for recording the detected image data and the analysis results in a database for use in subsequent analysis.

Images