System

Abstract

A system is provided.SOLUTION: A monitoring system includes video acquisition means, video analysis means, abnormality detection means, notification means, automatic handling means, and recording means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The challenge is to ensure the safety of socially vulnerable people, such as the elderly and babies, by quickly and reliably detecting abnormalities (e.g., falls and abnormal behavior) that they may encounter in their daily lives. Existing fall detection systems and abnormal behavior detection systems require users to wear specific devices, which is inconvenient and expensive. Furthermore, even if these systems detect an abnormality, they often lack the functionality to immediately take appropriate action. The present invention aims to solve these challenges. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a surveillance system that includes a video acquisition means, a video analysis means, an anomaly detection means, a notification means, an automatic response means, and a recording means. The video acquisition means acquires video in real time, and the video analysis means analyzes the acquired video using a deep learning algorithm. The anomaly detection means detects a subject's fall or abnormal behavior, and the notification means notifies the user of the abnormality via push notification, SMS, voice alarm, or other means. Furthermore, the automatic response means automatically contacts designated emergency contacts when an abnormality is detected. The recording means saves all abnormal events and notification history as a log, which can be reviewed later. This system allows users to quickly and reliably detect a variety of abnormalities and take immediate appropriate action without having to wear a specific device.

[0006] "Video acquisition means" refers to a device that captures video data in real time using a camera, sensor, etc.

[0007] "Video analysis means" refers to a device or software that analyzes acquired video data using a deep learning algorithm or the like to recognize objects and actions.

[0008] The "anomaly detection means" is a device or software that detects specific anomalies (e.g., falls or abnormal behavior) from the data analyzed by the video analysis means.

[0009] The "notification means" refers to a communication means for notifying the user of an abnormality detected by the abnormality detection means, and includes push notification, SMS, voice alarm, etc.

[0010] An "automatic response means" is a device or software that automatically contacts emergency contacts based on a pre-set protocol when an abnormality is detected.

[0011] The "recording means" is a device or software that stores the history of abnormality detection and notification as a log so that it can be checked later.

[0012] A "surveillance system" is a system that consists of video acquisition means, video analysis means, abnormality detection means, notification means, automatic response means, and recording means, and that detects abnormalities in socially vulnerable people such as the elderly and babies and takes appropriate action. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and is configured as follows:

[0035] System configuration

[0036] This monitoring system includes video acquisition means, video analysis means, anomaly detection means, notification means, automatic response means, and recording means, enabling a wide range of anomaly detection and rapid response.

[0037] Program processing

[0038] First, the user installs the camera (video acquisition means) in an appropriate location and configures the system. The camera continues to capture video in real time and transmits it to the terminal. The terminal then streams this video data to the server.

[0039] On the server, the video analytics uses deep learning algorithms to analyze the video data. Through this analysis, the server monitors the subject's movement patterns in real time, detecting, for example, an elderly person falling or an abnormal baby behavior (such as excessively violent movements or abnormally quiet and motionless behavior).

[0040] If an anomaly is detected, the server's anomaly detection means will immediately identify it and send an alert to the user via the notification means, which can be push notification, SMS, or voice alarm, depending on the user's settings.

[0041] Furthermore, based on pre-set conditions, the server's automatic response means will execute appropriate actions in the event of an abnormality (for example, contacting emergency contacts, requesting an ambulance, etc.) At this time, all abnormal events and response actions are saved as logs by the recording means and can be checked later.

[0042] Specific examples

[0043] Example 1: Detecting falls in elderly people

[0044] User: Install the camera in the living room and configure fall detection using the app.

[0045] Terminal: The camera captures video in real time 24 hours a day and sends it to the server.

[0046] Server: The video analysis means analyzes the movements of the elderly person in the video, and when a fall is detected, the anomaly detection means immediately identifies it.

[0047] Server: When a fall is detected, the notification means sends a push notification to the user's smartphone and also sends an SMS to emergency contacts.

[0048] User: Checks the notification, checks the video in the app, and confirms the elderly person's condition. Based on that information, the user can take necessary action (for example, contact the elderly person directly, call an ambulance, etc.).

[0049] Example 2: Detecting abnormal baby behavior

[0050] User: Install a camera in the baby's room and set up baby behavior monitoring.

[0051] Device: The camera captures the baby's movements in real time and sends the video to the server.

[0052] Server: The server's video analysis means analyzes the baby's movements and detects abnormal behavior (for example, unusually violent movements or abnormal quietness).

[0053] Server: When an abnormality is detected, the notification means sends an alert to the user.

[0054] User: Receives notification, checks camera footage in the app, monitors the baby's condition, and intervenes directly to respond to any abnormalities if necessary.

[0055] This method allows users to monitor the condition of elderly people or babies in real time, reliably detect abnormalities, and quickly take appropriate action.

[0056] The processing flow will be explained below.

[0057] Step 1:

[0058] User: Installs cameras in the rooms or areas to be monitored. Connects the cameras to the network and configures the system using a dedicated app or web interface. Configuration includes information about the people to be monitored (elderly, babies, etc.) and anomaly detection requirements (fall detection, abnormal behavior detection, etc.).

[0059] Step 2:

[0060] Terminal: Turns on the camera and starts capturing video in real time, appropriately encodes the captured video data, and streams it over the network to the server.

[0061] Step 3:

[0062] Server: Stores the received video data and prepares it for collaboration with deep learning algorithms. It processes the data frame by frame for video analysis.

[0063] Step 4:

[0064] Server: Uses video analytics to analyze the movements of subjects in the video. This analysis includes estimating a person's pose and classifying their movements. For example, it can identify whether an elderly person is sitting, standing, or walking.

[0065] Step 5:

[0066] Server: The anomaly detection means detects anomalies (e.g., falls, abnormal stillness, abnormal movements) based on data from the video analysis means. It determines the action to be taken based on the type and severity of the detected anomaly.

[0067] Step 6:

[0068] Server: When an anomaly is detected, the server alerts the user using notification methods, such as push notification, SMS, email, and audio alarm.

[0069] Step 7:

[0070] User: Receives notification and checks the video footage via a dedicated app or web interface, checks the details of the anomaly, and takes emergency action if necessary.

[0071] Step 8:

[0072] Server: Furthermore, the automatic response means automatically executes actions based on pre-set conditions (e.g., notifying emergency contacts when a fall is detected), such as calling an ambulance or contacting family members or caregivers.

[0073] Step 9:

[0074] Server: All abnormal events and response actions are saved in a log file by a recording method, including the date and time the abnormality occurred, the type of abnormality detected, and the response taken.

[0075] Step 10:

[0076] Users: Review past event logs via the app or web interface, analyze anomaly frequency and trends, and fine-tune settings as needed to optimize system accuracy and effectiveness.

[0077] By performing the operations in each processing step, the present invention can effectively detect abnormalities in elderly people and babies and take prompt and appropriate measures.

[0078] Example 1

[0079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0080] In today's society, vulnerable members of society, such as the elderly and babies, often spend their time in vulnerable situations, and a rapid response is required, especially when accidents or abnormal behavior occur at home. However, conventional surveillance systems often lack the accuracy of anomaly detection and the ability to respond in real time, preventing appropriate countermeasures from being implemented. The present invention aims to solve these problems and provide an advanced surveillance system to ensure the safety of vulnerable members of society.

[0081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0082] In this invention, the server includes a video analysis means, anomaly detection means, a notification means, an automatic response means, a recording means, a means for streaming video data to the server via a terminal, and a means for detecting anomalies based on the video data analyzed using a deep learning algorithm, thereby enabling highly accurate real-time anomaly detection and rapid response.

[0083] "Video acquisition means" refers to means for capturing video in real time using a device such as a camera.

[0084] The "video analysis means" is a means for analyzing acquired video data and recognizing specific patterns or abnormalities.

[0085] The "abnormality detection means" is a means for detecting specific abnormalities (e.g., falls or abnormal behavior) based on the analysis results of the video analysis means.

[0086] "Notification means" refers to the means of notifying the user when an abnormality is detected. Notification methods include push notification, SMS, and voice alarm.

[0087] An "automatic response means" is a means for automatically taking appropriate action in response to a detected abnormality based on preset conditions.

[0088] "Recording means" is a means for recording all abnormal events and corresponding actions and saving them for later review.

[0089] "Means for streaming video data to a server through a terminal" refers to means for a terminal to continuously transmit captured video data to a server via a network.

[0090] "Means for detecting anomalies based on video data analyzed using a deep learning algorithm" refers to a means for analyzing video data in real time using a deep learning model to detect anomalies with high accuracy.

[0091] This invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and is operated by combining the following hardware and software: Specifically, it utilizes a camera (video acquisition means), devices such as smartphones and tablets, and a deep learning algorithm installed on a server.

[0092] The user simply installs the camera in an appropriate location and configures the system using a dedicated app. For example, the user installs the camera in the living room and configures it for fall detection. This configuration is performed through the app by selecting whether to enable fall detection and the notification method (push notification, SMS, voice alarm, etc.).

[0093] The camera captures video in real time 24 hours a day, and the video data is sent to a terminal via communication methods such as Wi-Fi or Bluetooth. The terminal compresses the received video data and streams it to a server.

[0094] The server uses deep learning algorithms to analyze video data in real time. For example, frameworks such as TensorFlow and PyTorch are used to detect falls by elderly people or abnormal behavior in babies. The server's anomaly detection means immediately identifies these anomalies and sends an alert to the user via the notification means.

[0095] If an abnormality is detected, the server will take appropriate action through automated response measures based on pre-defined conditions, such as automatically calling emergency contacts or requesting an ambulance, enabling a prompt and appropriate response.

[0096] In addition, the server records all abnormal events and corresponding actions and saves them as logs for later review, allowing users to use the system to review past abnormal events and their responses at any time.

[0097] Specific examples

[0098] Example 1: Detecting falls in elderly people

[0099] User: Install the camera in the living room and configure fall detection settings in the app. For example, set "Fall detection on," "Notification method: push notification," and "Emergency contact: family member's phone number."

[0100] Camera: Captures real-time footage of your living room 24 hours a day.

[0101] Terminal: Receives camera footage, compresses it and streams it to the server.

[0102] Server: Analyzes video using a TensorFlow model to detect falls by elderly people.

[0103] Server: When a fall is detected, a push notification is sent to the user's smartphone using the notification method, and if necessary, a call is made to an emergency contact.

[0104] Example 2: Detecting abnormal baby behavior

[0105] User: Install a camera in the baby's room and configure the baby's behavior monitoring settings, such as "Abnormal behavior detection on," "Notification method: SMS," and "Emergency contact: parent's phone number."

[0106] Camera: Captures real-time footage of the baby's room 24 hours a day.

[0107] Terminal: Receives camera footage, compresses it and streams it to the server.

[0108] Server: Analyzes the video using a PyTorch model and detects abnormal baby behavior.

[0109] Server: When abnormal behavior is detected, an SMS is sent to the user using a notification method, and in the event of an emergency, an emergency contact is called.

[0110] Prompt Sentence Examples

[0111] Prompt to explain what the system should do if an elderly person falls:

[0112] "The user uses a camera installed in their living room and configures fall detection using a dedicated app. The camera captures video 24 hours a day and sends it to a device via Wi-Fi. The device compresses the video and streams it to a server. The server analyzes the video using a TensorFlow model and detects falls. If an abnormality is identified, the server immediately sends a push notification to the user's smartphone and, if necessary, calls an emergency contact. All events are recorded in a detailed log."

[0113] Prompt sentence that describes the system's behavior when detecting abnormal baby behavior:

[0114] "The user uses a camera installed in the baby's room and configures abnormal behavior detection using a dedicated app. The camera captures video 24 hours a day and transmits it via Wi-Fi to a device. The device compresses the video and streams it to a server. The server analyzes the video using a PyTorch model and detects abnormal behavior. If an abnormality is identified, the server immediately sends an SMS to the user's smartphone and, if necessary, calls an emergency contact. All events are recorded in a detailed log."

[0115] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0116] Step 1:

[0117] The user installs the camera in an appropriate location and configures the system using a dedicated app. Through the app, the user enables fall detection and abnormal behavior monitoring and sets the notification method. The input is the user's configuration information (e.g., fall detection on, notification method: push notification, emergency contact: family member's phone number), which is then output as configuration data to be reflected in the system.

[0118] Step 2:

[0119] The camera captures video in real time 24 hours a day and transmits the video data to a device using a communication method such as Wi-Fi or Bluetooth. The input is video data obtained from the physical environment, and the output is uncompressed video data sent to the device. The specific operation involves the camera periodically generating video frames and transmitting them to the device.

[0120] Step 3:

[0121] The video data received by the device is compressed for efficient transmission within a certain bandwidth and streamed to the server. The input is video data sent from the camera, and the output is compressed video data sent to the server. Specifically, the device compresses the video frames using a compression algorithm and sends them to the server as a stream.

[0122] Step 4:

[0123] The server analyzes the received video data using a deep learning algorithm. The input is the compressed video data sent from the device, and the output is the analysis results (e.g., whether or not a fall has occurred, or abnormal behavior has been detected). Specifically, the server uses libraries such as TensorFlow and PyTorch to analyze specific movement patterns within the video data.

[0124] Step 5:

[0125] The server's anomaly detection means identifies anomalies based on the results of deep learning analysis. The input is the analysis result data from the video analysis means, and the output is the anomaly detection result (e.g., an elderly person falling, or a baby behaving abnormally). Specifically, the server centralizes the analysis results and narrows down the data that is judged to be abnormal.

[0126] Step 6:

[0127] When the server detects an abnormality, it notifies the user through the notification means. The input is the abnormality detection result from the abnormality detection means, and the output is notification data (push notification, SMS, voice alarm). In concrete terms, the server sends a push notification to the user's smartphone based on the abnormality detection result, and if necessary, sends an SMS to emergency contacts.

[0128] Step 7:

[0129] The server's automatic response means automatically executes an appropriate response to the anomaly. The inputs are the anomaly detection results and pre-set response conditions, and the output is specific response actions (contacting emergency contacts, dispatching an ambulance). In concrete terms, the server determines that it is an emergency and starts a script to execute the response.

[0130] Step 8:

[0131] The server records all abnormal events and corresponding actions, saving them as logs for later review. The inputs are the anomaly detection results and corresponding actions, and the output is the recorded data. Specifically, the server records the timestamp, location information, and details of the response to the abnormal event in detail, saving them in a database.

[0132] (Application example 1)

[0133] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] To ensure the safety of vulnerable members of society, such as the elderly and babies, there is a need for real-time behavior monitoring, anomaly detection, and rapid response. However, conventional monitoring systems often have low anomaly detection accuracy, resulting in delayed notification and response. In addition, there are limited means of notifying users, making it difficult to respond quickly in emergencies.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0136] In this invention, the server includes means for analyzing video data using a deep learning algorithm, means for notifying users using push notifications, short message services, and audio alarms, and means for automatically contacting emergency contacts, enabling highly accurate detection of abnormalities and prompt notification and response.

[0137] "Video acquisition means" refers to means for acquiring video data in real time using a camera or video device.

[0138] "Video analysis means" refers to algorithms or software that analyze acquired video data and recognize specific actions or abnormalities.

[0139] An "abnormality detection means" is a mechanism that detects abnormal behavior or conditions based on the results of video analysis.

[0140] "Notification means" means means for sending an alert to a user when an abnormality is detected, including push notifications, short message services, and audio alarms.

[0141] "Automatic response means" refers to a means for automatically taking appropriate action in accordance with a set protocol when an abnormality is detected.

[0142] "Recording means" refers to a means for saving data related to anomaly detection and responses so that it can be checked later.

[0143] A "deep learning algorithm" is an algorithm that uses deep learning technology to analyze video data and recognize specific actions and abnormalities with high accuracy.

[0144] "Push notification" is a method of automatically sending information from a server to a user's device.

[0145] "Short Message Service" is a service that sends short text messages to mobile phones and smartphones.

[0146] An "audio alarm" is a means of notifying the user of an emergency using audio.

[0147] "Means for automatically contacting emergency contacts" refers to a system for automatically contacting the emergency contacts set up when an abnormality is detected.

[0148] The present invention is a system for ensuring the safety of socially vulnerable people such as the elderly and babies, and includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and different software and hardware configurations for linking these means.

[0149] System configuration

[0150] 1. Video acquisition method

[0151] The server uses a dedicated camera to capture the movements of elderly people and babies in real time, and the video data is encoded on the spot and sent to the server in streaming format.

[0152] 2. Video analysis methods

[0153] The server uses deep learning algorithms (such as TensorFlow) to analyze the video data. Specifically, it analyzes the received video in real time, monitoring the subject's movements and behavior, and uses models designed to identify specific movement patterns, such as falls or abnormal movements.

[0154] 3. Anomaly detection methods

[0155] The server detects abnormalities from the results of video analysis. The abnormality detection means detects deviations from the set movement patterns and judges them as abnormal. For example, an abnormality is detected when an elderly person falls or a baby is unusually quiet.

[0156] 4. Means of notification

[0157] The server will immediately notify the user when an abnormality is detected. Notification methods include push notifications, short message service, and audio alarms. The content of the notification is customized depending on the type and urgency of the abnormality.

[0158] 5. Automated Response Methods

[0159] If an abnormality is detected, the server automatically takes appropriate action based on pre-set conditions, such as contacting emergency contacts or calling an ambulance, enabling a prompt and appropriate response.

[0160] 6. Recording Method

[0161] The server logs all abnormal events and their responses. The log data is stored in a database service such as Firebase, allowing it to be viewed and analyzed at a later date.

[0162] Specific examples

[0163] Example 1: Detecting falls in elderly people

[0164] Users simply install the camera in their living room and set the fall detection mode in the app. The camera captures video in real time 24 hours a day and sends it to the server. The server then uses a deep learning algorithm to analyze the video and immediately alerts the user via push notification and short message service if a fall is detected. Emergency contacts are also notified at the same time.

[0165] Example 2: Detecting abnormal baby behavior

[0166] The user installs a camera in the baby's room and sets it to baby behavior monitoring mode. The camera captures the baby's movements in real time and sends them to the server. The server's video analysis means analyzes the baby's movements and, if it detects abnormal quietness or extreme movements, it sends a notification to the user's smartphone along with an audio alarm. Emergency response is also automatically implemented if necessary.

[0167] Prompt Sentence Examples

[0168] text

[0169] "When an elderly person falls, generate a code that detects the abnormality and immediately notifies the user. Use push notifications and short message services as notification methods, and use Firebase to record the data."

[0170] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0171] Step 1:

[0172] The user installs the camera in a suitable location and configures the system through the application. The camera captures video in real time and acquires video data. The input is the video data from the camera, which is then sent to the server in streaming format.

[0173] Step 2:

[0174] The server encodes the received video data and begins analyzing it using a deep learning algorithm. Specifically, the video data is preprocessed (e.g., frame resizing, color correction) and input into a deep learning model (using TensorFlow). The output returns a list of movement patterns of the elderly and babies.

[0175] Step 3:

[0176] The server uses an anomaly detection means to detect abnormalities based on the analysis of the movement patterns. For example, it detects falling or the baby standing still unnaturally. The input at this time is the analyzed movement pattern, and the output is a flag indicating whether an abnormality has been detected.

[0177] Step 4:

[0178] When an anomaly is detected, the server immediately notifies the user using the notification methods available: push notification, short message service (using Twilio API), and audio alarm. In this process, the anomaly detection flag is input and the notification message is output.

[0179] Step 5:

[0180] The server then uses automated response methods based on pre-configured settings to take appropriate action, such as automatically contacting emergency contacts and requesting an ambulance. The inputs are a flag indicating an anomaly has been detected and a pre-configured response protocol, and the output is a list of the response actions taken.

[0181] Step 6:

[0182] The server uses a recording method to store all of these abnormal events and their corresponding actions. The log data is sent to and stored in a cloud database such as Firebase. The input is detailed data on the abnormal event that occurred and the corresponding action, and the output is the log record stored in the database.

[0183] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0184] This invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and by combining it with an emotion engine in particular, it enables comprehensive surveillance including the psychological state of the target person and quick response. Specifically, the system configuration includes a video acquisition means, a video analysis means, anomaly detection means, a notification means, an automatic response means, a recording means, and an emotion engine.

[0185] System configuration

[0186] The monitoring system consists of the following elements:

[0187] Video acquisition means: A device that uses cameras and sensors to capture video data in real time.

[0188] Video analysis means: A device or software that uses deep learning algorithms to analyze the movements and facial expressions of subjects in captured video.

[0189] Anomaly detection means: A device or software that detects falls, abnormal behavior, and even emotional changes in a subject based on data from video analysis means.

[0190] Notification method: The method to notify the user of abnormalities through push notification, SMS, voice alarm, etc.

[0191] Automated response measures: Devices or software that automatically send notifications to emergency contacts when an abnormality is detected.

[0192] Recording means: A device or software that stores all abnormal events and notification history as a log.

[0193] Emotion engine: A device or software that contains algorithms to analyze a subject's facial expressions and voice and recognize their emotions.

[0194] Program processing

[0195] First, the user installs the camera (video acquisition means) in an appropriate location and configures the system. The camera continues to capture video in real time and transmits it to the terminal. The terminal then processes the video data to stream it to the server.

[0196] The server, which integrates the video analysis and emotion engine, analyzes the video data using a deep learning algorithm. This analysis allows the server to monitor the subject's emotions in real time from their movement patterns and facial expressions. For example, it can detect an elderly person's fall, a baby's abnormal behavior, or even changes in the subject's emotions such as sadness or stress.

[0197] If an anomaly is detected, the server's anomaly detection means will immediately identify it and send an alert to the user via the notification means. Notification methods include push notifications, SMS, and voice alarms, depending on the user's settings. Changes in emotions recognized by the emotion engine will also be included in the notification.

[0198] Furthermore, based on pre-defined conditions, the server's automated response mechanism will take appropriate action in the event of an abnormality, such as contacting emergency contacts, requesting an ambulance, or notifying mental health support contacts if necessary. All abnormal events and response actions are saved as logs by the recording mechanism and can be reviewed later.

[0199] Specific examples

[0200] Example 1: Detecting falls and emotional changes in the elderly

[0201] User: Installs a camera in the living room and configures fall detection and emotion recognition.

[0202] Terminal: The camera captures video in real time 24 hours a day and sends it to the server.

[0203] Server: The video analysis method and emotion engine analyze the movements and facial expressions of the elderly person in the video and detect falls and sad expressions.

[0204] Server: When a fall or grief is detected, the notification mechanism sends an alert to the user's smartphone and also sends an SMS to emergency contacts.

[0205] User: Checks the notification, checks the video in the app, and confirms the elderly person's condition. Based on that information, the user takes necessary action.

[0206] Example 2: Baby abnormal behavior and emotion recognition detection

[0207] User: Install a camera in the baby's room and set up baby behavior monitoring and emotion recognition.

[0208] Device: The camera captures the baby's movements in real time and sends the video to the server.

[0209] Server: The server's video analysis means and emotion engine analyze the baby's movements and facial expressions to detect abnormal behavior or signs of stress.

[0210] Server: When an abnormality is detected, the notification means sends an alert to the user, and if a response is required, the automatic response means contacts the designated emergency contact.

[0211] User: Receives notifications, checks the video in the app, understands the baby's condition, and intervenes directly if necessary to respond to any abnormalities.

[0212] As a result, the present invention can monitor and detect not only physical abnormalities in the elderly and babies, but also mental abnormalities and stress, enabling quick and effective responses.

[0213] The processing flow will be explained below.

[0214] Step 1:

[0215] User: Installs the camera in the room or area to be monitored and turns it on. Configures the system using a dedicated app or web interface. The configuration includes information about the person to be monitored (e.g., elderly, baby) and anomaly detection requirements (e.g., fall detection, abnormal behavior detection).

[0216] Step 2:

[0217] Terminal: Turns on the camera and starts capturing video in real time, encodes the captured video data appropriately, and sends it to the server via the network.

[0218] Step 3:

[0219] Server: Stores the received video data and prepares it for integration with deep learning algorithms. It processes the video data frame by frame using video analysis methods.

[0220] Step 4:

[0221] Server: Analyzes the subject's movement patterns and facial expressions using video analysis. This analysis includes estimating the person's posture, classifying their movements, and analyzing their facial expressions. For example, it can identify whether an elderly person is sitting, standing, or walking, and detect emotions from their facial expressions.

[0222] Step 5:

[0223] Server: Uses an emotion engine to recognize the subject's emotional state (e.g., joy, sadness, anger, stress) based on data obtained from video analysis means.

[0224] Step 6:

[0225] Server: The anomaly detection means detects anomalies (e.g., falls, abnormal stillness, abnormal movements, sudden changes in emotions) from the analysis results and determines how to respond based on the type and severity of the detected anomaly.

[0226] Step 7:

[0227] Server: When an anomaly is detected, the server sends an alert to the user using notification methods such as push notification, SMS, and voice alarm. The notification also includes any changes in emotions recognized by the emotion engine.

[0228] Step 8:

[0229] User: Receives notification and checks the video via a dedicated app or web interface. Checks the details of the abnormality and takes emergency action if necessary.

[0230] Step 9:

[0231] Server: When an anomaly is detected, automated response measures automatically take action based on pre-defined conditions, such as contacting emergency contacts, calling an ambulance, or notifying mental health support contacts.

[0232] Step 10:

[0233] Server: All abnormal events and response actions are saved in a log file using a recording method, including the date and time the abnormality occurred, the type of abnormality detected, and the response taken.

[0234] Step 11:

[0235] Users: Review past event logs via the app or web interface, analyze anomaly frequency and trends, and fine-tune settings as needed to optimize system accuracy and effectiveness.

[0236] With this detailed processing flow, the present invention can effectively detect abnormalities in elderly people and babies, and can take prompt and appropriate measures, including emotional changes.

[0237] Example 2

[0238] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0239] When monitoring vulnerable people such as the elderly and babies, comprehensive monitoring is required, including not only physical abnormalities but also psychological abnormalities, making it difficult to respond quickly and appropriately. Furthermore, there is a lack of technology that can detect abnormalities and recognize emotional changes in real time and provide effective notifications and responses. These issues must be resolved.

[0240] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0241] In this invention, the server includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and an emotion recognition means, which makes it possible to monitor the movements and emotional changes of socially vulnerable people such as the elderly and babies in real time, and to notify and respond quickly and effectively when an abnormality is detected.

[0242] "Video acquisition means" refers to a device that captures video data in real time using a camera or sensor.

[0243] "Video analysis means" refers to a device or software that uses a deep learning algorithm to analyze the movements and facial expressions of subjects in captured video.

[0244] An "abnormality detection means" is a device or software that detects a subject's falls, abnormal behavior, or even changes in emotion based on data from the video analysis means.

[0245] "Notification means" refers to the means of notifying the user of an abnormality via push notification, SMS, voice alarm, etc.

[0246] An "automatic response means" is a device or software that automatically sends a notification to an emergency contact when an abnormality is detected.

[0247] The "recording means" is a device or software that stores all abnormal events and notification history as a log.

[0248] An "emotion recognition means" is a device or software that includes an algorithm for analyzing a subject's facial expressions and voice and recognizing their emotions.

[0249] This invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and by combining it with an emotion recognition function, it enables comprehensive monitoring including the psychological state of the target person and quick response. Specifically, the system configuration includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and an emotion recognition means.

[0250] The user installs a camera (video acquisition means) in the area to be monitored (such as the living room or baby room). The camera captures video in real time and sends it to a device. The device then streams the video data to a server. The server uses video analysis means and emotion recognition means to analyze the video data with a deep learning algorithm and monitor the subject's movements and facial expressions in real time.

[0251] The server's video analysis means uses image recognition technologies such as Convolutional Neural Networks (CNN) to analyze the subject's movement patterns and facial expressions for each video frame. The emotion recognition means also combines voice and facial expression analysis to evaluate changes in emotion. For example, it can detect falls in elderly people, abnormal behavior in babies, and even changes in the subject's emotions such as sadness or stress.

[0252] If an abnormality is detected, the server's anomaly detection means immediately identifies it and sends an alert to the user via the notification means. Notification methods include push notifications, SMS, and voice alarms, depending on the user's settings. For example, a push notification saying "An elderly person has fallen" can be sent to a smartphone. Furthermore, any changes in emotions recognized by the emotion recognition means are also included in the notification.

[0253] Based on pre-set conditions, the server's automatic response means will take appropriate action in the event of an abnormality. For example, it can automatically contact emergency contacts, request an ambulance, or notify mental health support contacts if necessary. All abnormal events and response actions are saved as logs by the recording means, and can be viewed by the user later.

[0254] Specific examples

[0255] Example 1: Detecting falls and emotional changes in the elderly

[0256] User: Installs a camera in the living room and configures fall detection and emotion recognition.

[0257] Terminal: The camera captures video in real time 24 hours a day and sends it to the server.

[0258] Server: Video analysis and emotion recognition means analyze the movements and facial expressions of elderly people in the video and detect falls and sad expressions.

[0259] Server: When a fall or grief is detected, the notification mechanism sends an alert to the user's smartphone and also sends an SMS to emergency contacts.

[0260] User: Checks the notification, checks the video in the app, and confirms the elderly person's condition. Based on that information, the user takes necessary action.

[0261] Examples of prompt statements

[0262] (example)

[0263] Real-time alert prompt for elderly fall:

[0264] Video acquisition method: "A camera installed in the living room captures the elderly person's movements."

[0265] Video analysis method: "Analyzing video data in real time using deep learning algorithms"

[0266] Anomaly detection method: "Detecting falls in elderly people"

[0267] Notification method: "Send an alert notification to your smartphone"

[0268] Example 2: Baby abnormal behavior and emotion recognition detection

[0269] User: Install a camera in the baby's room and set up baby behavior monitoring and emotion recognition.

[0270] Device: The camera captures the baby's movements in real time and sends the video to the server.

[0271] Server: Video analysis and emotion recognition tools analyze the baby's movements and facial expressions to detect abnormal behavior or signs of stress.

[0272] Server: When an abnormality is detected, the notification means sends an alert to the user, and if a response is required, the automatic response means contacts the designated emergency contact.

[0273] User: Receives notifications, checks the video in the app, understands the baby's condition, and intervenes directly if necessary to respond to any abnormalities.

[0274] As a result, the present invention can monitor and detect not only physical abnormalities in the elderly and babies, but also mental abnormalities and stress, enabling quick and effective responses.

[0275] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0276] Step 1:

[0277] The user installs a camera (video capture means) in the monitoring area (such as a living room or baby room) and configures the system. Input information includes the camera's installation location and information about the monitored object (e.g., information about an elderly person or a baby). This configuration prepares the camera to capture the precise location and movement of the monitored object in real time.

[0278] input:

[0279] Camera installation position

[0280] Monitored information

[0281] output:

[0282] System initialization complete

[0283] Specific behavior:

[0284] Users simply sync the installed camera with the smartphone app and adjust the camera's position on the app's initial setup screen. Once setup is complete, the camera will enter surveillance mode.

[0285] Step 2:

[0286] The camera captures video in real time and sends it to the device. The camera takes multiple frames per second and sends them to the device.

[0287] input:

[0288] Real-time video data (multiple frames)

[0289] output:

[0290] Video data sent to the device

[0291] Specific behavior:

[0292] The camera operates in 24-hour surveillance mode, capturing video at 30 frames per second and transmitting it to the device.

[0293] Step 3:

[0294] The device compresses the received video data and streams it efficiently to the server using video compression algorithms such as H.264 and H.265.

[0295] input:

[0296] Real-time video data from cameras

[0297] output:

[0298] Compressed video data

[0299] Specific behavior:

[0300] The terminal compresses the video data and streams the data over the Internet to a server.

[0301] Step 4:

[0302] The server analyzes the received video data using video analysis and emotion recognition methods. Based on a deep learning algorithm, it analyzes the movements and facial expressions of the subjects in the video and evaluates changes in their emotions.

[0303] input:

[0304] Compressed video data

[0305] output:

[0306] Subject's motion analysis and emotion recognition results

[0307] Specific behavior:

[0308] The server uses Convolutional Neural Networks (CNN) to analyze movement patterns and facial expressions for each video frame, and also evaluates emotional changes by combining voice and facial analysis.

[0309] Step 5:

[0310] The server detects abnormalities based on the results of video analysis. The abnormality detection means identifies falls and abnormal behavior of the subject in real time.

[0311] input:

[0312] Motion analysis results

[0313] Emotion recognition results

[0314] output:

[0315] Anomaly detection results

[0316] Specific behavior:

[0317] The server uses the acquired analytical data to detect falls by elderly people or abnormal movements by babies, and also determines abnormalities based on emotion recognition results.

[0318] Step 6:

[0319] When an abnormality is detected, the server sends an alert to the user using a notification method, such as push notification, SMS, or voice alarm.

[0320] input:

[0321] Anomaly detection results

[0322] output:

[0323] Alert Notifications

[0324] Specific behavior:

[0325] When an abnormality is detected, the server sends a push notification to the user's smartphone and an SMS to the user's configured contacts.

[0326] Step 7:

[0327] The server uses automated response mechanisms to take appropriate action depending on the situation, such as contacting emergency contacts or calling an ambulance based on pre-defined conditions.

[0328] input:

[0329] Anomaly detection results

[0330] Predefined conditions

[0331] output:

[0332] Auto-response actions taken

[0333] Specific behavior:

[0334] If the server detects an abnormality, it will automatically call an emergency contact and, if necessary, will also take action such as calling an ambulance.

[0335] Step 8:

[0336] The server stores all abnormal events and notification history as a log using a recording means, which allows users to check later.

[0337] input:

[0338] Auto-response actions taken

[0339] Notification history

[0340] output:

[0341] Abnormal event logs

[0342] Specific behavior:

[0343] The server stores all anomaly detection and response data along with timestamps in a database, allowing users to view past event history through the application.

[0344] (Application example 2)

[0345] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] Conventional monitoring systems are limited to detecting only physical abnormalities in subjects, making it difficult to monitor and respond to psychological changes or emotional abnormalities in real time. In addition, security guards and staff lack the means to quickly identify and respond to abnormalities on-site, creating a need for efficient monitoring, especially at night or in complex environments.

[0347] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0348] In this invention, the server includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and a display means via a head-mounted display. This allows not only physical abnormalities but also emotional changes such as fear and anger to be monitored in real time, enabling security guards and other staff to respond quickly and appropriately.

[0349] "Video acquisition means" refers to a device that captures video data in real time using a camera or sensor.

[0350] "Video analysis means" refers to a device or software that uses a deep learning algorithm to analyze the movements and facial expressions of subjects in captured video.

[0351] An "abnormality detection means" is a device or software that detects a subject's falls, abnormal behavior, or even changes in emotion based on data from the video analysis means.

[0352] "Notification means" refers to the means of notifying the user of an abnormality via push notification, SMS, voice alarm, etc.

[0353] An "automatic response means" is a device or software that automatically sends a notification to an emergency contact when an abnormality is detected.

[0354] The "recording means" is a device or software that stores all abnormal events and notification history as a log.

[0355] A "head-mounted display" is a device that a user wears on their head and can display information within their field of vision.

[0356] The invention takes the form of a real-time monitoring system via a head-mounted display for use by guards and security staff. The system includes the following elements:

[0357] 1. Video acquisition means: A device that uses cameras and sensors to capture video data in real time, allowing for constant monitoring of the subject and their surroundings.

[0358] 2. Video analysis means: A device or software that uses deep learning algorithms to analyze the behavior and emotional changes of subjects in captured video. This analysis can accurately detect abnormal behavior and emotional changes of subjects.

[0359] 3. Anomaly detection means: This is a device or software that uses data from video analysis means to detect falls, abnormal behavior, and emotional changes such as fear or anger in the subject. This allows for real-time monitoring of not only physical abnormalities but also psychological abnormalities.

[0360] 4. Notification methods: This is a method of notifying users of abnormalities via push notifications, SMS, voice alarms, etc. When an abnormality is detected, a notification is sent immediately to security guards and staff, enabling a prompt response.

[0361] 5. Automatic response measures: Devices or software that automatically send notifications to emergency contacts when an abnormality is detected, enabling rapid response in emergency situations.

[0362] 6. Recording means: A device or software that stores all abnormal events and notification history as a log. This allows past data to be reviewed later, which is useful for troubleshooting and considering countermeasures.

[0363] 7. Display via head-mounted display: A device worn by the user on the head that displays information within the field of view, allowing security guards to grasp upcoming situations in real time without using their hands.

[0364] Examples:

[0365] While security guards are patrolling commercial facilities at night, they acquire real-time video data through a head-mounted display and analyze the movements and facial expressions of the target. If abnormal behavior or changes in emotions such as fear or anger are detected, a notification means immediately sends an alert to the security guard. In addition, an automatic response means contacts emergency contacts, enabling a prompt response. A recording means saves all abnormal events as a log, allowing the situation to be reviewed later and countermeasures to be considered.

[0366] Example prompts to input to the generative AI model:

[0367] "Design an application for a head-mounted display that captures and analyzes video data in real time and sends a notification to security guards if any abnormal emotions are detected. As a specific use case, please describe a scenario in which a security guard patrolling a commercial facility at night detects a suspicious individual's facial expression of fear or anger and immediately sends a notification to the security company."

[0368] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0369] Step 1:

[0370] The user places the camera in the appropriate location.

[0371] Input: Camera location information

[0372] Output: Video data of the monitored area

[0373] How it works: A user installs a camera in a location within a commercial facility that needs to be monitored, turns it on, and the camera begins capturing video in real time.

[0374] Step 2:

[0375] The device acquires video data from the camera in real time.

[0376] Input: Video stream from camera

[0377] Output: Streaming video data

[0378] Specific operation: The terminal (e.g., the security guard's mobile device) connects to the camera and streams video data to the server.

[0379] Step 3:

[0380] The server prepares data for analyzing the video data.

[0381] Input: Streaming video data

[0382] Output: Video frames for analysis

[0383] Specific operation: The server divides the video data and prepares for batch processing on a frame-by-frame basis.

[0384] Step 4:

[0385] The server uses deep learning algorithms to analyze the video frames.

[0386] Input: Video frame for analysis

[0387] Output: behavior and emotion data

[0388] Specific operation: The server inputs video frames into a deep learning model to analyze the subject's movements and facial expressions.

[0389] Step 5:

[0390] The server detects an abnormality using an abnormality detection means.

[0391] Input: Motion and emotion data

[0392] Output: Anomaly detection event

[0393] Specific operation: The server evaluates the analysis results and detects abnormal behavior and emotional changes such as falls, fear, or anger.

[0394] Step 6:

[0395] The server notifies you of the abnormality.

[0396] Input: Anomaly detection event

[0397] Output: Alert notification

[0398] Specific operation: The server sends a push notification or SMS to the user's device to inform them that an abnormality has been detected.

[0399] Step 7:

[0400] The server will handle the issue automatically.

[0401] Input: Anomaly detection event

[0402] Output: Automatic response action (emergency contact, ambulance request, etc.)

[0403] Specific operation: The server automatically notifies pre-configured emergency contacts and arranges for emergency vehicles if necessary.

[0404] Step 8:

[0405] The server records the logs.

[0406] Input: Anomaly detection events and corresponding actions

[0407] Output: Log data

[0408] Specific operation: The server uses a recording method to store all abnormal events and corresponding actions as logs so that they can be checked later.

[0409] Step 9:

[0410] The user checks the information on the head-mounted display.

[0411] Input: Alert notification, video data

[0412] Output: Real-time monitoring information

[0413] Specific operation: The user wears a head-mounted display and has real-time monitoring information displayed within their field of vision, allowing them to quickly grasp the situation.

[0414] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0415] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0416] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0417] [Second embodiment]

[0418] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0419] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0420] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0421] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0422] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0423] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0424] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0425] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0426] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0427] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0428] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0429] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0430] The present invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and is configured as follows:

[0431] System configuration

[0432] This monitoring system includes video acquisition means, video analysis means, anomaly detection means, notification means, automatic response means, and recording means, enabling a wide range of anomaly detection and rapid response.

[0433] Program processing

[0434] First, the user installs the camera (video acquisition means) in an appropriate location and configures the system. The camera continues to capture video in real time and transmits it to the terminal. The terminal then streams this video data to the server.

[0435] On the server, the video analytics uses deep learning algorithms to analyze the video data. Through this analysis, the server monitors the subject's movement patterns in real time, detecting, for example, an elderly person falling or an abnormal baby behavior (such as excessively violent movements or abnormally quiet and motionless behavior).

[0436] If an anomaly is detected, the server's anomaly detection means will immediately identify it and send an alert to the user via the notification means, which can be push notification, SMS, or voice alarm, depending on the user's settings.

[0437] Furthermore, based on pre-set conditions, the server's automatic response means will execute appropriate actions in the event of an abnormality (for example, contacting emergency contacts, requesting an ambulance, etc.) At this time, all abnormal events and response actions are saved as logs by the recording means and can be checked later.

[0438] Specific examples

[0439] Example 1: Detecting falls in elderly people

[0440] User: Install the camera in the living room and configure fall detection using the app.

[0441] Terminal: The camera captures video in real time 24 hours a day and sends it to the server.

[0442] Server: The video analysis means analyzes the movements of the elderly person in the video, and when a fall is detected, the anomaly detection means immediately identifies it.

[0443] Server: When a fall is detected, the notification means sends a push notification to the user's smartphone and also sends an SMS to emergency contacts.

[0444] User: Checks the notification, checks the video in the app, and confirms the elderly person's condition. Based on that information, the user can take necessary action (for example, contact the elderly person directly, call an ambulance, etc.).

[0445] Example 2: Detecting abnormal baby behavior

[0446] User: Install a camera in the baby's room and set up baby behavior monitoring.

[0447] Device: The camera captures the baby's movements in real time and sends the video to the server.

[0448] Server: The server's video analysis means analyzes the baby's movements and detects abnormal behavior (for example, unusually violent movements or abnormal quietness).

[0449] Server: When an abnormality is detected, the notification means sends an alert to the user.

[0450] User: Receives notification, checks camera footage in the app, monitors the baby's condition, and intervenes directly to respond to any abnormalities if necessary.

[0451] This method allows users to monitor the condition of elderly people or babies in real time, reliably detect abnormalities, and quickly take appropriate action.

[0452] The processing flow will be explained below.

[0453] Step 1:

[0454] User: Installs cameras in the rooms or areas to be monitored. Connects the cameras to the network and configures the system using a dedicated app or web interface. Configuration includes information about the people to be monitored (elderly, babies, etc.) and anomaly detection requirements (fall detection, abnormal behavior detection, etc.).

[0455] Step 2:

[0456] Terminal: Turns on the camera and starts capturing video in real time, appropriately encodes the captured video data, and streams it over the network to the server.

[0457] Step 3:

[0458] Server: Stores the received video data and prepares it for collaboration with deep learning algorithms. It processes the data frame by frame for video analysis.

[0459] Step 4:

[0460] Server: Uses video analytics to analyze the movements of subjects in the video. This analysis includes estimating a person's pose and classifying their movements. For example, it can identify whether an elderly person is sitting, standing, or walking.

[0461] Step 5:

[0462] Server: The anomaly detection means detects anomalies (e.g., falls, abnormal stillness, abnormal movements) based on data from the video analysis means. It determines the action to be taken based on the type and severity of the detected anomaly.

[0463] Step 6:

[0464] Server: When an anomaly is detected, the server alerts the user using notification methods, such as push notification, SMS, email, and audio alarm.

[0465] Step 7:

[0466] User: Receives notification and checks the video footage via a dedicated app or web interface, checks the details of the anomaly, and takes emergency action if necessary.

[0467] Step 8:

[0468] Server: Furthermore, the automatic response means automatically executes actions based on pre-set conditions (e.g., notifying emergency contacts when a fall is detected), such as calling an ambulance or contacting family members or caregivers.

[0469] Step 9:

[0470] Server: All abnormal events and response actions are saved in a log file by a recording method, including the date and time the abnormality occurred, the type of abnormality detected, and the response taken.

[0471] Step 10:

[0472] Users: Review past event logs via the app or web interface, analyze anomaly frequency and trends, and fine-tune settings as needed to optimize system accuracy and effectiveness.

[0473] By performing the operations in each processing step, the present invention can effectively detect abnormalities in elderly people and babies and take prompt and appropriate measures.

[0474] Example 1

[0475] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0476] In today's society, vulnerable members of society, such as the elderly and babies, often spend their time in vulnerable situations, and a rapid response is required, especially when accidents or abnormal behavior occur at home. However, conventional surveillance systems often lack the accuracy of anomaly detection and the ability to respond in real time, preventing appropriate countermeasures from being implemented. The present invention aims to solve these problems and provide an advanced surveillance system to ensure the safety of vulnerable members of society.

[0477] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0478] In this invention, the server includes a video analysis means, anomaly detection means, a notification means, an automatic response means, a recording means, a means for streaming video data to the server via a terminal, and a means for detecting anomalies based on the video data analyzed using a deep learning algorithm, thereby enabling highly accurate real-time anomaly detection and rapid response.

[0479] "Video acquisition means" refers to means for capturing video in real time using a device such as a camera.

[0480] The "video analysis means" is a means for analyzing acquired video data and recognizing specific patterns or abnormalities.

[0481] The "abnormality detection means" is a means for detecting specific abnormalities (e.g., falls or abnormal behavior) based on the analysis results of the video analysis means.

[0482] "Notification means" refers to the means of notifying the user when an abnormality is detected. Notification methods include push notification, SMS, and voice alarm.

[0483] An "automatic response means" is a means for automatically taking appropriate action in response to a detected abnormality based on preset conditions.

[0484] "Recording means" is a means for recording all abnormal events and corresponding actions and saving them for later review.

[0485] "Means for streaming video data to a server through a terminal" refers to means for a terminal to continuously transmit captured video data to a server via a network.

[0486] "Means for detecting anomalies based on video data analyzed using a deep learning algorithm" refers to a means for analyzing video data in real time using a deep learning model to detect anomalies with high accuracy.

[0487] This invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and is operated by combining the following hardware and software: Specifically, it utilizes a camera (video acquisition means), devices such as smartphones and tablets, and a deep learning algorithm installed on a server.

[0488] The user simply installs the camera in an appropriate location and configures the system using a dedicated app. For example, the user installs the camera in the living room and configures it for fall detection. This configuration is performed through the app by selecting whether to enable fall detection and the notification method (push notification, SMS, voice alarm, etc.).

[0489] The camera captures video in real time 24 hours a day, and the video data is sent to a terminal via communication methods such as Wi-Fi or Bluetooth. The terminal compresses the received video data and streams it to a server.

[0490] The server uses deep learning algorithms to analyze video data in real time. For example, frameworks such as TensorFlow and PyTorch are used to detect falls by elderly people or abnormal behavior in babies. The server's anomaly detection means immediately identifies these anomalies and sends an alert to the user via the notification means.

[0491] If an abnormality is detected, the server will take appropriate action through automated response measures based on pre-defined conditions, such as automatically calling emergency contacts or requesting an ambulance, enabling a prompt and appropriate response.

[0492] In addition, the server records all abnormal events and corresponding actions and saves them as logs for later review, allowing users to use the system to review past abnormal events and their responses at any time.

[0493] Specific examples

[0494] Example 1: Detecting falls in elderly people

[0495] User: Install the camera in the living room and configure fall detection settings in the app. For example, set "Fall detection on," "Notification method: push notification," and "Emergency contact: family member's phone number."

[0496] Camera: Captures real-time footage of your living room 24 hours a day.

[0497] Terminal: Receives camera footage, compresses it and streams it to the server.

[0498] Server: Analyzes video using a TensorFlow model to detect falls by elderly people.

[0499] Server: When a fall is detected, a push notification is sent to the user's smartphone using the notification method, and if necessary, a call is made to an emergency contact.

[0500] Example 2: Detecting abnormal baby behavior

[0501] User: Install a camera in the baby's room and configure the baby's behavior monitoring settings, such as "Abnormal behavior detection on," "Notification method: SMS," and "Emergency contact: parent's phone number."

[0502] Camera: Captures real-time footage of the baby's room 24 hours a day.

[0503] Terminal: Receives camera footage, compresses it and streams it to the server.

[0504] Server: Analyzes the video using a PyTorch model and detects abnormal baby behavior.

[0505] Server: When abnormal behavior is detected, an SMS is sent to the user using a notification method, and in the event of an emergency, an emergency contact is called.

[0506] Prompt Sentence Examples

[0507] Prompt to explain what the system should do if an elderly person falls:

[0508] "The user uses a camera installed in their living room and configures fall detection using a dedicated app. The camera captures video 24 hours a day and sends it to a device via Wi-Fi. The device compresses the video and streams it to a server. The server analyzes the video using a TensorFlow model and detects falls. If an abnormality is identified, the server immediately sends a push notification to the user's smartphone and, if necessary, calls an emergency contact. All events are recorded in a detailed log."

[0509] Prompt sentence that describes the system's behavior when detecting abnormal baby behavior:

[0510] "The user uses a camera installed in the baby's room and configures abnormal behavior detection using a dedicated app. The camera captures video 24 hours a day and transmits it via Wi-Fi to a device. The device compresses the video and streams it to a server. The server analyzes the video using a PyTorch model and detects abnormal behavior. If an abnormality is identified, the server immediately sends an SMS to the user's smartphone and, if necessary, calls an emergency contact. All events are recorded in a detailed log."

[0511] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0512] Step 1:

[0513] The user installs the camera in an appropriate location and configures the system using a dedicated app. Through the app, the user enables fall detection and abnormal behavior monitoring and sets the notification method. The input is the user's configuration information (e.g., fall detection on, notification method: push notification, emergency contact: family member's phone number), which is then output as configuration data to be reflected in the system.

[0514] Step 2:

[0515] The camera captures video in real time 24 hours a day and transmits the video data to a device using a communication method such as Wi-Fi or Bluetooth. The input is video data obtained from the physical environment, and the output is uncompressed video data sent to the device. The specific operation involves the camera periodically generating video frames and transmitting them to the device.

[0516] Step 3:

[0517] The video data received by the device is compressed for efficient transmission within a certain bandwidth and streamed to the server. The input is video data sent from the camera, and the output is compressed video data sent to the server. Specifically, the device compresses the video frames using a compression algorithm and sends them to the server as a stream.

[0518] Step 4:

[0519] The server analyzes the received video data using a deep learning algorithm. The input is the compressed video data sent from the device, and the output is the analysis results (e.g., whether or not a fall has occurred, or abnormal behavior has been detected). Specifically, the server uses libraries such as TensorFlow and PyTorch to analyze specific movement patterns within the video data.

[0520] Step 5:

[0521] The server's anomaly detection means identifies anomalies based on the results of deep learning analysis. The input is the analysis result data from the video analysis means, and the output is the anomaly detection result (e.g., an elderly person falling, or a baby behaving abnormally). Specifically, the server centralizes the analysis results and narrows down the data that is judged to be abnormal.

[0522] Step 6:

[0523] When the server detects an abnormality, it notifies the user through the notification means. The input is the abnormality detection result from the abnormality detection means, and the output is notification data (push notification, SMS, voice alarm). In concrete terms, the server sends a push notification to the user's smartphone based on the abnormality detection result, and if necessary, sends an SMS to emergency contacts.

[0524] Step 7:

[0525] The server's automatic response means automatically executes an appropriate response to the anomaly. The inputs are the anomaly detection results and pre-set response conditions, and the output is specific response actions (contacting emergency contacts, dispatching an ambulance). In concrete terms, the server determines that it is an emergency and starts a script to execute the response.

[0526] Step 8:

[0527] The server records all abnormal events and corresponding actions, saving them as logs for later review. The inputs are the anomaly detection results and corresponding actions, and the output is the recorded data. Specifically, the server records the timestamp, location information, and details of the response to the abnormal event in detail, saving them in a database.

[0528] (Application example 1)

[0529] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0530] To ensure the safety of vulnerable members of society, such as the elderly and babies, there is a need for real-time behavior monitoring, anomaly detection, and rapid response. However, conventional monitoring systems often have low anomaly detection accuracy, resulting in delayed notification and response. In addition, there are limited means of notifying users, making it difficult to respond quickly in emergencies.

[0531] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0532] In this invention, the server includes means for analyzing video data using a deep learning algorithm, means for notifying users using push notifications, short message services, and audio alarms, and means for automatically contacting emergency contacts, enabling highly accurate detection of abnormalities and prompt notification and response.

[0533] "Video acquisition means" refers to means for acquiring video data in real time using a camera or video device.

[0534] "Video analysis means" refers to algorithms or software that analyze acquired video data and recognize specific actions or abnormalities.

[0535] An "abnormality detection means" is a mechanism that detects abnormal behavior or conditions based on the results of video analysis.

[0536] "Notification means" means means for sending an alert to a user when an abnormality is detected, including push notifications, short message services, and audio alarms.

[0537] "Automatic response means" refers to a means for automatically taking appropriate action in accordance with a set protocol when an abnormality is detected.

[0538] "Recording means" refers to a means for saving data related to anomaly detection and responses so that it can be checked later.

[0539] A "deep learning algorithm" is an algorithm that uses deep learning technology to analyze video data and recognize specific actions and abnormalities with high accuracy.

[0540] "Push notification" is a method of automatically sending information from a server to a user's device.

[0541] "Short Message Service" is a service that sends short text messages to mobile phones and smartphones.

[0542] An "audio alarm" is a means of notifying the user of an emergency using audio.

[0543] "Means for automatically contacting emergency contacts" refers to a system for automatically contacting the emergency contacts set up when an abnormality is detected.

[0544] The present invention is a system for ensuring the safety of socially vulnerable people such as the elderly and babies, and includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and different software and hardware configurations for linking these means.

[0545] System configuration

[0546] 1. Video acquisition method

[0547] The server uses a dedicated camera to capture the movements of elderly people and babies in real time, and the video data is encoded on the spot and sent to the server in streaming format.

[0548] 2. Video analysis methods

[0549] The server uses deep learning algorithms (such as TensorFlow) to analyze the video data. Specifically, it analyzes the received video in real time, monitoring the subject's movements and behavior, and uses models designed to identify specific movement patterns, such as falls or abnormal movements.

[0550] 3. Anomaly detection methods

[0551] The server detects abnormalities from the results of video analysis. The abnormality detection means detects deviations from the set movement patterns and judges them as abnormal. For example, an abnormality is detected when an elderly person falls or a baby is unusually quiet.

[0552] 4. Means of notification

[0553] The server will immediately notify the user when an abnormality is detected. Notification methods include push notifications, short message service, and audio alarms. The content of the notification is customized depending on the type and urgency of the abnormality.

[0554] 5. Automated Response Methods

[0555] If an abnormality is detected, the server automatically takes appropriate action based on pre-set conditions, such as contacting emergency contacts or calling an ambulance, enabling a prompt and appropriate response.

[0556] 6. Recording Method

[0557] The server logs all abnormal events and their responses. The log data is stored in a database service such as Firebase, allowing it to be viewed and analyzed at a later date.

[0558] Specific examples

[0559] Example 1: Detecting falls in elderly people

[0560] Users simply install the camera in their living room and set the fall detection mode in the app. The camera captures video in real time 24 hours a day and sends it to the server. The server then uses a deep learning algorithm to analyze the video and immediately alerts the user via push notification and short message service if a fall is detected. Emergency contacts are also notified at the same time.

[0561] Example 2: Detecting abnormal baby behavior

[0562] The user installs a camera in the baby's room and sets it to baby behavior monitoring mode. The camera captures the baby's movements in real time and sends them to the server. The server's video analysis means analyzes the baby's movements and, if it detects abnormal quietness or extreme movements, it sends a notification to the user's smartphone along with an audio alarm. Emergency response is also automatically implemented if necessary.

[0563] Prompt Sentence Examples

[0564] text

[0565] "When an elderly person falls, generate a code that detects the abnormality and immediately notifies the user. Use push notifications and short message services as notification methods, and use Firebase to record the data."

[0566] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0567] Step 1:

[0568] The user installs the camera in a suitable location and configures the system through the application. The camera captures video in real time and acquires video data. The input is the video data from the camera, which is then sent to the server in streaming format.

[0569] Step 2:

[0570] The server encodes the received video data and begins analyzing it using a deep learning algorithm. Specifically, the video data is preprocessed (e.g., frame resizing, color correction) and input into a deep learning model (using TensorFlow). The output returns a list of movement patterns of the elderly and babies.

[0571] Step 3:

[0572] The server uses an anomaly detection means to detect abnormalities based on the analysis of the movement patterns. For example, it detects falling or the baby standing still unnaturally. The input at this time is the analyzed movement pattern, and the output is a flag indicating whether an abnormality has been detected.

[0573] Step 4:

[0574] When an anomaly is detected, the server immediately notifies the user using the notification methods available: push notification, short message service (using Twilio API), and audio alarm. In this process, the anomaly detection flag is input and the notification message is output.

[0575] Step 5:

[0576] The server then uses automated response methods based on pre-configured settings to take appropriate action, such as automatically contacting emergency contacts and requesting an ambulance. The inputs are a flag indicating an anomaly has been detected and a pre-configured response protocol, and the output is a list of the response actions taken.

[0577] Step 6:

[0578] The server uses a recording method to store all of these abnormal events and their corresponding actions. The log data is sent to and stored in a cloud database such as Firebase. The input is detailed data on the abnormal event that occurred and the corresponding action, and the output is the log record stored in the database.

[0579] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0580] This invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and by combining it with an emotion engine in particular, it enables comprehensive surveillance including the psychological state of the target person and quick response. Specifically, the system configuration includes a video acquisition means, a video analysis means, anomaly detection means, a notification means, an automatic response means, a recording means, and an emotion engine.

[0581] System configuration

[0582] The monitoring system consists of the following elements:

[0583] Video acquisition means: A device that uses cameras and sensors to capture video data in real time.

[0584] Video analysis means: A device or software that uses deep learning algorithms to analyze the movements and facial expressions of subjects in captured video.

[0585] Anomaly detection means: A device or software that detects falls, abnormal behavior, and even emotional changes in a subject based on data from video analysis means.

[0586] Notification method: The method to notify the user of abnormalities through push notification, SMS, voice alarm, etc.

[0587] Automated response measures: Devices or software that automatically send notifications to emergency contacts when an abnormality is detected.

[0588] Recording means: A device or software that stores all abnormal events and notification history as a log.

[0589] Emotion engine: A device or software that contains algorithms to analyze a subject's facial expressions and voice and recognize their emotions.

[0590] Program processing

[0591] First, the user installs the camera (video acquisition means) in an appropriate location and configures the system. The camera continues to capture video in real time and transmits it to the terminal. The terminal then processes the video data to stream it to the server.

[0592] The server, which integrates the video analysis and emotion engine, analyzes the video data using a deep learning algorithm. This analysis allows the server to monitor the subject's emotions in real time from their movement patterns and facial expressions. For example, it can detect an elderly person's fall, a baby's abnormal behavior, or even changes in the subject's emotions such as sadness or stress.

[0593] If an anomaly is detected, the server's anomaly detection means will immediately identify it and send an alert to the user via the notification means. Notification methods include push notifications, SMS, and voice alarms, depending on the user's settings. Changes in emotions recognized by the emotion engine will also be included in the notification.

[0594] Furthermore, based on pre-defined conditions, the server's automated response mechanism will take appropriate action in the event of an abnormality, such as contacting emergency contacts, requesting an ambulance, or notifying mental health support contacts if necessary. All abnormal events and response actions are saved as logs by the recording mechanism and can be reviewed later.

[0595] Specific examples

[0596] Example 1: Detecting falls and emotional changes in the elderly

[0597] User: Installs a camera in the living room and configures fall detection and emotion recognition.

[0598] Terminal: The camera captures video in real time 24 hours a day and sends it to the server.

[0599] Server: The video analysis method and emotion engine analyze the movements and facial expressions of the elderly person in the video and detect falls and sad expressions.

[0600] Server: When a fall or grief is detected, the notification mechanism sends an alert to the user's smartphone and also sends an SMS to emergency contacts.

[0601] User: Checks the notification, checks the video in the app, and confirms the elderly person's condition. Based on that information, the user takes necessary action.

[0602] Example 2: Baby abnormal behavior and emotion recognition detection

[0603] User: Install a camera in the baby's room and set up baby behavior monitoring and emotion recognition.

[0604] Device: The camera captures the baby's movements in real time and sends the video to the server.

[0605] Server: The server's video analysis means and emotion engine analyze the baby's movements and facial expressions to detect abnormal behavior or signs of stress.

[0606] Server: When an abnormality is detected, the notification means sends an alert to the user, and if a response is required, the automatic response means contacts the designated emergency contact.

[0607] User: Receives notifications, checks the video in the app, understands the baby's condition, and intervenes directly if necessary to respond to any abnormalities.

[0608] As a result, the present invention can monitor and detect not only physical abnormalities in the elderly and babies, but also mental abnormalities and stress, enabling quick and effective responses.

[0609] The processing flow will be explained below.

[0610] Step 1:

[0611] User: Installs the camera in the room or area to be monitored and turns it on. Configures the system using a dedicated app or web interface. The configuration includes information about the person to be monitored (e.g., elderly, baby) and anomaly detection requirements (e.g., fall detection, abnormal behavior detection).

[0612] Step 2:

[0613] Terminal: Turns on the camera and starts capturing video in real time, encodes the captured video data appropriately, and sends it to the server via the network.

[0614] Step 3:

[0615] Server: Stores the received video data and prepares it for integration with deep learning algorithms. It processes the video data frame by frame using video analysis methods.

[0616] Step 4:

[0617] Server: Analyzes the subject's movement patterns and facial expressions using video analysis. This analysis includes estimating the person's posture, classifying their movements, and analyzing their facial expressions. For example, it can identify whether an elderly person is sitting, standing, or walking, and detect emotions from their facial expressions.

[0618] Step 5:

[0619] Server: Uses an emotion engine to recognize the subject's emotional state (e.g., joy, sadness, anger, stress) based on data obtained from video analysis means.

[0620] Step 6:

[0621] Server: The anomaly detection means detects anomalies (e.g., falls, abnormal stillness, abnormal movements, sudden changes in emotions) from the analysis results and determines how to respond based on the type and severity of the detected anomaly.

[0622] Step 7:

[0623] Server: When an anomaly is detected, the server sends an alert to the user using notification methods such as push notification, SMS, and voice alarm. The notification also includes any changes in emotions recognized by the emotion engine.

[0624] Step 8:

[0625] User: Receives notification and checks the video via a dedicated app or web interface. Checks the details of the abnormality and takes emergency action if necessary.

[0626] Step 9:

[0627] Server: When an anomaly is detected, automated response measures automatically take action based on pre-defined conditions, such as contacting emergency contacts, calling an ambulance, or notifying mental health support contacts.

[0628] Step 10:

[0629] Server: All abnormal events and response actions are saved in a log file using a recording method, including the date and time the abnormality occurred, the type of abnormality detected, and the response taken.

[0630] Step 11:

[0631] Users: Review past event logs via the app or web interface, analyze anomaly frequency and trends, and fine-tune settings as needed to optimize system accuracy and effectiveness.

[0632] With this detailed processing flow, the present invention can effectively detect abnormalities in elderly people and babies, and can take prompt and appropriate measures, including emotional changes.

[0633] Example 2

[0634] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0635] When monitoring vulnerable people such as the elderly and babies, comprehensive monitoring is required, including not only physical abnormalities but also psychological abnormalities, making it difficult to respond quickly and appropriately. Furthermore, there is a lack of technology that can detect abnormalities and recognize emotional changes in real time and provide effective notifications and responses. These issues must be resolved.

[0636] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0637] In this invention, the server includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and an emotion recognition means, which makes it possible to monitor the movements and emotional changes of socially vulnerable people such as the elderly and babies in real time, and to notify and respond quickly and effectively when an abnormality is detected.

[0638] "Video acquisition means" refers to a device that captures video data in real time using a camera or sensor.

[0639] "Video analysis means" refers to a device or software that uses a deep learning algorithm to analyze the movements and facial expressions of subjects in captured video.

[0640] An "abnormality detection means" is a device or software that detects a subject's falls, abnormal behavior, or even changes in emotion based on data from the video analysis means.

[0641] "Notification means" refers to the means of notifying the user of an abnormality via push notification, SMS, voice alarm, etc.

[0642] An "automatic response means" is a device or software that automatically sends a notification to an emergency contact when an abnormality is detected.

[0643] The "recording means" is a device or software that stores all abnormal events and notification history as a log.

[0644] An "emotion recognition means" is a device or software that includes an algorithm for analyzing a subject's facial expressions and voice and recognizing their emotions.

[0645] This invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and by combining it with an emotion recognition function, it enables comprehensive monitoring including the psychological state of the target person and quick response. Specifically, the system configuration includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and an emotion recognition means.

[0646] The user installs a camera (video acquisition means) in the area to be monitored (such as the living room or baby room). The camera captures video in real time and sends it to a device. The device then streams the video data to a server. The server uses video analysis means and emotion recognition means to analyze the video data with a deep learning algorithm and monitor the subject's movements and facial expressions in real time.

[0647] The server's video analysis means uses image recognition technologies such as Convolutional Neural Networks (CNN) to analyze the subject's movement patterns and facial expressions for each video frame. The emotion recognition means also combines voice and facial expression analysis to evaluate changes in emotion. For example, it can detect falls in elderly people, abnormal behavior in babies, and even changes in the subject's emotions such as sadness or stress.

[0648] If an abnormality is detected, the server's anomaly detection means immediately identifies it and sends an alert to the user via the notification means. Notification methods include push notifications, SMS, and voice alarms, depending on the user's settings. For example, a push notification saying "An elderly person has fallen" can be sent to a smartphone. Furthermore, any changes in emotions recognized by the emotion recognition means are also included in the notification.

[0649] Based on pre-set conditions, the server's automatic response means will take appropriate action in the event of an abnormality. For example, it can automatically contact emergency contacts, request an ambulance, or notify mental health support contacts if necessary. All abnormal events and response actions are saved as logs by the recording means, and can be viewed by the user later.

[0650] Specific examples

[0651] Example 1: Detecting falls and emotional changes in the elderly

[0652] User: Installs a camera in the living room and configures fall detection and emotion recognition.

[0653] Terminal: The camera captures video in real time 24 hours a day and sends it to the server.

[0654] Server: Video analysis and emotion recognition means analyze the movements and facial expressions of elderly people in the video and detect falls and sad expressions.

[0655] Server: When a fall or grief is detected, the notification mechanism sends an alert to the user's smartphone and also sends an SMS to emergency contacts.

[0656] User: Checks the notification, checks the video in the app, and confirms the elderly person's condition. Based on that information, the user takes necessary action.

[0657] Examples of prompt statements

[0658] (example)

[0659] Real-time alert prompt for elderly fall:

[0660] Video acquisition method: "A camera installed in the living room captures the elderly person's movements."

[0661] Video analysis method: "Analyzing video data in real time using deep learning algorithms"

[0662] Anomaly detection method: "Detecting falls in elderly people"

[0663] Notification method: "Send an alert notification to your smartphone"

[0664] Example 2: Baby abnormal behavior and emotion recognition detection

[0665] User: Install a camera in the baby's room and set up baby behavior monitoring and emotion recognition.

[0666] Device: The camera captures the baby's movements in real time and sends the video to the server.

[0667] Server: Video analysis and emotion recognition tools analyze the baby's movements and facial expressions to detect abnormal behavior or signs of stress.

[0668] Server: When an abnormality is detected, the notification means sends an alert to the user, and if a response is required, the automatic response means contacts the designated emergency contact.

[0669] User: Receives notifications, checks the video in the app, understands the baby's condition, and intervenes directly if necessary to respond to any abnormalities.

[0670] As a result, the present invention can monitor and detect not only physical abnormalities in the elderly and babies, but also mental abnormalities and stress, enabling quick and effective responses.

[0671] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0672] Step 1:

[0673] The user installs a camera (video capture means) in the monitoring area (such as a living room or baby room) and configures the system. Input information includes the camera's installation location and information about the monitored object (e.g., information about an elderly person or a baby). This configuration prepares the camera to capture the precise location and movement of the monitored object in real time.

[0674] input:

[0675] Camera installation position

[0676] Monitored information

[0677] output:

[0678] System initialization complete

[0679] Specific behavior:

[0680] Users simply sync the installed camera with the smartphone app and adjust the camera's position on the app's initial setup screen. Once setup is complete, the camera will enter surveillance mode.

[0681] Step 2:

[0682] The camera captures video in real time and sends it to the device. The camera takes multiple frames per second and sends them to the device.

[0683] input:

[0684] Real-time video data (multiple frames)

[0685] output:

[0686] Video data sent to the device

[0687] Specific behavior:

[0688] The camera operates in 24-hour surveillance mode, capturing video at 30 frames per second and transmitting it to the device.

[0689] Step 3:

[0690] The device compresses the received video data and streams it efficiently to the server using video compression algorithms such as H.264 and H.265.

[0691] input:

[0692] Real-time video data from cameras

[0693] output:

[0694] Compressed video data

[0695] Specific behavior:

[0696] The terminal compresses the video data and streams the data over the Internet to a server.

[0697] Step 4:

[0698] The server analyzes the received video data using video analysis and emotion recognition methods. Based on a deep learning algorithm, it analyzes the movements and facial expressions of the subjects in the video and evaluates changes in their emotions.

[0699] input:

[0700] Compressed video data

[0701] output:

[0702] Subject's motion analysis and emotion recognition results

[0703] Specific behavior:

[0704] The server uses Convolutional Neural Networks (CNN) to analyze movement patterns and facial expressions for each video frame, and also evaluates emotional changes by combining voice and facial analysis.

[0705] Step 5:

[0706] The server detects abnormalities based on the results of video analysis. The abnormality detection means identifies falls and abnormal behavior of the subject in real time.

[0707] input:

[0708] Motion analysis results

[0709] Emotion recognition results

[0710] output:

[0711] Anomaly detection results

[0712] Specific behavior:

[0713] The server uses the acquired analytical data to detect falls by elderly people or abnormal movements by babies, and also determines abnormalities based on emotion recognition results.

[0714] Step 6:

[0715] When an abnormality is detected, the server sends an alert to the user using a notification method, such as push notification, SMS, or voice alarm.

[0716] input:

[0717] Anomaly detection results

[0718] output:

[0719] Alert Notifications

[0720] Specific behavior:

[0721] When an abnormality is detected, the server sends a push notification to the user's smartphone and an SMS to the user's configured contacts.

[0722] Step 7:

[0723] The server uses automated response mechanisms to take appropriate action depending on the situation, such as contacting emergency contacts or calling an ambulance based on pre-defined conditions.

[0724] input:

[0725] Anomaly detection results

[0726] Predefined conditions

[0727] output:

[0728] Auto-response actions taken

[0729] Specific behavior:

[0730] If the server detects an abnormality, it will automatically call an emergency contact and, if necessary, will also take action such as calling an ambulance.

[0731] Step 8:

[0732] The server stores all abnormal events and notification history as a log using a recording means, which allows users to check later.

[0733] input:

[0734] Auto-response actions taken

[0735] Notification history

[0736] output:

[0737] Abnormal event logs

[0738] Specific behavior:

[0739] The server stores all anomaly detection and response data along with timestamps in a database, allowing users to view past event history through the application.

[0740] (Application example 2)

[0741] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0742] Conventional monitoring systems are limited to detecting only physical abnormalities in subjects, making it difficult to monitor and respond to psychological changes or emotional abnormalities in real time. In addition, security guards and staff lack the means to quickly identify and respond to abnormalities on-site, creating a need for efficient monitoring, especially at night or in complex environments.

[0743] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0744] In this invention, the server includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and a display means via a head-mounted display. This allows not only physical abnormalities but also emotional changes such as fear and anger to be monitored in real time, enabling security guards and other staff to respond quickly and appropriately.

[0745] "Video acquisition means" refers to a device that captures video data in real time using a camera or sensor.

[0746] "Video analysis means" refers to a device or software that uses a deep learning algorithm to analyze the movements and facial expressions of subjects in captured video.

[0747] An "abnormality detection means" is a device or software that detects a subject's falls, abnormal behavior, or even changes in emotion based on data from the video analysis means.

[0748] "Notification means" refers to the means of notifying the user of an abnormality via push notification, SMS, voice alarm, etc.

[0749] An "automatic response means" is a device or software that automatically sends a notification to an emergency contact when an abnormality is detected.

[0750] The "recording means" is a device or software that stores all abnormal events and notification history as a log.

[0751] A "head-mounted display" is a device that a user wears on their head and can display information within their field of vision.

[0752] The invention takes the form of a real-time monitoring system via a head-mounted display for use by guards and security staff. The system includes the following elements:

[0753] 1. Video acquisition means: A device that uses cameras and sensors to capture video data in real time, allowing for constant monitoring of the subject and their surroundings.

[0754] 2. Video analysis means: A device or software that uses deep learning algorithms to analyze the behavior and emotional changes of subjects in captured video. This analysis can accurately detect abnormal behavior and emotional changes of subjects.

[0755] 3. Anomaly detection means: This is a device or software that uses data from video analysis means to detect falls, abnormal behavior, and emotional changes such as fear or anger in the subject. This allows for real-time monitoring of not only physical abnormalities but also psychological abnormalities.

[0756] 4. Notification methods: This is a method of notifying users of abnormalities via push notifications, SMS, voice alarms, etc. When an abnormality is detected, a notification is sent immediately to security guards and staff, enabling a prompt response.

[0757] 5. Automatic response measures: Devices or software that automatically send notifications to emergency contacts when an abnormality is detected, enabling rapid response in emergency situations.

[0758] 6. Recording means: A device or software that stores all abnormal events and notification history as a log. This allows past data to be reviewed later, which is useful for troubleshooting and considering countermeasures.

[0759] 7. Display via head-mounted display: A device worn by the user on the head that displays information within the field of view, allowing security guards to grasp upcoming situations in real time without using their hands.

[0760] Examples:

[0761] While security guards are patrolling commercial facilities at night, they acquire real-time video data through a head-mounted display and analyze the movements and facial expressions of the target. If abnormal behavior or changes in emotions such as fear or anger are detected, a notification means immediately sends an alert to the security guard. In addition, an automatic response means contacts emergency contacts, enabling a prompt response. A recording means saves all abnormal events as a log, allowing the situation to be reviewed later and countermeasures to be considered.

[0762] Example prompts to input to the generative AI model:

[0763] "Design an application for a head-mounted display that captures and analyzes video data in real time and sends a notification to security guards if any abnormal emotions are detected. As a specific use case, please describe a scenario in which a security guard patrolling a commercial facility at night detects a suspicious individual's facial expression of fear or anger and immediately sends a notification to the security company."

[0764] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0765] Step 1:

[0766] The user places the camera in the appropriate location.

[0767] Input: Camera location information

[0768] Output: Video data of the monitored area

[0769] How it works: A user installs a camera in a location within a commercial facility that needs to be monitored, turns it on, and the camera begins capturing video in real time.

[0770] Step 2:

[0771] The device acquires video data from the camera in real time.

[0772] Input: Video stream from camera

[0773] Output: Streaming video data

[0774] Specific operation: The terminal (e.g., the security guard's mobile device) connects to the camera and streams video data to the server.

[0775] Step 3:

[0776] The server prepares data for analyzing the video data.

[0777] Input: Streaming video data

[0778] Output: Video frames for analysis

[0779] Specific operation: The server divides the video data and prepares for batch processing on a frame-by-frame basis.

[0780] Step 4:

[0781] The server uses deep learning algorithms to analyze the video frames.

[0782] Input: Video frame for analysis

[0783] Output: behavior and emotion data

[0784] Specific operation: The server inputs video frames into a deep learning model to analyze the subject's movements and facial expressions.

[0785] Step 5:

[0786] The server detects an abnormality using an abnormality detection means.

[0787] Input: Motion and emotion data

[0788] Output: Anomaly detection event

[0789] Specific operation: The server evaluates the analysis results and detects abnormal behavior and emotional changes such as falls, fear, or anger.

[0790] Step 6:

[0791] The server notifies you of the abnormality.

[0792] Input: Anomaly detection event

[0793] Output: Alert notification

[0794] Specific operation: The server sends a push notification or SMS to the user's device to inform them that an abnormality has been detected.

[0795] Step 7:

[0796] The server will handle the issue automatically.

[0797] Input: Anomaly detection event

[0798] Output: Automatic response action (emergency contact, ambulance request, etc.)

[0799] Specific operation: The server automatically notifies pre-configured emergency contacts and arranges for emergency vehicles if necessary.

[0800] Step 8:

[0801] The server records the logs.

[0802] Input: Anomaly detection events and corresponding actions

[0803] Output: Log data

[0804] Specific operation: The server uses a recording method to store all abnormal events and corresponding actions as logs so that they can be checked later.

[0805] Step 9:

[0806] The user checks the information on the head-mounted display.

[0807] Input: Alert notification, video data

[0808] Output: Real-time monitoring information

[0809] Specific operation: The user wears a head-mounted display and has real-time monitoring information displayed within their field of vision, allowing them to quickly grasp the situation.

[0810] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0811] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0812] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0813] [Third embodiment]

[0814] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0815] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0816] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0817] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0818] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0819] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0820] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0821] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0822] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0823] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0824] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0825] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0826] The present invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and is configured as follows:

[0827] System configuration

[0828] This monitoring system includes video acquisition means, video analysis means, anomaly detection means, notification means, automatic response means, and recording means, enabling a wide range of anomaly detection and rapid response.

[0829] Program processing

[0830] First, the user installs the camera (video acquisition means) in an appropriate location and configures the system. The camera continues to capture video in real time and transmits it to the terminal. The terminal then streams this video data to the server.

[0831] On the server, the video analytics uses deep learning algorithms to analyze the video data. Through this analysis, the server monitors the subject's movement patterns in real time, detecting, for example, an elderly person falling or an abnormal baby behavior (such as excessively violent movements or abnormally quiet and motionless behavior).

[0832] If an anomaly is detected, the server's anomaly detection means will immediately identify it and send an alert to the user via the notification means, which can be push notification, SMS, or voice alarm, depending on the user's settings.

[0833] Furthermore, based on pre-set conditions, the server's automatic response means will execute appropriate actions in the event of an abnormality (for example, contacting emergency contacts, requesting an ambulance, etc.) At this time, all abnormal events and response actions are saved as logs by the recording means and can be checked later.

[0834] Specific examples

[0835] Example 1: Detecting falls in elderly people

[0836] User: Install the camera in the living room and configure fall detection using the app.

[0837] Terminal: The camera captures video in real time 24 hours a day and sends it to the server.

[0838] Server: The video analysis means analyzes the movements of the elderly person in the video, and when a fall is detected, the anomaly detection means immediately identifies it.

[0839] Server: When a fall is detected, the notification means sends a push notification to the user's smartphone and also sends an SMS to emergency contacts.

[0840] User: Checks the notification, checks the video in the app, and confirms the elderly person's condition. Based on that information, the user can take necessary action (for example, contact the elderly person directly, call an ambulance, etc.).

[0841] Example 2: Detecting abnormal baby behavior

[0842] User: Install a camera in the baby's room and set up baby behavior monitoring.

[0843] Device: The camera captures the baby's movements in real time and sends the video to the server.

[0844] Server: The server's video analysis means analyzes the baby's movements and detects abnormal behavior (for example, unusually violent movements or abnormal quietness).

[0845] Server: When an abnormality is detected, the notification means sends an alert to the user.

[0846] User: Receives notification, checks camera footage in the app, monitors the baby's condition, and intervenes directly to respond to any abnormalities if necessary.

[0847] This method allows users to monitor the condition of elderly people or babies in real time, reliably detect abnormalities, and quickly take appropriate action.

[0848] The processing flow will be explained below.

[0849] Step 1:

[0850] User: Installs cameras in the rooms or areas to be monitored. Connects the cameras to the network and configures the system using a dedicated app or web interface. Configuration includes information about the people to be monitored (elderly, babies, etc.) and anomaly detection requirements (fall detection, abnormal behavior detection, etc.).

[0851] Step 2:

[0852] Terminal: Turns on the camera and starts capturing video in real time, appropriately encodes the captured video data, and streams it over the network to the server.

[0853] Step 3:

[0854] Server: Stores the received video data and prepares it for collaboration with deep learning algorithms. It processes the data frame by frame for video analysis.

[0855] Step 4:

[0856] Server: Uses video analytics to analyze the movements of subjects in the video. This analysis includes estimating a person's pose and classifying their movements. For example, it can identify whether an elderly person is sitting, standing, or walking.

[0857] Step 5:

[0858] Server: The anomaly detection means detects anomalies (e.g., falls, abnormal stillness, abnormal movements) based on data from the video analysis means. It determines the action to be taken based on the type and severity of the detected anomaly.

[0859] Step 6:

[0860] Server: When an anomaly is detected, the server alerts the user using notification methods, such as push notification, SMS, email, and audio alarm.

[0861] Step 7:

[0862] User: Receives notification and checks the video footage via a dedicated app or web interface, checks the details of the anomaly, and takes emergency action if necessary.

[0863] Step 8:

[0864] Server: Furthermore, the automatic response means automatically executes actions based on pre-set conditions (e.g., notifying emergency contacts when a fall is detected), such as calling an ambulance or contacting family members or caregivers.

[0865] Step 9:

[0866] Server: All abnormal events and response actions are saved in a log file by a recording method, including the date and time the abnormality occurred, the type of abnormality detected, and the response taken.

[0867] Step 10:

[0868] Users: Review past event logs via the app or web interface, analyze anomaly frequency and trends, and fine-tune settings as needed to optimize system accuracy and effectiveness.

[0869] By performing the operations in each processing step, the present invention can effectively detect abnormalities in elderly people and babies and take prompt and appropriate measures.

[0870] Example 1

[0871] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0872] In today's society, vulnerable members of society, such as the elderly and babies, often spend their time in vulnerable situations, and a rapid response is required, especially when accidents or abnormal behavior occur at home. However, conventional surveillance systems often lack the accuracy of anomaly detection and the ability to respond in real time, preventing appropriate countermeasures from being implemented. The present invention aims to solve these problems and provide an advanced surveillance system to ensure the safety of vulnerable members of society.

[0873] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0874] In this invention, the server includes a video analysis means, anomaly detection means, a notification means, an automatic response means, a recording means, a means for streaming video data to the server via a terminal, and a means for detecting anomalies based on the video data analyzed using a deep learning algorithm, thereby enabling highly accurate real-time anomaly detection and rapid response.

[0875] "Video acquisition means" refers to means for capturing video in real time using a device such as a camera.

[0876] The "video analysis means" is a means for analyzing acquired video data and recognizing specific patterns or abnormalities.

[0877] The "abnormality detection means" is a means for detecting specific abnormalities (e.g., falls or abnormal behavior) based on the analysis results of the video analysis means.

[0878] "Notification means" refers to the means of notifying the user when an abnormality is detected. Notification methods include push notification, SMS, and voice alarm.

[0879] An "automatic response means" is a means for automatically taking appropriate action in response to a detected abnormality based on preset conditions.

[0880] "Recording means" is a means for recording all abnormal events and corresponding actions and saving them for later review.

[0881] "Means for streaming video data to a server through a terminal" refers to means for a terminal to continuously transmit captured video data to a server via a network.

[0882] "Means for detecting anomalies based on video data analyzed using a deep learning algorithm" refers to a means for analyzing video data in real time using a deep learning model to detect anomalies with high accuracy.

[0883] This invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and is operated by combining the following hardware and software: Specifically, it utilizes a camera (video acquisition means), devices such as smartphones and tablets, and a deep learning algorithm installed on a server.

[0884] The user simply installs the camera in an appropriate location and configures the system using a dedicated app. For example, the user installs the camera in the living room and configures it for fall detection. This configuration is performed through the app by selecting whether to enable fall detection and the notification method (push notification, SMS, voice alarm, etc.).

[0885] The camera captures video in real time 24 hours a day, and the video data is sent to a terminal via communication methods such as Wi-Fi or Bluetooth. The terminal compresses the received video data and streams it to a server.

[0886] The server uses deep learning algorithms to analyze video data in real time. For example, frameworks such as TensorFlow and PyTorch are used to detect falls by elderly people or abnormal behavior in babies. The server's anomaly detection means immediately identifies these anomalies and sends an alert to the user via the notification means.

[0887] If an abnormality is detected, the server will take appropriate action through automated response measures based on pre-defined conditions, such as automatically calling emergency contacts or requesting an ambulance, enabling a prompt and appropriate response.

[0888] In addition, the server records all abnormal events and corresponding actions and saves them as logs for later review, allowing users to use the system to review past abnormal events and their responses at any time.

[0889] Specific examples

[0890] Example 1: Detecting falls in elderly people

[0891] User: Install the camera in the living room and configure fall detection settings in the app. For example, set "Fall detection on," "Notification method: push notification," and "Emergency contact: family member's phone number."

[0892] Camera: Captures real-time footage of your living room 24 hours a day.

[0893] Terminal: Receives camera footage, compresses it and streams it to the server.

[0894] Server: Analyzes video using a TensorFlow model to detect falls by elderly people.

[0895] Server: When a fall is detected, a push notification is sent to the user's smartphone using the notification method, and if necessary, a call is made to an emergency contact.

[0896] Example 2: Detecting abnormal baby behavior

[0897] User: Install a camera in the baby's room and configure the baby's behavior monitoring settings, such as "Abnormal behavior detection on," "Notification method: SMS," and "Emergency contact: parent's phone number."

[0898] Camera: Captures real-time footage of the baby's room 24 hours a day.

[0899] Terminal: Receives camera footage, compresses it and streams it to the server.

[0900] Server: Analyzes the video using a PyTorch model and detects abnormal baby behavior.

[0901] Server: When abnormal behavior is detected, an SMS is sent to the user using a notification method, and in the event of an emergency, an emergency contact is called.

[0902] Prompt Sentence Examples

[0903] Prompt to explain what the system should do if an elderly person falls:

[0904] "The user uses a camera installed in their living room and configures fall detection using a dedicated app. The camera captures video 24 hours a day and sends it to a device via Wi-Fi. The device compresses the video and streams it to a server. The server analyzes the video using a TensorFlow model and detects falls. If an abnormality is identified, the server immediately sends a push notification to the user's smartphone and, if necessary, calls an emergency contact. All events are recorded in a detailed log."

[0905] Prompt sentence that describes the system's behavior when detecting abnormal baby behavior:

[0906] "The user uses a camera installed in the baby's room and configures abnormal behavior detection using a dedicated app. The camera captures video 24 hours a day and transmits it via Wi-Fi to a device. The device compresses the video and streams it to a server. The server analyzes the video using a PyTorch model and detects abnormal behavior. If an abnormality is identified, the server immediately sends an SMS to the user's smartphone and, if necessary, calls an emergency contact. All events are recorded in a detailed log."

[0907] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0908] Step 1:

[0909] The user installs the camera in an appropriate location and configures the system using a dedicated app. Through the app, the user enables fall detection and abnormal behavior monitoring and sets the notification method. The input is the user's configuration information (e.g., fall detection on, notification method: push notification, emergency contact: family member's phone number), which is then output as configuration data to be reflected in the system.

[0910] Step 2:

[0911] The camera captures video in real time 24 hours a day and transmits the video data to a device using a communication method such as Wi-Fi or Bluetooth. The input is video data obtained from the physical environment, and the output is uncompressed video data sent to the device. The specific operation involves the camera periodically generating video frames and transmitting them to the device.

[0912] Step 3:

[0913] The video data received by the device is compressed for efficient transmission within a certain bandwidth and streamed to the server. The input is video data sent from the camera, and the output is compressed video data sent to the server. Specifically, the device compresses the video frames using a compression algorithm and sends them to the server as a stream.

[0914] Step 4:

[0915] The server analyzes the received video data using a deep learning algorithm. The input is the compressed video data sent from the device, and the output is the analysis results (e.g., whether or not a fall has occurred, or abnormal behavior has been detected). Specifically, the server uses libraries such as TensorFlow and PyTorch to analyze specific movement patterns within the video data.

[0916] Step 5:

[0917] The server's anomaly detection means identifies anomalies based on the results of deep learning analysis. The input is the analysis result data from the video analysis means, and the output is the anomaly detection result (e.g., an elderly person falling, or a baby behaving abnormally). Specifically, the server centralizes the analysis results and narrows down the data that is judged to be abnormal.

[0918] Step 6:

[0919] When the server detects an abnormality, it notifies the user through the notification means. The input is the abnormality detection result from the abnormality detection means, and the output is notification data (push notification, SMS, voice alarm). In concrete terms, the server sends a push notification to the user's smartphone based on the abnormality detection result, and if necessary, sends an SMS to emergency contacts.

[0920] Step 7:

[0921] The server's automatic response means automatically executes an appropriate response to the anomaly. The inputs are the anomaly detection results and pre-set response conditions, and the output is specific response actions (contacting emergency contacts, dispatching an ambulance). In concrete terms, the server determines that it is an emergency and starts a script to execute the response.

[0922] Step 8:

[0923] The server records all abnormal events and corresponding actions, saving them as logs for later review. The inputs are the anomaly detection results and corresponding actions, and the output is the recorded data. Specifically, the server records the timestamp, location information, and details of the response to the abnormal event in detail, saving them in a database.

[0924] (Application example 1)

[0925] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0926] To ensure the safety of vulnerable members of society, such as the elderly and babies, there is a need for real-time behavior monitoring, anomaly detection, and rapid response. However, conventional monitoring systems often have low anomaly detection accuracy, resulting in delayed notification and response. In addition, there are limited means of notifying users, making it difficult to respond quickly in emergencies.

[0927] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0928] In this invention, the server includes means for analyzing video data using a deep learning algorithm, means for notifying users using push notifications, short message services, and audio alarms, and means for automatically contacting emergency contacts, enabling highly accurate detection of abnormalities and prompt notification and response.

[0929] "Video acquisition means" refers to means for acquiring video data in real time using a camera or video device.

[0930] "Video analysis means" refers to algorithms or software that analyze acquired video data and recognize specific actions or abnormalities.

[0931] An "abnormality detection means" is a mechanism that detects abnormal behavior or conditions based on the results of video analysis.

[0932] "Notification means" means means for sending an alert to a user when an abnormality is detected, including push notifications, short message services, and audio alarms.

[0933] "Automatic response means" refers to a means for automatically taking appropriate action in accordance with a set protocol when an abnormality is detected.

[0934] "Recording means" refers to a means for saving data related to anomaly detection and responses so that it can be checked later.

[0935] A "deep learning algorithm" is an algorithm that uses deep learning technology to analyze video data and recognize specific actions and abnormalities with high accuracy.

[0936] "Push notification" is a method of automatically sending information from a server to a user's device.

[0937] "Short Message Service" is a service that sends short text messages to mobile phones and smartphones.

[0938] An "audio alarm" is a means of notifying the user of an emergency using audio.

[0939] "Means for automatically contacting emergency contacts" refers to a system for automatically contacting the emergency contacts set up when an abnormality is detected.

[0940] The present invention is a system for ensuring the safety of socially vulnerable people such as the elderly and babies, and includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and different software and hardware configurations for linking these means.

[0941] System configuration

[0942] 1. Video acquisition method

[0943] The server uses a dedicated camera to capture the movements of elderly people and babies in real time, and the video data is encoded on the spot and sent to the server in streaming format.

[0944] 2. Video analysis methods

[0945] The server uses deep learning algorithms (such as TensorFlow) to analyze the video data. Specifically, it analyzes the received video in real time, monitoring the subject's movements and behavior, and uses models designed to identify specific movement patterns, such as falls or abnormal movements.

[0946] 3. Anomaly detection methods

[0947] The server detects abnormalities from the results of video analysis. The abnormality detection means detects deviations from the set movement patterns and judges them as abnormal. For example, an abnormality is detected when an elderly person falls or a baby is unusually quiet.

[0948] 4. Means of notification

[0949] The server will immediately notify the user when an abnormality is detected. Notification methods include push notifications, short message service, and audio alarms. The content of the notification is customized depending on the type and urgency of the abnormality.

[0950] 5. Automated Response Methods

[0951] If an abnormality is detected, the server automatically takes appropriate action based on pre-set conditions, such as contacting emergency contacts or calling an ambulance, enabling a prompt and appropriate response.

[0952] 6. Recording Method

[0953] The server logs all abnormal events and their responses. The log data is stored in a database service such as Firebase, allowing it to be viewed and analyzed at a later date.

[0954] Specific examples

[0955] Example 1: Detecting falls in elderly people

[0956] Users simply install the camera in their living room and set the fall detection mode in the app. The camera captures video in real time 24 hours a day and sends it to the server. The server then uses a deep learning algorithm to analyze the video and immediately alerts the user via push notification and short message service if a fall is detected. Emergency contacts are also notified at the same time.

[0957] Example 2: Detecting abnormal baby behavior

[0958] The user installs a camera in the baby's room and sets it to baby behavior monitoring mode. The camera captures the baby's movements in real time and sends them to the server. The server's video analysis means analyzes the baby's movements and, if it detects abnormal quietness or extreme movements, it sends a notification to the user's smartphone along with an audio alarm. Emergency response is also automatically implemented if necessary.

[0959] Prompt Sentence Examples

[0960] text

[0961] "When an elderly person falls, generate a code that detects the abnormality and immediately notifies the user. Use push notifications and short message services as notification methods, and use Firebase to record the data."

[0962] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0963] Step 1:

[0964] The user installs the camera in a suitable location and configures the system through the application. The camera captures video in real time and acquires video data. The input is the video data from the camera, which is then sent to the server in streaming format.

[0965] Step 2:

[0966] The server encodes the received video data and begins analyzing it using a deep learning algorithm. Specifically, the video data is preprocessed (e.g., frame resizing, color correction) and input into a deep learning model (using TensorFlow). The output returns a list of movement patterns of the elderly and babies.

[0967] Step 3:

[0968] The server uses an anomaly detection means to detect abnormalities based on the analysis of the movement patterns. For example, it detects falling or the baby standing still unnaturally. The input at this time is the analyzed movement pattern, and the output is a flag indicating whether an abnormality has been detected.

[0969] Step 4:

[0970] When an anomaly is detected, the server immediately notifies the user using the notification methods available: push notification, short message service (using Twilio API), and audio alarm. In this process, the anomaly detection flag is input and the notification message is output.

[0971] Step 5:

[0972] The server then uses automated response methods based on pre-configured settings to take appropriate action, such as automatically contacting emergency contacts and requesting an ambulance. The inputs are a flag indicating an anomaly has been detected and a pre-configured response protocol, and the output is a list of the response actions taken.

[0973] Step 6:

[0974] The server uses a recording method to store all of these abnormal events and their corresponding actions. The log data is sent to and stored in a cloud database such as Firebase. The input is detailed data on the abnormal event that occurred and the corresponding action, and the output is the log record stored in the database.

[0975] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0976] This invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and by combining it with an emotion engine in particular, it enables comprehensive surveillance including the psychological state of the target person and quick response. Specifically, the system configuration includes a video acquisition means, a video analysis means, anomaly detection means, a notification means, an automatic response means, a recording means, and an emotion engine.

[0977] System configuration

[0978] The monitoring system consists of the following elements:

[0979] Video acquisition means: A device that uses cameras and sensors to capture video data in real time.

[0980] Video analysis means: A device or software that uses deep learning algorithms to analyze the movements and facial expressions of subjects in captured video.

[0981] Anomaly detection means: A device or software that detects falls, abnormal behavior, and even emotional changes in a subject based on data from video analysis means.

[0982] Notification method: The method to notify the user of abnormalities through push notification, SMS, voice alarm, etc.

[0983] Automated response measures: Devices or software that automatically send notifications to emergency contacts when an abnormality is detected.

[0984] Recording means: A device or software that stores all abnormal events and notification history as a log.

[0985] Emotion engine: A device or software that contains algorithms to analyze a subject's facial expressions and voice and recognize their emotions.

[0986] Program processing

[0987] First, the user installs the camera (video acquisition means) in an appropriate location and configures the system. The camera continues to capture video in real time and transmits it to the terminal. The terminal then processes the video data to stream it to the server.

[0988] The server, which integrates the video analysis and emotion engine, analyzes the video data using a deep learning algorithm. This analysis allows the server to monitor the subject's emotions in real time from their movement patterns and facial expressions. For example, it can detect an elderly person's fall, a baby's abnormal behavior, or even changes in the subject's emotions such as sadness or stress.

[0989] If an anomaly is detected, the server's anomaly detection means will immediately identify it and send an alert to the user via the notification means. Notification methods include push notifications, SMS, and voice alarms, depending on the user's settings. Changes in emotions recognized by the emotion engine will also be included in the notification.

[0990] Furthermore, based on pre-defined conditions, the server's automated response mechanism will take appropriate action in the event of an abnormality, such as contacting emergency contacts, requesting an ambulance, or notifying mental health support contacts if necessary. All abnormal events and response actions are saved as logs by the recording mechanism and can be reviewed later.

[0991] Specific examples

[0992] Example 1: Detecting falls and emotional changes in the elderly

[0993] User: Installs a camera in the living room and configures fall detection and emotion recognition.

[0994] Terminal: The camera captures video in real time 24 hours a day and sends it to the server.

[0995] Server: The video analysis method and emotion engine analyze the movements and facial expressions of the elderly person in the video and detect falls and sad expressions.

[0996] Server: When a fall or grief is detected, the notification mechanism sends an alert to the user's smartphone and also sends an SMS to emergency contacts.

[0997] User: Checks the notification, checks the video in the app, and confirms the elderly person's condition. Based on that information, the user takes necessary action.

[0998] Example 2: Baby abnormal behavior and emotion recognition detection

[0999] User: Install a camera in the baby's room and set up baby behavior monitoring and emotion recognition.

[1000] Device: The camera captures the baby's movements in real time and sends the video to the server.

[1001] Server: The server's video analysis means and emotion engine analyze the baby's movements and facial expressions to detect abnormal behavior or signs of stress.

[1002] Server: When an abnormality is detected, the notification means sends an alert to the user, and if a response is required, the automatic response means contacts the designated emergency contact.

[1003] User: Receives notifications, checks the video in the app, understands the baby's condition, and intervenes directly if necessary to respond to any abnormalities.

[1004] As a result, the present invention can monitor and detect not only physical abnormalities in the elderly and babies, but also mental abnormalities and stress, enabling quick and effective responses.

[1005] The processing flow will be explained below.

[1006] Step 1:

[1007] User: Installs the camera in the room or area to be monitored and turns it on. Configures the system using a dedicated app or web interface. The configuration includes information about the person to be monitored (e.g., elderly, baby) and anomaly detection requirements (e.g., fall detection, abnormal behavior detection).

[1008] Step 2:

[1009] Terminal: Turns on the camera and starts capturing video in real time, encodes the captured video data appropriately, and sends it to the server via the network.

[1010] Step 3:

[1011] Server: Stores the received video data and prepares it for integration with deep learning algorithms. It processes the video data frame by frame using video analysis methods.

[1012] Step 4:

[1013] Server: Analyzes the subject's movement patterns and facial expressions using video analysis. This analysis includes estimating the person's posture, classifying their movements, and analyzing their facial expressions. For example, it can identify whether an elderly person is sitting, standing, or walking, and detect emotions from their facial expressions.

[1014] Step 5:

[1015] Server: Uses an emotion engine to recognize the subject's emotional state (e.g., joy, sadness, anger, stress) based on data obtained from video analysis means.

[1016] Step 6:

[1017] Server: The anomaly detection means detects anomalies (e.g., falls, abnormal stillness, abnormal movements, sudden changes in emotions) from the analysis results and determines how to respond based on the type and severity of the detected anomaly.

[1018] Step 7:

[1019] Server: When an anomaly is detected, the server sends an alert to the user using notification methods such as push notification, SMS, and voice alarm. The notification also includes any changes in emotions recognized by the emotion engine.

[1020] Step 8:

[1021] User: Receives notification and checks the video via a dedicated app or web interface. Checks the details of the abnormality and takes emergency action if necessary.

[1022] Step 9:

[1023] Server: When an anomaly is detected, automated response measures automatically take action based on pre-defined conditions, such as contacting emergency contacts, calling an ambulance, or notifying mental health support contacts.

[1024] Step 10:

[1025] Server: All abnormal events and response actions are saved in a log file using a recording method, including the date and time the abnormality occurred, the type of abnormality detected, and the response taken.

[1026] Step 11:

[1027] Users: Review past event logs via the app or web interface, analyze anomaly frequency and trends, and fine-tune settings as needed to optimize system accuracy and effectiveness.

[1028] With this detailed processing flow, the present invention can effectively detect abnormalities in elderly people and babies, and can take prompt and appropriate measures, including emotional changes.

[1029] Example 2

[1030] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1031] When monitoring vulnerable people such as the elderly and babies, comprehensive monitoring is required, including not only physical abnormalities but also psychological abnormalities, making it difficult to respond quickly and appropriately. Furthermore, there is a lack of technology that can detect abnormalities and recognize emotional changes in real time and provide effective notifications and responses. These issues must be resolved.

[1032] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1033] In this invention, the server includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and an emotion recognition means, which makes it possible to monitor the movements and emotional changes of socially vulnerable people such as the elderly and babies in real time, and to notify and respond quickly and effectively when an abnormality is detected.

[1034] "Video acquisition means" refers to a device that captures video data in real time using a camera or sensor.

[1035] "Video analysis means" refers to a device or software that uses a deep learning algorithm to analyze the movements and facial expressions of subjects in captured video.

[1036] An "abnormality detection means" is a device or software that detects a subject's falls, abnormal behavior, or even changes in emotion based on data from the video analysis means.

[1037] "Notification means" refers to the means of notifying the user of an abnormality via push notification, SMS, voice alarm, etc.

[1038] An "automatic response means" is a device or software that automatically sends a notification to an emergency contact when an abnormality is detected.

[1039] The "recording means" is a device or software that stores all abnormal events and notification history as a log.

[1040] An "emotion recognition means" is a device or software that includes an algorithm for analyzing a subject's facial expressions and voice and recognizing their emotions.

[1041] This invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and by combining it with an emotion recognition function, it enables comprehensive monitoring including the psychological state of the target person and quick response. Specifically, the system configuration includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and an emotion recognition means.

[1042] The user installs a camera (video acquisition means) in the area to be monitored (such as the living room or baby room). The camera captures video in real time and sends it to a device. The device then streams the video data to a server. The server uses video analysis means and emotion recognition means to analyze the video data with a deep learning algorithm and monitor the subject's movements and facial expressions in real time.

[1043] The server's video analysis means uses image recognition technologies such as Convolutional Neural Networks (CNN) to analyze the subject's movement patterns and facial expressions for each video frame. The emotion recognition means also combines voice and facial expression analysis to evaluate changes in emotion. For example, it can detect falls in elderly people, abnormal behavior in babies, and even changes in the subject's emotions such as sadness or stress.

[1044] If an abnormality is detected, the server's anomaly detection means immediately identifies it and sends an alert to the user via the notification means. Notification methods include push notifications, SMS, and voice alarms, depending on the user's settings. For example, a push notification saying "An elderly person has fallen" can be sent to a smartphone. Furthermore, any changes in emotions recognized by the emotion recognition means are also included in the notification.

[1045] Based on pre-set conditions, the server's automatic response means will take appropriate action in the event of an abnormality. For example, it can automatically contact emergency contacts, request an ambulance, or notify mental health support contacts if necessary. All abnormal events and response actions are saved as logs by the recording means, and can be viewed by the user later.

[1046] Specific examples

[1047] Example 1: Detecting falls and emotional changes in the elderly

[1048] User: Installs a camera in the living room and configures fall detection and emotion recognition.

[1049] Terminal: The camera captures video in real time 24 hours a day and sends it to the server.

[1050] Server: Video analysis and emotion recognition means analyze the movements and facial expressions of elderly people in the video and detect falls and sad expressions.

[1051] Server: When a fall or grief is detected, the notification mechanism sends an alert to the user's smartphone and also sends an SMS to emergency contacts.

[1052] User: Checks the notification, checks the video in the app, and confirms the elderly person's condition. Based on that information, the user takes necessary action.

[1053] Examples of prompt statements

[1054] (example)

[1055] Real-time alert prompt for elderly fall:

[1056] Video acquisition method: "A camera installed in the living room captures the elderly person's movements."

[1057] Video analysis method: "Analyzing video data in real time using deep learning algorithms"

[1058] Anomaly detection method: "Detecting falls in elderly people"

[1059] Notification method: "Send an alert notification to your smartphone"

[1060] Example 2: Baby abnormal behavior and emotion recognition detection

[1061] User: Install a camera in the baby's room and set up baby behavior monitoring and emotion recognition.

[1062] Device: The camera captures the baby's movements in real time and sends the video to the server.

[1063] Server: Video analysis and emotion recognition tools analyze the baby's movements and facial expressions to detect abnormal behavior or signs of stress.

[1064] Server: When an abnormality is detected, the notification means sends an alert to the user, and if a response is required, the automatic response means contacts the designated emergency contact.

[1065] User: Receives notifications, checks the video in the app, understands the baby's condition, and intervenes directly if necessary to respond to any abnormalities.

[1066] As a result, the present invention can monitor and detect not only physical abnormalities in the elderly and babies, but also mental abnormalities and stress, enabling quick and effective responses.

[1067] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1068] Step 1:

[1069] The user installs a camera (video capture means) in the monitoring area (such as a living room or baby room) and configures the system. Input information includes the camera's installation location and information about the monitored object (e.g., information about an elderly person or a baby). This configuration prepares the camera to capture the precise location and movement of the monitored object in real time.

[1070] input:

[1071] Camera installation position

[1072] Monitored information

[1073] output:

[1074] System initialization complete

[1075] Specific behavior:

[1076] Users simply sync the installed camera with the smartphone app and adjust the camera's position on the app's initial setup screen. Once setup is complete, the camera will enter surveillance mode.

[1077] Step 2:

[1078] The camera captures video in real time and sends it to the device. The camera takes multiple frames per second and sends them to the device.

[1079] input:

[1080] Real-time video data (multiple frames)

[1081] output:

[1082] Video data sent to the device

[1083] Specific behavior:

[1084] The camera operates in 24-hour surveillance mode, capturing video at 30 frames per second and transmitting it to the device.

[1085] Step 3:

[1086] The device compresses the received video data and streams it efficiently to the server using video compression algorithms such as H.264 and H.265.

[1087] input:

[1088] Real-time video data from cameras

[1089] output:

[1090] Compressed video data

[1091] Specific behavior:

[1092] The terminal compresses the video data and streams the data over the Internet to a server.

[1093] Step 4:

[1094] The server analyzes the received video data using video analysis and emotion recognition methods. Based on a deep learning algorithm, it analyzes the movements and facial expressions of the subjects in the video and evaluates changes in their emotions.

[1095] input:

[1096] Compressed video data

[1097] output:

[1098] Subject's motion analysis and emotion recognition results

[1099] Specific behavior:

[1100] The server uses Convolutional Neural Networks (CNN) to analyze movement patterns and facial expressions for each video frame, and also evaluates emotional changes by combining voice and facial analysis.

[1101] Step 5:

[1102] The server detects abnormalities based on the results of video analysis. The abnormality detection means identifies falls and abnormal behavior of the subject in real time.

[1103] input:

[1104] Motion analysis results

[1105] Emotion recognition results

[1106] output:

[1107] Anomaly detection results

[1108] Specific behavior:

[1109] The server uses the acquired analytical data to detect falls by elderly people or abnormal movements by babies, and also determines abnormalities based on emotion recognition results.

[1110] Step 6:

[1111] When an abnormality is detected, the server sends an alert to the user using a notification method, such as push notification, SMS, or voice alarm.

[1112] input:

[1113] Anomaly detection results

[1114] output:

[1115] Alert Notifications

[1116] Specific behavior:

[1117] When an abnormality is detected, the server sends a push notification to the user's smartphone and an SMS to the user's configured contacts.

[1118] Step 7:

[1119] The server uses automated response mechanisms to take appropriate action depending on the situation, such as contacting emergency contacts or calling an ambulance based on pre-defined conditions.

[1120] input:

[1121] Anomaly detection results

[1122] Predefined conditions

[1123] output:

[1124] Auto-response actions taken

[1125] Specific behavior:

[1126] If the server detects an abnormality, it will automatically call an emergency contact and, if necessary, will also take action such as calling an ambulance.

[1127] Step 8:

[1128] The server stores all abnormal events and notification history as a log using a recording means, which allows users to check later.

[1129] input:

[1130] Auto-response actions taken

[1131] Notification history

[1132] output:

[1133] Abnormal event logs

[1134] Specific behavior:

[1135] The server stores all anomaly detection and response data along with timestamps in a database, allowing users to view past event history through the application.

[1136] (Application example 2)

[1137] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1138] Conventional monitoring systems are limited to detecting only physical abnormalities in subjects, making it difficult to monitor and respond to psychological changes or emotional abnormalities in real time. In addition, security guards and staff lack the means to quickly identify and respond to abnormalities on-site, creating a need for efficient monitoring, especially at night or in complex environments.

[1139] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1140] In this invention, the server includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and a display means via a head-mounted display. This allows not only physical abnormalities but also emotional changes such as fear and anger to be monitored in real time, enabling security guards and other staff to respond quickly and appropriately.

[1141] "Video acquisition means" refers to a device that captures video data in real time using a camera or sensor.

[1142] "Video analysis means" refers to a device or software that uses a deep learning algorithm to analyze the movements and facial expressions of subjects in captured video.

[1143] An "abnormality detection means" is a device or software that detects a subject's falls, abnormal behavior, or even changes in emotion based on data from the video analysis means.

[1144] "Notification means" refers to the means of notifying the user of an abnormality via push notification, SMS, voice alarm, etc.

[1145] An "automatic response means" is a device or software that automatically sends a notification to an emergency contact when an abnormality is detected.

[1146] The "recording means" is a device or software that stores all abnormal events and notification history as a log.

[1147] A "head-mounted display" is a device that a user wears on their head and can display information within their field of vision.

[1148] The invention takes the form of a real-time monitoring system via a head-mounted display for use by guards and security staff. The system includes the following elements:

[1149] 1. Video acquisition means: A device that uses cameras and sensors to capture video data in real time, allowing for constant monitoring of the subject and their surroundings.

[1150] 2. Video analysis means: A device or software that uses deep learning algorithms to analyze the behavior and emotional changes of subjects in captured video. This analysis can accurately detect abnormal behavior and emotional changes of subjects.

[1151] 3. Anomaly detection means: This is a device or software that uses data from video analysis means to detect falls, abnormal behavior, and emotional changes such as fear or anger in the subject. This allows for real-time monitoring of not only physical abnormalities but also psychological abnormalities.

[1152] 4. Notification methods: This is a method of notifying users of abnormalities via push notifications, SMS, voice alarms, etc. When an abnormality is detected, a notification is sent immediately to security guards and staff, enabling a prompt response.

[1153] 5. Automatic response measures: Devices or software that automatically send notifications to emergency contacts when an abnormality is detected, enabling rapid response in emergency situations.

[1154] 6. Recording means: A device or software that stores all abnormal events and notification history as a log. This allows past data to be reviewed later, which is useful for troubleshooting and considering countermeasures.

[1155] 7. Display via head-mounted display: A device worn by the user on the head that displays information within the field of view, allowing security guards to grasp upcoming situations in real time without using their hands.

[1156] Examples:

[1157] While security guards are patrolling commercial facilities at night, they acquire real-time video data through a head-mounted display and analyze the movements and facial expressions of the target. If abnormal behavior or changes in emotions such as fear or anger are detected, a notification means immediately sends an alert to the security guard. In addition, an automatic response means contacts emergency contacts, enabling a prompt response. A recording means saves all abnormal events as a log, allowing the situation to be reviewed later and countermeasures to be considered.

[1158] Example prompts to input to the generative AI model:

[1159] "Design an application for a head-mounted display that captures and analyzes video data in real time and sends a notification to security guards if any abnormal emotions are detected. As a specific use case, please describe a scenario in which a security guard patrolling a commercial facility at night detects a suspicious individual's facial expression of fear or anger and immediately sends a notification to the security company."

[1160] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1161] Step 1:

[1162] The user places the camera in the appropriate location.

[1163] Input: Camera location information

[1164] Output: Video data of the monitored area

[1165] How it works: A user installs a camera in a location within a commercial facility that needs to be monitored, turns it on, and the camera begins capturing video in real time.

[1166] Step 2:

[1167] The device acquires video data from the camera in real time.

[1168] Input: Video stream from camera

[1169] Output: Streaming video data

[1170] Specific operation: The terminal (e.g., the security guard's mobile device) connects to the camera and streams video data to the server.

[1171] Step 3:

[1172] The server prepares data for analyzing the video data.

[1173] Input: Streaming video data

[1174] Output: Video frames for analysis

[1175] Specific operation: The server divides the video data and prepares for batch processing on a frame-by-frame basis.

[1176] Step 4:

[1177] The server uses deep learning algorithms to analyze the video frames.

[1178] Input: Video frame for analysis

[1179] Output: behavior and emotion data

[1180] Specific operation: The server inputs video frames into a deep learning model to analyze the subject's movements and facial expressions.

[1181] Step 5:

[1182] The server detects an abnormality using an abnormality detection means.

[1183] Input: Motion and emotion data

[1184] Output: Anomaly detection event

[1185] Specific operation: The server evaluates the analysis results and detects abnormal behavior and emotional changes such as falls, fear, or anger.

[1186] Step 6:

[1187] The server notifies you of the abnormality.

[1188] Input: Anomaly detection event

[1189] Output: Alert notification

[1190] Specific operation: The server sends a push notification or SMS to the user's device to inform them that an abnormality has been detected.

[1191] Step 7:

[1192] The server will handle the issue automatically.

[1193] Input: Anomaly detection event

[1194] Output: Automatic response action (emergency contact, ambulance request, etc.)

[1195] Specific operation: The server automatically notifies pre-configured emergency contacts and arranges for emergency vehicles if necessary.

[1196] Step 8:

[1197] The server records the logs.

[1198] Input: Anomaly detection events and corresponding actions

[1199] Output: Log data

[1200] Specific operation: The server uses a recording method to store all abnormal events and corresponding actions as logs so that they can be checked later.

[1201] Step 9:

[1202] The user checks the information on the head-mounted display.

[1203] Input: Alert notification, video data

[1204] Output: Real-time monitoring information

[1205] Specific operation: The user wears a head-mounted display and has real-time monitoring information displayed within their field of vision, allowing them to quickly grasp the situation.

[1206] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1207] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1208] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1209] [Fourth embodiment]

[1210] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1211] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1212] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1213] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1214] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1215] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1216] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1217] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1218] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1219] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1220] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1221] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1222] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1223] The present invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and is configured as follows:

[1224] System configuration

[1225] This monitoring system includes video acquisition means, video analysis means, anomaly detection means, notification means, automatic response means, and recording means, enabling a wide range of anomaly detection and rapid response.

[1226] Program processing

[1227] First, the user installs the camera (video acquisition means) in an appropriate location and configures the system. The camera continues to capture video in real time and transmits it to the terminal. The terminal then streams this video data to the server.

[1228] On the server, the video analytics uses deep learning algorithms to analyze the video data. Through this analysis, the server monitors the subject's movement patterns in real time, detecting, for example, an elderly person falling or an abnormal baby behavior (such as excessively violent movements or abnormally quiet and motionless behavior).

[1229] If an anomaly is detected, the server's anomaly detection means will immediately identify it and send an alert to the user via the notification means, which can be push notification, SMS, or voice alarm, depending on the user's settings.

[1230] Furthermore, based on pre-set conditions, the server's automatic response means will execute appropriate actions in the event of an abnormality (for example, contacting emergency contacts, requesting an ambulance, etc.) At this time, all abnormal events and response actions are saved as logs by the recording means and can be checked later.

[1231] Specific examples

[1232] Example 1: Detecting falls in elderly people

[1233] User: Install the camera in the living room and configure fall detection using the app.

[1234] Terminal: The camera captures video in real time 24 hours a day and sends it to the server.

[1235] Server: The video analysis means analyzes the movements of the elderly person in the video, and when a fall is detected, the anomaly detection means immediately identifies it.

[1236] Server: When a fall is detected, the notification means sends a push notification to the user's smartphone and also sends an SMS to emergency contacts.

[1237] User: Checks the notification, checks the video in the app, and confirms the elderly person's condition. Based on that information, the user can take necessary action (for example, contact the elderly person directly, call an ambulance, etc.).

[1238] Example 2: Detecting abnormal baby behavior

[1239] User: Install a camera in the baby's room and set up baby behavior monitoring.

[1240] Device: The camera captures the baby's movements in real time and sends the video to the server.

[1241] Server: The server's video analysis means analyzes the baby's movements and detects abnormal behavior (for example, unusually violent movements or abnormal quietness).

[1242] Server: When an abnormality is detected, the notification means sends an alert to the user.

[1243] User: Receives notification, checks camera footage in the app, monitors the baby's condition, and intervenes directly to respond to any abnormalities if necessary.

[1244] This method allows users to monitor the condition of elderly people or babies in real time, reliably detect abnormalities, and quickly take appropriate action.

[1245] The processing flow will be explained below.

[1246] Step 1:

[1247] User: Installs cameras in the rooms or areas to be monitored. Connects the cameras to the network and configures the system using a dedicated app or web interface. Configuration includes information about the people to be monitored (elderly, babies, etc.) and anomaly detection requirements (fall detection, abnormal behavior detection, etc.).

[1248] Step 2:

[1249] Terminal: Turns on the camera and starts capturing video in real time, appropriately encodes the captured video data, and streams it over the network to the server.

[1250] Step 3:

[1251] Server: Stores the received video data and prepares it for collaboration with deep learning algorithms. It processes the data frame by frame for video analysis.

[1252] Step 4:

[1253] Server: Uses video analytics to analyze the movements of subjects in the video. This analysis includes estimating a person's pose and classifying their movements. For example, it can identify whether an elderly person is sitting, standing, or walking.

[1254] Step 5:

[1255] Server: The anomaly detection means detects anomalies (e.g., falls, abnormal stillness, abnormal movements) based on data from the video analysis means. It determines the action to be taken based on the type and severity of the detected anomaly.

[1256] Step 6:

[1257] Server: When an anomaly is detected, the server alerts the user using notification methods, such as push notification, SMS, email, and audio alarm.

[1258] Step 7:

[1259] User: Receives notification and checks the video footage via a dedicated app or web interface, checks the details of the anomaly, and takes emergency action if necessary.

[1260] Step 8:

[1261] Server: Furthermore, the automatic response means automatically executes actions based on pre-set conditions (e.g., notifying emergency contacts when a fall is detected), such as calling an ambulance or contacting family members or caregivers.

[1262] Step 9:

[1263] Server: All abnormal events and response actions are saved in a log file by a recording method, including the date and time the abnormality occurred, the type of abnormality detected, and the response taken.

[1264] Step 10:

[1265] Users: Review past event logs via the app or web interface, analyze anomaly frequency and trends, and fine-tune settings as needed to optimize system accuracy and effectiveness.

[1266] By performing the operations in each processing step, the present invention can effectively detect abnormalities in elderly people and babies and take prompt and appropriate measures.

[1267] Example 1

[1268] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1269] In today's society, vulnerable members of society, such as the elderly and babies, often spend their time in vulnerable situations, and a rapid response is required, especially when accidents or abnormal behavior occur at home. However, conventional surveillance systems often lack the accuracy of anomaly detection and the ability to respond in real time, preventing appropriate countermeasures from being implemented. The present invention aims to solve these problems and provide an advanced surveillance system to ensure the safety of vulnerable members of society.

[1270] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1271] In this invention, the server includes a video analysis means, anomaly detection means, a notification means, an automatic response means, a recording means, a means for streaming video data to the server via a terminal, and a means for detecting anomalies based on the video data analyzed using a deep learning algorithm, thereby enabling highly accurate real-time anomaly detection and rapid response.

[1272] "Video acquisition means" refers to means for capturing video in real time using a device such as a camera.

[1273] The "video analysis means" is a means for analyzing acquired video data and recognizing specific patterns or abnormalities.

[1274] The "abnormality detection means" is a means for detecting specific abnormalities (e.g., falls or abnormal behavior) based on the analysis results of the video analysis means.

[1275] "Notification means" refers to the means of notifying the user when an abnormality is detected. Notification methods include push notification, SMS, and voice alarm.

[1276] An "automatic response means" is a means for automatically taking appropriate action in response to a detected abnormality based on preset conditions.

[1277] "Recording means" is a means for recording all abnormal events and corresponding actions and saving them for later review.

[1278] "Means for streaming video data to a server through a terminal" refers to means for a terminal to continuously transmit captured video data to a server via a network.

[1279] "Means for detecting anomalies based on video data analyzed using a deep learning algorithm" refers to a means for analyzing video data in real time using a deep learning model to detect anomalies with high accuracy.

[1280] This invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and is operated by combining the following hardware and software: Specifically, it utilizes a camera (video acquisition means), devices such as smartphones and tablets, and a deep learning algorithm installed on a server.

[1281] The user simply installs the camera in an appropriate location and configures the system using a dedicated app. For example, the user installs the camera in the living room and configures it for fall detection. This configuration is performed through the app by selecting whether to enable fall detection and the notification method (push notification, SMS, voice alarm, etc.).

[1282] The camera captures video in real time 24 hours a day, and the video data is sent to a terminal via communication methods such as Wi-Fi or Bluetooth. The terminal compresses the received video data and streams it to a server.

[1283] The server uses deep learning algorithms to analyze video data in real time. For example, frameworks such as TensorFlow and PyTorch are used to detect falls by elderly people or abnormal behavior in babies. The server's anomaly detection means immediately identifies these anomalies and sends an alert to the user via the notification means.

[1284] If an abnormality is detected, the server will take appropriate action through automated response measures based on pre-defined conditions, such as automatically calling emergency contacts or requesting an ambulance, enabling a prompt and appropriate response.

[1285] In addition, the server records all abnormal events and corresponding actions and saves them as logs for later review, allowing users to use the system to review past abnormal events and their responses at any time.

[1286] Specific examples

[1287] Example 1: Detecting falls in elderly people

[1288] User: Install the camera in the living room and configure fall detection settings in the app. For example, set "Fall detection on," "Notification method: push notification," and "Emergency contact: family member's phone number."

[1289] Camera: Captures real-time footage of your living room 24 hours a day.

[1290] Terminal: Receives camera footage, compresses it and streams it to the server.

[1291] Server: Analyzes video using a TensorFlow model to detect falls by elderly people.

[1292] Server: When a fall is detected, a push notification is sent to the user's smartphone using the notification method, and if necessary, a call is made to an emergency contact.

[1293] Example 2: Detecting abnormal baby behavior

[1294] User: Install a camera in the baby's room and configure the baby's behavior monitoring settings, such as "Abnormal behavior detection on," "Notification method: SMS," and "Emergency contact: parent's phone number."

[1295] Camera: Captures real-time footage of the baby's room 24 hours a day.

[1296] Terminal: Receives camera footage, compresses it and streams it to the server.

[1297] Server: Analyzes the video using a PyTorch model and detects abnormal baby behavior.

[1298] Server: When abnormal behavior is detected, an SMS is sent to the user using a notification method, and in the event of an emergency, an emergency contact is called.

[1299] Prompt Sentence Examples

[1300] Prompt to explain what the system should do if an elderly person falls:

[1301] "The user uses a camera installed in their living room and configures fall detection using a dedicated app. The camera captures video 24 hours a day and sends it to a device via Wi-Fi. The device compresses the video and streams it to a server. The server analyzes the video using a TensorFlow model and detects falls. If an abnormality is identified, the server immediately sends a push notification to the user's smartphone and, if necessary, calls an emergency contact. All events are recorded in a detailed log."

[1302] Prompt sentence that describes the system's behavior when detecting abnormal baby behavior:

[1303] "The user uses a camera installed in the baby's room and configures abnormal behavior detection using a dedicated app. The camera captures video 24 hours a day and transmits it via Wi-Fi to a device. The device compresses the video and streams it to a server. The server analyzes the video using a PyTorch model and detects abnormal behavior. If an abnormality is identified, the server immediately sends an SMS to the user's smartphone and, if necessary, calls an emergency contact. All events are recorded in a detailed log."

[1304] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1305] Step 1:

[1306] The user installs the camera in an appropriate location and configures the system using a dedicated app. Through the app, the user enables fall detection and abnormal behavior monitoring and sets the notification method. The input is the user's configuration information (e.g., fall detection on, notification method: push notification, emergency contact: family member's phone number), which is then output as configuration data to be reflected in the system.

[1307] Step 2:

[1308] The camera captures video in real time 24 hours a day and transmits the video data to a device using a communication method such as Wi-Fi or Bluetooth. The input is video data obtained from the physical environment, and the output is uncompressed video data sent to the device. The specific operation involves the camera periodically generating video frames and transmitting them to the device.

[1309] Step 3:

[1310] The video data received by the device is compressed for efficient transmission within a certain bandwidth and streamed to the server. The input is video data sent from the camera, and the output is compressed video data sent to the server. Specifically, the device compresses the video frames using a compression algorithm and sends them to the server as a stream.

[1311] Step 4:

[1312] The server analyzes the received video data using a deep learning algorithm. The input is the compressed video data sent from the device, and the output is the analysis results (e.g., whether or not a fall has occurred, or abnormal behavior has been detected). Specifically, the server uses libraries such as TensorFlow and PyTorch to analyze specific movement patterns within the video data.

[1313] Step 5:

[1314] The server's anomaly detection means identifies anomalies based on the results of deep learning analysis. The input is the analysis result data from the video analysis means, and the output is the anomaly detection result (e.g., an elderly person falling, or a baby behaving abnormally). Specifically, the server centralizes the analysis results and narrows down the data that is judged to be abnormal.

[1315] Step 6:

[1316] When the server detects an abnormality, it notifies the user through the notification means. The input is the abnormality detection result from the abnormality detection means, and the output is notification data (push notification, SMS, voice alarm). In concrete terms, the server sends a push notification to the user's smartphone based on the abnormality detection result, and if necessary, sends an SMS to emergency contacts.

[1317] Step 7:

[1318] The server's automatic response means automatically executes an appropriate response to the anomaly. The inputs are the anomaly detection results and pre-set response conditions, and the output is specific response actions (contacting emergency contacts, dispatching an ambulance). In concrete terms, the server determines that it is an emergency and starts a script to execute the response.

[1319] Step 8:

[1320] The server records all abnormal events and corresponding actions, saving them as logs for later review. The inputs are the anomaly detection results and corresponding actions, and the output is the recorded data. Specifically, the server records the timestamp, location information, and details of the response to the abnormal event in detail, saving them in a database.

[1321] (Application example 1)

[1322] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1323] To ensure the safety of vulnerable members of society, such as the elderly and babies, there is a need for real-time behavior monitoring, anomaly detection, and rapid response. However, conventional monitoring systems often have low anomaly detection accuracy, resulting in delayed notification and response. In addition, there are limited means of notifying users, making it difficult to respond quickly in emergencies.

[1324] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1325] In this invention, the server includes means for analyzing video data using a deep learning algorithm, means for notifying users using push notifications, short message services, and audio alarms, and means for automatically contacting emergency contacts, enabling highly accurate detection of abnormalities and prompt notification and response.

[1326] "Video acquisition means" refers to means for acquiring video data in real time using a camera or video device.

[1327] "Video analysis means" refers to algorithms or software that analyze acquired video data and recognize specific actions or abnormalities.

[1328] An "abnormality detection means" is a mechanism that detects abnormal behavior or conditions based on the results of video analysis.

[1329] "Notification means" means means for sending an alert to a user when an abnormality is detected, including push notifications, short message services, and audio alarms.

[1330] "Automatic response means" refers to a means for automatically taking appropriate action in accordance with a set protocol when an abnormality is detected.

[1331] "Recording means" refers to a means for saving data related to anomaly detection and responses so that it can be checked later.

[1332] A "deep learning algorithm" is an algorithm that uses deep learning technology to analyze video data and recognize specific actions and abnormalities with high accuracy.

[1333] "Push notification" is a method of automatically sending information from a server to a user's device.

[1334] "Short Message Service" is a service that sends short text messages to mobile phones and smartphones.

[1335] An "audio alarm" is a means of notifying the user of an emergency using audio.

[1336] "Means for automatically contacting emergency contacts" refers to a system for automatically contacting the emergency contacts set up when an abnormality is detected.

[1337] The present invention is a system for ensuring the safety of socially vulnerable people such as the elderly and babies, and includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and different software and hardware configurations for linking these means.

[1338] System configuration

[1339] 1. Video acquisition method

[1340] The server uses a dedicated camera to capture the movements of elderly people and babies in real time, and the video data is encoded on the spot and sent to the server in streaming format.

[1341] 2. Video analysis methods

[1342] The server uses deep learning algorithms (such as TensorFlow) to analyze the video data. Specifically, it analyzes the received video in real time, monitoring the subject's movements and behavior, and uses models designed to identify specific movement patterns, such as falls or abnormal movements.

[1343] 3. Anomaly detection methods

[1344] The server detects abnormalities from the results of video analysis. The abnormality detection means detects deviations from the set movement patterns and judges them as abnormal. For example, an abnormality is detected when an elderly person falls or a baby is unusually quiet.

[1345] 4. Means of notification

[1346] The server will immediately notify the user when an abnormality is detected. Notification methods include push notifications, short message service, and audio alarms. The content of the notification is customized depending on the type and urgency of the abnormality.

[1347] 5. Automated Response Methods

[1348] If an abnormality is detected, the server automatically takes appropriate action based on pre-set conditions, such as contacting emergency contacts or calling an ambulance, enabling a prompt and appropriate response.

[1349] 6. Recording Method

[1350] The server logs all abnormal events and their responses. The log data is stored in a database service such as Firebase, allowing it to be viewed and analyzed at a later date.

[1351] Specific examples

[1352] Example 1: Detecting falls in elderly people

[1353] Users simply install the camera in their living room and set the fall detection mode in the app. The camera captures video in real time 24 hours a day and sends it to the server. The server then uses a deep learning algorithm to analyze the video and immediately alerts the user via push notification and short message service if a fall is detected. Emergency contacts are also notified at the same time.

[1354] Example 2: Detecting abnormal baby behavior

[1355] The user installs a camera in the baby's room and sets it to baby behavior monitoring mode. The camera captures the baby's movements in real time and sends them to the server. The server's video analysis means analyzes the baby's movements and, if it detects abnormal quietness or extreme movements, it sends a notification to the user's smartphone along with an audio alarm. Emergency response is also automatically implemented if necessary.

[1356] Prompt Sentence Examples

[1357] text

[1358] "When an elderly person falls, generate a code that detects the abnormality and immediately notifies the user. Use push notifications and short message services as notification methods, and use Firebase to record the data."

[1359] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1360] Step 1:

[1361] The user installs the camera in a suitable location and configures the system through the application. The camera captures video in real time and acquires video data. The input is the video data from the camera, which is then sent to the server in streaming format.

[1362] Step 2:

[1363] The server encodes the received video data and begins analyzing it using a deep learning algorithm. Specifically, the video data is preprocessed (e.g., frame resizing, color correction) and input into a deep learning model (using TensorFlow). The output returns a list of movement patterns of the elderly and babies.

[1364] Step 3:

[1365] The server uses an anomaly detection means to detect abnormalities based on the analysis of the movement patterns. For example, it detects falling or the baby standing still unnaturally. The input at this time is the analyzed movement pattern, and the output is a flag indicating whether an abnormality has been detected.

[1366] Step 4:

[1367] When an anomaly is detected, the server immediately notifies the user using the notification methods available: push notification, short message service (using Twilio API), and audio alarm. In this process, the anomaly detection flag is input and the notification message is output.

[1368] Step 5:

[1369] The server then uses automated response methods based on pre-configured settings to take appropriate action, such as automatically contacting emergency contacts and requesting an ambulance. The inputs are a flag indicating an anomaly has been detected and a pre-configured response protocol, and the output is a list of the response actions taken.

[1370] Step 6:

[1371] The server uses a recording method to store all of these abnormal events and their corresponding actions. The log data is sent to and stored in a cloud database such as Firebase. The input is detailed data on the abnormal event that occurred and the corresponding action, and the output is the log record stored in the database.

[1372] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1373] This invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and by combining it with an emotion engine in particular, it enables comprehensive surveillance including the psychological state of the target person and quick response. Specifically, the system configuration includes a video acquisition means, a video analysis means, anomaly detection means, a notification means, an automatic response means, a recording means, and an emotion engine.

[1374] System configuration

[1375] The monitoring system consists of the following elements:

[1376] Video acquisition means: A device that uses cameras and sensors to capture video data in real time.

[1377] Video analysis means: A device or software that uses deep learning algorithms to analyze the movements and facial expressions of subjects in captured video.

[1378] Anomaly detection means: A device or software that detects falls, abnormal behavior, and even emotional changes in a subject based on data from video analysis means.

[1379] Notification method: The method to notify the user of abnormalities through push notification, SMS, voice alarm, etc.

[1380] Automated response measures: Devices or software that automatically send notifications to emergency contacts when an abnormality is detected.

[1381] Recording means: A device or software that stores all abnormal events and notification history as a log.

[1382] Emotion engine: A device or software that contains algorithms to analyze a subject's facial expressions and voice and recognize their emotions.

[1383] Program processing

[1384] First, the user installs the camera (video acquisition means) in an appropriate location and configures the system. The camera continues to capture video in real time and transmits it to the terminal. The terminal then processes the video data to stream it to the server.

[1385] The server, which integrates the video analysis and emotion engine, analyzes the video data using a deep learning algorithm. This analysis allows the server to monitor the subject's emotions in real time from their movement patterns and facial expressions. For example, it can detect an elderly person's fall, a baby's abnormal behavior, or even changes in the subject's emotions such as sadness or stress.

[1386] If an anomaly is detected, the server's anomaly detection means will immediately identify it and send an alert to the user via the notification means. Notification methods include push notifications, SMS, and voice alarms, depending on the user's settings. Changes in emotions recognized by the emotion engine will also be included in the notification.

[1387] Furthermore, based on pre-defined conditions, the server's automated response mechanism will take appropriate action in the event of an abnormality, such as contacting emergency contacts, requesting an ambulance, or notifying mental health support contacts if necessary. All abnormal events and response actions are saved as logs by the recording mechanism and can be reviewed later.

[1388] Specific examples

[1389] Example 1: Detecting falls and emotional changes in the elderly

[1390] User: Installs a camera in the living room and configures fall detection and emotion recognition.

[1391] Terminal: The camera captures video in real time 24 hours a day and sends it to the server.

[1392] Server: The video analysis method and emotion engine analyze the movements and facial expressions of the elderly person in the video and detect falls and sad expressions.

[1393] Server: When a fall or grief is detected, the notification mechanism sends an alert to the user's smartphone and also sends an SMS to emergency contacts.

[1394] User: Checks the notification, checks the video in the app, and confirms the elderly person's condition. Based on that information, the user takes necessary action.

[1395] Example 2: Baby abnormal behavior and emotion recognition detection

[1396] User: Install a camera in the baby's room and set up baby behavior monitoring and emotion recognition.

[1397] Device: The camera captures the baby's movements in real time and sends the video to the server.

[1398] Server: The server's video analysis means and emotion engine analyze the baby's movements and facial expressions to detect abnormal behavior or signs of stress.

[1399] Server: When an abnormality is detected, the notification means sends an alert to the user, and if a response is required, the automatic response means contacts the designated emergency contact.

[1400] User: Receives notifications, checks the video in the app, understands the baby's condition, and intervenes directly if necessary to respond to any abnormalities.

[1401] As a result, the present invention can monitor and detect not only physical abnormalities in the elderly and babies, but also mental abnormalities and stress, enabling quick and effective responses.

[1402] The processing flow will be explained below.

[1403] Step 1:

[1404] User: Installs the camera in the room or area to be monitored and turns it on. Configures the system using a dedicated app or web interface. The configuration includes information about the person to be monitored (e.g., elderly, baby) and anomaly detection requirements (e.g., fall detection, abnormal behavior detection).

[1405] Step 2:

[1406] Terminal: Turns on the camera and starts capturing video in real time, encodes the captured video data appropriately, and sends it to the server via the network.

[1407] Step 3:

[1408] Server: Stores the received video data and prepares it for integration with deep learning algorithms. It processes the video data frame by frame using video analysis methods.

[1409] Step 4:

[1410] Server: Analyzes the subject's movement patterns and facial expressions using video analysis. This analysis includes estimating the person's posture, classifying their movements, and analyzing their facial expressions. For example, it can identify whether an elderly person is sitting, standing, or walking, and detect emotions from their facial expressions.

[1411] Step 5:

[1412] Server: Uses an emotion engine to recognize the subject's emotional state (e.g., joy, sadness, anger, stress) based on data obtained from video analysis means.

[1413] Step 6:

[1414] Server: The anomaly detection means detects anomalies (e.g., falls, abnormal stillness, abnormal movements, sudden changes in emotions) from the analysis results and determines how to respond based on the type and severity of the detected anomaly.

[1415] Step 7:

[1416] Server: When an anomaly is detected, the server sends an alert to the user using notification methods such as push notification, SMS, and voice alarm. The notification also includes any changes in emotions recognized by the emotion engine.

[1417] Step 8:

[1418] User: Receives notification and checks the video via a dedicated app or web interface. Checks the details of the abnormality and takes emergency action if necessary.

[1419] Step 9:

[1420] Server: When an anomaly is detected, automated response measures automatically take action based on pre-defined conditions, such as contacting emergency contacts, calling an ambulance, or notifying mental health support contacts.

[1421] Step 10:

[1422] Server: All abnormal events and response actions are saved in a log file using a recording method, including the date and time the abnormality occurred, the type of abnormality detected, and the response taken.

[1423] Step 11:

[1424] Users: Review past event logs via the app or web interface, analyze anomaly frequency and trends, and fine-tune settings as needed to optimize system accuracy and effectiveness.

[1425] With this detailed processing flow, the present invention can effectively detect abnormalities in elderly people and babies, and can take prompt and appropriate measures, including emotional changes.

[1426] Example 2

[1427] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1428] When monitoring vulnerable people such as the elderly and babies, comprehensive monitoring is required, including not only physical abnormalities but also psychological abnormalities, making it difficult to respond quickly and appropriately. Furthermore, there is a lack of technology that can detect abnormalities and recognize emotional changes in real time and provide effective notifications and responses. These issues must be resolved.

[1429] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1430] In this invention, the server includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and an emotion recognition means, which makes it possible to monitor the movements and emotional changes of socially vulnerable people such as the elderly and babies in real time, and to notify and respond quickly and effectively when an abnormality is detected.

[1431] "Video acquisition means" refers to a device that captures video data in real time using a camera or sensor.

[1432] "Video analysis means" refers to a device or software that uses a deep learning algorithm to analyze the movements and facial expressions of subjects in captured video.

[1433] An "abnormality detection means" is a device or software that detects a subject's falls, abnormal behavior, or even changes in emotion based on data from the video analysis means.

[1434] "Notification means" refers to the means of notifying the user of an abnormality via push notification, SMS, voice alarm, etc.

[1435] An "automatic response means" is a device or software that automatically sends a notification to an emergency contact when an abnormality is detected.

[1436] The "recording means" is a device or software that stores all abnormal events and notification history as a log.

[1437] An "emotion recognition means" is a device or software that includes an algorithm for analyzing a subject's facial expressions and voice and recognizing their emotions.

[1438] This invention is a surveillance system for watching over socially vulnerable people such as the elderly and babies, and by combining it with an emotion recognition function, it enables comprehensive monitoring including the psychological state of the target person and quick response. Specifically, the system configuration includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and an emotion recognition means.

[1439] The user installs a camera (video acquisition means) in the area to be monitored (such as the living room or baby room). The camera captures video in real time and sends it to a device. The device then streams the video data to a server. The server uses video analysis means and emotion recognition means to analyze the video data with a deep learning algorithm and monitor the subject's movements and facial expressions in real time.

[1440] The server's video analysis means uses image recognition technologies such as Convolutional Neural Networks (CNN) to analyze the subject's movement patterns and facial expressions for each video frame. The emotion recognition means also combines voice and facial expression analysis to evaluate changes in emotion. For example, it can detect falls in elderly people, abnormal behavior in babies, and even changes in the subject's emotions such as sadness or stress.

[1441] If an abnormality is detected, the server's anomaly detection means immediately identifies it and sends an alert to the user via the notification means. Notification methods include push notifications, SMS, and voice alarms, depending on the user's settings. For example, a push notification saying "An elderly person has fallen" can be sent to a smartphone. Furthermore, any changes in emotions recognized by the emotion recognition means are also included in the notification.

[1442] Based on pre-set conditions, the server's automatic response means will take appropriate action in the event of an abnormality. For example, it can automatically contact emergency contacts, request an ambulance, or notify mental health support contacts if necessary. All abnormal events and response actions are saved as logs by the recording means, and can be viewed by the user later.

[1443] Specific examples

[1444] Example 1: Detecting falls and emotional changes in the elderly

[1445] User: Installs a camera in the living room and configures fall detection and emotion recognition.

[1446] Terminal: The camera captures video in real time 24 hours a day and sends it to the server.

[1447] Server: Video analysis and emotion recognition means analyze the movements and facial expressions of elderly people in the video and detect falls and sad expressions.

[1448] Server: When a fall or grief is detected, the notification mechanism sends an alert to the user's smartphone and also sends an SMS to emergency contacts.

[1449] User: Checks the notification, checks the video in the app, and confirms the elderly person's condition. Based on that information, the user takes necessary action.

[1450] Examples of prompt statements

[1451] (example)

[1452] Real-time alert prompt for elderly fall:

[1453] Video acquisition method: "A camera installed in the living room captures the elderly person's movements."

[1454] Video analysis method: "Analyzing video data in real time using deep learning algorithms"

[1455] Anomaly detection method: "Detecting falls in elderly people"

[1456] Notification method: "Send an alert notification to your smartphone"

[1457] Example 2: Baby abnormal behavior and emotion recognition detection

[1458] User: Install a camera in the baby's room and set up baby behavior monitoring and emotion recognition.

[1459] Device: The camera captures the baby's movements in real time and sends the video to the server.

[1460] Server: Video analysis and emotion recognition tools analyze the baby's movements and facial expressions to detect abnormal behavior or signs of stress.

[1461] Server: When an abnormality is detected, the notification means sends an alert to the user, and if a response is required, the automatic response means contacts the designated emergency contact.

[1462] User: Receives notifications, checks the video in the app, understands the baby's condition, and intervenes directly if necessary to respond to any abnormalities.

[1463] As a result, the present invention can monitor and detect not only physical abnormalities in the elderly and babies, but also mental abnormalities and stress, enabling quick and effective responses.

[1464] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1465] Step 1:

[1466] The user installs a camera (video capture means) in the monitoring area (such as a living room or baby room) and configures the system. Input information includes the camera's installation location and information about the monitored object (e.g., information about an elderly person or a baby). This configuration prepares the camera to capture the precise location and movement of the monitored object in real time.

[1467] input:

[1468] Camera installation position

[1469] Monitored information

[1470] output:

[1471] System initialization complete

[1472] Specific behavior:

[1473] Users simply sync the installed camera with the smartphone app and adjust the camera's position on the app's initial setup screen. Once setup is complete, the camera will enter surveillance mode.

[1474] Step 2:

[1475] The camera captures video in real time and sends it to the device. The camera takes multiple frames per second and sends them to the device.

[1476] input:

[1477] Real-time video data (multiple frames)

[1478] output:

[1479] Video data sent to the device

[1480] Specific behavior:

[1481] The camera operates in 24-hour surveillance mode, capturing video at 30 frames per second and transmitting it to the device.

[1482] Step 3:

[1483] The device compresses the received video data and streams it efficiently to the server using video compression algorithms such as H.264 and H.265.

[1484] input:

[1485] Real-time video data from cameras

[1486] output:

[1487] Compressed video data

[1488] Specific behavior:

[1489] The terminal compresses the video data and streams the data over the Internet to a server.

[1490] Step 4:

[1491] The server analyzes the received video data using video analysis and emotion recognition methods. Based on a deep learning algorithm, it analyzes the movements and facial expressions of the subjects in the video and evaluates changes in their emotions.

[1492] input:

[1493] Compressed video data

[1494] output:

[1495] Subject's motion analysis and emotion recognition results

[1496] Specific behavior:

[1497] The server uses Convolutional Neural Networks (CNN) to analyze movement patterns and facial expressions for each video frame, and also evaluates emotional changes by combining voice and facial analysis.

[1498] Step 5:

[1499] The server detects abnormalities based on the results of video analysis. The abnormality detection means identifies falls and abnormal behavior of the subject in real time.

[1500] input:

[1501] Motion analysis results

[1502] Emotion recognition results

[1503] output:

[1504] Anomaly detection results

[1505] Specific behavior:

[1506] The server uses the acquired analytical data to detect falls by elderly people or abnormal movements by babies, and also determines abnormalities based on emotion recognition results.

[1507] Step 6:

[1508] When an abnormality is detected, the server sends an alert to the user using a notification method, such as push notification, SMS, or voice alarm.

[1509] input:

[1510] Anomaly detection results

[1511] output:

[1512] Alert Notifications

[1513] Specific behavior:

[1514] When an abnormality is detected, the server sends a push notification to the user's smartphone and an SMS to the user's configured contacts.

[1515] Step 7:

[1516] The server uses automated response mechanisms to take appropriate action depending on the situation, such as contacting emergency contacts or calling an ambulance based on pre-defined conditions.

[1517] input:

[1518] Anomaly detection results

[1519] Predefined conditions

[1520] output:

[1521] Auto-response actions taken

[1522] Specific behavior:

[1523] If the server detects an abnormality, it will automatically call an emergency contact and, if necessary, will also take action such as calling an ambulance.

[1524] Step 8:

[1525] The server stores all abnormal events and notification history as a log using a recording means, which allows users to check later.

[1526] input:

[1527] Auto-response actions taken

[1528] Notification history

[1529] output:

[1530] Abnormal event logs

[1531] Specific behavior:

[1532] The server stores all anomaly detection and response data along with timestamps in a database, allowing users to view past event history through the application.

[1533] (Application example 2)

[1534] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1535] Conventional monitoring systems are limited to detecting only physical abnormalities in subjects, making it difficult to monitor and respond to psychological changes or emotional abnormalities in real time. In addition, security guards and staff lack the means to quickly identify and respond to abnormalities on-site, creating a need for efficient monitoring, especially at night or in complex environments.

[1536] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1537] In this invention, the server includes a video acquisition means, a video analysis means, an abnormality detection means, a notification means, an automatic response means, a recording means, and a display means via a head-mounted display. This allows not only physical abnormalities but also emotional changes such as fear and anger to be monitored in real time, enabling security guards and other staff to respond quickly and appropriately.

[1538] "Video acquisition means" refers to a device that captures video data in real time using a camera or sensor.

[1539] "Video analysis means" refers to a device or software that uses a deep learning algorithm to analyze the movements and facial expressions of subjects in captured video.

[1540] An "abnormality detection means" is a device or software that detects a subject's falls, abnormal behavior, or even changes in emotion based on data from the video analysis means.

[1541] "Notification means" refers to the means of notifying the user of an abnormality via push notification, SMS, voice alarm, etc.

[1542] An "automatic response means" is a device or software that automatically sends a notification to an emergency contact when an abnormality is detected.

[1543] The "recording means" is a device or software that stores all abnormal events and notification history as a log.

[1544] A "head-mounted display" is a device that a user wears on their head and can display information within their field of vision.

[1545] The invention takes the form of a real-time monitoring system via a head-mounted display for use by guards and security staff. The system includes the following elements:

[1546] 1. Video acquisition means: A device that uses cameras and sensors to capture video data in real time, allowing for constant monitoring of the subject and their surroundings.

[1547] 2. Video analysis means: A device or software that uses deep learning algorithms to analyze the behavior and emotional changes of subjects in captured video. This analysis can accurately detect abnormal behavior and emotional changes of subjects.

[1548] 3. Anomaly detection means: This is a device or software that uses data from video analysis means to detect falls, abnormal behavior, and emotional changes such as fear or anger in the subject. This allows for real-time monitoring of not only physical abnormalities but also psychological abnormalities.

[1549] 4. Notification methods: This is a method of notifying users of abnormalities via push notifications, SMS, voice alarms, etc. When an abnormality is detected, a notification is sent immediately to security guards and staff, enabling a prompt response.

[1550] 5. Automatic response measures: Devices or software that automatically send notifications to emergency contacts when an abnormality is detected, enabling rapid response in emergency situations.

[1551] 6. Recording means: A device or software that stores all abnormal events and notification history as a log. This allows past data to be reviewed later, which is useful for troubleshooting and considering countermeasures.

[1552] 7. Display via head-mounted display: A device worn by the user on the head that displays information within the field of view, allowing security guards to grasp upcoming situations in real time without using their hands.

[1553] Examples:

[1554] While security guards are patrolling commercial facilities at night, they acquire real-time video data through a head-mounted display and analyze the movements and facial expressions of the target. If abnormal behavior or changes in emotions such as fear or anger are detected, a notification means immediately sends an alert to the security guard. In addition, an automatic response means contacts emergency contacts, enabling a prompt response. A recording means saves all abnormal events as a log, allowing the situation to be reviewed later and countermeasures to be considered.

[1555] Example prompts to input to the generative AI model:

[1556] "Design an application for a head-mounted display that captures and analyzes video data in real time and sends a notification to security guards if any abnormal emotions are detected. As a specific use case, please describe a scenario in which a security guard patrolling a commercial facility at night detects a suspicious individual's facial expression of fear or anger and immediately sends a notification to the security company."

[1557] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1558] Step 1:

[1559] The user places the camera in the appropriate location.

[1560] Input: Camera location information

[1561] Output: Video data of the monitored area

[1562] How it works: A user installs a camera in a location within a commercial facility that needs to be monitored, turns it on, and the camera begins capturing video in real time.

[1563] Step 2:

[1564] The device acquires video data from the camera in real time.

[1565] Input: Video stream from camera

[1566] Output: Streaming video data

[1567] Specific operation: The terminal (e.g., the security guard's mobile device) connects to the camera and streams video data to the server.

[1568] Step 3:

[1569] The server prepares data for analyzing the video data.

[1570] Input: Streaming video data

[1571] Output: Video frames for analysis

[1572] Specific operation: The server divides the video data and prepares for batch processing on a frame-by-frame basis.

[1573] Step 4:

[1574] The server uses deep learning algorithms to analyze the video frames.

[1575] Input: Video frame for analysis

[1576] Output: behavior and emotion data

[1577] Specific operation: The server inputs video frames into a deep learning model to analyze the subject's movements and facial expressions.

[1578] Step 5:

[1579] The server detects an abnormality using an abnormality detection means.

[1580] Input: Motion and emotion data

[1581] Output: Anomaly detection event

[1582] Specific operation: The server evaluates the analysis results and detects abnormal behavior and emotional changes such as falls, fear, or anger.

[1583] Step 6:

[1584] The server notifies you of the abnormality.

[1585] Input: Anomaly detection event

[1586] Output: Alert notification

[1587] Specific operation: The server sends a push notification or SMS to the user's device to inform them that an abnormality has been detected.

[1588] Step 7:

[1589] The server will handle the issue automatically.

[1590] Input: Anomaly detection event

[1591] Output: Automatic response action (emergency contact, ambulance request, etc.)

[1592] Specific operation: The server automatically notifies pre-configured emergency contacts and arranges for emergency vehicles if necessary.

[1593] Step 8:

[1594] The server records the logs.

[1595] Input: Anomaly detection events and corresponding actions

[1596] Output: Log data

[1597] Specific operation: The server uses a recording method to store all abnormal events and corresponding actions as logs so that they can be checked later.

[1598] Step 9:

[1599] The user checks the information on the head-mounted display.

[1600] Input: Alert notification, video data

[1601] Output: Real-time monitoring information

[1602] Specific operation: The user wears a head-mounted display and has real-time monitoring information displayed within their field of vision, allowing them to quickly grasp the situation.

[1603] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1604] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1605] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1606] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1607] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1608] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1609] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1610] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1611] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1612] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1613] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1614] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1615] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1616] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1617] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1618] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1619] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1620] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1621] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1622] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1623] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1624] The following is further disclosed regarding the above embodiment.

[1625] (Claim 1)

[1626] An image acquisition means;

[1627] A video analysis means;

[1628] An anomaly detection means;

[1629] Notification means;

[1630] Automatic response means;

[1631] a recording means;

[1632] Monitoring system including.

[1633] (Claim 2)

[1634] The surveillance system according to claim 1, characterized in that the video analysis means uses a deep learning algorithm to analyze the movements of subjects in the video.

[1635] (Claim 3)

[1636] 2. The monitoring system according to claim 1, wherein the abnormality detection means detects a fall of the subject.

[1637] (Claim 4)

[1638] 2. The monitoring system according to claim 1, wherein the notification means notifies the user by at least one of a push notification, an SMS, and an audio alarm.

[1639] (Claim 5)

[1640] 2. The monitoring system according to claim 1, wherein the automatic response means automatically contacts a designated emergency contact when an abnormality is detected.

[1641] "Example 1"

[1642] (Claim 1)

[1643] An image acquisition means;

[1644] A video analysis means;

[1645] An anomaly detection means;

[1646] Notification means;

[1647] Automatic response means;

[1648] a recording means;

[1649] means for streaming video data to a server through the terminal;

[1650] A means for detecting anomalies based on video data analyzed using a deep learning algorithm;

[1651] A system including:

[1652] (Claim 2)

[1653] 2. The system of claim 1, wherein the video analysis means uses a deep learning algorithm to analyze the behavior of subjects in the video in real time and identify anomalies.

[1654] (Claim 3)

[1655] The system according to claim 1, characterized in that the abnormality detection means immediately detects a fall or abnormal behavior of the subject and sends an alert to the user via the notification means.

[1656] "Application Example 1"

[1657] (Claim 1)

[1658] An image acquisition means;

[1659] A video analysis means;

[1660] An anomaly detection means;

[1661] Notification means;

[1662] Automatic response means;

[1663] a recording means;

[1664] a means for analyzing the video data using a deep learning algorithm;

[1665] a means of notifying the user using push notifications, short message service, and audio alarms;

[1666] A means of automatically contacting emergency contacts; and

[1667] How to request an ambulance;

[1668] A system including:

[1669] (Claim 2)

[1670] 10. The system of claim 1, further comprising: a deep learning algorithm for analyzing the behavior of subjects in the video.

[1671] (Claim 3)

[1672] 10. The system of claim 1, further comprising: a notification sent to a user when an anomaly is detected.

[1673] "Example 2: Combining Emotion Engines"

[1674] (Claim 1)

[1675] An image acquisition means;

[1676] A video analysis means;

[1677] An anomaly detection means;

[1678] Notification means;

[1679] Automatic response means;

[1680] a recording means;

[1681] An emotion recognition means;

[1682] A system including:

[1683] (Claim 2)

[1684] 2. The system of claim 1, wherein the video analysis means uses a deep learning algorithm to analyze the movements and facial expressions of subjects in the video.

[1685] (Claim 3)

[1686] 2. The system according to claim 1, wherein the abnormality detection means detects a fall, abnormal behavior, or emotional change of the subject.

[1687] "Application example 2 when combining emotion engines"

[1688] (Claim 1)

[1689] An image acquisition means;

[1690] A video analysis means;

[1691] An anomaly detection means;

[1692] Notification means;

[1693] Automatic response means;

[1694] a recording means;

[1695] A display means via a head-mounted display;

[1696] A system including:

[1697] (Claim 2)

[1698] 2. The system of claim 1, wherein the video analysis means uses a deep learning algorithm to analyze the movements and emotional changes of subjects in the video.

[1699] (Claim 3)

[1700] 2. The system according to claim 1, wherein the abnormality detection means detects abnormal emotional changes such as a fall in the subject or fear or anger. [Explanation of symbols]

[1701] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. An image acquisition means; A video analysis means; An anomaly detection means; Notification means; Automatic response means; a recording means; Monitoring system including.

2. The surveillance system according to claim 1, wherein the video analysis means uses a deep learning algorithm to analyze the movements of the subject in the video.

3. 2. The monitoring system according to claim 1, wherein the abnormality detection means detects a fall of the subject.

4. The monitoring system according to claim 1, wherein the notification means notifies the user by at least one of push notification, SMS, and audio alarm.

5. 2. The monitoring system according to claim 1, wherein the automatic response means automatically contacts a designated emergency contact when an abnormality is detected.

Images