system

Abstract

We provide the system. [Solution] A means for dividing visual information received from a monitoring device and converting it into a format that can be immediately analyzed, A means for performing individual recognition and motion analysis on the converted visual information to detect abnormal behavior, A means for generating an alarm and sending a notification to an external device based on detected abnormal behavior, A means for summarizing the analysis results using generation technology and storing the summarized data on a recording medium, Means for providing details and visual information of detected abnormal behavior to a display device, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional security monitoring systems have placed emphasis on the post-analysis of video data, and there are problems that it is difficult to detect abnormalities in real time and to respond quickly. As a result, criminal acts that require a quick response cannot be immediately deterred, and the security effect is limited. Also, it is difficult to efficiently extract significant information from a huge amount of video data, and the efficiency of analysis and recording is low, which has also been a problem.

Means for Solving the Problems

[0005] This invention introduces a device that converts video data from a monitoring device into a format that can be divided and analyzed in real time, and provides a means for immediately detecting abnormal behavior using facial recognition and motion analysis. Furthermore, it realizes a system that automatically issues an alarm in response to detected abnormal behavior and sends a notification to an external terminal. In addition, by utilizing generation technology to summarize the analysis results and saving them to a recording device, rapid information collection and analysis at a later date is made possible. The aim is to simultaneously achieve improved security efficiency and enhanced safety.

[0006] A "monitoring device" is a device used to continuously capture video and activity within a monitored area.

[0007] "Video data" refers to digital data of images and video information acquired from surveillance devices.

[0008] "Real-time" refers to a time frame in which data is processed immediately and analysis results can be obtained without delay.

[0009] "Facial recognition" is a process that includes technology for detecting human faces from video data and identifying specific individuals.

[0010] "Dynamic analysis" is a technology that analyzes movement and actions within video footage to identify abnormalities and specific actions.

[0011] "Abnormal behavior" refers to actions that deviate from normal behavioral patterns or standards, and in the context of crime prevention, it refers to behavior that indicates a potential threat.

[0012] An "alarm" is an audio or visual signal issued to warn of an abnormality or danger.

[0013] An "external terminal" refers to a device, such as a smartphone or PC, that is connected to the system and is primarily accessible to the user.

[0014] "Notification" refers to digital communication used to transmit warnings or information to external devices.

[0015] "The "generation technology" is an artificial intelligence technology used to summarize data analysis results or generate new data."

[0016] "The "recording device" is a device such as a hard disk or SSD for storing digital data."

Brief Description of Drawings

[0017] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment." [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment." [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment." [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment." [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment." [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment." [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment." [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment." [Figure 9] It shows an emotion map to which a plurality of emotions are mapped." [Figure 10] It shows an emotion map to which a plurality of emotions are mapped." [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1." [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1." [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Embodiment for Implementing the Invention

[0018] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0021] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disk (e.g., hard disk), or magnetic tape, etc.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] The system of this invention aims to process video data acquired from a monitoring device in real time and to quickly detect abnormal behavior. This system is configured based on the respective roles of server, terminal, and user.

[0039] The server receives video data in streaming format from multiple monitoring devices. The received data is divided into frames in real time and pre-processed. Specifically, noise in the video is removed and the resolution is adjusted as needed to improve the efficiency of the subsequent analysis process. Next, the server activates a facial recognition algorithm to detect human faces frame by frame and compare them with a database. This process enables the identification of known suspicious individuals and the detection of new anomaly patterns.

[0040] Furthermore, the server uses dynamic analysis technology to detect abnormal behavior. For example, it identifies unnatural movements and behavioral patterns that differ from normal human movement. This enables a rapid response to prevent criminal activity.

[0041] If an anomaly is detected, the server issues an alarm and immediately sends a notification to the terminal. The terminal is responsible for providing these notifications to the user in real time. Based on the information provided, the user can take the most appropriate security measures. The terminal also provides access to detailed information about the detected anomaly and live video, allowing the user to understand the situation more accurately.

[0042] Furthermore, the server utilizes generation technology to summarize the analyzed data. This summary includes a timeline of specific events and highlights of important scenes, and is saved to a recording device. The saved data can be accessed by users at a later date and used as reference material for detailed analysis and re-evaluation.

[0043] For example, when a user is away from home for an extended period, activating the system will detect suspicious activity during the night and immediately send a notification to the user's device. This allows the user to contact external organizations such as the police or security companies and take swift action. In this way, the system provides a practical solution that contributes to the efficiency of crime prevention and the improvement of security.

[0044] The following describes the processing flow.

[0045] Step 1:

[0046] The server continuously receives video data from the monitoring device. Upon reception, the data is divided into frames for real-time processing and converted into a format that facilitates analysis.

[0047] Step 2:

[0048] The server applies a face recognition algorithm to the divided video frames. This algorithm detects human faces within the frames and compares them to registered known targets.

[0049] Step 3:

[0050] The server uses dynamic analysis technology to detect abnormal movements by comparing them to normal operating patterns. It identifies unnatural movements, such as rapid movement or intrusion into unauthorized areas.

[0051] Step 4:

[0052] When abnormal behavior is detected, the server immediately issues an alarm and sends a corresponding notification to an external terminal. This action is taken to quickly draw attention to potential threats.

[0053] Step 5:

[0054] The device displays alert notifications to the user in real time. Users are provided with the ability to check detailed information and live video on the device and determine whether an emergency response is necessary.

[0055] Step 6:

[0056] Based on the displayed information, users can decide whether to contact a security company or the police as needed. This ensures that an on-site response is carried out as quickly as possible.

[0057] Step 7:

[0058] The server uses generation technology to summarize the analysis results and stores them in a recording device. This summarized data is used later for further detailed analysis and reporting as needed.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] In recent years, with the increasing need for crime prevention and surveillance, there is a demand for systems that can detect abnormal behavior in real time and respond quickly. However, current systems suffer from problems such as delays in data processing and false positives, resulting in a lack of immediacy and accuracy. Furthermore, even when an anomaly is detected, there are insufficient means to quickly and clearly communicate that information to the user, raising concerns that effective crime prevention measures may be delayed. Therefore, a new crime prevention and surveillance system is needed to solve these problems.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes means for dividing video information received from a monitoring device and converting it into a format that can be analyzed in real time; means for performing object recognition and motion analysis on the converted video information to detect abnormal behavior; and means for issuing an alarm and transmitting a notification to an external computing device based on the detected abnormal behavior. This enables highly accurate detection of abnormal behavior in real time and rapid response notification.

[0064] A "surveillance device" is a device that captures images of a specific area and transmits that video information to a server in real time.

[0065] "Video information" refers to visual data acquired from surveillance devices, and is the element that will be analyzed.

[0066] "Object recognition" is a technology that identifies and judges specific patterns or shapes from video information.

[0067] "Dynamic analysis" is a technique that analyzes movement patterns in video information to detect anomalies.

[0068] "Abnormal behavior" refers to suspicious movements or actions that deviate from normal behavioral patterns, and is what the system detects.

[0069] "External computing device" refers to a digital device that the user can access after receiving a notification, and includes smartphones, personal computers, and other similar devices.

[0070] "Summary information" refers to information that concisely summarizes the results of the analysis and includes important data points.

[0071] A "storage device" is a digital device used to store analysis results and summary information for long periods of time, and includes hard disks and cloud storage.

[0072] "Real-time" refers to processing that occurs instantly, without any delay in the actual time that is happening.

[0073] In the system of this invention, the server, terminal, and user each play a specific role, and multi-layered processing is performed in real time.

[0074] The server receives video information acquired by the monitoring device in streaming format. The received data is divided into frames in real time using an open-source media framework (e.g., FFmpeg). During this process, noise is removed using an image processing library (e.g., OpenCV), and the resolution is adjusted as needed to achieve efficient data processing.

[0075] Next, the server employs an object recognition algorithm. Specifically, it uses a deep learning model utilizing a machine learning library (e.g., TENSORFLOW®) to detect human faces from the field within the video and identify known suspicious individuals by comparing them with a database. Furthermore, it applies motion analysis technology to detect irregular behavioral patterns within the video. Here, it also performs time-based data analysis to identify unusual movements.

[0076] When an anomaly is detected, the server immediately sends a notification to an external computing device. The terminal receives this notification in real time and displays the information on the user interface. The notification includes details of the detected anomaly and its location, allowing the user to quickly understand the situation and take appropriate action.

[0077] Furthermore, the server uses generation technology to summarize the analysis results and saves this summary information to a storage device. Users can access this information later and perform more detailed analysis as needed.

[0078] For example, if the system is activated while the user is away, and an intruder enters the property at night, the system will automatically detect this and immediately notify an external party. Upon receiving this notification, the user can view live video via their smartphone and report the incident to the security agency.

[0079] An example of a prompt when using a generative AI model is, "Please explain the real-time notification process when suspicious behavior is detected and how the user should respond afterward." This prompt is used to obtain detailed information about the system's operation and how to use it.

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] The server receives video information from the monitoring device in streaming format. The input is the video data provided by the monitoring device. The server uses an open-source media framework to divide this data into frames. The output is frame data that has undergone noise reduction and resolution adjustment, which streamlines the subsequent analysis process. Specifically, the server extracts important frames from the video and saves them in an optimized state.

[0083] Step 2:

[0084] The server performs object recognition on the divided frames. The input is the frame data obtained in step 1. The server detects faces in the frames using a deep learning model with a machine learning library and compares the face information with the database. The output is the detected face information and the matching result. Specifically, if a known suspicious person is found, the information is immediately recorded and passed on to the next process.

[0085] Step 3:

[0086] The server uses dynamic analysis technology to detect unnatural movements and behavioral patterns. The input consists of frame data adjusted in step 1 and face recognition results obtained in step 2. The server utilizes optical flow technology and neural networks to identify abnormal behavior. The output is a detailed description of the identified abnormal behavior. Specifically, upon detection of abnormal behavior, the server immediately prepares for the next action.

[0087] Step 4:

[0088] The server issues an alarm based on the detected anomaly. The input is the details of the abnormal behavior obtained in step 3. The server uses this to send a notification to an external computing device. The output is notification data containing details of the anomaly. In terms of specific actions, information suggesting concrete actions is delivered to the external computing device along with the alarm.

[0089] Step 5:

[0090] The terminal receives notifications sent from the server in real time and displays them on the user interface. The input is the notification data sent from the server in step 4. The terminal converts this into a user-friendly format and displays it. The output is information related to the abnormal behavior displayed to the user, allowing the user to take quick and appropriate action. Specifically, the terminal notifies the user of notifications with sound and vibration, and facilitates checking live video.

[0091] (Application Example 1)

[0092] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0093] Conventional monitoring systems have not adequately detected abnormal behavior in real time or provided rapid notification, posing a challenge to improving security. Furthermore, it was difficult for users to immediately obtain detailed information about abnormal behavior, potentially leading to delays in response. Therefore, there is a need for a system that improves monitoring accuracy and provides information to users quickly.

[0094] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0095] In this invention, the server includes means for segmenting visual information received from monitoring equipment and converting it into a format that can be immediately analyzed; means for performing individual recognition and motion analysis on the converted visual information to detect abnormal behavior; and means for providing details of the detected abnormal behavior and visual information to a display device. This makes it possible to quickly and accurately detect abnormal behavior by combining visual information and detection data, and to immediately convey detailed information to the user.

[0096] "Surveillance equipment" refers to devices used to acquire visual information, and includes cameras, etc.

[0097] "Visual information" refers to video and image data acquired from surveillance equipment.

[0098] "Real-time analysis" refers to the process of processing acquired data and making decisions immediately.

[0099] "Individual recognition" is a technology that identifies specific people or objects within visual information.

[0100] "Motion analysis" is a technique that analyzes visual information and evaluates the movement and behavior of objects depicted in it.

[0101] "Abnormal behavior" refers to irregular movements or actions that are not seen under normal circumstances, and may include criminal acts.

[0102] A "display device" is a device used to visually present information to a user, and includes smartphones and monitors.

[0103] "User interface" refers to the functions and screens that allow users to operate a system and obtain information.

[0104] "Instant display" means that information is presented to the user in some form the moment it is obtained.

[0105] This system aims to detect abnormal behavior by acquiring visual information through monitoring equipment and analyzing it in real time. In its implementation, the server, terminals, and users each play specific roles.

[0106] The server receives visual information from monitoring equipment and identifies faces using individual recognition software such as OpenCV. Furthermore, it utilizes motion analysis technology to analyze each frame of the acquired visual information and detect abnormal behavior. When an anomaly is detected, the server immediately generates an alarm and saves detailed data to a recording device. The analysis results are summarized using generation technology, and important information is organized.

[0107] The terminal provides users with notifications sent from the server. Specifically, it uses a display device such as a smartphone to present users with details and visual information about detected abnormal behavior. Through the user interface, users can quickly check the situation.

[0108] Based on the information provided through the device, users can take appropriate action by coordinating with external security managers and the police as needed.

[0109] As a concrete example, in one household, by activating the system while they are away for an extended period, they can receive a notification the moment a suspicious person approaches. The user can then check the visual information from the camera installed in their home on their smartphone and promptly report it to the police.

[0110] An example of a prompt message for a generated AI model would be: "If a suspicious person is detected in the surveillance camera footage, please advise on how to notify the user so that they can respond more quickly."

[0111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0112] Step 1:

[0113] The server receives visual information from surveillance equipment in real time. It takes a video stream from surveillance cameras as input. The received video is divided into frames, and this data is prepared for processing. The divided frame data is denoised and the resolution is adjusted.

[0114] Step 2:

[0115] The server applies individual recognition technology to the adjusted frame data. Specifically, it uses face recognition algorithms such as OpenCV to identify faces within each frame. The corrected frame data is used as input, and the output is the result of matching the detected face positions and features against a database.

[0116] Step 3:

[0117] The server uses motion analysis technology to analyze motion patterns within frames. It uses frame data as input and performs data calculations to detect unnatural movements and abnormal behavior. The output is a determination of whether or not an anomaly was detected.

[0118] Step 4:

[0119] When an anomaly is detected, the server immediately generates an alarm and creates notification data. Specifically, it constructs an alarm message containing information such as the type of anomaly, the location and time of detection, etc. The output is a notification message.

[0120] Step 5:

[0121] The server sends the generated alarm to an external terminal. The user's terminal receives the notification message. It receives the alarm message as input and provides notification information to the user as output.

[0122] Step 6:

[0123] The device displays notification details and real-time visual information to the user. It uses alarm messages and live video streams as inputs and visually presents the information in the user interface as output.

[0124] Step 7:

[0125] The user makes quick decisions based on the information presented. If necessary, they can take action to collaborate with external security organizations. The input involves referencing information provided by the terminal, and the output involves making decisions that lead to appropriate countermeasures.

[0126] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0127] The system of this invention analyzes video data from a monitoring device to detect abnormal behavior and incorporates an emotion engine to recognize the user's emotional state. This aims to support the user from both a security and safety perspective by monitoring not only abnormal behavior but also changes in the user's emotions.

[0128] The server processes video data acquired from the monitoring device in real time, dividing it into frames. This processing enables efficient face recognition and motion analysis. The server uses these analyses to identify abnormal behavior and simultaneously analyzes the user's facial expressions using an emotion engine. The emotion engine extracts the user's facial features from the video frames and executes an algorithm to evaluate their emotional state.

[0129] If an emotional state exceeds a certain threshold, for example, if high levels of anxiety or tension are detected, the server immediately issues an alarm and sends an alert notification to the terminal. This notification includes not only information about abnormal behavior but also the results of the emotional analysis, helping the user to gain a more comprehensive understanding of the situation.

[0130] The device displays received real-time notifications on the user interface, providing information to the user. This allows the user to take quick and appropriate action based on the emotional information obtained from the system. For example, if an emotionally unstable state is detected, the user can consider ways to relax or, if necessary, take steps to seek professional help.

[0131] As a concrete example, when a user is home alone at night, the system monitors the user's emotions in addition to normal surveillance. If a sudden anomaly is detected and the user becomes surprised and anxious, the emotion engine immediately recognizes this and a notification is displayed on the device. As a result, the user can regain their composure and quickly take appropriate countermeasures. In this way, the system can improve security and psychological safety in real time.

[0132] The following describes the processing flow.

[0133] Step 1:

[0134] The server receives video data sent from the monitoring device. The received data is divided into frames in real time and preprocessed for analysis.

[0135] Step 2:

[0136] The server applies a face recognition algorithm to the divided video frames. This algorithm detects facial features within the frames and identifies specific individuals by comparing them with an existing database.

[0137] Step 3:

[0138] The server uses dynamic analysis technology to detect abnormal behavior. This includes a process of identifying unusual movements and unexpected behavioral patterns and comparing them to normal behavior.

[0139] Step 4:

[0140] The server uses an emotion engine to analyze the user's emotional state. It analyzes subtle changes in the user's facial expressions from video frames to identify their emotional state.

[0141] Step 5:

[0142] When abnormal behavior or specific emotional states are detected, the server issues an alarm and sends a notification to an external terminal. The notification includes information about the detected anomaly and the results of an analysis of the emotional state.

[0143] Step 6:

[0144] The device displays notifications sent from the server to the user in real time. The user can understand the situation by reviewing the presented information and take appropriate action.

[0145] Step 7:

[0146] Users adjust their behavior based on their emotional state to ensure security measures are in place. For example, if a state of tension persists, they can contact an external professional organization, enabling a swift response.

[0147] (Example 2)

[0148] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0149] In recent years, monitoring systems have been required to detect abnormal behavior quickly and accurately, but simple motion analysis alone is insufficient to detect all anomalies. Furthermore, monitoring changes in users' emotional states is necessary to further enhance safety. However, there is a lack of efficient methods for doing this in real time.

[0150] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0151] In this invention, the server includes means for dividing video signals acquired from a monitoring device into frames and converting them into an analyzable format, means for performing face recognition and motion analysis using a machine learning model to detect abnormal behavior, and means for evaluating the user's emotional state in real time using a deep learning model. This improves the accuracy of detecting abnormal behavior and simultaneously allows for monitoring changes in the user's emotional state.

[0152] A "monitoring device" is a device used to acquire video signals and plays a role in monitoring the surrounding situation in real time.

[0153] A "video signal" is digital data containing visual information acquired by devices such as cameras.

[0154] A "frame" refers to each individual still image that makes up a video, and is the smallest unit that can be individually analyzed.

[0155] An "analyzable format" refers to a data format in which the data has been processed and converted into a state suitable for facial recognition and motion analysis.

[0156] A "machine learning model" is an algorithm that learns patterns based on past data and makes predictions and decisions based on new data.

[0157] "Facial recognition" is a technology that identifies the faces of people in a video signal and extracts their features.

[0158] "Motion analysis" is the process of analyzing a person's movements and behavior from video signals to detect abnormal patterns.

[0159] "Abnormal behavior" refers to behavior or actions that deviate from a predetermined normal pattern.

[0160] A "deep learning model" is a machine learning technique that uses multi-layered neural networks to extract features from data and recognize patterns.

[0161] "User's emotional state" refers to the psychological state and emotional changes inferred from the user's facial expressions and mucosal movements.

[0162] "Real-time" refers to a time frame in which data is processed and information is provided with extremely little delay, almost instantaneously.

[0163] This invention is a system that processes video signals acquired from a monitoring device in real time and monitors both safety and the emotional state of the user.

[0164] The server first receives a video signal from the monitoring device. This video signal is then divided into frames and converted into an analyzable format. Based on the divided frames, the server uses a machine learning model to perform face recognition and motion analysis. If abnormal behavior is detected, a deep learning model is used to extract the user's facial features and evaluate their emotional state. This process utilizes common open-source libraries such as TensorFlow and OpenCV.

[0165] If a user's emotional state exceeds a certain threshold, the server immediately issues an alarm and sends a notification to the terminal. At the same time, the generating AI model summarizes the analysis results and records them in a data storage device. The terminal displays the received notification in a graphical user interface, providing the user with the information. This allows the user to quickly and appropriately choose an action based on the information obtained from the system.

[0166] As a concrete example, while a user is home alone overnight, the system monitors the user's emotions in addition to normal surveillance. If any abnormality is detected and the user becomes emotionally unstable, the emotion engine immediately senses this and displays an appropriate notification on the device, allowing the user to regain a sense of security and take appropriate action quickly.

[0167] Examples of prompts include "Please describe the conditions for identifying abnormal behavior" and "Please provide information on technologies for evaluating a user's emotional state in real time."

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] The server receives video signals from the monitoring device. The input consists of real-time video signals from multiple cameras. The server receives these video signals and converts them into an analyzable format by dividing them into frames. This prepares reference data for detailed analysis of each frame.

[0171] Step 2:

[0172] The server receives video data divided into frames as input and performs face recognition and motion analysis using a machine learning model. This process extracts the facial features and motion patterns of people in each frame as output. Specifically, the server uses TensorFlow or similar tools to run the model and generate the data necessary for detecting abnormal behavior.

[0173] Step 3:

[0174] The server uses a deep learning model to evaluate the user's emotional state, taking the results of motion analysis as input. Specifically, the server extracts facial expressions and subtle body movements and classifies them into emotional categories. The output of this process is detailed data about the user's emotional state.

[0175] Step 4:

[0176] The server triggers an alarm when certain criteria are exceeded based on the evaluation results of emotional state and abnormal behavior. The input here is emotional state and behavioral analysis data as analysis results, and the server uses this data to determine the need for an alarm. As output, an alarm is decided to be issued and a notification is generated.

[0177] Step 5:

[0178] The server sends the generated alarm notification to the mobile terminal. The input here is the alarm notification data, which the server sends without delay using a real-time protocol. The output is the alarm message displayed on the user interface.

[0179] Step 6:

[0180] The terminal visualizes alarm notifications received from the server on a user interface and provides information to the user. The input is alarm notification data, and the terminal displays this as output on the screen. Specifically, it presents various countermeasures to the user based on the content of the notification.

[0181] Step 7:

[0182] The user selects an appropriate response based on the notification displayed on the device's user interface. The input is the alert information on the user interface, and based on this, the user decides on an action as the output. Specific actions include considering ways to promote relaxation and contacting a specialist if necessary.

[0183] (Application Example 2)

[0184] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0185] Modern security and safety systems often focus solely on detecting abnormal behavior, failing to consider changes in the user's emotional state. As a result, unstable emotional states can lead to system errors, potentially compromising user safety. This invention aims to provide a system that, in addition to detecting abnormal behavior, monitors changes in the user's emotions in real time, enabling appropriate responses.

[0186] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0187] In this invention, the server includes means for dividing video data received from a monitoring device and converting it into a format that can be analyzed in real time; means for performing face recognition and motion analysis on the converted video data to detect abnormal behavior; means for evaluating the emotional state based on the user's facial information and detecting changes in emotion; means for issuing an alarm and sending a notification to an external terminal based on the detected abnormal behavior and changes in emotion; means for summarizing the analysis results using generation technology and storing the summarized data in a recording device; and means for providing a user interface that provides information to help the user understand the situation and encourage safe behavior. This enables rapid and appropriate security measures that respond not only to abnormal behavior but also to changes in the user's emotions.

[0188] A "monitoring device" is a device that acquires video data and transmits it to a server.

[0189] "Video data" refers to a collection of visual information acquired from surveillance devices, which is used as material for evaluating abnormal behavior and emotional states through analysis.

[0190] "Analyzable format" refers to a state in which video data has been converted into a data format necessary for efficient analysis.

[0191] "Facial recognition" is a technology that identifies a person's face from video data and extracts its features.

[0192] "Dynamic analysis" is a technique for capturing movement and changes in video data to identify abnormal behavior.

[0193] "Abnormal behavior" refers to unnatural movements or actions in a subject under surveillance that differ from normal behavior.

[0194] "Emotional state" refers to the psychological condition inferred from a person's facial expressions and body movements.

[0195] "Emotional change" refers to an emotional state that changes over time.

[0196] An "alarm" is a notification issued to warn of an anomaly that has been detected.

[0197] An "external terminal" is a device used to receive and display alarms and notifications.

[0198] "Generative technology" refers to all technologies used to summarize analysis results and record data.

[0199] "Summary data" refers to data that concisely summarizes the analyzed information.

[0200] A "recording device" is a medium or device used to store summary data.

[0201] A "user interface" refers to the screens and means of operation that a user uses to interact with a device.

[0202] This invention constructs a system for detecting abnormal behavior and changes in emotional state. The system consists of the following components:

[0203] The server receives real-time video data acquired from the monitoring device and divides the data into frames. This video data is converted into an analyzable format and processed using video analysis software such as OpenCV. A face recognition algorithm identifies faces in each frame and extracts their features. Then, using an emotion recognition engine such as Microsoft® Azure® Face API or Google® Cloud Vision API, the system analyzes the user's emotional state and detects emotional changes such as anxiety and tension.

[0204] The server then performs further dynamic analysis to identify abnormal behavior. If the set thresholds for abnormal behavior or emotion are exceeded, the system quickly issues an alarm and sends an alert to an external device via a notification service such as Firebase Cloud Messaging. Users can receive the notification on their device and view detailed information through a dedicated user interface.

[0205] This system allows users to understand the situation appropriately and quickly, and take safety measures as needed. A concrete example is the use of smart glasses for safety assistance when returning home late at night. If a user encounters an unexpected situation, the system immediately detects their emotional state and abnormal behavior, and provides information to encourage a calm response.

[0206] An example of a prompt for a generative AI model is: "Design an AI assistant that checks the surroundings when returning home late at night and detects abnormal behavior or emotional changes. Notifications should be sent to the smart device to encourage safe behavior." In this way, it is expected that safety and psychological reassurance will be improved.

[0207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0208] Step 1:

[0209] The server acquires video data in real time from the monitoring device. It receives raw video data as input. This data is divided into frames and converted into an analyzable format to produce output. Specifically, a video editing library is used to perform the frame division.

[0210] Step 2:

[0211] The server performs face recognition on the converted frame-by-frame video data using OpenCV. The input is video data divided into frames. The server analyzes this data, identifies face information within each frame, and extracts its features. The output is the extracted face feature data. Specifically, it detects face regions and records their coordinates and feature points.

[0212] Step 3:

[0213] The server inputs the extracted facial feature data into the emotion recognition engine to evaluate the user's emotional state. The input is facial feature data. Using the Microsoft Azure Face API, the emotional state is identified, and emotions such as anxiety and tension are specified. The output is the evaluation result of the emotional state. Specifically, the API is called and the returned emotion score is analyzed.

[0214] Step 4:

[0215] The server simultaneously performs dynamic analysis on the video data. The input is video data frame by frame. It analyzes movements and changes to identify abnormal behavior and detects actions that exceed the judgment criteria. The output is the result of detecting abnormal behavior. Specifically, it performs vector analysis of movement to identify patterns of movement that are different from normal.

[0216] Step 5:

[0217] The server sends an alert to the device via Firebase Cloud Messaging based on the results of emotional changes and abnormal behavior. The inputs are the results of the emotional state assessment and the abnormal behavior detection. This information is aggregated, an alert is generated, and sent to the device. The output is the alert notification to the device. Specifically, the information is embedded in an alert template to construct the notification message.

[0218] Step 6:

[0219] The terminal displays received alarms on the user interface. The input is the alarm notification sent from the server. The terminal analyzes the alarm content and presents it to the user in an easy-to-understand manner. The output is the displayed alarm information. Specifically, it displays a pop-up on the screen to provide detailed information.

[0220] Step 7:

[0221] The user understands the situation based on the displayed information and considers countermeasures. The input is the alarm information displayed on the terminal. The user can evaluate the options and take additional actions as needed to take safe actions. The output is the selection of safety measures. Specifically, the user follows the displayed instructions and takes appropriate action.

[0222] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0223] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0224] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0225] [Second Embodiment]

[0226] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0227] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0228] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0229] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0230] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0232] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0233] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0234] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0235] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0236] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0237] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0238] The system of this invention aims to process video data acquired from a monitoring device in real time and to quickly detect abnormal behavior. This system is configured based on the respective roles of server, terminal, and user.

[0239] The server receives video data in streaming format from multiple monitoring devices. The received data is divided into frames in real time and pre-processed. Specifically, noise in the video is removed and the resolution is adjusted as needed to improve the efficiency of the subsequent analysis process. Next, the server activates a facial recognition algorithm to detect human faces frame by frame and compare them with a database. This process enables the identification of known suspicious individuals and the detection of new anomaly patterns.

[0240] Furthermore, the server uses dynamic analysis technology to detect abnormal behavior. For example, it identifies unnatural movements and behavioral patterns that differ from normal human movement. This enables a rapid response to prevent criminal activity.

[0241] If an anomaly is detected, the server issues an alarm and immediately sends a notification to the terminal. The terminal is responsible for providing these notifications to the user in real time. Based on the information provided, the user can take the most appropriate security measures. The terminal also provides access to detailed information about the detected anomaly and live video, allowing the user to understand the situation more accurately.

[0242] Furthermore, the server utilizes generation technology to summarize the analyzed data. This summary includes a timeline of specific events and highlights of important scenes, and is saved to a recording device. The saved data can be accessed by users at a later date and used as reference material for detailed analysis and re-evaluation.

[0243] For example, when a user is away from home for an extended period, activating the system will detect suspicious activity during the night and immediately send a notification to the user's device. This allows the user to contact external organizations such as the police or security companies and take swift action. In this way, the system provides a practical solution that contributes to the efficiency of crime prevention and the improvement of security.

[0244] The following describes the processing flow.

[0245] Step 1:

[0246] The server continuously receives video data from the monitoring device. Upon reception, the data is divided into frames for real-time processing and converted into a format that facilitates analysis.

[0247] Step 2:

[0248] The server applies a face recognition algorithm to the divided video frames. This algorithm detects human faces within the frames and compares them to registered known targets.

[0249] Step 3:

[0250] The server uses dynamic analysis technology to detect abnormal movements by comparing them to normal operating patterns. It identifies unnatural movements, such as rapid movement or intrusion into unauthorized areas.

[0251] Step 4:

[0252] When abnormal behavior is detected, the server immediately issues an alarm and sends a corresponding notification to an external terminal. This action is taken to quickly draw attention to potential threats.

[0253] Step 5:

[0254] The device displays alert notifications to the user in real time. Users are provided with the ability to check detailed information and live video on the device and determine whether an emergency response is necessary.

[0255] Step 6:

[0256] Based on the displayed information, users can decide whether to contact a security company or the police as needed. This ensures that an on-site response is carried out as quickly as possible.

[0257] Step 7:

[0258] The server uses generation technology to summarize the analysis results and stores them in a recording device. This summarized data is used later for further detailed analysis and reporting as needed.

[0259] (Example 1)

[0260] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0261] In recent years, with the increasing need for crime prevention and surveillance, there is a demand for systems that can detect abnormal behavior in real time and respond quickly. However, current systems suffer from problems such as delays in data processing and false positives, resulting in a lack of immediacy and accuracy. Furthermore, even when an anomaly is detected, there are insufficient means to quickly and clearly communicate that information to the user, raising concerns that effective crime prevention measures may be delayed. Therefore, a new crime prevention and surveillance system is needed to solve these problems.

[0262] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0263] In this invention, the server includes means for dividing video information received from a monitoring device and converting it into a format that can be analyzed in real time; means for performing object recognition and motion analysis on the converted video information to detect abnormal behavior; and means for issuing an alarm and transmitting a notification to an external computing device based on the detected abnormal behavior. This enables highly accurate detection of abnormal behavior in real time and rapid response notification.

[0264] A "surveillance device" is a device that captures images of a specific area and transmits that video information to a server in real time.

[0265] "Video information" refers to visual data acquired from surveillance devices, and is the element that will be analyzed.

[0266] "Object recognition" is a technology that identifies and judges specific patterns or shapes from video information.

[0267] "Dynamic analysis" is a technique that analyzes movement patterns in video information to detect anomalies.

[0268] "Abnormal behavior" refers to suspicious movements or actions that deviate from normal behavioral patterns, and is what the system detects.

[0269] "External computing device" refers to a digital device that the user can access after receiving a notification, and includes smartphones, personal computers, and other similar devices.

[0270] "Summary information" refers to information that concisely summarizes the results of the analysis and includes important data points.

[0271] A "storage device" is a digital device used to store analysis results and summary information for long periods of time, and includes hard disks and cloud storage.

[0272] "Real-time" refers to processing that occurs instantly, without any delay in the actual time that is happening.

[0273] In the system of this invention, the server, terminal, and user each play a specific role, and multi-layered processing is performed in real time.

[0274] The server receives video information acquired by the monitoring device in streaming format. The received data is divided into frames in real time using an open-source media framework (e.g., FFmpeg). During this process, noise is removed using an image processing library (e.g., OpenCV), and the resolution is adjusted as needed to achieve efficient data processing.

[0275] Next, the server employs an object recognition algorithm. Specifically, it uses a deep learning model powered by a machine learning library (e.g., TensorFlow) to detect human faces from the video field and identify known suspicious individuals by comparing them with a database. Furthermore, it applies motion analysis technology to detect irregular behavioral patterns in the video. Here, it also performs time-based data analysis to identify unusual movements.

[0276] When an anomaly is detected, the server immediately sends a notification to an external computing device. The terminal receives this notification in real time and displays the information on the user interface. The notification includes details of the detected anomaly and its location, allowing the user to quickly understand the situation and take appropriate action.

[0277] Furthermore, the server uses generation technology to summarize the analysis results and saves this summary information to a storage device. Users can access this information later and perform more detailed analysis as needed.

[0278] For example, if the system is activated while the user is away, and an intruder enters the property at night, the system will automatically detect this and immediately notify an external party. Upon receiving this notification, the user can view live video via their smartphone and report the incident to the security agency.

[0279] An example of a prompt when using a generative AI model is, "Please explain the real-time notification process when suspicious behavior is detected and how the user should respond afterward." This prompt is used to obtain detailed information about the system's operation and how to use it.

[0280] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0281] Step 1:

[0282] The server receives video information in streaming format from the monitoring device. The input is the video data provided by the monitoring device. The server uses an open-source media framework to split this data into frames. The output is frame data that has been subjected to noise removal and resolution adjustment, thereby streamlining the subsequent analysis process. As a specific operation, the server extracts important frames from the video and stores them in an optimized state.

[0283] Step 2:

[0284] The server performs object recognition on the split frames. The input is the frame data obtained in Step 1. The server uses a deep learning model with a machine learning library to detect faces within the frames and match the face information with a database. The output is the detected face information and the matching result. As a specific operation, if a known suspicious person is found, the information is immediately recorded and passed on to subsequent processing.

[0285] Step 3:

[0286] The server uses dynamic analysis technology to detect unnatural movements and behavioral patterns. The input is the frame data adjusted in Step 1 and the face recognition result obtained in Step 2. The server utilizes optical flow technology and neural networks to distinguish abnormal behaviors. The output is the details of the identified abnormal behaviors. As a specific operation, when an abnormal behavior is detected, the actions for the next step are immediately prepared.

[0287] Step 4:

[0288] The server issues an alarm based on the detected abnormality. The input is the details of the abnormal behavior obtained in Step 3. The server uses this as a basis to send a notification to an external computing device. The output is notification data containing the details of the abnormality. As a specific operation, information proposing specific actions along with the alarm is delivered to the external computing device.

[0289] Step 5:

[0290] The terminal receives notifications sent from the server in real time and displays them on the user interface. The input is the notification data sent from the server in step 4. The terminal converts this into a user-friendly format and displays it. The output is information related to the abnormal behavior displayed to the user, allowing the user to take quick and appropriate action. Specifically, the terminal notifies the user of notifications with sound and vibration, and facilitates checking live video.

[0291] (Application Example 1)

[0292] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0293] Conventional monitoring systems have not adequately detected abnormal behavior in real time or provided rapid notification, posing a challenge to improving security. Furthermore, it was difficult for users to immediately obtain detailed information about abnormal behavior, potentially leading to delays in response. Therefore, there is a need for a system that improves monitoring accuracy and provides information to users quickly.

[0294] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0295] In this invention, the server includes means for segmenting visual information received from monitoring equipment and converting it into a format that can be immediately analyzed; means for performing individual recognition and motion analysis on the converted visual information to detect abnormal behavior; and means for providing details of the detected abnormal behavior and visual information to a display device. This makes it possible to quickly and accurately detect abnormal behavior by combining visual information and detection data, and to immediately convey detailed information to the user.

[0296] "Surveillance equipment" refers to devices used to acquire visual information, and includes cameras, etc.

[0297] "Visual information" refers to video and image data acquired from surveillance equipment.

[0298] "Real-time analysis" refers to the process of processing acquired data and making decisions immediately.

[0299] "Individual recognition" is a technology that identifies specific people or objects within visual information.

[0300] "Motion analysis" is a technique that analyzes visual information and evaluates the movement and behavior of objects depicted in it.

[0301] "Abnormal behavior" refers to irregular movements or actions that are not seen under normal circumstances, and may include criminal acts.

[0302] A "display device" is a device used to visually present information to a user, and includes smartphones and monitors.

[0303] "User interface" refers to the functions and screens that allow users to operate a system and obtain information.

[0304] "Instant display" means that information is presented to the user in some form the moment it is obtained.

[0305] This system aims to detect abnormal behavior by acquiring visual information through monitoring equipment and analyzing it in real time. In its implementation, the server, terminals, and users each play specific roles.

[0306] The server receives visual information from monitoring equipment and identifies faces using individual recognition software such as OpenCV. Furthermore, it utilizes motion analysis technology to analyze each frame of the acquired visual information and detect abnormal behavior. When an anomaly is detected, the server immediately generates an alarm and saves detailed data to a recording device. The analysis results are summarized using generation technology, and important information is organized.

[0307] The terminal provides the user with the notifications sent from the server. Specifically, it plays the role of presenting the user with details of the detected abnormal behavior and visual information using a display device such as a smartphone. Through the user interface, the user can quickly check the situation.

[0308] Based on the information provided through the terminal, the user can, if necessary, cooperate with external security administrators or the police and take appropriate actions.

[0309] As a specific example, in a certain household, by operating the system while being away for a long time, a notification can be received the moment a suspicious person approaches. The user can check the visual information of the camera installed at home on the smartphone and promptly report it to the police.

[0310] As an example of the prompt text for the generative AI model, it is used in the form of "When a suspicious person is detected from the video of the surveillance camera, please advise how to notify the user more quickly for a response."

[0311] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0312] Step 1:

[0313] The server receives visual information from the monitoring device in real time. As input, it acquires the video stream from the surveillance camera. The received video is split into frames, and preliminary preparations for processing this data are made. The split frame data is denoised and the resolution is adjusted.

[0314] Step 2:

[0315] The server applies individual recognition technology to the adjusted frame data. Specifically, it uses face recognition algorithms such as OpenCV to identify faces within each frame. The corrected frame data is used as input, and the output is the result of matching the detected face positions and features against a database.

[0316] Step 3:

[0317] The server uses motion analysis technology to analyze motion patterns within frames. It uses frame data as input and performs data calculations to detect unnatural movements and abnormal behavior. The output is a determination of whether or not an anomaly was detected.

[0318] Step 4:

[0319] When an anomaly is detected, the server immediately generates an alarm and creates notification data. Specifically, it constructs an alarm message containing information such as the type of anomaly, the location and time of detection, etc. The output is a notification message.

[0320] Step 5:

[0321] The server sends the generated alarm to an external terminal. The user's terminal receives the notification message. It receives the alarm message as input and provides notification information to the user as output.

[0322] Step 6:

[0323] The device displays notification details and real-time visual information to the user. It uses alarm messages and live video streams as inputs and visually presents the information in the user interface as output.

[0324] Step 7:

[0325] The user makes quick decisions based on the information presented. If necessary, they can take action to collaborate with external security organizations. The input involves referencing information provided by the terminal, and the output involves making decisions that lead to appropriate countermeasures.

[0326] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0327] The system of this invention analyzes video data from a monitoring device to detect abnormal behavior and incorporates an emotion engine to recognize the user's emotional state. This aims to support the user from both a security and safety perspective by monitoring not only abnormal behavior but also changes in the user's emotions.

[0328] The server processes video data acquired from the monitoring device in real time, dividing it into frames. This processing enables efficient face recognition and motion analysis. The server uses these analyses to identify abnormal behavior and simultaneously analyzes the user's facial expressions using an emotion engine. The emotion engine extracts the user's facial features from the video frames and executes an algorithm to evaluate their emotional state.

[0329] If an emotional state exceeds a certain threshold, for example, if high levels of anxiety or tension are detected, the server immediately issues an alarm and sends an alert notification to the terminal. This notification includes not only information about abnormal behavior but also the results of the emotional analysis, helping the user to gain a more comprehensive understanding of the situation.

[0330] The device displays received real-time notifications on the user interface, providing information to the user. This allows the user to take quick and appropriate action based on the emotional information obtained from the system. For example, if an emotionally unstable state is detected, the user can consider ways to relax or, if necessary, take steps to seek professional help.

[0331] As a concrete example, when a user is home alone at night, the system monitors the user's emotions in addition to normal surveillance. If a sudden anomaly is detected and the user becomes surprised and anxious, the emotion engine immediately recognizes this and a notification is displayed on the device. As a result, the user can regain their composure and quickly take appropriate countermeasures. In this way, the system can improve security and psychological safety in real time.

[0332] The following describes the processing flow.

[0333] Step 1:

[0334] The server receives video data sent from the monitoring device. The received data is divided into frames in real time and preprocessed for analysis.

[0335] Step 2:

[0336] The server applies a face recognition algorithm to the divided video frames. This algorithm detects facial features within the frames and identifies specific individuals by comparing them with an existing database.

[0337] Step 3:

[0338] The server uses dynamic analysis technology to detect abnormal behavior. This includes a process of identifying unusual movements and unexpected behavioral patterns and comparing them to normal behavior.

[0339] Step 4:

[0340] The server uses an emotion engine to analyze the user's emotional state. It analyzes subtle changes in the user's facial expressions from video frames to identify their emotional state.

[0341] Step 5:

[0342] When abnormal behavior or specific emotional states are detected, the server issues an alarm and sends a notification to an external terminal. The notification includes information about the detected anomaly and the results of an analysis of the emotional state.

[0343] Step 6:

[0344] The device displays notifications sent from the server to the user in real time. The user can understand the situation by reviewing the presented information and take appropriate action.

[0345] Step 7:

[0346] Users adjust their behavior based on their emotional state to ensure security measures are in place. For example, if a state of tension persists, they can contact an external professional organization, enabling a swift response.

[0347] (Example 2)

[0348] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0349] In recent years, monitoring systems have been required to detect abnormal behavior quickly and accurately, but simple motion analysis alone is insufficient to detect all anomalies. Furthermore, monitoring changes in users' emotional states is necessary to further enhance safety. However, there is a lack of efficient methods for doing this in real time.

[0350] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0351] In this invention, the server includes means for dividing video signals acquired from a monitoring device into frames and converting them into an analyzable format, means for performing face recognition and motion analysis using a machine learning model to detect abnormal behavior, and means for evaluating the user's emotional state in real time using a deep learning model. This improves the accuracy of detecting abnormal behavior and simultaneously allows for monitoring changes in the user's emotional state.

[0352] A "monitoring device" is a device used to acquire video signals and plays a role in monitoring the surrounding situation in real time.

[0353] A "video signal" is digital data containing visual information acquired by devices such as cameras.

[0354] A "frame" refers to each individual still image that makes up a video, and is the smallest unit that can be individually analyzed.

[0355] An "analyzable format" refers to a data format in which the data has been processed and converted into a state suitable for facial recognition and motion analysis.

[0356] A "machine learning model" is an algorithm that learns patterns based on past data and makes predictions and decisions based on new data.

[0357] "Facial recognition" is a technology that identifies the faces of people in a video signal and extracts their features.

[0358] "Motion analysis" is the process of analyzing a person's movements and behavior from video signals to detect abnormal patterns.

[0359] "Abnormal behavior" refers to behavior or actions that deviate from a predetermined normal pattern.

[0360] A "deep learning model" is a machine learning technique that uses multi-layered neural networks to extract features from data and recognize patterns.

[0361] "User's emotional state" refers to the psychological state and emotional changes inferred from the user's facial expressions and mucosal movements.

[0362] "Real-time" refers to a time frame in which data is processed and information is provided with extremely little delay, almost instantaneously.

[0363] This invention is a system that processes video signals acquired from a monitoring device in real time and monitors both safety and the emotional state of the user.

[0364] The server first receives a video signal from the monitoring device. This video signal is then divided into frames and converted into an analyzable format. Based on the divided frames, the server uses a machine learning model to perform face recognition and motion analysis. If abnormal behavior is detected, a deep learning model is used to extract the user's facial features and evaluate their emotional state. This process utilizes common open-source libraries such as TensorFlow and OpenCV.

[0365] If a user's emotional state exceeds a certain threshold, the server immediately issues an alarm and sends a notification to the terminal. At the same time, the generating AI model summarizes the analysis results and records them in a data storage device. The terminal displays the received notification in a graphical user interface, providing the user with the information. This allows the user to quickly and appropriately choose an action based on the information obtained from the system.

[0366] As a concrete example, while a user is home alone overnight, the system monitors the user's emotions in addition to normal surveillance. If any abnormality is detected and the user becomes emotionally unstable, the emotion engine immediately senses this and displays an appropriate notification on the device, allowing the user to regain a sense of security and take appropriate action quickly.

[0367] Examples of prompts include "Please describe the conditions for identifying abnormal behavior" and "Please provide information on technologies for evaluating a user's emotional state in real time."

[0368] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0369] Step 1:

[0370] The server receives video signals from the monitoring device. The input consists of real-time video signals from multiple cameras. The server receives these video signals and converts them into an analyzable format by dividing them into frames. This prepares reference data for detailed analysis of each frame.

[0371] Step 2:

[0372] The server receives video data divided into frames as input and performs face recognition and motion analysis using a machine learning model. This process extracts the facial features and motion patterns of people in each frame as output. Specifically, the server uses TensorFlow or similar tools to run the model and generate the data necessary for detecting abnormal behavior.

[0373] Step 3:

[0374] The server uses a deep learning model to evaluate the user's emotional state, taking the results of motion analysis as input. Specifically, the server extracts facial expressions and subtle body movements and classifies them into emotional categories. The output of this process is detailed data about the user's emotional state.

[0375] Step 4:

[0376] The server triggers an alarm when certain criteria are exceeded based on the evaluation results of emotional state and abnormal behavior. The input here is emotional state and behavioral analysis data as analysis results, and the server uses this data to determine the need for an alarm. As output, an alarm is decided to be issued and a notification is generated.

[0377] Step 5:

[0378] The server sends the generated alarm notification to the mobile terminal. The input here is the alarm notification data, which the server sends without delay using a real-time protocol. The output is the alarm message displayed on the user interface.

[0379] Step 6:

[0380] The terminal visualizes alarm notifications received from the server on a user interface and provides information to the user. The input is alarm notification data, and the terminal displays this as output on the screen. Specifically, it presents various countermeasures to the user based on the content of the notification.

[0381] Step 7:

[0382] The user selects an appropriate response based on the notification displayed on the device's user interface. The input is the alert information on the user interface, and based on this, the user decides on an action as the output. Specific actions include considering ways to promote relaxation and contacting a specialist if necessary.

[0383] (Application Example 2)

[0384] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0385] Modern security and safety systems often focus solely on detecting abnormal behavior, failing to consider changes in the user's emotional state. As a result, unstable emotional states can lead to system errors, potentially compromising user safety. This invention aims to provide a system that, in addition to detecting abnormal behavior, monitors changes in the user's emotions in real time, enabling appropriate responses.

[0386] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0387] In this invention, the server includes means for dividing video data received from a monitoring device and converting it into a format that can be analyzed in real time; means for performing face recognition and motion analysis on the converted video data to detect abnormal behavior; means for evaluating the emotional state based on the user's facial information and detecting changes in emotion; means for issuing an alarm and sending a notification to an external terminal based on the detected abnormal behavior and changes in emotion; means for summarizing the analysis results using generation technology and storing the summarized data in a recording device; and means for providing a user interface that provides information to help the user understand the situation and encourage safe behavior. This enables rapid and appropriate security measures that respond not only to abnormal behavior but also to changes in the user's emotions.

[0388] A "monitoring device" is a device that acquires video data and transmits it to a server.

[0389] "Video data" refers to a collection of visual information acquired from surveillance devices, which is used as material for evaluating abnormal behavior and emotional states through analysis.

[0390] "Analyzable format" refers to a state in which video data has been converted into a data format necessary for efficient analysis.

[0391] "Facial recognition" is a technology that identifies a person's face from video data and extracts its features.

[0392] "Dynamic analysis" is a technique for capturing movement and changes in video data to identify abnormal behavior.

[0393] "Abnormal behavior" refers to unnatural movements or actions in a subject under surveillance that differ from normal behavior.

[0394] "Emotional state" refers to the psychological condition inferred from a person's facial expressions and body movements.

[0395] "Emotional change" refers to an emotional state that changes over time.

[0396] An "alarm" is a notification issued to warn of an anomaly that has been detected.

[0397] An "external terminal" is a device used to receive and display alarms and notifications.

[0398] "Generative technology" refers to all technologies used to summarize analysis results and record data.

[0399] "Summary data" refers to data that concisely summarizes the analyzed information.

[0400] A "recording device" is a medium or device used to store summary data.

[0401] A "user interface" refers to the screens and means of operation that a user uses to interact with a device.

[0402] This invention constructs a system for detecting abnormal behavior and changes in emotional state. The system consists of the following components:

[0403] The server receives real-time video data acquired from the monitoring device and divides the data into frames. This video data is converted into an analyzable format and processed using video analysis software such as OpenCV. A face recognition algorithm identifies faces in each frame and extracts their features. Then, using an emotion recognition engine such as Microsoft Azure Face API or Google Cloud Vision API, the system analyzes the user's emotional state and detects emotional changes such as anxiety or tension.

[0404] The server then performs further dynamic analysis to identify abnormal behavior. If the set thresholds for abnormal behavior or emotion are exceeded, the system quickly issues an alarm and sends an alert to an external device via a notification service such as Firebase Cloud Messaging. Users can receive the notification on their device and view detailed information through a dedicated user interface.

[0405] This system allows users to understand the situation appropriately and quickly, and take safety measures as needed. A concrete example is the use of smart glasses for safety assistance when returning home late at night. If a user encounters an unexpected situation, the system immediately detects their emotional state and abnormal behavior, and provides information to encourage a calm response.

[0406] An example of a prompt for a generative AI model is: "Design an AI assistant that checks the surroundings when returning home late at night and detects abnormal behavior or emotional changes. Notifications should be sent to the smart device to encourage safe behavior." In this way, it is expected that safety and psychological reassurance will be improved.

[0407] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0408] Step 1:

[0409] The server acquires video data in real time from the monitoring device. It receives raw video data as input. This data is divided into frames and converted into an analyzable format to produce output. Specifically, a video editing library is used to perform the frame division.

[0410] Step 2:

[0411] The server performs face recognition on the converted frame-by-frame video data using OpenCV. The input is video data divided into frames. The server analyzes this data, identifies face information within each frame, and extracts its features. The output is the extracted face feature data. Specifically, it detects face regions and records their coordinates and feature points.

[0412] Step 3:

[0413] The server inputs the extracted facial feature data into the emotion recognition engine to evaluate the user's emotional state. The input is facial feature data. Using the Microsoft Azure Face API, the emotional state is identified, and emotions such as anxiety and tension are specified. The output is the evaluation result of the emotional state. Specifically, the API is called and the returned emotion score is analyzed.

[0414] Step 4:

[0415] The server simultaneously performs dynamic analysis on the video data. The input is video data frame by frame. It analyzes movements and changes to identify abnormal behavior and detects actions that exceed the judgment criteria. The output is the result of detecting abnormal behavior. Specifically, it performs vector analysis of movement to identify patterns of movement that are different from normal.

[0416] Step 5:

[0417] The server sends an alert to the device via Firebase Cloud Messaging based on the results of emotional changes and abnormal behavior. The inputs are the results of the emotional state assessment and the abnormal behavior detection. This information is aggregated, an alert is generated, and sent to the device. The output is the alert notification to the device. Specifically, the information is embedded in an alert template to construct the notification message.

[0418] Step 6:

[0419] The terminal displays received alarms on the user interface. The input is the alarm notification sent from the server. The terminal analyzes the alarm content and presents it to the user in an easy-to-understand manner. The output is the displayed alarm information. Specifically, it displays a pop-up on the screen to provide detailed information.

[0420] Step 7:

[0421] The user understands the situation based on the displayed information and considers countermeasures. The input is the alarm information displayed on the terminal. The user can evaluate the options and take additional actions as needed to take safe actions. The output is the selection of safety measures. Specifically, the user follows the displayed instructions and takes appropriate action.

[0422] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0423] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0424] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0425] [Third Embodiment]

[0426] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0427] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0428] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0429] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0430] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0431] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0432] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0433] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0434] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0435] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0436] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0437] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0438] The system of this invention aims to process video data acquired from a monitoring device in real time and to quickly detect abnormal behavior. This system is configured based on the respective roles of server, terminal, and user.

[0439] The server receives video data in streaming format from multiple monitoring devices. The received data is divided into frames in real time and pre-processed. Specifically, noise in the video is removed and the resolution is adjusted as needed to improve the efficiency of the subsequent analysis process. Next, the server activates a facial recognition algorithm to detect human faces frame by frame and compare them with a database. This process enables the identification of known suspicious individuals and the detection of new anomaly patterns.

[0440] Furthermore, the server uses dynamic analysis technology to detect abnormal behavior. For example, it identifies unnatural movements and behavioral patterns that differ from normal human movement. This enables a rapid response to prevent criminal activity.

[0441] If an anomaly is detected, the server issues an alarm and immediately sends a notification to the terminal. The terminal is responsible for providing these notifications to the user in real time. Based on the information provided, the user can take the most appropriate security measures. The terminal also provides access to detailed information about the detected anomaly and live video, allowing the user to understand the situation more accurately.

[0442] Furthermore, the server utilizes generation technology to summarize the analyzed data. This summary includes a timeline of specific events and highlights of important scenes, and is saved to a recording device. The saved data can be accessed by users at a later date and used as reference material for detailed analysis and re-evaluation.

[0443] For example, when a user is away from home for an extended period, activating the system will detect suspicious activity during the night and immediately send a notification to the user's device. This allows the user to contact external organizations such as the police or security companies and take swift action. In this way, the system provides a practical solution that contributes to the efficiency of crime prevention and the improvement of security.

[0444] The following describes the processing flow.

[0445] Step 1:

[0446] The server continuously receives video data from the monitoring device. Upon reception, the data is divided into frames for real-time processing and converted into a format that facilitates analysis.

[0447] Step 2:

[0448] The server applies a face recognition algorithm to the divided video frames. This algorithm detects human faces within the frames and compares them to registered known targets.

[0449] Step 3:

[0450] The server uses dynamic analysis technology to detect abnormal movements by comparing them to normal operating patterns. It identifies unnatural movements, such as rapid movement or intrusion into unauthorized areas.

[0451] Step 4:

[0452] When abnormal behavior is detected, the server immediately issues an alarm and sends a corresponding notification to an external terminal. This action is taken to quickly draw attention to potential threats.

[0453] Step 5:

[0454] The device displays alert notifications to the user in real time. Users are provided with the ability to check detailed information and live video on the device and determine whether an emergency response is necessary.

[0455] Step 6:

[0456] Based on the displayed information, users can decide whether to contact a security company or the police as needed. This ensures that an on-site response is carried out as quickly as possible.

[0457] Step 7:

[0458] The server uses generation technology to summarize the analysis results and stores them in a recording device. This summarized data is used later for further detailed analysis and reporting as needed.

[0459] (Example 1)

[0460] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0461] In recent years, with the increasing need for crime prevention and surveillance, there is a demand for systems that can detect abnormal behavior in real time and respond quickly. However, current systems suffer from problems such as delays in data processing and false positives, resulting in a lack of immediacy and accuracy. Furthermore, even when an anomaly is detected, there are insufficient means to quickly and clearly communicate that information to the user, raising concerns that effective crime prevention measures may be delayed. Therefore, a new crime prevention and surveillance system is needed to solve these problems.

[0462] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0463] In this invention, the server includes means for dividing video information received from a monitoring device and converting it into a format that can be analyzed in real time; means for performing object recognition and motion analysis on the converted video information to detect abnormal behavior; and means for issuing an alarm and transmitting a notification to an external computing device based on the detected abnormal behavior. This enables highly accurate detection of abnormal behavior in real time and rapid response notification.

[0464] A "surveillance device" is a device that captures images of a specific area and transmits that video information to a server in real time.

[0465] "Video information" refers to visual data acquired from surveillance devices, and is the element that will be analyzed.

[0466] "Object recognition" is a technology that identifies and judges specific patterns or shapes from video information.

[0467] "Dynamic analysis" is a technique that analyzes movement patterns in video information to detect anomalies.

[0468] "Abnormal behavior" refers to suspicious movements or actions that deviate from normal behavioral patterns, and is what the system detects.

[0469] "External computing device" refers to a digital device that the user can access after receiving a notification, and includes smartphones, personal computers, and other similar devices.

[0470] "Summary information" refers to information that concisely summarizes the results of the analysis and includes important data points.

[0471] A "storage device" is a digital device used to store analysis results and summary information for long periods of time, and includes hard disks and cloud storage.

[0472] "Real-time" refers to processing that occurs instantly, without any delay in the actual time that is happening.

[0473] In the system of this invention, the server, terminal, and user each play a specific role, and multi-layered processing is performed in real time.

[0474] The server receives video information acquired by the monitoring device in streaming format. The received data is divided into frames in real time using an open-source media framework (e.g., FFmpeg). During this process, noise is removed using an image processing library (e.g., OpenCV), and the resolution is adjusted as needed to achieve efficient data processing.

[0475] Next, the server employs an object recognition algorithm. Specifically, it uses a deep learning model powered by a machine learning library (e.g., TensorFlow) to detect human faces from the video field and identify known suspicious individuals by comparing them with a database. Furthermore, it applies motion analysis technology to detect irregular behavioral patterns in the video. Here, it also performs time-based data analysis to identify unusual movements.

[0476] When an anomaly is detected, the server immediately sends a notification to an external computing device. The terminal receives this notification in real time and displays the information on the user interface. The notification includes details of the detected anomaly and its location, allowing the user to quickly understand the situation and take appropriate action.

[0477] Furthermore, the server uses generation technology to summarize the analysis results and saves this summary information to a storage device. Users can access this information later and perform more detailed analysis as needed.

[0478] For example, if the system is activated while the user is away, and an intruder enters the property at night, the system will automatically detect this and immediately notify an external party. Upon receiving this notification, the user can view live video via their smartphone and report the incident to the security agency.

[0479] An example of a prompt when using a generative AI model is, "Please explain the real-time notification process when suspicious behavior is detected and how the user should respond afterward." This prompt is used to obtain detailed information about the system's operation and how to use it.

[0480] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0481] Step 1:

[0482] The server receives video information from the monitoring device in streaming format. The input is the video data provided by the monitoring device. The server uses an open-source media framework to divide this data into frames. The output is frame data that has undergone noise reduction and resolution adjustment, which streamlines the subsequent analysis process. Specifically, the server extracts important frames from the video and saves them in an optimized state.

[0483] Step 2:

[0484] The server performs object recognition on the divided frames. The input is the frame data obtained in step 1. The server detects faces in the frames using a deep learning model with a machine learning library and compares the face information with the database. The output is the detected face information and the matching result. Specifically, if a known suspicious person is found, the information is immediately recorded and passed on to the next process.

[0485] Step 3:

[0486] The server uses dynamic analysis technology to detect unnatural movements and behavioral patterns. The input consists of frame data adjusted in step 1 and face recognition results obtained in step 2. The server utilizes optical flow technology and neural networks to identify abnormal behavior. The output is a detailed description of the identified abnormal behavior. Specifically, upon detection of abnormal behavior, the server immediately prepares for the next action.

[0487] Step 4:

[0488] The server issues an alarm based on the detected anomaly. The input is the details of the abnormal behavior obtained in step 3. The server uses this to send a notification to an external computing device. The output is notification data containing details of the anomaly. In terms of specific actions, information suggesting concrete actions is delivered to the external computing device along with the alarm.

[0489] Step 5:

[0490] The terminal receives notifications sent from the server in real time and displays them on the user interface. The input is the notification data sent from the server in step 4. The terminal converts this into a user-friendly format and displays it. The output is information related to the abnormal behavior displayed to the user, allowing the user to take quick and appropriate action. Specifically, the terminal notifies the user of notifications with sound and vibration, and facilitates checking live video.

[0491] (Application Example 1)

[0492] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0493] Conventional monitoring systems have not adequately detected abnormal behavior in real time or provided rapid notification, posing a challenge to improving security. Furthermore, it was difficult for users to immediately obtain detailed information about abnormal behavior, potentially leading to delays in response. Therefore, there is a need for a system that improves monitoring accuracy and provides information to users quickly.

[0494] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0495] In this invention, the server includes means for segmenting visual information received from monitoring equipment and converting it into a format that can be immediately analyzed; means for performing individual recognition and motion analysis on the converted visual information to detect abnormal behavior; and means for providing details of the detected abnormal behavior and visual information to a display device. This makes it possible to quickly and accurately detect abnormal behavior by combining visual information and detection data, and to immediately convey detailed information to the user.

[0496] "Surveillance equipment" refers to devices used to acquire visual information, and includes cameras, etc.

[0497] "Visual information" refers to video and image data acquired from surveillance equipment.

[0498] "Real-time analysis" refers to the process of processing acquired data and making decisions immediately.

[0499] "Individual recognition" is a technology that identifies specific people or objects within visual information.

[0500] "Motion analysis" is a technique that analyzes visual information and evaluates the movement and behavior of objects depicted in it.

[0501] "Abnormal behavior" refers to irregular movements or actions that are not seen under normal circumstances, and may include criminal acts.

[0502] A "display device" is a device used to visually present information to a user, and includes smartphones and monitors.

[0503] "User interface" refers to the functions and screens that allow users to operate a system and obtain information.

[0504] "Instant display" means that information is presented to the user in some form the moment it is obtained.

[0505] This system aims to detect abnormal behavior by acquiring visual information through monitoring equipment and analyzing it in real time. In its implementation, the server, terminals, and users each play specific roles.

[0506] The server receives visual information from monitoring equipment and identifies faces using individual recognition software such as OpenCV. Furthermore, it utilizes motion analysis technology to analyze each frame of the acquired visual information and detect abnormal behavior. When an anomaly is detected, the server immediately generates an alarm and saves detailed data to a recording device. The analysis results are summarized using generation technology, and important information is organized.

[0507] The terminal provides users with notifications sent from the server. Specifically, it uses a display device such as a smartphone to present users with details and visual information about detected abnormal behavior. Through the user interface, users can quickly check the situation.

[0508] Based on the information provided through the device, users can take appropriate action by coordinating with external security managers and the police as needed.

[0509] As a concrete example, in one household, by activating the system while they are away for an extended period, they can receive a notification the moment a suspicious person approaches. The user can then check the visual information from the camera installed in their home on their smartphone and promptly report it to the police.

[0510] An example of a prompt message for a generated AI model would be: "If a suspicious person is detected in the surveillance camera footage, please advise on how to notify the user so that they can respond more quickly."

[0511] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0512] Step 1:

[0513] The server receives visual information from surveillance equipment in real time. It takes a video stream from surveillance cameras as input. The received video is divided into frames, and this data is prepared for processing. The divided frame data is denoised and the resolution is adjusted.

[0514] Step 2:

[0515] The server applies individual recognition technology to the adjusted frame data. Specifically, it uses face recognition algorithms such as OpenCV to identify faces within each frame. The corrected frame data is used as input, and the output is the result of matching the detected face positions and features against a database.

[0516] Step 3:

[0517] The server uses motion analysis technology to analyze motion patterns within frames. It uses frame data as input and performs data calculations to detect unnatural movements and abnormal behavior. The output is a determination of whether or not an anomaly was detected.

[0518] Step 4:

[0519] When an anomaly is detected, the server immediately generates an alarm and creates notification data. Specifically, it constructs an alarm message containing information such as the type of anomaly, the location and time of detection, etc. The output is a notification message.

[0520] Step 5:

[0521] The server sends the generated alarm to an external terminal. The user's terminal receives the notification message. It receives the alarm message as input and provides notification information to the user as output.

[0522] Step 6:

[0523] The device displays notification details and real-time visual information to the user. It uses alarm messages and live video streams as inputs and visually presents the information in the user interface as output.

[0524] Step 7:

[0525] The user makes quick decisions based on the information presented. If necessary, they can take action to collaborate with external security organizations. The input involves referencing information provided by the terminal, and the output involves making decisions that lead to appropriate countermeasures.

[0526] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0527] The system of this invention analyzes video data from a monitoring device to detect abnormal behavior and incorporates an emotion engine to recognize the user's emotional state. This aims to support the user from both a security and safety perspective by monitoring not only abnormal behavior but also changes in the user's emotions.

[0528] The server processes video data acquired from the monitoring device in real time, dividing it into frames. This processing enables efficient face recognition and motion analysis. The server uses these analyses to identify abnormal behavior and simultaneously analyzes the user's facial expressions using an emotion engine. The emotion engine extracts the user's facial features from the video frames and executes an algorithm to evaluate their emotional state.

[0529] If an emotional state exceeds a certain threshold, for example, if high levels of anxiety or tension are detected, the server immediately issues an alarm and sends an alert notification to the terminal. This notification includes not only information about abnormal behavior but also the results of the emotional analysis, helping the user to gain a more comprehensive understanding of the situation.

[0530] The device displays received real-time notifications on the user interface, providing information to the user. This allows the user to take quick and appropriate action based on the emotional information obtained from the system. For example, if an emotionally unstable state is detected, the user can consider ways to relax or, if necessary, take steps to seek professional help.

[0531] As a concrete example, when a user is home alone at night, the system monitors the user's emotions in addition to normal surveillance. If a sudden anomaly is detected and the user becomes surprised and anxious, the emotion engine immediately recognizes this and a notification is displayed on the device. As a result, the user can regain their composure and quickly take appropriate countermeasures. In this way, the system can improve security and psychological safety in real time.

[0532] The following describes the processing flow.

[0533] Step 1:

[0534] The server receives video data sent from the monitoring device. The received data is divided into frames in real time and preprocessed for analysis.

[0535] Step 2:

[0536] The server applies a face recognition algorithm to the divided video frames. This algorithm detects facial features within the frames and identifies specific individuals by comparing them with an existing database.

[0537] Step 3:

[0538] The server uses dynamic analysis technology to detect abnormal behavior. This includes a process of identifying unusual movements and unexpected behavioral patterns and comparing them to normal behavior.

[0539] Step 4:

[0540] The server uses an emotion engine to analyze the user's emotional state. It analyzes subtle changes in the user's facial expressions from video frames to identify their emotional state.

[0541] Step 5:

[0542] When abnormal behavior or specific emotional states are detected, the server issues an alarm and sends a notification to an external terminal. The notification includes information about the detected anomaly and the results of an analysis of the emotional state.

[0543] Step 6:

[0544] The device displays notifications sent from the server to the user in real time. The user can understand the situation by reviewing the presented information and take appropriate action.

[0545] Step 7:

[0546] Users adjust their behavior based on their emotional state to ensure security measures are in place. For example, if a state of tension persists, they can contact an external professional organization, enabling a swift response.

[0547] (Example 2)

[0548] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0549] In recent years, monitoring systems have been required to detect abnormal behavior quickly and accurately, but simple motion analysis alone is insufficient to detect all anomalies. Furthermore, monitoring changes in users' emotional states is necessary to further enhance safety. However, there is a lack of efficient methods for doing this in real time.

[0550] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0551] In this invention, the server includes means for dividing video signals acquired from a monitoring device into frames and converting them into an analyzable format, means for performing face recognition and motion analysis using a machine learning model to detect abnormal behavior, and means for evaluating the user's emotional state in real time using a deep learning model. This improves the accuracy of detecting abnormal behavior and simultaneously allows for monitoring changes in the user's emotional state.

[0552] A "monitoring device" is a device used to acquire video signals and plays a role in monitoring the surrounding situation in real time.

[0553] A "video signal" is digital data containing visual information acquired by devices such as cameras.

[0554] A "frame" refers to each individual still image that makes up a video, and is the smallest unit that can be individually analyzed.

[0555] An "analyzable format" refers to a data format in which the data has been processed and converted into a state suitable for facial recognition and motion analysis.

[0556] A "machine learning model" is an algorithm that learns patterns based on past data and makes predictions and decisions based on new data.

[0557] "Facial recognition" is a technology that identifies the faces of people in a video signal and extracts their features.

[0558] "Motion analysis" is the process of analyzing a person's movements and behavior from video signals to detect abnormal patterns.

[0559] "Abnormal behavior" refers to behavior or actions that deviate from a predetermined normal pattern.

[0560] A "deep learning model" is a machine learning technique that uses multi-layered neural networks to extract features from data and recognize patterns.

[0561] "User's emotional state" refers to the psychological state and emotional changes inferred from the user's facial expressions and mucosal movements.

[0562] "Real-time" refers to a time frame in which data is processed and information is provided with extremely little delay, almost instantaneously.

[0563] This invention is a system that processes video signals acquired from a monitoring device in real time and monitors both safety and the emotional state of the user.

[0564] The server first receives a video signal from the monitoring device. This video signal is then divided into frames and converted into an analyzable format. Based on the divided frames, the server uses a machine learning model to perform face recognition and motion analysis. If abnormal behavior is detected, a deep learning model is used to extract the user's facial features and evaluate their emotional state. This process utilizes common open-source libraries such as TensorFlow and OpenCV.

[0565] If a user's emotional state exceeds a certain threshold, the server immediately issues an alarm and sends a notification to the terminal. At the same time, the generating AI model summarizes the analysis results and records them in a data storage device. The terminal displays the received notification in a graphical user interface, providing the user with the information. This allows the user to quickly and appropriately choose an action based on the information obtained from the system.

[0566] As a concrete example, while a user is home alone overnight, the system monitors the user's emotions in addition to normal surveillance. If any abnormality is detected and the user becomes emotionally unstable, the emotion engine immediately senses this and displays an appropriate notification on the device, allowing the user to regain a sense of security and take appropriate action quickly.

[0567] Examples of prompts include "Please describe the conditions for identifying abnormal behavior" and "Please provide information on technologies for evaluating a user's emotional state in real time."

[0568] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0569] Step 1:

[0570] The server receives video signals from the monitoring device. The input consists of real-time video signals from multiple cameras. The server receives these video signals and converts them into an analyzable format by dividing them into frames. This prepares reference data for detailed analysis of each frame.

[0571] Step 2:

[0572] The server receives video data divided into frames as input and performs face recognition and motion analysis using a machine learning model. This process extracts the facial features and motion patterns of people in each frame as output. Specifically, the server uses TensorFlow or similar tools to run the model and generate the data necessary for detecting abnormal behavior.

[0573] Step 3:

[0574] The server uses a deep learning model to evaluate the user's emotional state, taking the results of motion analysis as input. Specifically, the server extracts facial expressions and subtle body movements and classifies them into emotional categories. The output of this process is detailed data about the user's emotional state.

[0575] Step 4:

[0576] The server triggers an alarm when certain criteria are exceeded based on the evaluation results of emotional state and abnormal behavior. The input here is emotional state and behavioral analysis data as analysis results, and the server uses this data to determine the need for an alarm. As output, an alarm is decided to be issued and a notification is generated.

[0577] Step 5:

[0578] The server sends the generated alarm notification to the mobile terminal. The input here is the alarm notification data, which the server sends without delay using a real-time protocol. The output is the alarm message displayed on the user interface.

[0579] Step 6:

[0580] The terminal visualizes alarm notifications received from the server on a user interface and provides information to the user. The input is alarm notification data, and the terminal displays this as output on the screen. Specifically, it presents various countermeasures to the user based on the content of the notification.

[0581] Step 7:

[0582] The user selects an appropriate response based on the notification displayed on the device's user interface. The input is the alert information on the user interface, and based on this, the user decides on an action as the output. Specific actions include considering ways to promote relaxation and contacting a specialist if necessary.

[0583] (Application Example 2)

[0584] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0585] Modern security and safety systems often focus solely on detecting abnormal behavior, failing to consider changes in the user's emotional state. As a result, unstable emotional states can lead to system errors, potentially compromising user safety. This invention aims to provide a system that, in addition to detecting abnormal behavior, monitors changes in the user's emotions in real time, enabling appropriate responses.

[0586] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0587] In this invention, the server includes means for dividing video data received from a monitoring device and converting it into a format that can be analyzed in real time; means for performing face recognition and motion analysis on the converted video data to detect abnormal behavior; means for evaluating the emotional state based on the user's facial information and detecting changes in emotion; means for issuing an alarm and sending a notification to an external terminal based on the detected abnormal behavior and changes in emotion; means for summarizing the analysis results using generation technology and storing the summarized data in a recording device; and means for providing a user interface that provides information to help the user understand the situation and encourage safe behavior. This enables rapid and appropriate security measures that respond not only to abnormal behavior but also to changes in the user's emotions.

[0588] A "monitoring device" is a device that acquires video data and transmits it to a server.

[0589] "Video data" refers to a collection of visual information acquired from surveillance devices, which is used as material for evaluating abnormal behavior and emotional states through analysis.

[0590] "Analyzable format" refers to a state in which video data has been converted into a data format necessary for efficient analysis.

[0591] "Facial recognition" is a technology that identifies a person's face from video data and extracts its features.

[0592] "Dynamic analysis" is a technique for capturing movement and changes in video data to identify abnormal behavior.

[0593] "Abnormal behavior" refers to unnatural movements or actions in a subject under surveillance that differ from normal behavior.

[0594] "Emotional state" refers to the psychological condition inferred from a person's facial expressions and body movements.

[0595] "Emotional change" refers to an emotional state that changes over time.

[0596] An "alarm" is a notification issued to warn of an anomaly that has been detected.

[0597] An "external terminal" is a device used to receive and display alarms and notifications.

[0598] "Generative technology" refers to all technologies used to summarize analysis results and record data.

[0599] "Summary data" refers to data that concisely summarizes the analyzed information.

[0600] A "recording device" is a medium or device used to store summary data.

[0601] A "user interface" refers to the screens and means of operation that a user uses to interact with a device.

[0602] This invention constructs a system for detecting abnormal behavior and changes in emotional state. The system consists of the following components:

[0603] The server receives real-time video data acquired from the monitoring device and divides the data into frames. This video data is converted into an analyzable format and processed using video analysis software such as OpenCV. A face recognition algorithm identifies faces in each frame and extracts their features. Then, using an emotion recognition engine such as Microsoft Azure Face API or Google Cloud Vision API, the system analyzes the user's emotional state and detects emotional changes such as anxiety or tension.

[0604] The server then performs further dynamic analysis to identify abnormal behavior. If the set thresholds for abnormal behavior or emotion are exceeded, the system quickly issues an alarm and sends an alert to an external device via a notification service such as Firebase Cloud Messaging. Users can receive the notification on their device and view detailed information through a dedicated user interface.

[0605] This system allows users to understand the situation appropriately and quickly, and take safety measures as needed. A concrete example is the use of smart glasses for safety assistance when returning home late at night. If a user encounters an unexpected situation, the system immediately detects their emotional state and abnormal behavior, and provides information to encourage a calm response.

[0606] An example of a prompt for a generative AI model is: "Design an AI assistant that checks the surroundings when returning home late at night and detects abnormal behavior or emotional changes. Notifications should be sent to the smart device to encourage safe behavior." In this way, it is expected that safety and psychological reassurance will be improved.

[0607] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0608] Step 1:

[0609] The server acquires video data in real time from the monitoring device. It receives raw video data as input. This data is divided into frames and converted into an analyzable format to produce output. Specifically, a video editing library is used to perform the frame division.

[0610] Step 2:

[0611] The server performs face recognition on the converted frame-by-frame video data using OpenCV. The input is video data divided into frames. The server analyzes this data, identifies face information within each frame, and extracts its features. The output is the extracted face feature data. Specifically, it detects face regions and records their coordinates and feature points.

[0612] Step 3:

[0613] The server inputs the extracted facial feature data into the emotion recognition engine to evaluate the user's emotional state. The input is facial feature data. Using the Microsoft Azure Face API, the emotional state is identified, and emotions such as anxiety and tension are specified. The output is the evaluation result of the emotional state. Specifically, the API is called and the returned emotion score is analyzed.

[0614] Step 4:

[0615] The server simultaneously performs dynamic analysis on the video data. The input is video data frame by frame. It analyzes movements and changes to identify abnormal behavior and detects actions that exceed the judgment criteria. The output is the result of detecting abnormal behavior. Specifically, it performs vector analysis of movement to identify patterns of movement that are different from normal.

[0616] Step 5:

[0617] The server sends an alert to the device via Firebase Cloud Messaging based on the results of emotional changes and abnormal behavior. The inputs are the results of the emotional state assessment and the abnormal behavior detection. This information is aggregated, an alert is generated, and sent to the device. The output is the alert notification to the device. Specifically, the information is embedded in an alert template to construct the notification message.

[0618] Step 6:

[0619] The terminal displays received alarms on the user interface. The input is the alarm notification sent from the server. The terminal analyzes the alarm content and presents it to the user in an easy-to-understand manner. The output is the displayed alarm information. Specifically, it displays a pop-up on the screen to provide detailed information.

[0620] Step 7:

[0621] The user understands the situation based on the displayed information and considers countermeasures. The input is the alarm information displayed on the terminal. The user can evaluate the options and take additional actions as needed to take safe actions. The output is the selection of safety measures. Specifically, the user follows the displayed instructions and takes appropriate action.

[0622] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0623] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0624] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0625] [Fourth Embodiment]

[0626] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0627] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0628] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0629] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0630] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0631] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0632] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0633] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0634] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0635] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0636] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0637] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0638] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0639] The system of this invention aims to process video data acquired from a monitoring device in real time and to quickly detect abnormal behavior. This system is configured based on the respective roles of server, terminal, and user.

[0640] The server receives video data in streaming format from multiple monitoring devices. The received data is divided into frames in real time and pre-processed. Specifically, noise in the video is removed and the resolution is adjusted as needed to improve the efficiency of the subsequent analysis process. Next, the server activates a facial recognition algorithm to detect human faces frame by frame and compare them with a database. This process enables the identification of known suspicious individuals and the detection of new anomaly patterns.

[0641] Furthermore, the server uses dynamic analysis technology to detect abnormal behavior. For example, it identifies unnatural movements and behavioral patterns that differ from normal human movement. This enables a rapid response to prevent criminal activity.

[0642] If an anomaly is detected, the server issues an alarm and immediately sends a notification to the terminal. The terminal is responsible for providing these notifications to the user in real time. Based on the information provided, the user can take the most appropriate security measures. The terminal also provides access to detailed information about the detected anomaly and live video, allowing the user to understand the situation more accurately.

[0643] Furthermore, the server utilizes generation technology to summarize the analyzed data. This summary includes a timeline of specific events and highlights of important scenes, and is saved to a recording device. The saved data can be accessed by users at a later date and used as reference material for detailed analysis and re-evaluation.

[0644] For example, when a user is away from home for an extended period, activating the system will detect suspicious activity during the night and immediately send a notification to the user's device. This allows the user to contact external organizations such as the police or security companies and take swift action. In this way, the system provides a practical solution that contributes to the efficiency of crime prevention and the improvement of security.

[0645] The following describes the processing flow.

[0646] Step 1:

[0647] The server continuously receives video data from the monitoring device. Upon reception, the data is divided into frames for real-time processing and converted into a format that facilitates analysis.

[0648] Step 2:

[0649] The server applies a face recognition algorithm to the divided video frames. This algorithm detects human faces within the frames and compares them to registered known targets.

[0650] Step 3:

[0651] The server uses dynamic analysis technology to detect abnormal movements by comparing them to normal operating patterns. It identifies unnatural movements, such as rapid movement or intrusion into unauthorized areas.

[0652] Step 4:

[0653] When abnormal behavior is detected, the server immediately issues an alarm and sends a corresponding notification to an external terminal. This action is taken to quickly draw attention to potential threats.

[0654] Step 5:

[0655] The device displays alert notifications to the user in real time. Users are provided with the ability to check detailed information and live video on the device and determine whether an emergency response is necessary.

[0656] Step 6:

[0657] Based on the displayed information, users can decide whether to contact a security company or the police as needed. This ensures that an on-site response is carried out as quickly as possible.

[0658] Step 7:

[0659] The server uses generation technology to summarize the analysis results and stores them in a recording device. This summarized data is used later for further detailed analysis and reporting as needed.

[0660] (Example 1)

[0661] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0662] In recent years, with the increasing need for crime prevention and surveillance, there is a demand for systems that can detect abnormal behavior in real time and respond quickly. However, current systems suffer from problems such as delays in data processing and false positives, resulting in a lack of immediacy and accuracy. Furthermore, even when an anomaly is detected, there are insufficient means to quickly and clearly communicate that information to the user, raising concerns that effective crime prevention measures may be delayed. Therefore, a new crime prevention and surveillance system is needed to solve these problems.

[0663] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0664] In this invention, the server includes means for dividing video information received from a monitoring device and converting it into a format that can be analyzed in real time; means for performing object recognition and motion analysis on the converted video information to detect abnormal behavior; and means for issuing an alarm and transmitting a notification to an external computing device based on the detected abnormal behavior. This enables highly accurate detection of abnormal behavior in real time and rapid response notification.

[0665] A "surveillance device" is a device that captures images of a specific area and transmits that video information to a server in real time.

[0666] "Video information" refers to visual data acquired from surveillance devices, and is the element that will be analyzed.

[0667] "Object recognition" is a technology that identifies and judges specific patterns or shapes from video information.

[0668] "Dynamic analysis" is a technique that analyzes movement patterns in video information to detect anomalies.

[0669] "Abnormal behavior" refers to suspicious movements or actions that deviate from normal behavioral patterns, and is what the system detects.

[0670] "External computing device" refers to a digital device that the user can access after receiving a notification, and includes smartphones, personal computers, and other similar devices.

[0671] "Summary information" refers to information that concisely summarizes the results of the analysis and includes important data points.

[0672] A "storage device" is a digital device used to store analysis results and summary information for long periods of time, and includes hard disks and cloud storage.

[0673] "Real-time" refers to processing that occurs instantly, without any delay in the actual time that is happening.

[0674] In the system of this invention, the server, terminal, and user each play a specific role, and multi-layered processing is performed in real time.

[0675] The server receives video information acquired by the monitoring device in streaming format. The received data is divided into frames in real time using an open-source media framework (e.g., FFmpeg). During this process, noise is removed using an image processing library (e.g., OpenCV), and the resolution is adjusted as needed to achieve efficient data processing.

[0676] Next, the server employs an object recognition algorithm. Specifically, it uses a deep learning model powered by a machine learning library (e.g., TensorFlow) to detect human faces from the video field and identify known suspicious individuals by comparing them with a database. Furthermore, it applies motion analysis technology to detect irregular behavioral patterns in the video. Here, it also performs time-based data analysis to identify unusual movements.

[0677] When an anomaly is detected, the server immediately sends a notification to an external computing device. The terminal receives this notification in real time and displays the information on the user interface. The notification includes details of the detected anomaly and its location, allowing the user to quickly understand the situation and take appropriate action.

[0678] Furthermore, the server uses generation technology to summarize the analysis results and saves this summary information to a storage device. Users can access this information later and perform more detailed analysis as needed.

[0679] For example, if the system is activated while the user is away, and an intruder enters the property at night, the system will automatically detect this and immediately notify an external party. Upon receiving this notification, the user can view live video via their smartphone and report the incident to the security agency.

[0680] An example of a prompt when using a generative AI model is, "Please explain the real-time notification process when suspicious behavior is detected and how the user should respond afterward." This prompt is used to obtain detailed information about the system's operation and how to use it.

[0681] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0682] Step 1:

[0683] The server receives video information from the monitoring device in streaming format. The input is the video data provided by the monitoring device. The server uses an open-source media framework to divide this data into frames. The output is frame data that has undergone noise reduction and resolution adjustment, which streamlines the subsequent analysis process. Specifically, the server extracts important frames from the video and saves them in an optimized state.

[0684] Step 2:

[0685] The server performs object recognition on the divided frames. The input is the frame data obtained in step 1. The server detects faces in the frames using a deep learning model with a machine learning library and compares the face information with the database. The output is the detected face information and the matching result. Specifically, if a known suspicious person is found, the information is immediately recorded and passed on to the next process.

[0686] Step 3:

[0687] The server uses dynamic analysis technology to detect unnatural movements and behavioral patterns. The input consists of frame data adjusted in step 1 and face recognition results obtained in step 2. The server utilizes optical flow technology and neural networks to identify abnormal behavior. The output is a detailed description of the identified abnormal behavior. Specifically, upon detection of abnormal behavior, the server immediately prepares for the next action.

[0688] Step 4:

[0689] The server issues an alarm based on the detected anomaly. The input is the details of the abnormal behavior obtained in step 3. The server uses this to send a notification to an external computing device. The output is notification data containing details of the anomaly. In terms of specific actions, information suggesting concrete actions is delivered to the external computing device along with the alarm.

[0690] Step 5:

[0691] The terminal receives notifications sent from the server in real time and displays them on the user interface. The input is the notification data sent from the server in step 4. The terminal converts this into a user-friendly format and displays it. The output is information related to the abnormal behavior displayed to the user, allowing the user to take quick and appropriate action. Specifically, the terminal notifies the user of notifications with sound and vibration, and facilitates checking live video.

[0692] (Application Example 1)

[0693] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0694] Conventional monitoring systems have not adequately detected abnormal behavior in real time or provided rapid notification, posing a challenge to improving security. Furthermore, it was difficult for users to immediately obtain detailed information about abnormal behavior, potentially leading to delays in response. Therefore, there is a need for a system that improves monitoring accuracy and provides information to users quickly.

[0695] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0696] In this invention, the server includes means for segmenting visual information received from monitoring equipment and converting it into a format that can be immediately analyzed; means for performing individual recognition and motion analysis on the converted visual information to detect abnormal behavior; and means for providing details of the detected abnormal behavior and visual information to a display device. This makes it possible to quickly and accurately detect abnormal behavior by combining visual information and detection data, and to immediately convey detailed information to the user.

[0697] "Surveillance equipment" refers to devices used to acquire visual information, and includes cameras, etc.

[0698] "Visual information" refers to video and image data acquired from surveillance equipment.

[0699] "Real-time analysis" refers to the process of processing acquired data and making decisions immediately.

[0700] "Individual recognition" is a technology that identifies specific people or objects within visual information.

[0701] "Motion analysis" is a technique that analyzes visual information and evaluates the movement and behavior of objects depicted in it.

[0702] "Abnormal behavior" refers to irregular movements or actions that are not seen under normal circumstances, and may include criminal acts.

[0703] A "display device" is a device used to visually present information to a user, and includes smartphones and monitors.

[0704] "User interface" refers to the functions and screens that allow users to operate a system and obtain information.

[0705] "Instant display" means that information is presented to the user in some form the moment it is obtained.

[0706] This system aims to detect abnormal behavior by acquiring visual information through monitoring equipment and analyzing it in real time. In its implementation, the server, terminals, and users each play specific roles.

[0707] The server receives visual information from monitoring equipment and identifies faces using individual recognition software such as OpenCV. Furthermore, it utilizes motion analysis technology to analyze each frame of the acquired visual information and detect abnormal behavior. When an anomaly is detected, the server immediately generates an alarm and saves detailed data to a recording device. The analysis results are summarized using generation technology, and important information is organized.

[0708] The terminal provides users with notifications sent from the server. Specifically, it uses a display device such as a smartphone to present users with details and visual information about detected abnormal behavior. Through the user interface, users can quickly check the situation.

[0709] Based on the information provided through the device, users can take appropriate action by coordinating with external security managers and the police as needed.

[0710] As a concrete example, in one household, by activating the system while they are away for an extended period, they can receive a notification the moment a suspicious person approaches. The user can then check the visual information from the camera installed in their home on their smartphone and promptly report it to the police.

[0711] An example of a prompt message for a generated AI model would be: "If a suspicious person is detected in the surveillance camera footage, please advise on how to notify the user so that they can respond more quickly."

[0712] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0713] Step 1:

[0714] The server receives visual information from surveillance equipment in real time. It takes a video stream from surveillance cameras as input. The received video is divided into frames, and this data is prepared for processing. The divided frame data is denoised and the resolution is adjusted.

[0715] Step 2:

[0716] The server applies individual recognition technology to the adjusted frame data. Specifically, it uses face recognition algorithms such as OpenCV to identify faces within each frame. The corrected frame data is used as input, and the output is the result of matching the detected face positions and features against a database.

[0717] Step 3:

[0718] The server uses motion analysis technology to analyze motion patterns within frames. It uses frame data as input and performs data calculations to detect unnatural movements and abnormal behavior. The output is a determination of whether or not an anomaly was detected.

[0719] Step 4:

[0720] When an anomaly is detected, the server immediately generates an alarm and creates notification data. Specifically, it constructs an alarm message containing information such as the type of anomaly, the location and time of detection, etc. The output is a notification message.

[0721] Step 5:

[0722] The server sends the generated alarm to an external terminal. The user's terminal receives the notification message. It receives the alarm message as input and provides notification information to the user as output.

[0723] Step 6:

[0724] The device displays notification details and real-time visual information to the user. It uses alarm messages and live video streams as inputs and visually presents the information in the user interface as output.

[0725] Step 7:

[0726] The user makes quick decisions based on the information presented. If necessary, they can take action to collaborate with external security organizations. The input involves referencing information provided by the terminal, and the output involves making decisions that lead to appropriate countermeasures.

[0727] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0728] The system of this invention analyzes video data from a monitoring device to detect abnormal behavior and incorporates an emotion engine to recognize the user's emotional state. This aims to support the user from both a security and safety perspective by monitoring not only abnormal behavior but also changes in the user's emotions.

[0729] The server processes video data acquired from the monitoring device in real time, dividing it into frames. This processing enables efficient face recognition and motion analysis. The server uses these analyses to identify abnormal behavior and simultaneously analyzes the user's facial expressions using an emotion engine. The emotion engine extracts the user's facial features from the video frames and executes an algorithm to evaluate their emotional state.

[0730] If an emotional state exceeds a certain threshold, for example, if high levels of anxiety or tension are detected, the server immediately issues an alarm and sends an alert notification to the terminal. This notification includes not only information about abnormal behavior but also the results of the emotional analysis, helping the user to gain a more comprehensive understanding of the situation.

[0731] The device displays received real-time notifications on the user interface, providing information to the user. This allows the user to take quick and appropriate action based on the emotional information obtained from the system. For example, if an emotionally unstable state is detected, the user can consider ways to relax or, if necessary, take steps to seek professional help.

[0732] As a concrete example, when a user is home alone at night, the system monitors the user's emotions in addition to normal surveillance. If a sudden anomaly is detected and the user becomes surprised and anxious, the emotion engine immediately recognizes this and a notification is displayed on the device. As a result, the user can regain their composure and quickly take appropriate countermeasures. In this way, the system can improve security and psychological safety in real time.

[0733] The following describes the processing flow.

[0734] Step 1:

[0735] The server receives video data sent from the monitoring device. The received data is divided into frames in real time and preprocessed for analysis.

[0736] Step 2:

[0737] The server applies a face recognition algorithm to the divided video frames. This algorithm detects facial features within the frames and identifies specific individuals by comparing them with an existing database.

[0738] Step 3:

[0739] The server uses dynamic analysis technology to detect abnormal behavior. This includes a process of identifying unusual movements and unexpected behavioral patterns and comparing them to normal behavior.

[0740] Step 4:

[0741] The server uses an emotion engine to analyze the user's emotional state. It analyzes subtle changes in the user's facial expressions from video frames to identify their emotional state.

[0742] Step 5:

[0743] When abnormal behavior or specific emotional states are detected, the server issues an alarm and sends a notification to an external terminal. The notification includes information about the detected anomaly and the results of an analysis of the emotional state.

[0744] Step 6:

[0745] The device displays notifications sent from the server to the user in real time. The user can understand the situation by reviewing the presented information and take appropriate action.

[0746] Step 7:

[0747] Users adjust their behavior based on their emotional state to ensure security measures are in place. For example, if a state of tension persists, they can contact an external professional organization, enabling a swift response.

[0748] (Example 2)

[0749] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0750] In recent years, monitoring systems have been required to detect abnormal behavior quickly and accurately, but simple motion analysis alone is insufficient to detect all anomalies. Furthermore, monitoring changes in users' emotional states is necessary to further enhance safety. However, there is a lack of efficient methods for doing this in real time.

[0751] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0752] In this invention, the server includes means for dividing video signals acquired from a monitoring device into frames and converting them into an analyzable format, means for performing face recognition and motion analysis using a machine learning model to detect abnormal behavior, and means for evaluating the user's emotional state in real time using a deep learning model. This improves the accuracy of detecting abnormal behavior and simultaneously allows for monitoring changes in the user's emotional state.

[0753] A "monitoring device" is a device used to acquire video signals and plays a role in monitoring the surrounding situation in real time.

[0754] A "video signal" is digital data containing visual information acquired by devices such as cameras.

[0755] A "frame" refers to each individual still image that makes up a video, and is the smallest unit that can be individually analyzed.

[0756] An "analyzable format" refers to a data format in which the data has been processed and converted into a state suitable for facial recognition and motion analysis.

[0757] A "machine learning model" is an algorithm that learns patterns based on past data and makes predictions and decisions based on new data.

[0758] "Facial recognition" is a technology that identifies the faces of people in a video signal and extracts their features.

[0759] "Motion analysis" is the process of analyzing a person's movements and behavior from video signals to detect abnormal patterns.

[0760] "Abnormal behavior" refers to behavior or actions that deviate from a predetermined normal pattern.

[0761] A "deep learning model" is a machine learning technique that uses multi-layered neural networks to extract features from data and recognize patterns.

[0762] "User's emotional state" refers to the psychological state and emotional changes inferred from the user's facial expressions and mucosal movements.

[0763] "Real-time" refers to a time frame in which data is processed and information is provided with extremely little delay, almost instantaneously.

[0764] This invention is a system that processes video signals acquired from a monitoring device in real time and monitors both safety and the emotional state of the user.

[0765] The server first receives a video signal from the monitoring device. This video signal is then divided into frames and converted into an analyzable format. Based on the divided frames, the server uses a machine learning model to perform face recognition and motion analysis. If abnormal behavior is detected, a deep learning model is used to extract the user's facial features and evaluate their emotional state. This process utilizes common open-source libraries such as TensorFlow and OpenCV.

[0766] If a user's emotional state exceeds a certain threshold, the server immediately issues an alarm and sends a notification to the terminal. At the same time, the generating AI model summarizes the analysis results and records them in a data storage device. The terminal displays the received notification in a graphical user interface, providing the user with the information. This allows the user to quickly and appropriately choose an action based on the information obtained from the system.

[0767] As a concrete example, while a user is home alone overnight, the system monitors the user's emotions in addition to normal surveillance. If any abnormality is detected and the user becomes emotionally unstable, the emotion engine immediately senses this and displays an appropriate notification on the device, allowing the user to regain a sense of security and take appropriate action quickly.

[0768] Examples of prompts include "Please describe the conditions for identifying abnormal behavior" and "Please provide information on technologies for evaluating a user's emotional state in real time."

[0769] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0770] Step 1:

[0771] The server receives video signals from the monitoring device. The input consists of real-time video signals from multiple cameras. The server receives these video signals and converts them into an analyzable format by dividing them into frames. This prepares reference data for detailed analysis of each frame.

[0772] Step 2:

[0773] The server receives video data divided into frames as input and performs face recognition and motion analysis using a machine learning model. This process extracts the facial features and motion patterns of people in each frame as output. Specifically, the server uses TensorFlow or similar tools to run the model and generate the data necessary for detecting abnormal behavior.

[0774] Step 3:

[0775] The server uses a deep learning model to evaluate the user's emotional state, taking the results of motion analysis as input. Specifically, the server extracts facial expressions and subtle body movements and classifies them into emotional categories. The output of this process is detailed data about the user's emotional state.

[0776] Step 4:

[0777] The server triggers an alarm when certain criteria are exceeded based on the evaluation results of emotional state and abnormal behavior. The input here is emotional state and behavioral analysis data as analysis results, and the server uses this data to determine the need for an alarm. As output, an alarm is decided to be issued and a notification is generated.

[0778] Step 5:

[0779] The server sends the generated alarm notification to the mobile terminal. The input here is the alarm notification data, which the server sends without delay using a real-time protocol. The output is the alarm message displayed on the user interface.

[0780] Step 6:

[0781] The terminal visualizes alarm notifications received from the server on a user interface and provides information to the user. The input is alarm notification data, and the terminal displays this as output on the screen. Specifically, it presents various countermeasures to the user based on the content of the notification.

[0782] Step 7:

[0783] The user selects an appropriate response based on the notification displayed on the device's user interface. The input is the alert information on the user interface, and based on this, the user decides on an action as the output. Specific actions include considering ways to promote relaxation and contacting a specialist if necessary.

[0784] (Application Example 2)

[0785] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0786] Modern security and safety systems often focus solely on detecting abnormal behavior, failing to consider changes in the user's emotional state. As a result, unstable emotional states can lead to system errors, potentially compromising user safety. This invention aims to provide a system that, in addition to detecting abnormal behavior, monitors changes in the user's emotions in real time, enabling appropriate responses.

[0787] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0788] In this invention, the server includes means for dividing video data received from a monitoring device and converting it into a format that can be analyzed in real time; means for performing face recognition and motion analysis on the converted video data to detect abnormal behavior; means for evaluating the emotional state based on the user's facial information and detecting changes in emotion; means for issuing an alarm and sending a notification to an external terminal based on the detected abnormal behavior and changes in emotion; means for summarizing the analysis results using generation technology and storing the summarized data in a recording device; and means for providing a user interface that provides information to help the user understand the situation and encourage safe behavior. This enables rapid and appropriate security measures that respond not only to abnormal behavior but also to changes in the user's emotions.

[0789] A "monitoring device" is a device that acquires video data and transmits it to a server.

[0790] "Video data" refers to a collection of visual information acquired from surveillance devices, which is used as material for evaluating abnormal behavior and emotional states through analysis.

[0791] "Analyzable format" refers to a state in which video data has been converted into a data format necessary for efficient analysis.

[0792] "Facial recognition" is a technology that identifies a person's face from video data and extracts its features.

[0793] "Dynamic analysis" is a technique for capturing movement and changes in video data to identify abnormal behavior.

[0794] "Abnormal behavior" refers to unnatural movements or actions in a subject under surveillance that differ from normal behavior.

[0795] "Emotional state" refers to the psychological condition inferred from a person's facial expressions and body movements.

[0796] "Emotional change" refers to an emotional state that changes over time.

[0797] An "alarm" is a notification issued to warn of an anomaly that has been detected.

[0798] An "external terminal" is a device used to receive and display alarms and notifications.

[0799] "Generative technology" refers to all technologies used to summarize analysis results and record data.

[0800] "Summary data" refers to data that concisely summarizes the analyzed information.

[0801] A "recording device" is a medium or device used to store summary data.

[0802] A "user interface" refers to the screens and means of operation that a user uses to interact with a device.

[0803] This invention constructs a system for detecting abnormal behavior and changes in emotional state. The system consists of the following components:

[0804] The server receives real-time video data acquired from the monitoring device and divides the data into frames. This video data is converted into an analyzable format and processed using video analysis software such as OpenCV. A face recognition algorithm identifies faces in each frame and extracts their features. Then, using an emotion recognition engine such as Microsoft Azure Face API or Google Cloud Vision API, the system analyzes the user's emotional state and detects emotional changes such as anxiety or tension.

[0805] The server then performs further dynamic analysis to identify abnormal behavior. If the set thresholds for abnormal behavior or emotion are exceeded, the system quickly issues an alarm and sends an alert to an external device via a notification service such as Firebase Cloud Messaging. Users can receive the notification on their device and view detailed information through a dedicated user interface.

[0806] This system allows users to understand the situation appropriately and quickly, and take safety measures as needed. A concrete example is the use of smart glasses for safety assistance when returning home late at night. If a user encounters an unexpected situation, the system immediately detects their emotional state and abnormal behavior, and provides information to encourage a calm response.

[0807] An example of a prompt for a generative AI model is: "Design an AI assistant that checks the surroundings when returning home late at night and detects abnormal behavior or emotional changes. Notifications should be sent to the smart device to encourage safe behavior." In this way, it is expected that safety and psychological reassurance will be improved.

[0808] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0809] Step 1:

[0810] The server acquires video data in real time from the monitoring device. It receives raw video data as input. This data is divided into frames and converted into an analyzable format to produce output. Specifically, a video editing library is used to perform the frame division.

[0811] Step 2:

[0812] The server performs face recognition on the converted frame-by-frame video data using OpenCV. The input is video data divided into frames. The server analyzes this data, identifies face information within each frame, and extracts its features. The output is the extracted face feature data. Specifically, it detects face regions and records their coordinates and feature points.

[0813] Step 3:

[0814] The server inputs the extracted facial feature data into the emotion recognition engine to evaluate the user's emotional state. The input is facial feature data. Using the Microsoft Azure Face API, the emotional state is identified, and emotions such as anxiety and tension are specified. The output is the evaluation result of the emotional state. Specifically, the API is called and the returned emotion score is analyzed.

[0815] Step 4:

[0816] The server simultaneously performs dynamic analysis on the video data. The input is video data frame by frame. It analyzes movements and changes to identify abnormal behavior and detects actions that exceed the judgment criteria. The output is the result of detecting abnormal behavior. Specifically, it performs vector analysis of movement to identify patterns of movement that are different from normal.

[0817] Step 5:

[0818] The server sends an alert to the device via Firebase Cloud Messaging based on the results of emotional changes and abnormal behavior. The inputs are the results of the emotional state assessment and the abnormal behavior detection. This information is aggregated, an alert is generated, and sent to the device. The output is the alert notification to the device. Specifically, the information is embedded in an alert template to construct the notification message.

[0819] Step 6:

[0820] The terminal displays received alarms on the user interface. The input is the alarm notification sent from the server. The terminal analyzes the alarm content and presents it to the user in an easy-to-understand manner. The output is the displayed alarm information. Specifically, it displays a pop-up on the screen to provide detailed information.

[0821] Step 7:

[0822] The user understands the situation based on the displayed information and considers countermeasures. The input is the alarm information displayed on the terminal. The user can evaluate the options and take additional actions as needed to take safe actions. The output is the selection of safety measures. Specifically, the user follows the displayed instructions and takes appropriate action.

[0823] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0824] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0825] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0826] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0827] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0828] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0829] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0830] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0831] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0832] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0833] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0834] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0835] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0836] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0837] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0838] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0839] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0840] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0841] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0842] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0843] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0844] The following is further disclosed regarding the embodiments described above.

[0845] (Claim 1)

[0846] A means for splitting video data received from a monitoring device and converting it into a format that can be analyzed in real time,

[0847] A means for performing facial recognition and motion analysis on the converted video data to detect abnormal behavior,

[0848] A means for issuing an alarm and sending a notification to an external terminal based on detected abnormal behavior,

[0849] A means for summarizing the analysis results using generation technology and storing the summarized data in a recording device,

[0850] A system that includes this.

[0851] (Claim 2)

[0852] The system according to claim 1, further comprising means for improving the accuracy of anomaly detection by synchronizing video data and sensor data and analyzing them in an integrated manner.

[0853] (Claim 3)

[0854] The system according to claim 1, further comprising means for displaying an alert generated when abnormal behavior is detected on the user interface of a mobile terminal in real time.

[0855] "Example 1"

[0856] (Claim 1)

[0857] A means for dividing video information received from a monitoring device and converting it into a format that can be analyzed in real time,

[0858] A means for detecting abnormal behavior by performing object recognition and motion analysis on the converted video information,

[0859] A means for issuing an alarm and sending a notification to an external computing device based on detected abnormal behavior,

[0860] A means for summarizing the analysis results using generation technology and storing the summarized information in a storage device,

[0861] A means of immediately notifying an external computing device when an anomaly is detected and displaying the information in real time via a user interface,

[0862] A system that includes this.

[0863] (Claim 2)

[0864] The system according to claim 1, further comprising means for improving the accuracy of anomaly detection by synchronizing video information and detection device information and analyzing them in an integrated manner.

[0865] (Claim 3)

[0866] The system according to claim 1, further comprising means for summarizing the analyzed information as a time series and important parts, and saving it as material that can be used for future analysis.

[0867] "Application Example 1"

[0868] (Claim 1)

[0869] A means for dividing visual information received from a monitoring device and converting it into a format that can be immediately analyzed,

[0870] A means for performing individual recognition and motion analysis on the converted visual information to detect abnormal behavior,

[0871] A means for generating an alarm and sending a notification to an external device based on detected abnormal behavior,

[0872] A means for summarizing the analysis results using generation technology and storing the summarized data on a recording medium,

[0873] Means for providing details and visual information of detected abnormal behavior to a display device,

[0874] A system that includes this.

[0875] (Claim 2)

[0876] The system according to claim 1, further comprising means for improving the accuracy of anomaly detection by synchronizing and integrating visual information and detection data.

[0877] (Claim 3)

[0878] The system according to claim 1, further comprising means for immediately displaying a warning generated when abnormal behavior is detected on the user interface of a mobile device.

[0879] "Example 2 of combining an emotion engine"

[0880] (Claim 1)

[0881] A means for dividing video signals acquired from a monitoring device into frame units and converting them into an analyzable format for face recognition and motion analysis,

[0882] A means for performing face recognition and motion analysis on the converted video signal using a machine learning model to detect abnormal behavior that meets specific conditions,

[0883] In addition to abnormal behavior detection results, a means for performing facial expression analysis processing using a deep learning model to evaluate the user's emotional state,

[0884] A means of issuing an alert and sending a notification to a mobile device when a user's sentiment rating exceeds a certain standard,

[0885] A means of summarizing the analysis results using a generative AI model and storing them in a data storage device,

[0886] A system that includes this.

[0887] (Claim 2)

[0888] The system according to claim 1, further comprising means for improving the accuracy of anomaly detection by synchronizing video signals and sensor signals in time and analyzing them integrally.

[0889] (Claim 3)

[0890] The system according to claim 1, further comprising means for displaying alarms based on abnormal behavior detection and emotion evaluation in a visible format in real time on the graphical user interface of a mobile terminal.

[0891] "Application example 2 when combining with an emotional engine"

[0892] (Claim 1)

[0893] A means for splitting video data received from a monitoring device and converting it into a format that can be analyzed in real time,

[0894] A means for performing facial recognition and motion analysis on the converted video data to detect abnormal behavior,

[0895] A means for evaluating emotional state and detecting emotional changes based on the user's facial information,

[0896] A means for issuing an alarm and sending a notification to an external terminal based on detected abnormal behavior and emotional changes,

[0897] A means for summarizing the analysis results using generation technology and storing the summarized data in a recording device,

[0898] A means of providing a user interface that provides information to help users understand the situation and take safe actions,

[0899] A system that includes this.

[0900] (Claim 2)

[0901] The system according to claim 1, further comprising means for improving the accuracy of anomaly detection by synchronizing video data and sensor data and analyzing them in an integrated manner.

[0902] (Claim 3)

[0903] The system according to claim 1, further comprising means for displaying an alert generated when abnormal behavior is detected on the user interface of a portable information processing device in real time. [Explanation of symbols]

[0904] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for dividing visual information received from a monitoring device and converting it into a format that can be immediately analyzed, A means for performing individual recognition and motion analysis on the converted visual information to detect abnormal behavior, A means for generating an alarm and sending a notification to an external device based on detected abnormal behavior, A means for summarizing the analysis results using generation technology and storing the summarized data on a recording medium, Means for providing details and visual information of detected abnormal behavior to a display device, A system that includes this.

2. The system according to claim 1, further comprising means for improving the accuracy of anomaly detection by synchronizing visual information and detection data and analyzing them in an integrated manner.

3. The system according to claim 1, further comprising means for immediately displaying a warning generated when abnormal behavior is detected on the user interface of a mobile device.

Images