system

Abstract

We provide the system. [Solution] A means for analyzing video data acquired by an image acquisition device in real time to detect suspicious activity, A means of identifying authorized persons using registered identification information, A means of automatically adjusting the security level by learning environmental data using a learning device, A means of generating alarms and reports using a generation device and notifying relevant parties, A method for predicting future abnormal behavior using past data, A means of receiving notifications on a mobile device and displaying video in real time, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional monitoring systems, it is difficult to accurately detect suspicious persons and abnormal behaviors in real time, and false alarms frequently occur. Also, there is a problem that the security level cannot be flexibly adjusted according to changes in the environment, making it difficult to take effective security measures. Furthermore, there is concern that the generated alarms and reports are difficult to understand, hindering prompt response.

Means for Solving the Problems

[0005] This invention provides a means for analyzing video data acquired by an image acquisition device in real time and detecting suspicious behavior. Furthermore, it includes a means for identifying authorized individuals using registered identification information, and implements a function to automatically adjust the security level by learning environmental data using a learning device. It also includes a means for generating alerts and reports using a generation device and notifying relevant parties, thereby realizing rapid and efficient crime prevention measures. Moreover, it has a function to predict future abnormal behavior using past data and a self-correction function to reduce false alarms, enabling effective monitoring.

[0006] An "image acquisition device" is a device used to collect video data, and includes equipment such as cameras.

[0007] "Real-time analysis" is a technical method that processes acquired data immediately and analyzes the current situation.

[0008] "Suspicious behavior" refers to actions or behaviors that deviate from normal activity patterns and are judged to be illegal or dangerous.

[0009] "Identification information" refers to information necessary to identify individual people, and includes facial recognition data and ID information.

[0010] A "learning device" is an artificial intelligence device that analyzes data, learns patterns, and uses them for future decisions and actions.

[0011] A "generation device" is a device that creates new reports or alerts based on specified information or data.

[0012] A "self-correction function" is a feature that, when an error or inconsistency is detected, analyzes it and automatically adjusts the system to prevent similar errors from occurring in the future.

[0013] "Abnormal behavior prediction" is an analytical technique that uses past data to predict suspicious behavior or anomalies that may occur in the future, allowing for proactive precautions. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention relates to a surveillance system that can detect suspicious activity by acquiring and analyzing video footage within a surveillance area in real time, and adjust the security level as needed. This system consists of multiple terminals, servers, and users.

[0036] ■ System Overview

[0037] Device (camera)

[0038] The terminal has the function of continuously acquiring video data from the monitoring area and sending it to the server.

[0039] server

[0040] The server receives video data sent from terminals and uses an AI agent to analyze the data in real time. This analysis detects the behavior patterns of moving objects and identifies suspicious individuals by comparing them with a registered database. Facial recognition technology is used to identify authorized individuals, thereby determining whether an alarm needs to be issued. Furthermore, the server is equipped with a learning device that automatically adjusts the security level based on environmental data. The server uses a generation device to generate alarms and reports and distribute them to relevant parties.

[0041] ■Example of operation

[0042] Detection of a suspicious person

[0043] For example, if an intruder enters an office building at night, the terminal captures the scene as video and sends it to the server. The server analyzes the video data, detects suspicious movements, and compares them with a registered facial database. If the person is determined to be an unauthorized individual, an alarm is triggered, and the building manager is simultaneously notified.

[0044] Adjusting security levels through learning

[0045] In areas that are usually crowded but see reduced foot traffic during specific events, the server can use environmental and historical data to increase security levels during certain time periods. This allows for adjustment of the sensitivity of suspicious person detection and reduces false alarms.

[0046] Reports and forecasts

[0047] The server automatically generates reports on detected abnormal behavior and, when sending them to administrators, uses generation technology to summarize the content and provide it in an easy-to-understand format. Furthermore, it predicts future abnormal behavior based on past data and provides information to enable timely security measures.

[0048] Thus, this system utilizes real-time analysis, facial recognition, learning capabilities, suspicious person prediction, and report generation functions to achieve effective and efficient security monitoring.

[0049] The following describes the processing flow.

[0050] Step 1:

[0051] The terminal continuously captures video data from the monitoring area and transmits it to the server in real time. The video data is encrypted with privacy in mind and transmitted in a secure manner.

[0052] Step 2:

[0053] The server receives video data transmitted from the terminal and immediately begins analysis using an AI agent. First, it evaluates the changes between frames and identifies moving objects.

[0054] Step 3:

[0055] The server analyzes the behavioral patterns of identified objects to detect abnormal behavior. Here, it compares this behavior against pre-configured alert conditions, and if suspicious behavior is detected, it proceeds to the next step.

[0056] Step 4:

[0057] The server analyzes the face of the detected person and compares it to authorized person information in the database. If the person is not registered, they are treated as an unknown person and an alarm is immediately issued.

[0058] Step 5:

[0059] Based on the detection results, the server generates alarms and status reports using a generator. The generated information is automatically notified to the user, enabling the user to take a quick response according to the situation.

[0060] Step 6:

[0061] The server collects environmental data (e.g., time of day, date, weather), and an AI agent compares this data with historical data to dynamically adjust security levels. This reduces false alarms and enables more precise monitoring.

[0062] Step 7:

[0063] Based on a history of abnormal behavior detected in the past, the server predicts future abnormal behavior. This predictive data is provided to administrators to help them develop security plans.

[0064] This process allows the system to quickly and accurately detect abnormal behavior and strengthen security measures.

[0065] (Example 1)

[0066] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0067] The aim is to solve the challenges of video surveillance systems, such as the difficulty in immediately detecting suspicious behavior, reducing false alarms, and predicting future suspicious behavior. Conventional systems have faced challenges such as the generation of incorrect alerts and the difficulty in appropriately adjusting security levels.

[0068] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0069] In this invention, the server includes means for immediately analyzing visual information acquired via data acquisition means to identify suspicious behavior, means for identifying authorized individuals using authentication information registered in a database, and means for analyzing environmental information using a learning module to dynamically adjust the alert level. This enables rapid detection and reporting of suspicious behavior, thereby improving the accuracy of security.

[0070] "Data acquisition means" refers to devices or protocols that have the function of collecting visual information from a specific area and transmitting it to other system components.

[0071] "Visual information" refers to video data and image information acquired using cameras or similar devices.

[0072] "Immediate analysis" refers to performing data analysis immediately after data is acquired, without any waiting time or delay.

[0073] "Suspicious behavior" refers to actions or movements that deviate from normal patterns or are otherwise questionable, and may pose a security risk.

[0074] A "database" is a system for organizing, storing, and managing specific information, and can include authentication information and historical data.

[0075] "Authentication information" refers to information or data used to identify individuals or objects and to verify their authorized status.

[0076] A "learning module" refers to an algorithm or process that recognizes patterns based on past data and uses them to predict future trends or adjust the system.

[0077] "Environmental information" refers to external conditions and circumstances under which the system operates, such as weather and time of day data.

[0078] "Alert level" refers to an indicator or setting that shows how much attention a system should pay to suspicious behavior.

[0079] "Dynamic adjustment" means changing the system's settings and operation in real time in response to conditions that are constantly changing.

[0080] "Rapid detection of suspicious behavior" means quickly identifying suspicious behavior or actions and recognizing them immediately in order to take appropriate action.

[0081] A "report" is a document or digital message created to notify relevant parties of detailed information about events or activities detected by the system.

[0082] To implement this invention, it is necessary to install multiple terminals in the monitoring area and connect them to a server via a network. Each terminal includes a high-resolution camera and is responsible for continuously collecting and transmitting visual information to the server. The server is equipped with a high-performance processor and graphics processing unit, such as an NVIDIA Tesla V100, enabling real-time data analysis.

[0083] The server preprocesses the video data transmitted from the terminal using the "OpenCV" library to remove noise and extract features. Then, it uses AI models such as "YOLOv5" to detect moving objects and analyze behavioral patterns. In addition, it utilizes a "Facial Recognition System" to identify individuals within the video and match them with authentication information to identify suspicious individuals.

[0084] Furthermore, the server uses the "Scikit-learn" library to analyze environmental data and learn from past data patterns. This allows for dynamic adjustment of security levels and reduction of false alarms. The generation module can be used to generate alerts and reports and send them to relevant parties. Alert messages are summarized using the "OpenAI® API," ensuring that relevant parties are notified quickly and easily.

[0085] As a concrete example, in one office building, if suspicious activity is detected at night, the terminal immediately sends the video to a server. The server uses an AI model to detect the suspicious activity, issues an alarm if necessary, and notifies the relevant parties. During events, the server can appropriately adjust the security level in response to unusual pedestrian traffic, maintaining the accuracy of monitoring.

[0086] As an example of a prompt message, if the system is instructed to "Start analyzing video data and identify whether there is a suspicious person," it can immediately begin taking appropriate action. In this way, this invention utilizes advanced AI technology and data analysis technology to achieve efficient and effective security monitoring.

[0087] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0088] Step 1:

[0089] The terminal continuously acquires video data within the monitoring area and transmits it to the server in real time. The input is raw video data from the camera, and the output is encrypted video data sent to the server over the network. Specifically, the terminal periodically generates data packets and transmits this data using a secure protocol.

[0090] Step 2:

[0091] The server immediately receives video data from the terminal and preprocesses it using the "OpenCV" library. The input is the received raw video data, and the output is analysis-ready data with noise removed and features extracted. The server performs filtering and edge detection on each frame of the video and formats it into a format suitable for analysis.

[0092] Step 3:

[0093] The server analyzes the prepared video data using an AI model to detect suspicious movements and behaviors. The input for this step is formatted video data, and the output is identification information for suspicious movements. The "YOLOv5" model is used to detect moving objects and analyze behavioral patterns to identify abnormal movements.

[0094] Step 4:

[0095] The server uses the "Facial Recognition System" to compare detected behavior with registered authentication information and identify the target. The input consists of identification information of the suspicious behavior and facial image data, and the output is the result of whether or not the user is authenticated. The server performs a rapid matching with the facial database to confirm whether or not the user is authorized.

[0096] Step 5:

[0097] Based on the analysis results, the server issues alarms if necessary, generates alert messages, and notifies relevant parties. Inputs are authentication judgment results and operational identification information, while outputs are the generated alarms and summarized alert messages. The server uses the OpenAI API to organize the messages and send them to relevant parties in an easily understandable format via email or notification systems.

[0098] Step 6:

[0099] The server uses the "Scikit-learn" library to analyze environmental data and adjust security levels. Input consists of historical environmental data and analyzed operational data, while output is the adjusted security level setting. The server then uses this to optimize alert levels based on specific time periods and conditions.

[0100] By executing each step sequentially in this manner, the system can effectively monitor suspicious individuals and take necessary actions quickly.

[0101] (Application Example 1)

[0102] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0103] In monitoring systems, conventional methods are prone to misidentification when detecting suspicious activity in real time, and there is a lack of adequate means to quickly grasp the situation from remote locations. Furthermore, typical security systems have the problem of being difficult to flexibly adjust to changes in the environment, and alert notifications are cumbersome and time-consuming to understand.

[0104] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0105] In this invention, the server includes means for analyzing video data acquired by an image acquisition device in real time and detecting suspicious activity, means for identifying authorized individuals using registered identification information, and means for receiving notifications on a mobile terminal and displaying the video in real time. This reduces misidentification and enables rapid situation assessment from remote locations. Furthermore, it enables automatic adjustment of security levels in response to environmental changes and allows for simple alert notifications using summary messages generated with generation technology.

[0106] An "image acquisition device" is a device used to acquire video data within a monitoring area in real time.

[0107] "Real-time analysis" is a technology that instantly analyzes acquired video data to quickly detect suspicious activity.

[0108] "Identification information" refers to information used to verify a registered person, and includes facial data, etc.

[0109] A "learning device" is a device that automatically adjusts security levels based on environmental data.

[0110] A "generation device" is a device used to generate alarms and reports and notify relevant parties.

[0111] A "mobile device" is a device used by a user to monitor a situation remotely and receive notifications.

[0112] The system of this invention consists of multiple image acquisition devices, a server, and a mobile terminal. The image acquisition devices continuously acquire video data within the monitoring area. The acquired video data is transmitted to the server, which analyzes the data in real time. AI technology is used for this analysis, making it possible to detect suspicious activity from the acquired video. The server identifies authorized individuals based on registered identification information, and if suspicious activity is detected, it generates an alarm and report using a generation device.

[0113] The generated alarms and reports are sent to the mobile devices of relevant personnel via the notification system. This allows users to monitor the situation in real time from a remote location and take immediate action as needed. Furthermore, the server incorporates a learning device that automatically adjusts security levels based on environmental data.

[0114] As a concrete example, a user who is away from home at night can access their home's security cameras through an application installed on their mobile device. If suspicious activity is detected near the front door, an alert is immediately sent. The user receives a notification with a prompt message such as "(Suspicious person detected) A new face has been found in the footage. Time: 23:45, Location: Front door," and can immediately ask a neighbor or acquaintance to take action if necessary.

[0115] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0116] Step 1:

[0117] The terminal acquires video data of the monitoring area in real time. This data is continuously collected using an image acquisition device. Once the collected video data is sent to the server, it is ready for analysis.

[0118] Step 2:

[0119] The server analyzes the received video data using AI technology. It applies a motion detection algorithm to the video as input data to detect suspicious activity. This method analyzes movement patterns, and if suspicious activity is detected, the process proceeds to the next step.

[0120] Step 3:

[0121] The server compares the analyzed data with registered identification information. Specifically, it compares facial data extracted from video as input with the identification database to obtain output that identifies unauthorized individuals.

[0122] Step 4:

[0123] If suspicious activity is detected, the server uses a generator to create an alarm and report. Based on the suspicious person information received as input, it generates a report and constructs the relevant prompt messages. This output is then sent to the notification system.

[0124] Step 5:

[0125] Users receive alerts on their mobile devices and immediately check the situation. The system can receive prompt messages from the server as input and generate output that displays video in real time. This allows users to quickly assess the situation remotely and take appropriate action.

[0126] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0127] This invention relates to a system that, in addition to normal video analysis, recognizes user emotions within a surveillance system and dynamically adjusts the security level accordingly. This system comprises a terminal (login device), a server, and an emotion recognition engine.

[0128] ■ System Overview

[0129] Terminal (login device)

[0130] The terminal acquires video data in real time from the monitoring area. This video includes the user's facial expressions and is transmitted to the server.

[0131] server

[0132] The server receives video footage from the terminal and uses a standard AI agent to analyze and compare it with the footage to determine if there is any suspicious behavior. In addition, an emotion recognition engine analyzes facial expression data to identify the user's emotional state. This emotional information is fed back into the overall system operation, and the security level is adjusted as needed.

[0133] Emotion recognition engine

[0134] The emotion recognition engine recognizes emotions from facial expressions by matching them against specific patterns. This makes it possible to issue appropriate alerts if the user is experiencing anxiety or fear.

[0135] ■Example of operation

[0136] Alarm issued due to user anxiety

[0137] For example, if video footage is captured of a user looking around anxiously in a shopping mall, the emotion recognition engine can detect anxiety from their facial expression. Based on this information, the server can increase the security level and send an alert to security guards.

[0138] Rapid response in emergencies

[0139] In the event of an incident or accident, if a user at the scene is distressed, the emotion recognition engine quickly detects this distress, and the server analyzes the information to take immediate action. This minimizes confusion at the scene and enables a rapid response.

[0140] This system goes beyond simple behavioral analysis, enabling flexible and advanced security monitoring that takes into account the emotional impact of users. As a result, it allows for optimal responses tailored to the specific situation on-site, thereby improving security.

[0141] The following describes the processing flow.

[0142] Step 1:

[0143] The terminal captures video data in real time from the monitoring area. The video data, which also includes information about the user's facial expressions, is streamed to the server.

[0144] Step 2:

[0145] The server receives video data transmitted from the terminal and starts normal motion analysis using an AI agent. In particular, it identifies moving objects and determines whether or not there is any abnormal behavior.

[0146] Step 3:

[0147] The server uses an emotion recognition engine to analyze the user's facial expressions in the video and identify their emotional state. The analysis results are then classified according to the emotional state (e.g., relief, excitement, anxiety, fear, etc.).

[0148] Step 4:

[0149] The server integrates the results of behavioral and sentiment analysis to adjust the overall system security level. If suspicious behavior or situations indicating anxiety are detected, it will consider increasing the security level and issuing an alarm.

[0150] Step 5:

[0151] The server uses a generator to produce alerts and detailed situation reports based on analyzed behavioral and emotional information, and immediately notifies relevant parties. This information also includes data to guide users towards the ideal response.

[0152] Step 6:

[0153] The server learns user behavior and emotional patterns based on past data. Through the training data, including the results of the emotion recognition engine, it predicts future abnormal behavior and potential risks, and uses this information in security planning.

[0154] This process enables the system to perform flexible monitoring and responses that take into account emotional factors depending on the situation.

[0155] (Example 2)

[0156] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0157] Modern surveillance systems are required not only to analyze behavior but also to accurately grasp the emotional state of users and dynamically adjust security measures based on that information. Existing technologies are limited to analyzing only individual behavior and react only to suspicious actions, which leads to the oversight of potential threats and false alarms. Furthermore, they are not good at detecting users' anxieties in advance and providing rapid alerts, highlighting the need for more advanced security surveillance systems.

[0158] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0159] In this invention, the server includes means for analyzing visual information acquired by an image acquisition device in real time and detecting abnormal operation, means for learning environmental information through a data analysis device and dynamically adjusting the maintenance level, and means for analyzing the user's emotions using an emotion recognition device and adjusting the system's operation based on the emotion information. This enables comprehensive security monitoring that combines both motion analysis and emotion analysis, improving the accuracy of anomaly detection, reducing false alarms, and enabling rapid maintenance response.

[0160] An "image acquisition device" is a device that acquires visual information from a monitoring area in real time.

[0161] "Abnormal operation" refers to behavior that deviates from normal operation and is an action that is subject to monitoring within a system.

[0162] A "data analysis device" is a device that has the function of learning and analyzing acquired environmental information and adjusting the system's maintenance level based on that information.

[0163] "Security level" refers to the level of security strength set within the system, which is dynamically changed according to the degree of the threat.

[0164] A "notification device" is a device that has the function of transmitting alarms and reports generated by the system to the relevant parties.

[0165] "Historical information" refers to past data and is used to predict future abnormal behavior.

[0166] An "emotion recognition device" is a device that analyzes the user's emotional state and adjusts the system's operation based on that information.

[0167] This invention provides advanced security by combining motion analysis and emotion analysis within a surveillance system. The system primarily consists of terminals, servers, an emotion recognition engine, and other related devices.

[0168] The terminals are placed in the monitoring area and collect visual information in real time. Specifically, they use cameras to capture people's actions and facial expressions and transmit the video data to a server.

[0169] The server is responsible for analyzing the video data transmitted from the terminal. Using AI agents, the server detects abnormal behavior from the acquired visual information and further identifies emotions by analyzing the user's facial expressions using an emotion recognition engine. The emotion recognition engine learns specific facial expression patterns and has the ability to identify emotions such as anxiety and tension in real time.

[0170] Based on this information, the server learns environmental data through a data analysis device and dynamically adjusts the overall system's maintenance level. This automatically triggers alarms and notifies relevant parties when necessary.

[0171] For example, in a shopping mall, if a problem has occurred in a specific area in the past, monitoring of that area can be strengthened based on the historical information. Furthermore, if user anxiety is detected through emotion recognition, measures such as deploying additional security guards can be taken quickly.

[0172] Examples of prompt messages are as follows:

[0173] "Please explain the procedure for analyzing user emotions based on surveillance footage from a shopping mall and automatically raising the security level if anxiety is detected."

[0174] This invention, by integrating individual devices and analysis techniques, can provide flexible and highly accurate security management that goes beyond normal operational monitoring.

[0175] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0176] Step 1:

[0177] The terminal acquires visual information in real time using cameras installed in the monitoring area and transmits the video data to the server. The input is raw visual information captured by the cameras, and the output is a compressed data stream of that information. This data stream is transferred to the server and used for subsequent analysis processing.

[0178] Step 2:

[0179] The server receives visual information transmitted from the terminal and analyzes abnormal behavior using an AI agent. The input is a compressed data stream from the terminal, and the output is the analyzed behavioral data. The server detects suspicious behavior and records the results in a database. In this process, a pre-trained algorithm is used to identify anomalies by comparing them with behavioral patterns.

[0180] Step 3:

[0181] The server uses an emotion recognition engine to analyze the user's facial expressions in the video and identify their emotions. The input is the video data after motion analysis, and the output is the analysis result indicating the user's emotional state. If the identified emotion is anxiety or tension, the server issues an alert. This analysis includes recognition of facial patterns and calculation of the degree of match by an AI model.

[0182] Step 4:

[0183] The server adjusts the overall system security level based on the analysis results. Inputs are behavioral data and sentiment analysis results, while outputs are the adjusted security level and corresponding alarms. This adjustment is performed according to system policies based on a set of scripted rules, and notifications are sent to relevant parties.

[0184] Step 5:

[0185] The server uses historical data to predict future abnormal behavior. The input is historical monitoring data, and the output is predicted data for future anomaly detection. This allows the system to take proactive measures before anticipated events occur, enabling rapid and accurate security management.

[0186] (Application Example 2)

[0187] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0188] In modern society, security measures are becoming increasingly important. However, conventional monitoring systems are primarily based on behavioral analysis and have the problem of not being able to take into account the user's emotions and psychological state. Because they cannot detect anxiety or fear early on and take appropriate security measures before it manifests, it can be difficult to respond quickly to emergencies. In addition, many false alarms and unnecessary alerts occur, causing confusion, which is also a problem. To solve these problems, there is a need for a new system that can dynamically adjust security levels based on the user's emotional state.

[0189] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0190] In this invention, the server includes means for analyzing video data acquired by an image acquisition device in real time and detecting suspicious behavior; means for recognizing the user's emotional state using an emotion recognition engine and adjusting security measures based on that information; and means for generating alerts and reports using a generation device and notifying relevant parties. This enables flexible and advanced security monitoring that takes user emotions into consideration, allowing for quick and optimal responses tailored to the situation on site.

[0191] An "image acquisition device" is a device that acquires video data from a monitoring area in real time.

[0192] "Video data" refers to a collection of visual information captured by an image acquisition device.

[0193] "Real-time analysis" refers to the process of immediately processing acquired data and outputting results.

[0194] "Suspicious behavior" refers to user actions that are unusual or considered abnormal.

[0195] "Identification information" refers to information used to recognize a specific person.

[0196] An "authorized person" is someone who has been identified through registered identification information.

[0197] A "learning device" is a device that takes in environmental data and performs analysis and pattern recognition.

[0198] "Security level" is a standard that indicates the degree of vigilance in a monitoring system.

[0199] An "emotion recognition engine" is a mechanism that determines a user's emotional state from their facial expressions and actions.

[0200] "Security measures" refer to specific steps taken to ensure the safety of a system.

[0201] A "generation device" is a device that creates appropriate alerts and reports and notifies relevant parties.

[0202] An "alert" is a warning signal issued when suspicious activity or anomalies are detected.

[0203] "Past data" refers to the accumulation of information recorded previously.

[0204] "Abnormal behavior" refers to actions that deviate from the normal behavioral patterns of an individual or group.

[0205] The system for realizing this invention mainly comprises an image acquisition device, a server, and an emotion recognition engine. First, the terminal uses the image acquisition device to acquire video data of the monitoring area in real time. This video data also includes the user's facial expressions, making it analyzable. Next, the server receives this video data and performs behavioral analysis using an internally installed AI agent. If suspicious behavior is detected, it acts as an alert trigger. The server also uses the emotion recognition engine to analyze the user's emotions from the video data and identify emotional states such as anxiety and fear. Based on this information, the system can dynamically adjust the security level.

[0206] The emotion recognition engine uses specific patterns to evaluate the user's facial expressions and recognize emotions in real time. This allows security alerts to be generated according to the individual's emotional state. Alerts and reports generated by the generator are notified to relevant parties in an appropriate manner. If anxiety or fear is detected, notifications are immediately sent to the user and their designated contacts.

[0207] In particular, AI-powered video analysis and emotion recognition processes commonly utilize image processing libraries such as OpenCV and machine learning frameworks such as TENSORFLOW®. These tools enable more precise and rapid analysis of image data.

[0208] For example, if a user experiences anxiety while walking alone in a dark place, this system can immediately send a notification to family or friends to prepare for an unforeseen situation. In such scenarios, a prompt such as, "Write a script that analyzes the user's facial expressions from the camera footage to determine if they are feeling fear or anxiety. If an emotion is detected, send an alert to the designated contacts," would be useful.

[0209] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0210] Step 1:

[0211] The terminal uses an image acquisition device to acquire video data in real time from the monitoring area. The input here is the physical environment, and digital video data is generated as the output. This video data is sent to the server.

[0212] Step 2:

[0213] The server analyzes the received video data. Here, an AI agent is used for motion analysis to detect suspicious behavior. The input is video data, and the output is the detection result of suspicious behavior. Based on this result, the next process is triggered.

[0214] Step 3:

[0215] The server uses an emotion recognition engine to analyze the user's emotional state from video data. The input here is the user's facial features, which are processed using pattern recognition technology to identify emotions such as anxiety and fear. The output is the result of the emotional state analysis, which is used to adjust the security level.

[0216] Step 4:

[0217] The server uses a generator to produce alerts and reports as needed. The inputs here are the results of suspicious behavior and emotional states, which are then integrated using generation technology. The output is an alert notification, which is sent to the relevant parties.

[0218] Step 5:

[0219] If a user expresses anxiety or fear in an unexpected situation, the server promptly sends a notification to the designated contacts. The input here is the identified emotional state, and the output is a notification message. This message includes summary information generated using a generative AI model, if necessary.

[0220] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0221] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0222] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0223] [Second Embodiment]

[0224] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0225] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0226] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0227] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0228] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0229] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0230] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0231] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0232] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0233] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0234] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0235] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0236] This invention relates to a surveillance system that can detect suspicious activity by acquiring and analyzing video footage within a surveillance area in real time, and adjust the security level as needed. This system consists of multiple terminals, servers, and users.

[0237] ■ System Overview

[0238] Device (camera)

[0239] The terminal has the function of continuously acquiring video data from the monitoring area and sending it to the server.

[0240] server

[0241] The server receives video data sent from terminals and uses an AI agent to analyze the data in real time. This analysis detects the behavior patterns of moving objects and identifies suspicious individuals by comparing them with a registered database. Facial recognition technology is used to identify authorized individuals, thereby determining whether an alarm needs to be issued. Furthermore, the server is equipped with a learning device that automatically adjusts the security level based on environmental data. The server uses a generation device to generate alarms and reports and distribute them to relevant parties.

[0242] ■Example of operation

[0243] Detection of a suspicious person

[0244] For example, if an intruder enters an office building at night, the terminal captures the scene as video and sends it to the server. The server analyzes the video data, detects suspicious movements, and compares them with a registered facial database. If the person is determined to be an unauthorized individual, an alarm is triggered, and the building manager is simultaneously notified.

[0245] Adjusting security levels through learning

[0246] In areas that are usually crowded but see reduced foot traffic during specific events, the server can use environmental and historical data to increase security levels during certain time periods. This allows for adjustment of the sensitivity of suspicious person detection and reduces false alarms.

[0247] Reports and forecasts

[0248] The server automatically generates reports on detected abnormal behavior and, when sending them to administrators, uses generation technology to summarize the content and provide it in an easy-to-understand format. Furthermore, it predicts future abnormal behavior based on past data and provides information to enable timely security measures.

[0249] Thus, this system utilizes real-time analysis, facial recognition, learning capabilities, suspicious person prediction, and report generation functions to achieve effective and efficient security monitoring.

[0250] The following describes the processing flow.

[0251] Step 1:

[0252] The terminal continuously captures video data from the monitoring area and transmits it to the server in real time. The video data is encrypted with privacy in mind and transmitted in a secure manner.

[0253] Step 2:

[0254] The server receives video data transmitted from the terminal and immediately begins analysis using an AI agent. First, it evaluates the changes between frames and identifies moving objects.

[0255] Step 3:

[0256] The server analyzes the behavioral patterns of identified objects to detect abnormal behavior. Here, it compares this behavior against pre-configured alert conditions, and if suspicious behavior is detected, it proceeds to the next step.

[0257] Step 4:

[0258] The server analyzes the face of the detected person and compares it to authorized person information in the database. If the person is not registered, they are treated as an unknown person and an alarm is immediately issued.

[0259] Step 5:

[0260] Based on the detection results, the server generates alarms and status reports using a generator. The generated information is automatically notified to the user, enabling the user to take a quick response according to the situation.

[0261] Step 6:

[0262] The server collects environmental data (e.g., time of day, date, weather), and an AI agent compares this data with historical data to dynamically adjust security levels. This reduces false alarms and enables more precise monitoring.

[0263] Step 7:

[0264] Based on a history of abnormal behavior detected in the past, the server predicts future abnormal behavior. This predictive data is provided to administrators to help them develop security plans.

[0265] This process allows the system to quickly and accurately detect abnormal behavior and strengthen security measures.

[0266] (Example 1)

[0267] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0268] The aim is to solve the challenges of video surveillance systems, such as the difficulty in immediately detecting suspicious behavior, reducing false alarms, and predicting future suspicious behavior. Conventional systems have faced challenges such as the generation of incorrect alerts and the difficulty in appropriately adjusting security levels.

[0269] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0270] In this invention, the server includes means for immediately analyzing visual information acquired via data acquisition means to identify suspicious behavior, means for identifying authorized individuals using authentication information registered in a database, and means for analyzing environmental information using a learning module to dynamically adjust the alert level. This enables rapid detection and reporting of suspicious behavior, thereby improving the accuracy of security.

[0271] "Data acquisition means" refers to devices or protocols that have the function of collecting visual information from a specific area and transmitting it to other system components.

[0272] "Visual information" refers to video data and image information acquired using cameras or similar devices.

[0273] "Immediate analysis" refers to performing data analysis immediately after data is acquired, without any waiting time or delay.

[0274] "Suspicious behavior" refers to actions or movements that deviate from normal patterns or are otherwise questionable, and may pose a security risk.

[0275] A "database" is a system for organizing, storing, and managing specific information, and can include authentication information and historical data.

[0276] "Authentication information" refers to information or data used to identify individuals or objects and to verify their authorized status.

[0277] A "learning module" refers to an algorithm or process that recognizes patterns based on past data and uses them to predict future trends or adjust the system.

[0278] "Environmental information" refers to external conditions and circumstances under which the system operates, such as weather and time of day data.

[0279] The "alert level" refers to an indicator or set value that shows how much attention the system should pay to suspicious behavior.

[0280] "Dynamically adjust" means to change the system settings and operations in real time according to the conditions that change every moment.

[0281] "Quick detection of suspicious behavior" means to discover suspicious behaviors or actions in a short time and immediately recognize them in order to take appropriate measures.

[0282] "Report" refers to a document or digital message constructed to notify relevant parties of detailed information about the events and activities detected by the system.

[0283] To implement this invention, it is necessary to install multiple terminals in the monitoring area and connect them to the server via a network. The terminals include high-resolution cameras and play the role of constantly collecting visual information and transmitting it to the server. The server is equipped with a high-performance processor and a graphics processing unit, and by using, for example, "NVIDIA Tesla V100", it enables real-time data analysis.

[0284] The server preprocesses the video data transmitted from the terminals using the "OpenCV" library, performing noise removal and feature extraction. Then, it uses an AI model such as "YOLOv5" to detect moving objects and analyze action patterns. In addition, by utilizing the "Facial Recognition System" to identify individuals in the video and compare them with authentication information, it identifies suspicious persons.

[0285] Furthermore, the server analyzes environmental data using the "Scikit-learn" library and learns past data patterns. As a result, the security level can be dynamically adjusted and false alarms can be reduced. By leveraging the generation module, alerts and reports can be generated and sent to relevant personnel. The alert messages are summarized using the "OpenAI API", so that relevant personnel are notified in a quick and easy-to-understand form.

[0286] As a specific example, in an office building, if there is suspicious movement at night, the terminal immediately sends the video to the server. The server uses an AI model to detect suspicious behavior and issues an alarm if necessary to notify relevant personnel. During an event, according to the flow of people different from normal, the server can appropriately adjust the security level and maintain the accuracy of monitoring.

[0287] As an example of a prompt sentence, when an instruction such as "Start analyzing video data and identify if there are any suspicious persons" is given, the system can immediately start corresponding operations. Thus, this invention can utilize advanced AI technology and data analysis technology to achieve efficient and effective security monitoring.

[0288] The flow of the specific process in Example 1 will be described using FIG. 11.

[0289] Step 1:

[0290] The terminal continuously acquires video data within the monitoring area and sends it to the server in real time. The input at this time is raw video data from the camera, and the output is encrypted video data sent to the server via the network. As a specific operation, the terminal periodically generates data packets and sends the data using a secure protocol.

[0291] Step 2:

[0292] The server immediately receives video data from the terminal and preprocesses it using the "OpenCV" library. The input is the received raw video data, and the output is analysis-ready data with noise removed and features extracted. The server performs filtering and edge detection on each frame of the video and formats it into a format suitable for analysis.

[0293] Step 3:

[0294] The server analyzes the prepared video data using an AI model to detect suspicious movements and behaviors. The input for this step is formatted video data, and the output is identification information for suspicious movements. The "YOLOv5" model is used to detect moving objects and analyze behavioral patterns to identify abnormal movements.

[0295] Step 4:

[0296] The server uses the "Facial Recognition System" to compare detected behavior with registered authentication information and identify the target. The input consists of identification information of the suspicious behavior and facial image data, and the output is the result of whether or not the user is authenticated. The server performs a rapid matching with the facial database to confirm whether or not the user is authorized.

[0297] Step 5:

[0298] Based on the analysis results, the server issues alarms if necessary, generates alert messages, and notifies relevant parties. Inputs are authentication judgment results and operational identification information, while outputs are the generated alarms and summarized alert messages. The server uses the OpenAI API to organize the messages and send them to relevant parties in an easily understandable format via email or notification systems.

[0299] Step 6:

[0300] The server uses the "Scikit-learn" library to analyze environmental data and adjust security levels. Input consists of historical environmental data and analyzed operational data, while output is the adjusted security level setting. The server then uses this to optimize alert levels based on specific time periods and conditions.

[0301] By executing each step sequentially in this manner, the system can effectively monitor suspicious individuals and take necessary actions quickly.

[0302] (Application Example 1)

[0303] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0304] In monitoring systems, conventional methods are prone to misidentification when detecting suspicious activity in real time, and there is a lack of adequate means to quickly grasp the situation from remote locations. Furthermore, typical security systems have the problem of being difficult to flexibly adjust to changes in the environment, and alert notifications are cumbersome and time-consuming to understand.

[0305] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0306] In this invention, the server includes means for analyzing video data acquired by an image acquisition device in real time and detecting suspicious activity, means for identifying authorized individuals using registered identification information, and means for receiving notifications on a mobile terminal and displaying the video in real time. This reduces misidentification and enables rapid situation assessment from remote locations. Furthermore, it enables automatic adjustment of security levels in response to environmental changes and allows for simple alert notifications using summary messages generated with generation technology.

[0307] The "image acquisition device" is a device for acquiring video data in the monitoring area in real time.

[0308] "Real-time analysis" is a technology that immediately analyzes the acquired video data and quickly detects suspicious actions.

[0309] "Identification information" is information used to confirm a registered person and includes face data and the like.

[0310] The "learning device" is a device for automatically adjusting the security level based on environmental data.

[0311] The "generation device" is a device used to generate alarms and reports and notify the relevant parties.

[0312] The "mobile terminal" is a device used by the user to monitor the situation remotely and receive notifications.

[0313] The system of this invention is composed of multiple image acquisition devices, a server, and a mobile terminal. The image acquisition device continuously acquires video data in the monitoring area. The acquired video data is transmitted to the server, and the server analyzes the data in real time. AI technology is used in this analysis, and it is possible to detect suspicious actions from the acquired video. The server identifies the permitted persons based on the registered identification information, and when a suspicious action is detected, the generation device generates alarms and reports.

[0314] The generated alarms and reports are transmitted to the mobile terminals of the relevant parties via the notification system. Thereby, the user can monitor the situation in real time from a remote location and take immediate action as needed. Also, a learning device is incorporated in the server, which has a function of automatically adjusting the security level based on environmental data.

[0315] As a concrete example, a user who is away from home at night can access their home's security cameras through an application installed on their mobile device. If suspicious activity is detected near the front door, an alert is immediately sent. The user receives a notification with a prompt message such as "(Suspicious person detected) A new face has been found in the footage. Time: 23:45, Location: Front door," and can immediately ask a neighbor or acquaintance to take action if necessary.

[0316] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0317] Step 1:

[0318] The terminal acquires video data of the monitoring area in real time. This data is continuously collected using an image acquisition device. Once the collected video data is sent to the server, it is ready for analysis.

[0319] Step 2:

[0320] The server analyzes the received video data using AI technology. It applies a motion detection algorithm to the video as input data to detect suspicious activity. This method analyzes movement patterns, and if suspicious activity is detected, the process proceeds to the next step.

[0321] Step 3:

[0322] The server compares the analyzed data with registered identification information. Specifically, it compares facial data extracted from video as input with the identification database to obtain output that identifies unauthorized individuals.

[0323] Step 4:

[0324] If suspicious activity is detected, the server uses a generator to create an alarm and report. Based on the suspicious person information received as input, it generates a report and constructs the relevant prompt messages. This output is then sent to the notification system.

[0325] Step 5:

[0326] Users receive alerts on their mobile devices and immediately check the situation. The system can receive prompt messages from the server as input and generate output that displays video in real time. This allows users to quickly assess the situation remotely and take appropriate action.

[0327] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0328] This invention relates to a system that, in addition to normal video analysis, recognizes user emotions within a surveillance system and dynamically adjusts the security level accordingly. This system comprises a terminal (login device), a server, and an emotion recognition engine.

[0329] ■ System Overview

[0330] Terminal (login device)

[0331] The terminal acquires video data in real time from the monitoring area. This video includes the user's facial expressions and is transmitted to the server.

[0332] server

[0333] The server receives video footage from the terminal and uses a standard AI agent to analyze and compare it with the footage to determine if there is any suspicious behavior. In addition, an emotion recognition engine analyzes facial expression data to identify the user's emotional state. This emotional information is fed back into the overall system operation, and the security level is adjusted as needed.

[0334] Emotion recognition engine

[0335] The emotion recognition engine recognizes emotions from facial expressions by matching them against specific patterns. This makes it possible to issue appropriate alerts if the user is experiencing anxiety or fear.

[0336] ■Example of operation

[0337] Alarm issued due to user anxiety

[0338] For example, if video footage is captured of a user looking around anxiously in a shopping mall, the emotion recognition engine can detect anxiety from their facial expression. Based on this information, the server can increase the security level and send an alert to security guards.

[0339] Rapid response in emergencies

[0340] In the event of an incident or accident, if a user at the scene is distressed, the emotion recognition engine quickly detects this distress, and the server analyzes the information to take immediate action. This minimizes confusion at the scene and enables a rapid response.

[0341] This system goes beyond simple behavioral analysis, enabling flexible and advanced security monitoring that takes into account the emotional impact of users. As a result, it allows for optimal responses tailored to the specific situation on-site, thereby improving security.

[0342] The following describes the processing flow.

[0343] Step 1:

[0344] The terminal captures video data in real time from the monitoring area. The video data, which also includes information about the user's facial expressions, is streamed to the server.

[0345] Step 2:

[0346] The server receives video data transmitted from the terminal and starts normal motion analysis using an AI agent. In particular, it identifies moving objects and determines whether or not there is any abnormal behavior.

[0347] Step 3:

[0348] The server uses an emotion recognition engine to analyze the user's facial expressions in the video and identify their emotional state. The analysis results are then classified according to the emotional state (e.g., relief, excitement, anxiety, fear, etc.).

[0349] Step 4:

[0350] The server integrates the results of behavioral and sentiment analysis to adjust the overall system security level. If suspicious behavior or situations indicating anxiety are detected, it will consider increasing the security level and issuing an alarm.

[0351] Step 5:

[0352] The server uses a generator to produce alerts and detailed situation reports based on analyzed behavioral and emotional information, and immediately notifies relevant parties. This information also includes data to guide users towards the ideal response.

[0353] Step 6:

[0354] The server learns user behavior and emotional patterns based on past data. Through the training data, including the results of the emotion recognition engine, it predicts future abnormal behavior and potential risks, and uses this information in security planning.

[0355] This process enables the system to perform flexible monitoring and responses that take into account emotional factors depending on the situation.

[0356] (Example 2)

[0357] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0358] Modern surveillance systems are required not only to analyze behavior but also to accurately grasp the emotional state of users and dynamically adjust security measures based on that information. Existing technologies are limited to analyzing only individual behavior and react only to suspicious actions, which leads to the oversight of potential threats and false alarms. Furthermore, they are not good at detecting users' anxieties in advance and providing rapid alerts, highlighting the need for more advanced security surveillance systems.

[0359] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0360] In this invention, the server includes means for analyzing visual information acquired by an image acquisition device in real time and detecting abnormal operation, means for learning environmental information through a data analysis device and dynamically adjusting the maintenance level, and means for analyzing the user's emotions using an emotion recognition device and adjusting the system's operation based on the emotion information. This enables comprehensive security monitoring that combines both motion analysis and emotion analysis, improving the accuracy of anomaly detection, reducing false alarms, and enabling rapid maintenance response.

[0361] An "image acquisition device" is a device that acquires visual information from a monitoring area in real time.

[0362] "Abnormal operation" refers to behavior that deviates from normal operation and is an action that is subject to monitoring within a system.

[0363] A "data analysis device" is a device that has the function of learning and analyzing acquired environmental information and adjusting the system's maintenance level based on that information.

[0364] "Security level" refers to the level of security strength set within the system, which is dynamically changed according to the degree of the threat.

[0365] A "notification device" is a device that has the function of transmitting alarms and reports generated by the system to the relevant parties.

[0366] "Historical information" refers to past data and is used to predict future abnormal behavior.

[0367] An "emotion recognition device" is a device that analyzes the user's emotional state and adjusts the system's operation based on that information.

[0368] This invention provides advanced security by combining motion analysis and emotion analysis within a surveillance system. The system primarily consists of terminals, servers, an emotion recognition engine, and other related devices.

[0369] The terminals are placed in the monitoring area and collect visual information in real time. Specifically, they use cameras to capture people's actions and facial expressions and transmit the video data to a server.

[0370] The server is responsible for analyzing the video data transmitted from the terminal. Using AI agents, the server detects abnormal behavior from the acquired visual information and further identifies emotions by analyzing the user's facial expressions using an emotion recognition engine. The emotion recognition engine learns specific facial expression patterns and has the ability to identify emotions such as anxiety and tension in real time.

[0371] Based on this information, the server learns environmental data through a data analysis device and dynamically adjusts the overall system's maintenance level. This automatically triggers alarms and notifies relevant parties when necessary.

[0372] For example, in a shopping mall, if a problem has occurred in a specific area in the past, monitoring of that area can be strengthened based on the historical information. Furthermore, if user anxiety is detected through emotion recognition, measures such as deploying additional security guards can be taken quickly.

[0373] Examples of prompt messages are as follows:

[0374] "Please explain the procedure for analyzing user emotions based on surveillance footage from a shopping mall and automatically raising the security level if anxiety is detected."

[0375] This invention, by integrating individual devices and analysis techniques, can provide flexible and highly accurate security management that goes beyond normal operational monitoring.

[0376] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0377] Step 1:

[0378] The terminal acquires visual information in real time using cameras installed in the monitoring area and transmits the video data to the server. The input is raw visual information captured by the cameras, and the output is a compressed data stream of that information. This data stream is transferred to the server and used for subsequent analysis processing.

[0379] Step 2:

[0380] The server receives visual information transmitted from the terminal and analyzes abnormal behavior using an AI agent. The input is a compressed data stream from the terminal, and the output is the analyzed behavioral data. The server detects suspicious behavior and records the results in a database. In this process, a pre-trained algorithm is used to identify anomalies by comparing them with behavioral patterns.

[0381] Step 3:

[0382] The server uses an emotion recognition engine to analyze the user's facial expressions in the video and identify their emotions. The input is the video data after motion analysis, and the output is the analysis result indicating the user's emotional state. If the identified emotion is anxiety or tension, the server issues an alert. This analysis includes recognition of facial patterns and calculation of the degree of match by an AI model.

[0383] Step 4:

[0384] The server adjusts the overall system security level based on the analysis results. Inputs are behavioral data and sentiment analysis results, while outputs are the adjusted security level and corresponding alarms. This adjustment is performed according to system policies based on a set of scripted rules, and notifications are sent to relevant parties.

[0385] Step 5:

[0386] The server uses historical data to predict future abnormal behavior. The input is historical monitoring data, and the output is predicted data for future anomaly detection. This allows the system to take proactive measures before anticipated events occur, enabling rapid and accurate security management.

[0387] (Application Example 2)

[0388] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0389] In modern society, security measures are becoming increasingly important. However, conventional monitoring systems are primarily based on behavioral analysis and have the problem of not being able to take into account the user's emotions and psychological state. Because they cannot detect anxiety or fear early on and take appropriate security measures before it manifests, it can be difficult to respond quickly to emergencies. In addition, many false alarms and unnecessary alerts occur, causing confusion, which is also a problem. To solve these problems, there is a need for a new system that can dynamically adjust security levels based on the user's emotional state.

[0390] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0391] In this invention, the server includes means for analyzing video data acquired by an image acquisition device in real time and detecting suspicious behavior; means for recognizing the user's emotional state using an emotion recognition engine and adjusting security measures based on that information; and means for generating alerts and reports using a generation device and notifying relevant parties. This enables flexible and advanced security monitoring that takes user emotions into consideration, allowing for quick and optimal responses tailored to the situation on site.

[0392] An "image acquisition device" is a device that acquires video data from a monitoring area in real time.

[0393] "Video data" refers to a collection of visual information captured by an image acquisition device.

[0394] "Real-time analysis" refers to the process of immediately processing acquired data and outputting results.

[0395] "Suspicious behavior" refers to user actions that are unusual or considered abnormal.

[0396] "Identification information" refers to information used to recognize a specific person.

[0397] An "authorized person" is someone who has been identified through registered identification information.

[0398] A "learning device" is a device that takes in environmental data and performs analysis and pattern recognition.

[0399] "Security level" is a standard that indicates the degree of vigilance in a monitoring system.

[0400] An "emotion recognition engine" is a mechanism that determines a user's emotional state from their facial expressions and actions.

[0401] "Security measures" refer to specific steps taken to ensure the safety of a system.

[0402] A "generation device" is a device that creates appropriate alerts and reports and notifies relevant parties.

[0403] An "alert" is a warning signal issued when suspicious activity or anomalies are detected.

[0404] "Past data" refers to the accumulation of information recorded previously.

[0405] "Abnormal behavior" refers to actions that deviate from the normal behavioral patterns of an individual or group.

[0406] The system for realizing this invention mainly comprises an image acquisition device, a server, and an emotion recognition engine. First, the terminal uses the image acquisition device to acquire video data of the monitoring area in real time. This video data also includes the user's facial expressions, making it analyzable. Next, the server receives this video data and performs behavioral analysis using an internally installed AI agent. If suspicious behavior is detected, it acts as an alert trigger. The server also uses the emotion recognition engine to analyze the user's emotions from the video data and identify emotional states such as anxiety and fear. Based on this information, the system can dynamically adjust the security level.

[0407] The emotion recognition engine uses specific patterns to evaluate the user's facial expressions and recognize emotions in real time. This allows security alerts to be generated according to the individual's emotional state. Alerts and reports generated by the generator are notified to relevant parties in an appropriate manner. If anxiety or fear is detected, notifications are immediately sent to the user and their designated contacts.

[0408] In particular, AI-powered video analysis and emotion recognition processes commonly utilize image processing libraries such as OpenCV and machine learning frameworks such as TensorFlow. These tools enable more precise and rapid analysis of image data.

[0409] For example, if a user experiences anxiety while walking alone in a dark place, this system can immediately send a notification to family or friends to prepare for an unforeseen situation. In such scenarios, a prompt such as, "Write a script that analyzes the user's facial expressions from the camera footage to determine if they are feeling fear or anxiety. If an emotion is detected, send an alert to the designated contacts," would be useful.

[0410] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0411] Step 1:

[0412] The terminal uses an image acquisition device to acquire video data in real time from the monitoring area. The input here is the physical environment, and digital video data is generated as the output. This video data is sent to the server.

[0413] Step 2:

[0414] The server analyzes the received video data. Here, an AI agent is used for motion analysis to detect suspicious behavior. The input is video data, and the output is the detection result of suspicious behavior. Based on this result, the next process is triggered.

[0415] Step 3:

[0416] The server uses an emotion recognition engine to analyze the user's emotional state from video data. The input here is the user's facial features, which are processed using pattern recognition technology to identify emotions such as anxiety and fear. The output is the result of the emotional state analysis, which is used to adjust the security level.

[0417] Step 4:

[0418] The server uses a generator to produce alerts and reports as needed. The inputs here are the results of suspicious behavior and emotional states, which are then integrated using generation technology. The output is an alert notification, which is sent to the relevant parties.

[0419] Step 5:

[0420] If a user expresses anxiety or fear in an unexpected situation, the server promptly sends a notification to the designated contacts. The input here is the identified emotional state, and the output is a notification message. This message includes summary information generated using a generative AI model, if necessary.

[0421] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0422] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0423] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0424] [Third Embodiment]

[0425] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0426] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0427] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0428] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0429] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0430] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0431] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0432] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0433] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0434] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0435] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0436] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0437] This invention relates to a surveillance system that can detect suspicious activity by acquiring and analyzing video footage within a surveillance area in real time, and adjust the security level as needed. This system consists of multiple terminals, servers, and users.

[0438] ■ System Overview

[0439] Device (camera)

[0440] The terminal has the function of continuously acquiring video data from the monitoring area and sending it to the server.

[0441] server

[0442] The server receives video data sent from terminals and uses an AI agent to analyze the data in real time. This analysis detects the behavior patterns of moving objects and identifies suspicious individuals by comparing them with a registered database. Facial recognition technology is used to identify authorized individuals, thereby determining whether an alarm needs to be issued. Furthermore, the server is equipped with a learning device that automatically adjusts the security level based on environmental data. The server uses a generation device to generate alarms and reports and distribute them to relevant parties.

[0443] ■Example of operation

[0444] Detection of a suspicious person

[0445] For example, if an intruder enters an office building at night, the terminal captures the scene as video and sends it to the server. The server analyzes the video data, detects suspicious movements, and compares them with a registered facial database. If the person is determined to be an unauthorized individual, an alarm is triggered, and the building manager is simultaneously notified.

[0446] Adjusting security levels through learning

[0447] In areas that are usually crowded but see reduced foot traffic during specific events, the server can use environmental and historical data to increase security levels during certain time periods. This allows for adjustment of the sensitivity of suspicious person detection and reduces false alarms.

[0448] Reports and forecasts

[0449] The server automatically generates reports on detected abnormal behavior and, when sending them to administrators, uses generation technology to summarize the content and provide it in an easy-to-understand format. Furthermore, it predicts future abnormal behavior based on past data and provides information to enable timely security measures.

[0450] Thus, this system utilizes real-time analysis, facial recognition, learning capabilities, suspicious person prediction, and report generation functions to achieve effective and efficient security monitoring.

[0451] The following describes the processing flow.

[0452] Step 1:

[0453] The terminal continuously captures video data from the monitoring area and transmits it to the server in real time. The video data is encrypted with privacy in mind and transmitted in a secure manner.

[0454] Step 2:

[0455] The server receives video data transmitted from the terminal and immediately begins analysis using an AI agent. First, it evaluates the changes between frames and identifies moving objects.

[0456] Step 3:

[0457] The server analyzes the behavioral patterns of identified objects to detect abnormal behavior. Here, it compares this behavior against pre-configured alert conditions, and if suspicious behavior is detected, it proceeds to the next step.

[0458] Step 4:

[0459] The server analyzes the face of the detected person and compares it to authorized person information in the database. If the person is not registered, they are treated as an unknown person and an alarm is immediately issued.

[0460] Step 5:

[0461] Based on the detection results, the server generates alarms and status reports using a generator. The generated information is automatically notified to the user, enabling the user to take a quick response according to the situation.

[0462] Step 6:

[0463] The server collects environmental data (e.g., time of day, date, weather), and an AI agent compares this data with historical data to dynamically adjust security levels. This reduces false alarms and enables more precise monitoring.

[0464] Step 7:

[0465] Based on a history of abnormal behavior detected in the past, the server predicts future abnormal behavior. This predictive data is provided to administrators to help them develop security plans.

[0466] This process allows the system to quickly and accurately detect abnormal behavior and strengthen security measures.

[0467] (Example 1)

[0468] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0469] The aim is to solve the challenges of video surveillance systems, such as the difficulty in immediately detecting suspicious behavior, reducing false alarms, and predicting future suspicious behavior. Conventional systems have faced challenges such as the generation of incorrect alerts and the difficulty in appropriately adjusting security levels.

[0470] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0471] In this invention, the server includes means for immediately analyzing visual information acquired via data acquisition means to identify suspicious behavior, means for identifying authorized individuals using authentication information registered in a database, and means for analyzing environmental information using a learning module to dynamically adjust the alert level. This enables rapid detection and reporting of suspicious behavior, thereby improving the accuracy of security.

[0472] "Data acquisition means" refers to devices or protocols that have the function of collecting visual information from a specific area and transmitting it to other system components.

[0473] "Visual information" refers to video data and image information acquired using cameras or similar devices.

[0474] "Immediate analysis" refers to performing data analysis immediately after data is acquired, without any waiting time or delay.

[0475] "Suspicious behavior" refers to actions or movements that deviate from normal patterns or are otherwise questionable, and may pose a security risk.

[0476] A "database" is a system for organizing, storing, and managing specific information, and can include authentication information and historical data.

[0477] "Authentication information" refers to information or data used to identify individuals or objects and to verify their authorized status.

[0478] A "learning module" refers to an algorithm or process that recognizes patterns based on past data and uses them to predict future trends or adjust the system.

[0479] "Environmental information" refers to external conditions and circumstances under which the system operates, such as weather and time of day data.

[0480] "Alert level" refers to an indicator or setting that shows how much attention a system should pay to suspicious behavior.

[0481] "Dynamic adjustment" means changing the system's settings and operation in real time in response to conditions that are constantly changing.

[0482] "Rapid detection of suspicious behavior" means quickly identifying suspicious behavior or actions and recognizing them immediately in order to take appropriate action.

[0483] A "report" is a document or digital message created to notify relevant parties of detailed information about events or activities detected by the system.

[0484] To implement this invention, it is necessary to install multiple terminals in the monitoring area and connect them to a server via a network. Each terminal includes a high-resolution camera and is responsible for continuously collecting and transmitting visual information to the server. The server is equipped with a high-performance processor and graphics processing unit, such as an NVIDIA Tesla V100, enabling real-time data analysis.

[0485] The server preprocesses the video data transmitted from the terminal using the "OpenCV" library to remove noise and extract features. Then, it uses AI models such as "YOLOv5" to detect moving objects and analyze behavioral patterns. In addition, it utilizes a "Facial Recognition System" to identify individuals within the video and match them with authentication information to identify suspicious individuals.

[0486] Furthermore, the server uses the "Scikit-learn" library to analyze environmental data and learn from past data patterns. This allows for dynamic adjustment of security levels and reduction of false alarms. The generation module can be used to generate alerts and reports and send them to relevant parties. Alert messages are summarized using the "OpenAI API," ensuring that relevant parties are notified quickly and easily.

[0487] As a concrete example, in one office building, if suspicious activity is detected at night, the terminal immediately sends the video to a server. The server uses an AI model to detect the suspicious activity, issues an alarm if necessary, and notifies the relevant parties. During events, the server can appropriately adjust the security level in response to unusual pedestrian traffic, maintaining the accuracy of monitoring.

[0488] As an example of a prompt message, if the system is instructed to "Start analyzing video data and identify whether there is a suspicious person," it can immediately begin taking appropriate action. In this way, this invention utilizes advanced AI technology and data analysis technology to achieve efficient and effective security monitoring.

[0489] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0490] Step 1:

[0491] The terminal continuously acquires video data within the monitoring area and transmits it to the server in real time. The input is raw video data from the camera, and the output is encrypted video data sent to the server over the network. Specifically, the terminal periodically generates data packets and transmits this data using a secure protocol.

[0492] Step 2:

[0493] The server immediately receives video data from the terminal and preprocesses it using the "OpenCV" library. The input is the received raw video data, and the output is analysis-ready data with noise removed and features extracted. The server performs filtering and edge detection on each frame of the video and formats it into a format suitable for analysis.

[0494] Step 3:

[0495] The server analyzes the prepared video data using an AI model to detect suspicious movements and behaviors. The input for this step is formatted video data, and the output is identification information for suspicious movements. The "YOLOv5" model is used to detect moving objects and analyze behavioral patterns to identify abnormal movements.

[0496] Step 4:

[0497] The server uses the "Facial Recognition System" to compare detected behavior with registered authentication information and identify the target. The input consists of identification information of the suspicious behavior and facial image data, and the output is the result of whether or not the user is authenticated. The server performs a rapid matching with the facial database to confirm whether or not the user is authorized.

[0498] Step 5:

[0499] Based on the analysis results, the server issues alarms if necessary, generates alert messages, and notifies relevant parties. Inputs are authentication judgment results and operational identification information, while outputs are the generated alarms and summarized alert messages. The server uses the OpenAI API to organize the messages and send them to relevant parties in an easily understandable format via email or notification systems.

[0500] Step 6:

[0501] The server uses the "Scikit-learn" library to analyze environmental data and adjust security levels. Input consists of historical environmental data and analyzed operational data, while output is the adjusted security level setting. The server then uses this to optimize alert levels based on specific time periods and conditions.

[0502] By executing each step sequentially in this manner, the system can effectively monitor suspicious individuals and take necessary actions quickly.

[0503] (Application Example 1)

[0504] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0505] In monitoring systems, conventional methods are prone to misidentification when detecting suspicious activity in real time, and there is a lack of adequate means to quickly grasp the situation from remote locations. Furthermore, typical security systems have the problem of being difficult to flexibly adjust to changes in the environment, and alert notifications are cumbersome and time-consuming to understand.

[0506] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0507] In this invention, the server includes means for analyzing video data acquired by an image acquisition device in real time and detecting suspicious activity, means for identifying authorized individuals using registered identification information, and means for receiving notifications on a mobile terminal and displaying the video in real time. This reduces misidentification and enables rapid situation assessment from remote locations. Furthermore, it enables automatic adjustment of security levels in response to environmental changes and allows for simple alert notifications using summary messages generated with generation technology.

[0508] An "image acquisition device" is a device used to acquire video data within a monitoring area in real time.

[0509] "Real-time analysis" is a technology that instantly analyzes acquired video data to quickly detect suspicious activity.

[0510] "Identification information" refers to information used to verify a registered person, and includes facial data, etc.

[0511] A "learning device" is a device that automatically adjusts security levels based on environmental data.

[0512] A "generation device" is a device used to generate alarms and reports and notify relevant parties.

[0513] A "mobile device" is a device used by a user to monitor a situation remotely and receive notifications.

[0514] The system of this invention consists of multiple image acquisition devices, a server, and a mobile terminal. The image acquisition devices continuously acquire video data within the monitoring area. The acquired video data is transmitted to the server, which analyzes the data in real time. AI technology is used for this analysis, making it possible to detect suspicious activity from the acquired video. The server identifies authorized individuals based on registered identification information, and if suspicious activity is detected, it generates an alarm and report using a generation device.

[0515] The generated alarms and reports are sent to the mobile devices of relevant personnel via the notification system. This allows users to monitor the situation in real time from a remote location and take immediate action as needed. Furthermore, the server incorporates a learning device that automatically adjusts security levels based on environmental data.

[0516] As a concrete example, a user who is away from home at night can access their home's security cameras through an application installed on their mobile device. If suspicious activity is detected near the front door, an alert is immediately sent. The user receives a notification with a prompt message such as "(Suspicious person detected) A new face has been found in the footage. Time: 23:45, Location: Front door," and can immediately ask a neighbor or acquaintance to take action if necessary.

[0517] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0518] Step 1:

[0519] The terminal acquires video data of the monitoring area in real time. This data is continuously collected using an image acquisition device. Once the collected video data is sent to the server, it is ready for analysis.

[0520] Step 2:

[0521] The server analyzes the received video data using AI technology. It applies a motion detection algorithm to the video as input data to detect suspicious activity. This method analyzes movement patterns, and if suspicious activity is detected, the process proceeds to the next step.

[0522] Step 3:

[0523] The server compares the analyzed data with registered identification information. Specifically, it compares facial data extracted from video as input with the identification database to obtain output that identifies unauthorized individuals.

[0524] Step 4:

[0525] If suspicious activity is detected, the server uses a generator to create an alarm and report. Based on the suspicious person information received as input, it generates a report and constructs the relevant prompt messages. This output is then sent to the notification system.

[0526] Step 5:

[0527] Users receive alerts on their mobile devices and immediately check the situation. The system can receive prompt messages from the server as input and generate output that displays video in real time. This allows users to quickly assess the situation remotely and take appropriate action.

[0528] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0529] This invention relates to a system that, in addition to normal video analysis, recognizes user emotions within a surveillance system and dynamically adjusts the security level accordingly. This system comprises a terminal (login device), a server, and an emotion recognition engine.

[0530] ■ System Overview

[0531] Terminal (login device)

[0532] The terminal acquires video data in real time from the monitoring area. This video includes the user's facial expressions and is transmitted to the server.

[0533] server

[0534] The server receives video footage from the terminal and uses a standard AI agent to analyze and compare it with the footage to determine if there is any suspicious behavior. In addition, an emotion recognition engine analyzes facial expression data to identify the user's emotional state. This emotional information is fed back into the overall system operation, and the security level is adjusted as needed.

[0535] Emotion recognition engine

[0536] The emotion recognition engine recognizes emotions from facial expressions by matching them against specific patterns. This makes it possible to issue appropriate alerts if the user is experiencing anxiety or fear.

[0537] ■Example of operation

[0538] Alarm issued due to user anxiety

[0539] For example, if video footage is captured of a user looking around anxiously in a shopping mall, the emotion recognition engine can detect anxiety from their facial expression. Based on this information, the server can increase the security level and send an alert to security guards.

[0540] Rapid response in emergencies

[0541] In the event of an incident or accident, if a user at the scene is distressed, the emotion recognition engine quickly detects this distress, and the server analyzes the information to take immediate action. This minimizes confusion at the scene and enables a rapid response.

[0542] This system goes beyond simple behavioral analysis, enabling flexible and advanced security monitoring that takes into account the emotional impact of users. As a result, it allows for optimal responses tailored to the specific situation on-site, thereby improving security.

[0543] The following describes the processing flow.

[0544] Step 1:

[0545] The terminal captures video data in real time from the monitoring area. The video data, which also includes information about the user's facial expressions, is streamed to the server.

[0546] Step 2:

[0547] The server receives video data transmitted from the terminal and starts normal motion analysis using an AI agent. In particular, it identifies moving objects and determines whether or not there is any abnormal behavior.

[0548] Step 3:

[0549] The server uses an emotion recognition engine to analyze the user's facial expressions in the video and identify their emotional state. The analysis results are then classified according to the emotional state (e.g., relief, excitement, anxiety, fear, etc.).

[0550] Step 4:

[0551] The server integrates the results of behavioral and sentiment analysis to adjust the overall system security level. If suspicious behavior or situations indicating anxiety are detected, it will consider increasing the security level and issuing an alarm.

[0552] Step 5:

[0553] The server uses a generator to produce alerts and detailed situation reports based on analyzed behavioral and emotional information, and immediately notifies relevant parties. This information also includes data to guide users towards the ideal response.

[0554] Step 6:

[0555] The server learns user behavior and emotional patterns based on past data. Through the training data, including the results of the emotion recognition engine, it predicts future abnormal behavior and potential risks, and uses this information in security planning.

[0556] This process enables the system to perform flexible monitoring and responses that take into account emotional factors depending on the situation.

[0557] (Example 2)

[0558] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0559] Modern surveillance systems are required not only to analyze behavior but also to accurately grasp the emotional state of users and dynamically adjust security measures based on that information. Existing technologies are limited to analyzing only individual behavior and react only to suspicious actions, which leads to the oversight of potential threats and false alarms. Furthermore, they are not good at detecting users' anxieties in advance and providing rapid alerts, highlighting the need for more advanced security surveillance systems.

[0560] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0561] In this invention, the server includes means for analyzing visual information acquired by an image acquisition device in real time and detecting abnormal operation, means for learning environmental information through a data analysis device and dynamically adjusting the maintenance level, and means for analyzing the user's emotions using an emotion recognition device and adjusting the system's operation based on the emotion information. This enables comprehensive security monitoring that combines both motion analysis and emotion analysis, improving the accuracy of anomaly detection, reducing false alarms, and enabling rapid maintenance response.

[0562] An "image acquisition device" is a device that acquires visual information from a monitoring area in real time.

[0563] "Abnormal operation" refers to behavior that deviates from normal operation and is an action that is subject to monitoring within a system.

[0564] A "data analysis device" is a device that has the function of learning and analyzing acquired environmental information and adjusting the system's maintenance level based on that information.

[0565] "Security level" refers to the level of security strength set within the system, which is dynamically changed according to the degree of the threat.

[0566] A "notification device" is a device that has the function of transmitting alarms and reports generated by the system to the relevant parties.

[0567] "Historical information" refers to past data and is used to predict future abnormal behavior.

[0568] An "emotion recognition device" is a device that analyzes the user's emotional state and adjusts the system's operation based on that information.

[0569] This invention provides advanced security by combining motion analysis and emotion analysis within a surveillance system. The system primarily consists of terminals, servers, an emotion recognition engine, and other related devices.

[0570] The terminals are placed in the monitoring area and collect visual information in real time. Specifically, they use cameras to capture people's actions and facial expressions and transmit the video data to a server.

[0571] The server is responsible for analyzing the video data transmitted from the terminal. Using AI agents, the server detects abnormal behavior from the acquired visual information and further identifies emotions by analyzing the user's facial expressions using an emotion recognition engine. The emotion recognition engine learns specific facial expression patterns and has the ability to identify emotions such as anxiety and tension in real time.

[0572] Based on this information, the server learns environmental data through a data analysis device and dynamically adjusts the overall system's maintenance level. This automatically triggers alarms and notifies relevant parties when necessary.

[0573] For example, in a shopping mall, if a problem has occurred in a specific area in the past, monitoring of that area can be strengthened based on the historical information. Furthermore, if user anxiety is detected through emotion recognition, measures such as deploying additional security guards can be taken quickly.

[0574] Examples of prompt messages are as follows:

[0575] "Please explain the procedure for analyzing user emotions based on surveillance footage from a shopping mall and automatically raising the security level if anxiety is detected."

[0576] This invention, by integrating individual devices and analysis techniques, can provide flexible and highly accurate security management that goes beyond normal operational monitoring.

[0577] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0578] Step 1:

[0579] The terminal acquires visual information in real time using cameras installed in the monitoring area and transmits the video data to the server. The input is raw visual information captured by the cameras, and the output is a compressed data stream of that information. This data stream is transferred to the server and used for subsequent analysis processing.

[0580] Step 2:

[0581] The server receives visual information transmitted from the terminal and analyzes abnormal behavior using an AI agent. The input is a compressed data stream from the terminal, and the output is the analyzed behavioral data. The server detects suspicious behavior and records the results in a database. In this process, a pre-trained algorithm is used to identify anomalies by comparing them with behavioral patterns.

[0582] Step 3:

[0583] The server uses an emotion recognition engine to analyze the user's facial expressions in the video and identify their emotions. The input is the video data after motion analysis, and the output is the analysis result indicating the user's emotional state. If the identified emotion is anxiety or tension, the server issues an alert. This analysis includes recognition of facial patterns and calculation of the degree of match by an AI model.

[0584] Step 4:

[0585] The server adjusts the overall system security level based on the analysis results. Inputs are behavioral data and sentiment analysis results, while outputs are the adjusted security level and corresponding alarms. This adjustment is performed according to system policies based on a set of scripted rules, and notifications are sent to relevant parties.

[0586] Step 5:

[0587] The server uses historical data to predict future abnormal behavior. The input is historical monitoring data, and the output is predicted data for future anomaly detection. This allows the system to take proactive measures before anticipated events occur, enabling rapid and accurate security management.

[0588] (Application Example 2)

[0589] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0590] In modern society, security measures are becoming increasingly important. However, conventional monitoring systems are primarily based on behavioral analysis and have the problem of not being able to take into account the user's emotions and psychological state. Because they cannot detect anxiety or fear early on and take appropriate security measures before it manifests, it can be difficult to respond quickly to emergencies. In addition, many false alarms and unnecessary alerts occur, causing confusion, which is also a problem. To solve these problems, there is a need for a new system that can dynamically adjust security levels based on the user's emotional state.

[0591] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0592] In this invention, the server includes means for analyzing video data acquired by an image acquisition device in real time and detecting suspicious behavior; means for recognizing the user's emotional state using an emotion recognition engine and adjusting security measures based on that information; and means for generating alerts and reports using a generation device and notifying relevant parties. This enables flexible and advanced security monitoring that takes user emotions into consideration, allowing for quick and optimal responses tailored to the situation on site.

[0593] An "image acquisition device" is a device that acquires video data from a monitoring area in real time.

[0594] "Video data" refers to a collection of visual information captured by an image acquisition device.

[0595] "Real-time analysis" refers to the process of immediately processing acquired data and outputting results.

[0596] "Suspicious behavior" refers to user actions that are unusual or considered abnormal.

[0597] "Identification information" refers to information used to recognize a specific person.

[0598] An "authorized person" is someone who has been identified through registered identification information.

[0599] A "learning device" is a device that takes in environmental data and performs analysis and pattern recognition.

[0600] "Security level" is a standard that indicates the degree of vigilance in a monitoring system.

[0601] An "emotion recognition engine" is a mechanism that determines a user's emotional state from their facial expressions and actions.

[0602] "Security measures" refer to specific steps taken to ensure the safety of a system.

[0603] A "generation device" is a device that creates appropriate alerts and reports and notifies relevant parties.

[0604] An "alert" is a warning signal issued when suspicious activity or anomalies are detected.

[0605] "Past data" refers to the accumulation of information recorded previously.

[0606] "Abnormal behavior" refers to actions that deviate from the normal behavioral patterns of an individual or group.

[0607] The system for realizing this invention mainly comprises an image acquisition device, a server, and an emotion recognition engine. First, the terminal uses the image acquisition device to acquire video data of the monitoring area in real time. This video data also includes the user's facial expressions, making it analyzable. Next, the server receives this video data and performs behavioral analysis using an internally installed AI agent. If suspicious behavior is detected, it acts as an alert trigger. The server also uses the emotion recognition engine to analyze the user's emotions from the video data and identify emotional states such as anxiety and fear. Based on this information, the system can dynamically adjust the security level.

[0608] The emotion recognition engine uses specific patterns to evaluate the user's facial expressions and recognize emotions in real time. This allows security alerts to be generated according to the individual's emotional state. Alerts and reports generated by the generator are notified to relevant parties in an appropriate manner. If anxiety or fear is detected, notifications are immediately sent to the user and their designated contacts.

[0609] In particular, AI-powered video analysis and emotion recognition processes commonly utilize image processing libraries such as OpenCV and machine learning frameworks such as TensorFlow. These tools enable more precise and rapid analysis of image data.

[0610] For example, if a user experiences anxiety while walking alone in a dark place, this system can immediately send a notification to family or friends to prepare for an unforeseen situation. In such scenarios, a prompt such as, "Write a script that analyzes the user's facial expressions from the camera footage to determine if they are feeling fear or anxiety. If an emotion is detected, send an alert to the designated contacts," would be useful.

[0611] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0612] Step 1:

[0613] The terminal uses an image acquisition device to acquire video data in real time from the monitoring area. The input here is the physical environment, and digital video data is generated as the output. This video data is sent to the server.

[0614] Step 2:

[0615] The server analyzes the received video data. Here, an AI agent is used for motion analysis to detect suspicious behavior. The input is video data, and the output is the detection result of suspicious behavior. Based on this result, the next process is triggered.

[0616] Step 3:

[0617] The server uses an emotion recognition engine to analyze the user's emotional state from video data. The input here is the user's facial features, which are processed using pattern recognition technology to identify emotions such as anxiety and fear. The output is the result of the emotional state analysis, which is used to adjust the security level.

[0618] Step 4:

[0619] The server uses a generator to produce alerts and reports as needed. The inputs here are the results of suspicious behavior and emotional states, which are then integrated using generation technology. The output is an alert notification, which is sent to the relevant parties.

[0620] Step 5:

[0621] If a user expresses anxiety or fear in an unexpected situation, the server promptly sends a notification to the designated contacts. The input here is the identified emotional state, and the output is a notification message. This message includes summary information generated using a generative AI model, if necessary.

[0622] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0623] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0624] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0625] [Fourth Embodiment]

[0626] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0627] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0628] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0629] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0630] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0631] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0632] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0633] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0634] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0635] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0636] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0637] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0638] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0639] This invention relates to a surveillance system that can detect suspicious activity by acquiring and analyzing video footage within a surveillance area in real time, and adjust the security level as needed. This system consists of multiple terminals, servers, and users.

[0640] ■ System Overview

[0641] Device (camera)

[0642] The terminal has the function of continuously acquiring video data from the monitoring area and sending it to the server.

[0643] server

[0644] The server receives video data sent from terminals and uses an AI agent to analyze the data in real time. This analysis detects the behavior patterns of moving objects and identifies suspicious individuals by comparing them with a registered database. Facial recognition technology is used to identify authorized individuals, thereby determining whether an alarm needs to be issued. Furthermore, the server is equipped with a learning device that automatically adjusts the security level based on environmental data. The server uses a generation device to generate alarms and reports and distribute them to relevant parties.

[0645] ■Example of operation

[0646] Detection of a suspicious person

[0647] For example, if an intruder enters an office building at night, the terminal captures the scene as video and sends it to the server. The server analyzes the video data, detects suspicious movements, and compares them with a registered facial database. If the person is determined to be an unauthorized individual, an alarm is triggered, and the building manager is simultaneously notified.

[0648] Adjusting security levels through learning

[0649] In areas that are usually crowded but see reduced foot traffic during specific events, the server can use environmental and historical data to increase security levels during certain time periods. This allows for adjustment of the sensitivity of suspicious person detection and reduces false alarms.

[0650] Reports and forecasts

[0651] The server automatically generates reports on detected abnormal behavior and, when sending them to administrators, uses generation technology to summarize the content and provide it in an easy-to-understand format. Furthermore, it predicts future abnormal behavior based on past data and provides information to enable timely security measures.

[0652] Thus, this system utilizes real-time analysis, facial recognition, learning capabilities, suspicious person prediction, and report generation functions to achieve effective and efficient security monitoring.

[0653] The following describes the processing flow.

[0654] Step 1:

[0655] The terminal continuously captures video data from the monitoring area and transmits it to the server in real time. The video data is encrypted with privacy in mind and transmitted in a secure manner.

[0656] Step 2:

[0657] The server receives video data transmitted from the terminal and immediately begins analysis using an AI agent. First, it evaluates the changes between frames and identifies moving objects.

[0658] Step 3:

[0659] The server analyzes the behavioral patterns of identified objects to detect abnormal behavior. Here, it compares this behavior against pre-configured alert conditions, and if suspicious behavior is detected, it proceeds to the next step.

[0660] Step 4:

[0661] The server analyzes the face of the detected person and compares it to authorized person information in the database. If the person is not registered, they are treated as an unknown person and an alarm is immediately issued.

[0662] Step 5:

[0663] Based on the detection results, the server generates alarms and status reports using a generator. The generated information is automatically notified to the user, enabling the user to take a quick response according to the situation.

[0664] Step 6:

[0665] The server collects environmental data (e.g., time of day, date, weather), and an AI agent compares this data with historical data to dynamically adjust security levels. This reduces false alarms and enables more precise monitoring.

[0666] Step 7:

[0667] Based on a history of abnormal behavior detected in the past, the server predicts future abnormal behavior. This predictive data is provided to administrators to help them develop security plans.

[0668] This process allows the system to quickly and accurately detect abnormal behavior and strengthen security measures.

[0669] (Example 1)

[0670] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0671] The aim is to solve the challenges of video surveillance systems, such as the difficulty in immediately detecting suspicious behavior, reducing false alarms, and predicting future suspicious behavior. Conventional systems have faced challenges such as the generation of incorrect alerts and the difficulty in appropriately adjusting security levels.

[0672] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0673] In this invention, the server includes means for immediately analyzing visual information acquired via data acquisition means to identify suspicious behavior, means for identifying authorized individuals using authentication information registered in a database, and means for analyzing environmental information using a learning module to dynamically adjust the alert level. This enables rapid detection and reporting of suspicious behavior, thereby improving the accuracy of security.

[0674] "Data acquisition means" refers to devices or protocols that have the function of collecting visual information from a specific area and transmitting it to other system components.

[0675] "Visual information" refers to video data and image information acquired using cameras or similar devices.

[0676] "Immediate analysis" refers to performing data analysis immediately after data is acquired, without any waiting time or delay.

[0677] "Suspicious behavior" refers to actions or movements that deviate from normal patterns or are otherwise questionable, and may pose a security risk.

[0678] A "database" is a system for organizing, storing, and managing specific information, and can include authentication information and historical data.

[0679] "Authentication information" refers to information or data used to identify individuals or objects and to verify their authorized status.

[0680] A "learning module" refers to an algorithm or process that recognizes patterns based on past data and uses them to predict future trends or adjust the system.

[0681] "Environmental information" refers to external conditions and circumstances under which the system operates, such as weather and time of day data.

[0682] "Alert level" refers to an indicator or setting that shows how much attention a system should pay to suspicious behavior.

[0683] "Dynamic adjustment" means changing the system's settings and operation in real time in response to conditions that are constantly changing.

[0684] "Rapid detection of suspicious behavior" means quickly identifying suspicious behavior or actions and recognizing them immediately in order to take appropriate action.

[0685] A "report" is a document or digital message created to notify relevant parties of detailed information about events or activities detected by the system.

[0686] To implement this invention, it is necessary to install multiple terminals in the monitoring area and connect them to a server via a network. Each terminal includes a high-resolution camera and is responsible for continuously collecting and transmitting visual information to the server. The server is equipped with a high-performance processor and graphics processing unit, such as an NVIDIA Tesla V100, enabling real-time data analysis.

[0687] The server preprocesses the video data transmitted from the terminal using the "OpenCV" library to remove noise and extract features. Then, it uses AI models such as "YOLOv5" to detect moving objects and analyze behavioral patterns. In addition, it utilizes a "Facial Recognition System" to identify individuals within the video and match them with authentication information to identify suspicious individuals.

[0688] Furthermore, the server uses the "Scikit-learn" library to analyze environmental data and learn from past data patterns. This allows for dynamic adjustment of security levels and reduction of false alarms. The generation module can be used to generate alerts and reports and send them to relevant parties. Alert messages are summarized using the "OpenAI API," ensuring that relevant parties are notified quickly and easily.

[0689] As a concrete example, in one office building, if suspicious activity is detected at night, the terminal immediately sends the video to a server. The server uses an AI model to detect the suspicious activity, issues an alarm if necessary, and notifies the relevant parties. During events, the server can appropriately adjust the security level in response to unusual pedestrian traffic, maintaining the accuracy of monitoring.

[0690] As an example of a prompt message, if the system is instructed to "Start analyzing video data and identify whether there is a suspicious person," it can immediately begin taking appropriate action. In this way, this invention utilizes advanced AI technology and data analysis technology to achieve efficient and effective security monitoring.

[0691] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0692] Step 1:

[0693] The terminal continuously acquires video data within the monitoring area and transmits it to the server in real time. The input is raw video data from the camera, and the output is encrypted video data sent to the server over the network. Specifically, the terminal periodically generates data packets and transmits this data using a secure protocol.

[0694] Step 2:

[0695] The server immediately receives video data from the terminal and preprocesses it using the "OpenCV" library. The input is the received raw video data, and the output is analysis-ready data with noise removed and features extracted. The server performs filtering and edge detection on each frame of the video and formats it into a format suitable for analysis.

[0696] Step 3:

[0697] The server analyzes the prepared video data using an AI model to detect suspicious movements and behaviors. The input for this step is formatted video data, and the output is identification information for suspicious movements. The "YOLOv5" model is used to detect moving objects and analyze behavioral patterns to identify abnormal movements.

[0698] Step 4:

[0699] The server uses the "Facial Recognition System" to compare detected behavior with registered authentication information and identify the target. The input consists of identification information of the suspicious behavior and facial image data, and the output is the result of whether or not the user is authenticated. The server performs a rapid matching with the facial database to confirm whether or not the user is authorized.

[0700] Step 5:

[0701] Based on the analysis results, the server issues alarms if necessary, generates alert messages, and notifies relevant parties. Inputs are authentication judgment results and operational identification information, while outputs are the generated alarms and summarized alert messages. The server uses the OpenAI API to organize the messages and send them to relevant parties in an easily understandable format via email or notification systems.

[0702] Step 6:

[0703] The server uses the "Scikit-learn" library to analyze environmental data and adjust security levels. Input consists of historical environmental data and analyzed operational data, while output is the adjusted security level setting. The server then uses this to optimize alert levels based on specific time periods and conditions.

[0704] By executing each step sequentially in this manner, the system can effectively monitor suspicious individuals and take necessary actions quickly.

[0705] (Application Example 1)

[0706] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0707] In monitoring systems, conventional methods are prone to misidentification when detecting suspicious activity in real time, and there is a lack of adequate means to quickly grasp the situation from remote locations. Furthermore, typical security systems have the problem of being difficult to flexibly adjust to changes in the environment, and alert notifications are cumbersome and time-consuming to understand.

[0708] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0709] In this invention, the server includes means for analyzing video data acquired by an image acquisition device in real time and detecting suspicious activity, means for identifying authorized individuals using registered identification information, and means for receiving notifications on a mobile terminal and displaying the video in real time. This reduces misidentification and enables rapid situation assessment from remote locations. Furthermore, it enables automatic adjustment of security levels in response to environmental changes and allows for simple alert notifications using summary messages generated with generation technology.

[0710] An "image acquisition device" is a device used to acquire video data within a monitoring area in real time.

[0711] "Real-time analysis" is a technology that instantly analyzes acquired video data to quickly detect suspicious activity.

[0712] "Identification information" refers to information used to verify a registered person, and includes facial data, etc.

[0713] A "learning device" is a device that automatically adjusts security levels based on environmental data.

[0714] A "generation device" is a device used to generate alarms and reports and notify relevant parties.

[0715] A "mobile device" is a device used by a user to monitor a situation remotely and receive notifications.

[0716] The system of this invention consists of multiple image acquisition devices, a server, and a mobile terminal. The image acquisition devices continuously acquire video data within the monitoring area. The acquired video data is transmitted to the server, which analyzes the data in real time. AI technology is used for this analysis, making it possible to detect suspicious activity from the acquired video. The server identifies authorized individuals based on registered identification information, and if suspicious activity is detected, it generates an alarm and report using a generation device.

[0717] The generated alarms and reports are sent to the mobile devices of relevant personnel via the notification system. This allows users to monitor the situation in real time from a remote location and take immediate action as needed. Furthermore, the server incorporates a learning device that automatically adjusts security levels based on environmental data.

[0718] As a concrete example, a user who is away from home at night can access their home's security cameras through an application installed on their mobile device. If suspicious activity is detected near the front door, an alert is immediately sent. The user receives a notification with a prompt message such as "(Suspicious person detected) A new face has been found in the footage. Time: 23:45, Location: Front door," and can immediately ask a neighbor or acquaintance to take action if necessary.

[0719] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0720] Step 1:

[0721] The terminal acquires video data of the monitoring area in real time. This data is continuously collected using an image acquisition device. Once the collected video data is sent to the server, it is ready for analysis.

[0722] Step 2:

[0723] The server analyzes the received video data using AI technology. It applies a motion detection algorithm to the video as input data to detect suspicious activity. This method analyzes movement patterns, and if suspicious activity is detected, the process proceeds to the next step.

[0724] Step 3:

[0725] The server compares the analyzed data with registered identification information. Specifically, it compares facial data extracted from video as input with the identification database to obtain output that identifies unauthorized individuals.

[0726] Step 4:

[0727] If suspicious activity is detected, the server uses a generator to create an alarm and report. Based on the suspicious person information received as input, it generates a report and constructs the relevant prompt messages. This output is then sent to the notification system.

[0728] Step 5:

[0729] Users receive alerts on their mobile devices and immediately check the situation. The system can receive prompt messages from the server as input and generate output that displays video in real time. This allows users to quickly assess the situation remotely and take appropriate action.

[0730] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0731] This invention relates to a system that, in addition to normal video analysis, recognizes user emotions within a surveillance system and dynamically adjusts the security level accordingly. This system comprises a terminal (login device), a server, and an emotion recognition engine.

[0732] ■ System Overview

[0733] Terminal (login device)

[0734] The terminal acquires video data in real time from the monitoring area. This video includes the user's facial expressions and is transmitted to the server.

[0735] server

[0736] The server receives video footage from the terminal and uses a standard AI agent to analyze and compare it with the footage to determine if there is any suspicious behavior. In addition, an emotion recognition engine analyzes facial expression data to identify the user's emotional state. This emotional information is fed back into the overall system operation, and the security level is adjusted as needed.

[0737] Emotion recognition engine

[0738] The emotion recognition engine recognizes emotions from facial expressions by matching them against specific patterns. This makes it possible to issue appropriate alerts if the user is experiencing anxiety or fear.

[0739] ■Example of operation

[0740] Alarm issued due to user anxiety

[0741] For example, if video footage is captured of a user looking around anxiously in a shopping mall, the emotion recognition engine can detect anxiety from their facial expression. Based on this information, the server can increase the security level and send an alert to security guards.

[0742] Rapid response in emergencies

[0743] In the event of an incident or accident, if a user at the scene is distressed, the emotion recognition engine quickly detects this distress, and the server analyzes the information to take immediate action. This minimizes confusion at the scene and enables a rapid response.

[0744] This system goes beyond simple behavioral analysis, enabling flexible and advanced security monitoring that takes into account the emotional impact of users. As a result, it allows for optimal responses tailored to the specific situation on-site, thereby improving security.

[0745] The following describes the processing flow.

[0746] Step 1:

[0747] The terminal captures video data in real time from the monitoring area. The video data, which also includes information about the user's facial expressions, is streamed to the server.

[0748] Step 2:

[0749] The server receives video data transmitted from the terminal and starts normal motion analysis using an AI agent. In particular, it identifies moving objects and determines whether or not there is any abnormal behavior.

[0750] Step 3:

[0751] The server uses an emotion recognition engine to analyze the user's facial expressions in the video and identify their emotional state. The analysis results are then classified according to the emotional state (e.g., relief, excitement, anxiety, fear, etc.).

[0752] Step 4:

[0753] The server integrates the results of behavioral and sentiment analysis to adjust the overall system security level. If suspicious behavior or situations indicating anxiety are detected, it will consider increasing the security level and issuing an alarm.

[0754] Step 5:

[0755] The server uses a generator to produce alerts and detailed situation reports based on analyzed behavioral and emotional information, and immediately notifies relevant parties. This information also includes data to guide users towards the ideal response.

[0756] Step 6:

[0757] The server learns user behavior and emotional patterns based on past data. Through the training data, including the results of the emotion recognition engine, it predicts future abnormal behavior and potential risks, and uses this information in security planning.

[0758] This process enables the system to perform flexible monitoring and responses that take into account emotional factors depending on the situation.

[0759] (Example 2)

[0760] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0761] Modern surveillance systems are required not only to analyze behavior but also to accurately grasp the emotional state of users and dynamically adjust security measures based on that information. Existing technologies are limited to analyzing only individual behavior and react only to suspicious actions, which leads to the oversight of potential threats and false alarms. Furthermore, they are not good at detecting users' anxieties in advance and providing rapid alerts, highlighting the need for more advanced security surveillance systems.

[0762] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0763] In this invention, the server includes means for analyzing visual information acquired by an image acquisition device in real time and detecting abnormal operation, means for learning environmental information through a data analysis device and dynamically adjusting the maintenance level, and means for analyzing the user's emotions using an emotion recognition device and adjusting the system's operation based on the emotion information. This enables comprehensive security monitoring that combines both motion analysis and emotion analysis, improving the accuracy of anomaly detection, reducing false alarms, and enabling rapid maintenance response.

[0764] An "image acquisition device" is a device that acquires visual information from a monitoring area in real time.

[0765] "Abnormal operation" refers to behavior that deviates from normal operation and is an action that is subject to monitoring within a system.

[0766] A "data analysis device" is a device that has the function of learning and analyzing acquired environmental information and adjusting the system's maintenance level based on that information.

[0767] "Security level" refers to the level of security strength set within the system, which is dynamically changed according to the degree of the threat.

[0768] A "notification device" is a device that has the function of transmitting alarms and reports generated by the system to the relevant parties.

[0769] "Historical information" refers to past data and is used to predict future abnormal behavior.

[0770] An "emotion recognition device" is a device that analyzes the user's emotional state and adjusts the system's operation based on that information.

[0771] This invention provides advanced security by combining motion analysis and emotion analysis within a surveillance system. The system primarily consists of terminals, servers, an emotion recognition engine, and other related devices.

[0772] The terminals are placed in the monitoring area and collect visual information in real time. Specifically, they use cameras to capture people's actions and facial expressions and transmit the video data to a server.

[0773] The server is responsible for analyzing the video data transmitted from the terminal. Using AI agents, the server detects abnormal behavior from the acquired visual information and further identifies emotions by analyzing the user's facial expressions using an emotion recognition engine. The emotion recognition engine learns specific facial expression patterns and has the ability to identify emotions such as anxiety and tension in real time.

[0774] Based on this information, the server learns environmental data through a data analysis device and dynamically adjusts the overall system's maintenance level. This automatically triggers alarms and notifies relevant parties when necessary.

[0775] For example, in a shopping mall, if a problem has occurred in a specific area in the past, monitoring of that area can be strengthened based on the historical information. Furthermore, if user anxiety is detected through emotion recognition, measures such as deploying additional security guards can be taken quickly.

[0776] Examples of prompt messages are as follows:

[0777] "Please explain the procedure for analyzing user emotions based on surveillance footage from a shopping mall and automatically raising the security level if anxiety is detected."

[0778] This invention, by integrating individual devices and analysis techniques, can provide flexible and highly accurate security management that goes beyond normal operational monitoring.

[0779] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0780] Step 1:

[0781] The terminal acquires visual information in real time using cameras installed in the monitoring area and transmits the video data to the server. The input is raw visual information captured by the cameras, and the output is a compressed data stream of that information. This data stream is transferred to the server and used for subsequent analysis processing.

[0782] Step 2:

[0783] The server receives visual information transmitted from the terminal and analyzes abnormal behavior using an AI agent. The input is a compressed data stream from the terminal, and the output is the analyzed behavioral data. The server detects suspicious behavior and records the results in a database. In this process, a pre-trained algorithm is used to identify anomalies by comparing them with behavioral patterns.

[0784] Step 3:

[0785] The server uses an emotion recognition engine to analyze the user's facial expressions in the video and identify their emotions. The input is the video data after motion analysis, and the output is the analysis result indicating the user's emotional state. If the identified emotion is anxiety or tension, the server issues an alert. This analysis includes recognition of facial patterns and calculation of the degree of match by an AI model.

[0786] Step 4:

[0787] The server adjusts the overall system security level based on the analysis results. Inputs are behavioral data and sentiment analysis results, while outputs are the adjusted security level and corresponding alarms. This adjustment is performed according to system policies based on a set of scripted rules, and notifications are sent to relevant parties.

[0788] Step 5:

[0789] The server uses historical data to predict future abnormal behavior. The input is historical monitoring data, and the output is predicted data for future anomaly detection. This allows the system to take proactive measures before anticipated events occur, enabling rapid and accurate security management.

[0790] (Application Example 2)

[0791] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0792] In modern society, security measures are becoming increasingly important. However, conventional monitoring systems are primarily based on behavioral analysis and have the problem of not being able to take into account the user's emotions and psychological state. Because they cannot detect anxiety or fear early on and take appropriate security measures before it manifests, it can be difficult to respond quickly to emergencies. In addition, many false alarms and unnecessary alerts occur, causing confusion, which is also a problem. To solve these problems, there is a need for a new system that can dynamically adjust security levels based on the user's emotional state.

[0793] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0794] In this invention, the server includes means for analyzing video data acquired by an image acquisition device in real time and detecting suspicious behavior; means for recognizing the user's emotional state using an emotion recognition engine and adjusting security measures based on that information; and means for generating alerts and reports using a generation device and notifying relevant parties. This enables flexible and advanced security monitoring that takes user emotions into consideration, allowing for quick and optimal responses tailored to the situation on site.

[0795] An "image acquisition device" is a device that acquires video data from a monitoring area in real time.

[0796] "Video data" refers to a collection of visual information captured by an image acquisition device.

[0797] "Real-time analysis" refers to the process of immediately processing acquired data and outputting results.

[0798] "Suspicious behavior" refers to user actions that are unusual or considered abnormal.

[0799] "Identification information" refers to information used to recognize a specific person.

[0800] An "authorized person" is someone who has been identified through registered identification information.

[0801] A "learning device" is a device that takes in environmental data and performs analysis and pattern recognition.

[0802] "Security level" is a standard that indicates the degree of vigilance in a monitoring system.

[0803] An "emotion recognition engine" is a mechanism that determines a user's emotional state from their facial expressions and actions.

[0804] "Security measures" refer to specific steps taken to ensure the safety of a system.

[0805] A "generation device" is a device that creates appropriate alerts and reports and notifies relevant parties.

[0806] An "alert" is a warning signal issued when suspicious activity or anomalies are detected.

[0807] "Past data" refers to the accumulation of information recorded previously.

[0808] "Abnormal behavior" refers to actions that deviate from the normal behavioral patterns of an individual or group.

[0809] The system for realizing this invention mainly comprises an image acquisition device, a server, and an emotion recognition engine. First, the terminal uses the image acquisition device to acquire video data of the monitoring area in real time. This video data also includes the user's facial expressions, making it analyzable. Next, the server receives this video data and performs behavioral analysis using an internally installed AI agent. If suspicious behavior is detected, it acts as an alert trigger. The server also uses the emotion recognition engine to analyze the user's emotions from the video data and identify emotional states such as anxiety and fear. Based on this information, the system can dynamically adjust the security level.

[0810] The emotion recognition engine uses specific patterns to evaluate the user's facial expressions and recognize emotions in real time. This allows security alerts to be generated according to the individual's emotional state. Alerts and reports generated by the generator are notified to relevant parties in an appropriate manner. If anxiety or fear is detected, notifications are immediately sent to the user and their designated contacts.

[0811] In particular, AI-powered video analysis and emotion recognition processes commonly utilize image processing libraries such as OpenCV and machine learning frameworks such as TensorFlow. These tools enable more precise and rapid analysis of image data.

[0812] For example, if a user experiences anxiety while walking alone in a dark place, this system can immediately send a notification to family or friends to prepare for an unforeseen situation. In such scenarios, a prompt such as, "Write a script that analyzes the user's facial expressions from the camera footage to determine if they are feeling fear or anxiety. If an emotion is detected, send an alert to the designated contacts," would be useful.

[0813] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0814] Step 1:

[0815] The terminal uses an image acquisition device to acquire video data in real time from the monitoring area. The input here is the physical environment, and digital video data is generated as the output. This video data is sent to the server.

[0816] Step 2:

[0817] The server analyzes the received video data. Here, an AI agent is used for motion analysis to detect suspicious behavior. The input is video data, and the output is the detection result of suspicious behavior. Based on this result, the next process is triggered.

[0818] Step 3:

[0819] The server uses an emotion recognition engine to analyze the user's emotional state from video data. The input here is the user's facial features, which are processed using pattern recognition technology to identify emotions such as anxiety and fear. The output is the result of the emotional state analysis, which is used to adjust the security level.

[0820] Step 4:

[0821] The server uses a generator to produce alerts and reports as needed. The inputs here are the results of suspicious behavior and emotional states, which are then integrated using generation technology. The output is an alert notification, which is sent to the relevant parties.

[0822] Step 5:

[0823] If a user expresses anxiety or fear in an unexpected situation, the server promptly sends a notification to the designated contacts. The input here is the identified emotional state, and the output is a notification message. This message includes summary information generated using a generative AI model, if necessary.

[0824] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0825] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0826] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0827] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0828] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0829] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0830] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0831] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0832] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0833] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0834] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0835] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0836] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0837] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0838] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0839] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0840] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0841] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0842] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0843] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0844] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0845] The following is further disclosed regarding the embodiments described above.

[0846] (Claim 1)

[0847] A means for detecting suspicious behavior by analyzing video data acquired by an image acquisition device in real time,

[0848] A means of identifying authorized persons using registered identification information,

[0849] A means of automatically adjusting the security level by learning environmental data using a learning device,

[0850] A means of generating alerts and reports using a generation device and notifying relevant parties,

[0851] A method for predicting future abnormal behavior using past data,

[0852] A system that includes this.

[0853] (Claim 2)

[0854] The system according to claim 1, which has a self-correction function that learns the cause of a false detection and prevents future false alarms.

[0855] (Claim 3)

[0856] The system according to claim 1, which summarizes an alert message generated by a generating device using generation technology.

[0857] "Example 1"

[0858] (Claim 1)

[0859] A means for immediately analyzing visual information acquired through data acquisition means to identify suspicious behavior,

[0860] A means of identifying authorized individuals using authentication information registered in a database,

[0861] A means of analyzing environmental information using a learning module and dynamically adjusting the alert level,

[0862] A means of generating alarms and reports using a generation module and sending them to relevant parties,

[0863] A means of predicting future abnormal events using past information,

[0864] A system that includes this.

[0865] (Claim 2)

[0866] The system according to claim 1, which includes a self-correction function that analyzes the cause of a misjudgment and prevents future misjudgments.

[0867] (Claim 3)

[0868] The system according to claim 1, wherein a generation module summarizes an alarm message created by the generation module using generation technology.

[0869] "Application Example 1"

[0870] (Claim 1)

[0871] A means for analyzing video data acquired by an image acquisition device in real time to detect suspicious activity,

[0872] A means of identifying authorized persons using registered identification information,

[0873] A means of automatically adjusting the security level by learning environmental data using a learning device,

[0874] A means of generating alarms and reports using a generation device and notifying relevant parties,

[0875] A method for predicting future abnormal behavior using past data,

[0876] A means of receiving notifications on a mobile device and displaying video in real time,

[0877] A system that includes this.

[0878] (Claim 2)

[0879] The system according to claim 1, which has a self-correction function that learns the cause of a misrecognition when it occurs and prevents future false alarms.

[0880] (Claim 3)

[0881] The system according to claim 1, wherein an alarm message generated by a generating device is summarized using generation technology.

[0882] "Example 2 of combining an emotion engine"

[0883] (Claim 1)

[0884] A means for analyzing visual information acquired by an image acquisition device in real time and detecting abnormal operation,

[0885] A means for identifying an authenticated individual using registered identification information,

[0886] A means of learning environmental information through a data analysis device and dynamically adjusting the conservation level,

[0887] A means of generating alarms and reports using a notification device and transmitting them to relevant parties,

[0888] A means of predicting future abnormal behavior using historical information,

[0889] A means for analyzing the user's emotions using an emotion recognition device and adjusting the system's operation based on the emotional information,

[0890] A system that includes this.

[0891] (Claim 2)

[0892] The system according to claim 1, which has a self-correction function that learns the cause of a misrecognition when it occurs and prevents future false alarms.

[0893] (Claim 3)

[0894] The system according to claim 1, which summarizes alarm information generated by a notification device using generation technology.

[0895] "Application example 2 when combining with an emotional engine"

[0896] (Claim 1)

[0897] A means for detecting suspicious behavior by analyzing video data acquired by an image acquisition device in real time,

[0898] A means of identifying authorized persons using registered identification information,

[0899] A means of automatically adjusting the security level by learning environmental data using a learning device,

[0900] A means of recognizing the user's emotional state using an emotion recognition engine and adjusting security measures based on that information,

[0901] A means of generating alerts and reports using a generation device and notifying relevant parties,

[0902] A method for predicting future abnormal behavior using past data,

[0903] A system that includes this.

[0904] (Claim 2)

[0905] The system according to claim 1, which has a self-correction function that learns the cause of a false detection and prevents future false alarms.

[0906] (Claim 3)

[0907] The system according to claim 1, which summarizes an alert message generated by a generating device using generation technology. [Explanation of symbols]

[0908] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for analyzing video data acquired by an image acquisition device in real time to detect suspicious activity, A means of identifying authorized persons using registered identification information, A means of automatically adjusting the security level by learning environmental data using a learning device, A means of generating alarms and reports using a generation device and notifying relevant parties, A method for predicting future abnormal behavior using past data, A means of receiving notifications on a mobile device and displaying video in real time, A system that includes this.

2. The system according to claim 1, which has a self-correction function that learns the cause of misrecognition when it occurs and prevents future false alarms.

3. The system according to claim 1, wherein the alarm message generated by the generating device is summarized using generation technology.

Images