system

Abstract

We provide the system. [Solution] The primary method for acquiring camera footage in real time, A second method involves preprocessing the acquired video data to convert it into an analyzable state. A third method for extracting features from pre-processed video data, A fourth method for detecting abnormal behavior based on extracted features, A fifth means of issuing a notification when abnormal behavior is detected, A sixth method for recording events as logs, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a situation where the aging society is progressing, traffic safety is ensured, and security in schools is improved, there is a social demand for a system that can efficiently and effectively detect abnormal behavior and quickly take countermeasures in a situation where human resources are limited. Conventional monitoring systems relied on individual devices or manpower specialized for each environment, so they lacked comprehensive abnormal detection ability and immediate response ability. Therefore, it is necessary to develop an integrated system for real-time video analysis and abnormal detection.

Means for Solving the Problems

[0005] The present invention solves the aforementioned problems with a monitoring system that can handle multiple different environments. Specifically, it includes means for acquiring camera images in real time, means for pre-processing the acquired images, and means for extracting features from the images and detecting abnormal behavior. Furthermore, it provides means for immediately sending notifications when an anomaly is detected and includes means for recording the occurrence of various anomalies as logs, enabling a comprehensive and rapid response. This system allows for efficient safety management beyond the limits of human resources.

[0006] "Real-time" refers to a state where data acquisition and processing occur instantly, and results are obtained without delay.

[0007] "Camera footage" refers to video data obtained from optical equipment positioned to capture images of the environment being monitored.

[0008] "Preprocessing" refers to the initial data processing operations performed to facilitate data analysis, and includes noise reduction and resolution adjustment.

[0009] "Features" are the attributes and metrics of data that machine learning models need to analyze data and perform pattern recognition.

[0010] "Abnormal behavior" refers to actions that deviate from normal patterns or expected behaviors and may be a sign of an accident or problem.

[0011] "Notification" refers to a means of communication used to send a rapid alert to users about detected anomalies and provide them with necessary information.

[0012] A "log" is a data description that stores records of events and states that occur within a system, and is used for later analysis and auditing. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc. 2>

[0019] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applicable to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention specifically describes the construction and operation of a monitoring system that acquires video data in real time, detects abnormal behavior, and immediately provides notifications. The system consists of a server, cameras, and user terminals.

[0035] Server Role

[0036] The server functions as a central processing unit that centrally manages and analyzes video data acquired from multiple cameras in real time. The server preprocesses the data and extracts features using machine learning algorithms. This allows the server to quickly detect abnormal behavior and send necessary alerts to the appropriate users and stakeholders.

[0037] The server has a database that stores log information and historical detection data, enabling later analysis and system improvements. To improve the system's capabilities, the server periodically updates its machine learning models with new data.

[0038] The role of the camera

[0039] The cameras are installed in the monitored environment and continuously acquire video data, which is then transmitted to a server. The cameras have the capability to capture high-resolution video and are equipped with infrared capabilities to provide appropriate images even at night or in low-light conditions.

[0040] The role of the user terminal

[0041] User devices (e.g., smartphones, tablets, computers) are devices that receive notifications from the server. Users can check notifications, monitor the system's real-time status through applications or web interfaces, and take quick action as needed.

[0042] Specific example

[0043] For example, in a facility for the elderly, a server could be used to detect falls at night, and if an abnormality is detected, a system could be provided that immediately notifies the care staff's smartphones. This would allow staff to quickly rush to the scene and take appropriate action.

[0044] In a traffic safety scenario, a server detects unusual pedestrian movements from cameras at intersections and sends this information as a warning to nearby drivers. This allows drivers to slow down quickly and prevent accidents.

[0045] In this way, an integrated system combining servers, cameras, and user terminals can address a wide range of societal safety needs. This system provides advanced monitoring capabilities that overcome the limitations of human resources and other factors, creating a safe and secure environment.

[0046] The following describes the processing flow.

[0047] Step 1:

[0048] The server monitors signals from network-connected cameras and acquires video data in real time. The video is immediately reflected in the monitored environment, and processing begins to minimize background noise and other effects.

[0049] Step 2:

[0050] The server performs preprocessing on the acquired video data. Preprocessing includes noise reduction, frame correction, and resolution adjustment, which together create a dataset suitable for analysis.

[0051] Step 3:

[0052] The server extracts features from the pre-processed video data. Using a machine learning model, it analyzes the characteristics of people and vehicles, such as their movement, speed, and direction, for each frame and extracts them as numerical values.

[0053] Step 4:

[0054] The server inputs the extracted features into the abnormal behavior detection algorithm. The algorithm uses past training data to evaluate whether the detected features match any abnormal behavior patterns.

[0055] Step 5:

[0056] When abnormal behavior is detected, the server immediately sends an alert to the user's terminal according to a pre-configured notification protocol. This notification is sent as a text message, push notification, or email containing the necessary information.

[0057] Step 6:

[0058] Users check notifications received through their devices to understand the situation in real time. They take appropriate action as needed, such as rushing to the scene or issuing instructions remotely.

[0059] Step 7:

[0060] The server logs anomaly detection events and stores them as data points for analysis and adjustment. The accumulated logs can be used to improve the system and enhance the accuracy of predictive models.

[0061] (Example 1)

[0062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0063] Real-time detection of abnormal behavior requires rapid and accurate data analysis, but current technology makes it difficult to respond effectively in all situations. Furthermore, a notification system to appropriately respond to detected abnormal behavior and timely updates of models according to the situation are also necessary. This is required to enhance the security of the monitored entities.

[0064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0065] In this invention, the server includes means for receiving visual information in real time from a video receiving device, means for preprocessing the acquired visual information to convert it into an analyzable state, and means for extracting specific features from the preprocessed visual information. This enables real-time, highly accurate detection of abnormal behavior and rapid notification.

[0066] A "video receiving device" is a device used to acquire visual information in real time from devices such as cameras.

[0067] "Visual information" refers to image data or video data acquired from cameras or other sensors.

[0068] "Preprocessing" refers to initial processing such as data manipulation and noise reduction performed to convert acquired visual information into a state that can be analyzed.

[0069] "Specific features" refer to elements that are important patterns or shapes for identifying abnormal behavior from visual information.

[0070] "Abnormal behavior" refers to movements or conditions in a subject that differ from normal behavior and require vigilance.

[0071] A "warning" refers to a message or alert sent to external users when abnormal behavior is detected.

[0072] An "event" refers to abnormal behavior or other related occurrences detected by the system.

[0073] A "learning model" refers to a process that includes machine learning algorithms and datasets used to analyze visual information and detect abnormal behavior.

[0074] "Updating" refers to the periodic adjustment process performed to improve the performance of machine learning models by incorporating new data.

[0075] This invention provides a specific method for constructing and operating a system that acquires visual information in real time and detects abnormal behavior. This system mainly consists of a server, a video receiving device, and a user terminal.

[0076] The server plays a central role in the system, aggregating and analyzing visual information in real time from multiple video receiving devices. The server incorporates a high-performance central processing unit (CPC), which performs preprocessing such as noise reduction and data normalization after receiving the visual information. This transforms the visual information into an analyzable state. Next, machine learning algorithms are used to extract specific features and rapidly detect abnormal behavior. For this purpose, specialized data analysis platforms and libraries are used.

[0077] Furthermore, the server periodically updates its learning model with new information to improve the system's accuracy and reliability. The server also has the ability to quickly generate appropriate warnings about detected abnormal behavior and send notifications to the relevant user terminals.

[0078] User terminals are devices that receive notifications sent from the server, and include smartphones, tablets, and computers. Users can use these terminals to check notifications through applications or web interfaces and understand the details of abnormal behavior in real time.

[0079] As a concrete example, a scenario in which this system can be used in elderly care facilities can be cited. The server analyzes information from video receiving devices within the facility, and if it detects abnormal behavior such as a fall by an elderly person, a notification is immediately sent to the terminal of the care staff. This allows staff to quickly rush to the scene and take appropriate action.

[0080] A concrete example of a prompt to a generative AI model is the question, "Please tell me the details of how the warning system processes when a camera installed at an intersection detects abnormal pedestrian behavior at night." By using such prompts, the AI ​​can provide answers detailing the system's operation.

[0081] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0082] Step 1:

[0083] The server acquires visual information from the video receiving device. This visual information is high-resolution video data and is transmitted to the server in real time. The visual information input is first broken down into frames, and then converted into a state where processing can proceed on an individual image basis.

[0084] Step 2:

[0085] The server preprocesses the acquired visual information. Specifically, it performs image noise reduction and normalization. This makes the visual information clear enough for analysis, improving accuracy in the next processing step. The input here is the decomposed frames, and the output is clear image data with noise removed.

[0086] Step 3:

[0087] The server extracts specific features from pre-processed image data. Here, machine learning algorithms are used to identify, for example, movement patterns or shapes representative of specific actions. Clear image data is taken as input, and the output is a dataset of the extracted features.

[0088] Step 4:

[0089] The server detects abnormal behavior based on extracted features. A machine learning model analyzes this feature data and identifies unusual behavioral patterns. The input is a feature dataset, and the output is an analysis report containing the results of the abnormal behavior.

[0090] Step 5:

[0091] The server immediately generates a warning and sends a notification to the relevant user terminal when abnormal behavior is detected. This notification is structured as an alert message, which the user receives in real time. The input is the result of the abnormal behavior detection, and the output is the warning notification message.

[0092] Step 6:

[0093] The server logs all detection events and processing data. This log includes the time and location of abnormal behavior, as well as more detailed analysis results. This builds a database that contributes to long-term analysis and future model improvements. Input is the overall processing results, and output is stored as log entries.

[0094] (Application Example 1)

[0095] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0096] In home and office surveillance, there are challenges in quickly detecting intruders or abnormal behavior and taking appropriate action promptly. Furthermore, conventional surveillance systems can only monitor limited areas, and real-time notifications and warnings may be delayed. This results in insufficient security for users to live and work with peace of mind.

[0097] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0098] In this invention, the server includes means for acquiring video information in real time, means for preprocessing and structuring the acquired video information based on its characteristics, and means for analyzing the extracted characteristics to detect abnormal behavior. This makes it possible to immediately detect intruders or abnormal behavior and quickly send warnings to the user's communication terminal.

[0099] "Video information" refers to visual data acquired in real time from surveillance cameras and other recording devices.

[0100] "Preprocessing" refers to the initial data processing step performed to convert acquired video information into an analyzable format.

[0101] "Characteristics" refer to important features or patterns extracted from video information using a specific algorithm.

[0102] "Abnormal behavior" refers to unexpected actions detected by characteristics that exhibit movements or behaviors different from the norm.

[0103] A "warning" is a notification sent to a user or administrator to alert them when abnormal behavior is detected.

[0104] A "communication terminal" refers to an electronic device used by a user, such as a smartphone or computer, that receives information from an external source.

[0105] This invention is a system that detects intruders and abnormal behavior and sends out a rapid warning. This system mainly consists of a server, cameras, and communication terminals. The following describes each of its main components in detail.

[0106] The server is responsible for acquiring real-time video information from multiple cameras installed in remote locations, and for centrally managing and analyzing it. The server preprocesses the video information using software such as Python and OpenCV, and then analyzes the information using machine learning algorithms such as TENSORFLOW (registered trademark) to extract characteristics. Based on the extracted characteristics, if abnormal behavior is detected, the server sends a warning to a communication terminal using a communication API such as Twilio.

[0107] The cameras are installed in the area to be monitored and are designed to continuously acquire high-resolution video information day and night. This makes it possible to transmit clear images to the server even in low-light conditions.

[0108] Users can receive warnings sent from the server via their communication devices (smartphones or computers) and respond quickly to abnormal situations. For example, if a suspicious person crosses the front door in the middle of the night, a warning along with an image of the garden will be immediately sent to the user's device. This system allows users to manage their living space with peace of mind.

[0109] Example prompt: "Design an algorithm for detecting abnormal behavior in a home security system using a generative AI model. The model should process video footage of the garden and front door in real time and send a notification to the user's smartphone when suspicious behavior is detected."

[0110] In this way, by implementing this embodiment of the invention, it is possible to provide a system that allows users to effectively ensure safety.

[0111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0112] Step 1:

[0113] The server acquires video information in real time from surveillance cameras installed in remote locations. The input is video information from the cameras, and the output is the raw video data sent to the server. In this step, the camera's video is transferred to the server in digital format and temporarily stored.

[0114] Step 2:

[0115] The server preprocesses the acquired video information. The input is raw video data, and the output is video data converted into a format suitable for analysis. In this step, OpenCV is used to convert the image to grayscale, remove noise, and resize it to a size suitable for analysis.

[0116] Step 3:

[0117] The server extracts features from pre-processed video data using a machine learning algorithm. The input is the pre-processed video data, and the output is data representing the extracted features. In this step, TensorFlow is used to identify important features related to people and movements in the video data and extract them as data necessary for detecting abnormal behavior.

[0118] Step 4:

[0119] The server detects abnormal behavior based on the extracted characteristics. The input is the extracted feature data, and the output is the result indicating whether or not abnormal behavior occurred. In this step, a generative AI model is used to determine abnormal behavior by comparing it with normal behavior patterns and to record the results.

[0120] Step 5:

[0121] If abnormal behavior is detected, the server uses an external communication API such as Twilio to send a warning to the terminal. The input is the result of the abnormal behavior detection, and the output is the warning notification sent to the user's terminal. In this step, the specific actions of constructing the warning message and quickly sending the notification to the user's smartphone are performed.

[0122] Step 6:

[0123] The user checks the warning notification received on their device and takes the necessary action. The input is the warning message displayed on the device, and the output is the user's response action. In this step, the user performs specific actions through the application, such as checking based on the warning and taking a quick response on-site.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] This invention relates to an advanced abnormal behavior detection system that combines emotion recognition technology to enhance surveillance systems. The system consists of a server, cameras, a user terminal, and an emotion engine.

[0126] Server Role

[0127] The server is the central device responsible for processing and managing data across the entire system. The server acquires video data from cameras and first performs preprocessing on it, such as noise reduction and frame rate improvement. Next, it analyzes the content using machine learning algorithms to extract features that could indicate abnormal behavior. Furthermore, the server aggregates data from the emotion engine and analyzes changes in the user's facial expressions and voice to improve overall anomaly detection capabilities.

[0128] The role of the emotional engine

[0129] The emotion engine is a component for analyzing the user's emotional state. It analyzes facial and voice data acquired from the camera and microphone in real time to instantly determine the user's emotions. This information is fed back to the server and used to improve the accuracy of abnormal behavior detection and customize notification content.

[0130] The role of the user terminal

[0131] The user terminal is a device that receives notifications when abnormal behavior is detected. The notifications are adjusted according to the user's emotional state, as determined by the emotion engine, allowing the user to understand the information most effectively and take appropriate action.

[0132] Specific example

[0133] Specifically, in elderly care facilities, the server analyzes footage from in-room cameras to detect not only falls but also increases in stress and anxiety among residents through an emotional intelligence engine. In this way, the server can simultaneously notify of not only physical abnormalities but also emotional abnormalities. Facility staff receive these notifications and can respond quickly to the mental health needs of the elderly residents.

[0134] Furthermore, in school bullying prevention scenarios, the emotional engine detects students' stress levels and fears, which are then analyzed by a server and reported as abnormal behavior. Teachers and counselors can immediately take appropriate intervention measures based on this information.

[0135] By incorporating an emotion engine in this way, abnormal behavior detection systems can provide more multidimensional and comprehensive safety monitoring than before, achieving a higher level of reassurance and security.

[0136] The following describes the processing flow.

[0137] Step 1:

[0138] The server acquires video data of the monitored area in real time via cameras. The video data includes detailed information about the actions and background of the monitored area.

[0139] Step 2:

[0140] The server performs preprocessing on the acquired video data. During the preprocessing stage, frame adjustments and resolution settings are performed to remove noise and improve image quality.

[0141] Step 3:

[0142] The server extracts specific features from pre-processed video data. This process uses machine learning models to extract important data points such as human and object movements.

[0143] Step 4:

[0144] The server uses an emotion engine to analyze the user's emotional state from video and audio data. It analyzes changes in facial expressions and tone of voice to understand the user's emotional condition.

[0145] Step 5:

[0146] The server integrates extracted features and sentiment data to determine if abnormal behavior is occurring. Based on this combined data, if behavior deviating from the norm is detected, it is considered abnormal.

[0147] Step 6:

[0148] If an anomaly is detected, the server immediately sends an appropriate alert to the user's terminal. The notification is tailored based on sentiment data to help the user take the best possible action quickly.

[0149] Step 7:

[0150] Users can check notifications provided via their terminals and take on-site or remote action as needed. The terminals can instantly provide users with detailed system status information.

[0151] Step 8:

[0152] The server logs any abnormal events that occur and stores them in a database for analysis and future system improvements. The recorded data helps improve the accuracy of the system and refine the model.

[0153] (Example 2)

[0154] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0155] In recent years, improving safety and security in public facilities, elderly care facilities, and educational institutions has become crucial. However, conventional anomaly detection systems have focused only on physically abnormal behavior and have failed to take into account the emotional state of users. As a result, timely and appropriate responses have been difficult, and there is a need for systems that enable more detailed monitoring and prompt responses.

[0156] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0157] In this invention, the server includes means for collecting visual information from a video device in real time, means for denoising and improving the image quality of the collected visual information to convert it into an analyzable state, and means for extracting feature quantities from the analyzable visual information and audio information. This makes it possible to identify abnormal behavior more quickly and accurately, taking into account the user's emotional state.

[0158] "Real-time" refers to a state where processing and responses occur the instant information is generated, resulting in extremely low latency.

[0159] "Visual information" refers to image and video data acquired by devices such as cameras.

[0160] "Noise reduction" refers to the process of removing unwanted errors and distortions contained in video and audio data to improve quality.

[0161] "Image quality enhancement" refers to the process of improving the visual quality of videos and images, specifically by adjusting clarity and contrast.

[0162] "Analyzable state" refers to a state where data has been properly formatted and is available for analysis and interpretation.

[0163] "Features" refer to important indicators or patterns extracted from data, which are elements used to identify information through analysis.

[0164] "Abnormal behavior" refers to actions or behaviors that deviate from normal patterns or standards, and is identified through analysis.

[0165] "Generating notifications" refers to the process of automatically creating alerts and information and communicating them to users.

[0166] An "information terminal" refers to an electronic device used to display data and for users to receive and manipulate information.

[0167] "Arithmetic method" refers to a technical method that defines a set of procedures and rules for performing calculations and processing.

[0168] The system according to this invention is a monitoring system designed to quickly and accurately identify abnormal behavior. This system mainly consists of a server, a user terminal, and an emotion analysis engine.

[0169] The server acquires video data from multiple cameras via the network. This video data is preprocessed in the initial stages through noise reduction and image quality improvement processes. During this process, algorithms are applied to improve the clarity and sharpness of the image.

[0170] Next, the server fuses the pre-processed video data with the audio and emotion data obtained from the emotion analysis engine. The emotion analysis engine analyzes the user's facial expressions and voice to identify their current emotional state. This process is performed in real time and fed back to the server.

[0171] The collected data is analyzed in detail through a computational method performed on the server. This method is designed to identify abnormal behavior, extracting features from the data and evaluating the degree of anomaly. Based on the identified abnormal behavior, the server promptly sends a notification to the user's terminal. This notification is customized according to the nature of the anomaly and the emotional state of the user.

[0172] The user's terminal receives this notification and immediately issues a warning to the user. This allows the user to respond quickly and take appropriate measures if necessary.

[0173] As a concrete example, in elderly care facilities, cameras constantly monitor activities in residents' rooms, and if any abnormality is detected, a notification is immediately sent to facility staff. This notification is particularly emphasized if there is a potential risk of falls or injuries. In schools, if stress or anxiety in students is detected, this information is notified to teachers and counselors, and necessary support is provided promptly.

[0174] Examples of prompt statements include the following:

[0175] "The emotional engine detects signs that suggest high stress levels among students."

[0176] "We will analyze changes in voice that may indicate elderly people are experiencing anxiety."

[0177] This system allows users to enjoy multi-dimensional security and use the environment with peace of mind.

[0178] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0179] Step 1:

[0180] The server receives visual information in real time from the video equipment. This information includes raw data from various viewpoints within the monitoring area. The input is video data from the camera, and the output is raw video frames that are temporarily stored in memory for preprocessing.

[0181] Step 2:

[0182] The server performs noise reduction and image quality enhancement on the received visual information. Noise reduction uses techniques to filter out errors and interference within the video. Image quality enhancement involves adjusting brightness and contrast to ensure quality suitable for analysis. The input is the raw video frame saved in step 1, and the output is a high-quality video frame suitable for analysis.

[0183] Step 3:

[0184] The server extracts features from analyzable visual information. This process utilizes a generative AI model to identify specific patterns that indicate abnormal behavior. The input is high-quality video frames, and the output is feature data related to position and motion.

[0185] Step 4:

[0186] The server receives user voice and facial expression information from the emotion analysis engine and analyzes the emotional state in real time. This is achieved by extracting features of voice tone and facial expressions. The input is voice and facial expression data obtained from the microphone and camera, and the output is metrics indicating the emotional state.

[0187] Step 5:

[0188] The server integrates the obtained features and emotional states to identify abnormal behavior. Here, it evaluates the likelihood of abnormality using comparisons with past data and computational methods. The input is the features and emotional metrics obtained in steps 3 and 4, and the output is the evaluation result of the abnormal behavior.

[0189] Step 6:

[0190] The server generates a notification on the user's terminal when abnormal behavior is detected. This notification includes detailed information such as the type, time, and location of the abnormality. The input is the evaluation result of the abnormal behavior, and the output is a customized notification message.

[0191] Step 7:

[0192] The user's terminal receives notifications from the server and immediately warns the user. Notifications are presented in various formats, such as screen pop-ups and audio alerts. The input is the notification message from the server, and the output is a real-time warning display to the user.

[0193] (Application Example 2)

[0194] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0195] In recent years, abnormal behavior detection systems have become increasingly important from the perspective of public safety and privacy. However, these systems are primarily focused on detecting physical abnormal behavior and are insufficient in understanding potential threats based on emotions. Therefore, it is necessary to incorporate more advanced emotion recognition to enable early detection of abnormal behavior and appropriate responses.

[0196] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0197] In this invention, the server includes means for acquiring visual information in real time, means for preprocessing the acquired visual information and converting it into an analyzable state, and means for extracting features from the preprocessed visual information. This makes it possible to analyze emotional states and improve the accuracy of anomaly detection.

[0198] "Visual information" refers to images and video data acquired by cameras and video equipment.

[0199] "Preprocessing" refers to processes such as noise reduction and image quality improvement performed to convert acquired data into a state that can be analyzed.

[0200] "Features" are statistical or morphological information extracted from data, and are the attributes that form the basis for algorithms to identify anomalies.

[0201] "Abnormal behavior" refers to actions or behaviors that deviate from normal patterns or expected behaviors, and is particularly identified through detection systems.

[0202] "Emotional state" refers to a psychological state that is analyzed from an individual's facial expressions, tone of voice, etc., and indicates emotions such as anxiety and anger.

[0203] "Recorded information" refers to data used to store abnormal events detected by the system and their details, which are used for later analysis and reference.

[0204] A "user" refers to an external individual or organization that receives notifications from the system and takes appropriate action.

[0205] A "method" is a set of algorithms and processes used within a system to achieve a specific objective.

[0206] The system for realizing this application works as follows: The server acquires visual information in real time and performs preprocessing such as noise reduction and sharpening on the data. Next, it extracts features from the preprocessed visual information and uses a machine learning algorithm to detect abnormal behavior. The acquired data is analyzed using an emotion recognition library (e.g., EmotionAPI) to improve the accuracy of abnormality detection based on changes in emotional state. This makes it possible to notify of abnormalities not only based on physical abnormal behavior but also on emotional state.

[0207] The device immediately notifies the user when abnormal behavior or emotional changes are detected. This notification includes a risk assessment based on specific emotional states, enabling the user to take quick and appropriate action. As a result, early warning and response to abnormal situations are improved.

[0208] As a concrete example, in public transportation, a server can use facial recognition technology to analyze passengers' expressions and detect signs of tension or anxiety at an early stage. Based on these detection results, an alert is sent to the administrator, and countermeasures are taken as necessary.

[0209] An example of a prompt to input into the generating AI model is, "Develop an application that analyzes camera video and audio data in real time to detect emotions such as anger and anxiety with high accuracy." This prompt helps the AI ​​model automatically generate code to execute the analysis process based on the appropriate instructions.

[0210] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0211] Step 1:

[0212] The server acquires visual information in real time through the camera. It continuously captures frames from the camera video stream as input, thereby obtaining current visual data.

[0213] Step 2:

[0214] The server performs preprocessing on the acquired visual information, including noise reduction and image sharpening. The input is the visual data obtained in step 1, and an image filter is applied to this data to reduce noise. The output is the visual data prepared for analysis.

[0215] Step 3:

[0216] The server extracts features from pre-processed visual data. The input is formatted visual data, and an image analysis algorithm is used to identify specific patterns related to shape and movement. The output is a list of detected features.

[0217] Step 4:

[0218] The server detects abnormal behavior using the extracted features. The feature list from step 3 is used as input, and a machine learning model performs pattern matching based on this. The output is a determination indicating whether or not abnormal behavior has occurred.

[0219] Step 5:

[0220] The server analyzes emotional states based on additional data from the camera and microphone. It takes facial expressions and voice tone as input and analyzes emotional characteristics using an emotion recognition library. The output is an evaluation result indicating the current emotional state.

[0221] Step 6:

[0222] The device notifies the user when abnormal behavior or emotional changes are detected. The input is the judgment results from steps 4 and 5, and an immediate warning is sent using various notification methods (text, alert sound, etc.). The output is the notification information displayed on the user's screen or device.

[0223] Step 7:

[0224] The server logs the events that occur as recorded information. As input, it integrates the results of all processing steps to generate a dataset containing information about the anomalies and emotional changes that occurred. The output is time-series data that can be used for future analysis and review.

[0225] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0226] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0227] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0228] [Second Embodiment]

[0229] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0230] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0231] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0232] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0233] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0234] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0235] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0236] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0237] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0238] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0239] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0240] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0241] This invention specifically describes the construction and operation of a monitoring system that acquires video data in real time, detects abnormal behavior, and immediately provides notifications. The system consists of a server, cameras, and user terminals.

[0242] Server Role

[0243] The server functions as a central processing unit that centrally manages and analyzes video data acquired from multiple cameras in real time. The server preprocesses the data and extracts features using machine learning algorithms. This allows the server to quickly detect abnormal behavior and send necessary alerts to the appropriate users and stakeholders.

[0244] The server has a database that stores log information and historical detection data, enabling later analysis and system improvements. To improve the system's capabilities, the server periodically updates its machine learning models with new data.

[0245] The role of the camera

[0246] The cameras are installed in the monitored environment and continuously acquire video data, which is then transmitted to a server. The cameras have the capability to capture high-resolution video and are equipped with infrared capabilities to provide appropriate images even at night or in low-light conditions.

[0247] The role of the user terminal

[0248] User devices (e.g., smartphones, tablets, computers) are devices that receive notifications from the server. Users can check notifications, monitor the system's real-time status through applications or web interfaces, and take quick action as needed.

[0249] Specific example

[0250] For example, in a facility for the elderly, a server could be used to detect falls at night, and if an abnormality is detected, a system could be provided that immediately notifies the care staff's smartphones. This would allow staff to quickly rush to the scene and take appropriate action.

[0251] In a traffic safety scenario, a server detects unusual pedestrian movements from cameras at intersections and sends this information as a warning to nearby drivers. This allows drivers to slow down quickly and prevent accidents.

[0252] In this way, an integrated system combining servers, cameras, and user terminals can address a wide range of societal safety needs. This system provides advanced monitoring capabilities that overcome the limitations of human resources and other factors, creating a safe and secure environment.

[0253] The following describes the processing flow.

[0254] Step 1:

[0255] The server monitors signals from network-connected cameras and acquires video data in real time. The video is immediately reflected in the monitored environment, and processing begins to minimize background noise and other effects.

[0256] Step 2:

[0257] The server performs preprocessing on the acquired video data. Preprocessing includes noise reduction, frame correction, and resolution adjustment, which together create a dataset suitable for analysis.

[0258] Step 3:

[0259] The server extracts features from the pre-processed video data. Using a machine learning model, it analyzes the characteristics of people and vehicles, such as their movement, speed, and direction, for each frame and extracts them as numerical values.

[0260] Step 4:

[0261] The server inputs the extracted features into the abnormal behavior detection algorithm. The algorithm uses past training data to evaluate whether the detected features match any abnormal behavior patterns.

[0262] Step 5:

[0263] When abnormal behavior is detected, the server immediately sends an alert to the user's terminal according to a pre-configured notification protocol. This notification is sent as a text message, push notification, or email containing the necessary information.

[0264] Step 6:

[0265] Users check notifications received through their devices to understand the situation in real time. They take appropriate action as needed, such as rushing to the scene or issuing instructions remotely.

[0266] Step 7:

[0267] The server logs anomaly detection events and stores them as data points for analysis and adjustment. The accumulated logs can be used to improve the system and enhance the accuracy of predictive models.

[0268] (Example 1)

[0269] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0270] Real-time detection of abnormal behavior requires rapid and accurate data analysis, but current technology makes it difficult to respond effectively in all situations. Furthermore, a notification system to appropriately respond to detected abnormal behavior and timely updates of models according to the situation are also necessary. This is required to enhance the security of the monitored entities.

[0271] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0272] In this invention, the server includes means for receiving visual information in real time from a video receiving device, means for preprocessing the acquired visual information to convert it into an analyzable state, and means for extracting specific features from the preprocessed visual information. This enables real-time, highly accurate detection of abnormal behavior and rapid notification.

[0273] A "video receiving device" is a device used to acquire visual information in real time from devices such as cameras.

[0274] "Visual information" refers to image data or video data acquired from cameras or other sensors.

[0275] "Preprocessing" refers to initial processing such as data manipulation and noise reduction performed to convert acquired visual information into a state that can be analyzed.

[0276] "Specific features" refer to elements that are important patterns or shapes for identifying abnormal behavior from visual information.

[0277] "Abnormal behavior" refers to movements or conditions in a subject that differ from normal behavior and require vigilance.

[0278] A "warning" refers to a message or alert sent to external users when abnormal behavior is detected.

[0279] An "event" refers to abnormal behavior or other related occurrences detected by the system.

[0280] A "learning model" refers to a process that includes machine learning algorithms and datasets used to analyze visual information and detect abnormal behavior.

[0281] "Updating" refers to the periodic adjustment process performed to improve the performance of machine learning models by incorporating new data.

[0282] This invention shows a specific construction and operation method of a system that acquires visual information in real time and detects abnormal behaviors. This system mainly consists of a server, a video receiving device, and a user terminal.

[0283] The server plays a central role in the system, aggregates visual information from multiple video receiving devices in real time, and conducts analysis. A high-performance central processing unit is incorporated in the server. After receiving the visual information, noise removal and data normalization are performed as preprocessing. Thereby, the visual information is converted into an analyzable state. Next, specific features are extracted using machine learning algorithms to quickly detect abnormal behaviors. As software used for this purpose, platforms and libraries specialized in data analysis are employed.

[0284] In addition, the server regularly updates the learning model with new information to improve the accuracy and reliability of the system. The server has a function of quickly generating appropriate warnings for the detected abnormal behaviors and sending notifications to the relevant user terminals.

[0285] The user terminal is a device that receives notifications sent from the server, including smartphones, tablets, computers, etc. Users can use these terminals to check the notifications through applications or web interfaces and grasp the details of abnormal behaviors in real time.

[0286] As a specific example, a scenario of applying this system in an elderly care facility can be cited. When the server analyzes the information from the video receiving devices in the facility and detects an abnormal behavior such as an elderly person falling, a notification is immediately sent to the terminals of the care staff. Thereby, the staff can quickly rush to the scene and take appropriate actions.

[0287] A concrete example of a prompt to a generative AI model is the question, "Please tell me the details of how the warning system processes when a camera installed at an intersection detects abnormal pedestrian behavior at night." By using such prompts, the AI ​​can provide answers detailing the system's operation.

[0288] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0289] Step 1:

[0290] The server acquires visual information from the video receiving device. This visual information is high-resolution video data and is transmitted to the server in real time. The visual information input is first broken down into frames, and then converted into a state where processing can proceed on an individual image basis.

[0291] Step 2:

[0292] The server preprocesses the acquired visual information. Specifically, it performs image noise reduction and normalization. This makes the visual information clear enough for analysis, improving accuracy in the next processing step. The input here is the decomposed frames, and the output is clear image data with noise removed.

[0293] Step 3:

[0294] The server extracts specific features from pre-processed image data. Here, machine learning algorithms are used to identify, for example, movement patterns or shapes representative of specific actions. Clear image data is taken as input, and the output is a dataset of the extracted features.

[0295] Step 4:

[0296] The server detects abnormal behavior based on extracted features. A machine learning model analyzes this feature data and identifies unusual behavioral patterns. The input is a feature dataset, and the output is an analysis report containing the results of the abnormal behavior.

[0297] Step 5:

[0298] The server immediately generates a warning and sends a notification to the relevant user terminal when abnormal behavior is detected. This notification is structured as an alert message, which the user receives in real time. The input is the result of the abnormal behavior detection, and the output is the warning notification message.

[0299] Step 6:

[0300] The server logs all detection events and processing data. This log includes the time and location of abnormal behavior, as well as more detailed analysis results. This builds a database that contributes to long-term analysis and future model improvements. Input is the overall processing results, and output is stored as log entries.

[0301] (Application Example 1)

[0302] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0303] In home and office surveillance, there are challenges in quickly detecting intruders or abnormal behavior and taking appropriate action promptly. Furthermore, conventional surveillance systems can only monitor limited areas, and real-time notifications and warnings may be delayed. This results in insufficient security for users to live and work with peace of mind.

[0304] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0305] In this invention, the server includes means for acquiring video information in real time, means for preprocessing and structuring the acquired video information based on its characteristics, and means for analyzing the extracted characteristics to detect abnormal behavior. As a result, the intrusion and abnormal behavior of suspicious persons can be immediately detected, and it becomes possible to quickly send a warning to the user's communication terminal.

[0306] "Video information" refers to visual data acquired in real time from surveillance cameras and other imaging devices.

[0307] "Preprocessing" refers to the first data processing step performed to convert the acquired video information into an analyzable format.

[0308] "Characteristics" refer to important features and patterns extracted from video information using specific algorithms.

[0309] "Abnormal behavior" refers to unexpected behavior detected by characteristics indicating movements and behaviors different from normal.

[0310] "Warning" refers to a notice to arouse attention sent to the user or administrator when abnormal behavior is detected.

[0311] "Communication terminal" refers to an electronic device such as a smartphone or computer used by the user for receiving information from the outside.

[0312] This invention is a system for detecting the intrusion and abnormal behavior of suspicious persons and quickly sending a warning. This system is mainly composed of a server, a camera, and a communication terminal. The following specifically describes each main element.

[0313] The server is responsible for acquiring real-time video information from multiple cameras installed in remote locations, and for centrally managing and analyzing it. The server preprocesses the video information using software such as Python and OpenCV, and then analyzes the information using machine learning algorithms such as TensorFlow to extract characteristics. Based on the extracted characteristics, if abnormal behavior is detected, the server sends a warning to a communication terminal using a communication API such as Twilio.

[0314] The cameras are installed in the area to be monitored and are designed to continuously acquire high-resolution video information day and night. This makes it possible to transmit clear images to the server even in low-light conditions.

[0315] Users can receive warnings sent from the server via their communication devices (smartphones or computers) and respond quickly to abnormal situations. For example, if a suspicious person crosses the front door in the middle of the night, a warning along with an image of the garden will be immediately sent to the user's device. This system allows users to manage their living space with peace of mind.

[0316] Example prompt: "Design an algorithm for detecting abnormal behavior in a home security system using a generative AI model. The model should process video footage of the garden and front door in real time and send a notification to the user's smartphone when suspicious behavior is detected."

[0317] In this way, by implementing this embodiment of the invention, it is possible to provide a system that allows users to effectively ensure safety.

[0318] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0319] Step 1:

[0320] The server acquires video information in real time from surveillance cameras installed in remote locations. The input is video information from the cameras, and the output is the raw video data sent to the server. In this step, the camera's video is transferred to the server in digital format and temporarily stored.

[0321] Step 2:

[0322] The server preprocesses the acquired video information. The input is raw video data, and the output is video data converted into a format suitable for analysis. In this step, OpenCV is used to convert the image to grayscale, remove noise, and resize it to a size suitable for analysis.

[0323] Step 3:

[0324] The server extracts features from pre-processed video data using a machine learning algorithm. The input is the pre-processed video data, and the output is data representing the extracted features. In this step, TensorFlow is used to identify important features related to people and movements in the video data and extract them as data necessary for detecting abnormal behavior.

[0325] Step 4:

[0326] The server detects abnormal behavior based on the extracted characteristics. The input is the extracted feature data, and the output is the result indicating whether or not abnormal behavior occurred. In this step, a generative AI model is used to determine abnormal behavior by comparing it with normal behavior patterns and to record the results.

[0327] Step 5:

[0328] If abnormal behavior is detected, the server uses an external communication API such as Twilio to send a warning to the terminal. The input is the result of the abnormal behavior detection, and the output is the warning notification sent to the user's terminal. In this step, the specific actions of constructing the warning message and quickly sending the notification to the user's smartphone are performed.

[0329] Step 6:

[0330] The user checks the warning notification received on their device and takes the necessary action. The input is the warning message displayed on the device, and the output is the user's response action. In this step, the user performs specific actions through the application, such as checking based on the warning and taking a quick response on-site.

[0331] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0332] This invention relates to an advanced abnormal behavior detection system that combines emotion recognition technology to enhance surveillance systems. The system consists of a server, cameras, a user terminal, and an emotion engine.

[0333] Server Role

[0334] The server is the central device responsible for processing and managing data across the entire system. The server acquires video data from cameras and first performs preprocessing on it, such as noise reduction and frame rate improvement. Next, it analyzes the content using machine learning algorithms to extract features that could indicate abnormal behavior. Furthermore, the server aggregates data from the emotion engine and analyzes changes in the user's facial expressions and voice to improve overall anomaly detection capabilities.

[0335] The role of the emotional engine

[0336] The emotion engine is a component for analyzing the user's emotional state. It analyzes facial and voice data acquired from the camera and microphone in real time to instantly determine the user's emotions. This information is fed back to the server and used to improve the accuracy of abnormal behavior detection and customize notification content.

[0337] The role of the user terminal

[0338] The user terminal is a device that receives notifications when abnormal behavior is detected. The notifications are adjusted according to the user's emotional state, as determined by the emotion engine, allowing the user to understand the information most effectively and take appropriate action.

[0339] Specific example

[0340] Specifically, in elderly care facilities, the server analyzes footage from in-room cameras to detect not only falls but also increases in stress and anxiety among residents through an emotional intelligence engine. In this way, the server can simultaneously notify of not only physical abnormalities but also emotional abnormalities. Facility staff receive these notifications and can respond quickly to the mental health needs of the elderly residents.

[0341] Furthermore, in school bullying prevention scenarios, the emotional engine detects students' stress levels and fears, which are then analyzed by a server and reported as abnormal behavior. Teachers and counselors can immediately take appropriate intervention measures based on this information.

[0342] By incorporating an emotion engine in this way, abnormal behavior detection systems can provide more multidimensional and comprehensive safety monitoring than before, achieving a higher level of reassurance and security.

[0343] The following describes the processing flow.

[0344] Step 1:

[0345] The server acquires video data of the monitored area in real time via cameras. The video data includes detailed information about the actions and background of the monitored area.

[0346] Step 2:

[0347] The server performs preprocessing on the acquired video data. During the preprocessing stage, frame adjustments and resolution settings are performed to remove noise and improve image quality.

[0348] Step 3:

[0349] The server extracts specific features from pre-processed video data. This process uses machine learning models to extract important data points such as human and object movements.

[0350] Step 4:

[0351] The server uses an emotion engine to analyze the user's emotional state from video and audio data. It analyzes changes in facial expressions and tone of voice to understand the user's emotional condition.

[0352] Step 5:

[0353] The server integrates extracted features and sentiment data to determine if abnormal behavior is occurring. Based on this combined data, if behavior deviating from the norm is detected, it is considered abnormal.

[0354] Step 6:

[0355] If an anomaly is detected, the server immediately sends an appropriate alert to the user's terminal. The notification is tailored based on sentiment data to help the user take the best possible action quickly.

[0356] Step 7:

[0357] Users can check notifications provided via their terminals and take on-site or remote action as needed. The terminals can instantly provide users with detailed system status information.

[0358] Step 8:

[0359] The server logs any abnormal events that occur and stores them in a database for analysis and future system improvements. The recorded data helps improve the accuracy of the system and refine the model.

[0360] (Example 2)

[0361] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0362] In recent years, improving safety and security in public facilities, elderly care facilities, and educational institutions has become crucial. However, conventional anomaly detection systems have focused only on physically abnormal behavior and have failed to take into account the emotional state of users. As a result, timely and appropriate responses have been difficult, and there is a need for systems that enable more detailed monitoring and prompt responses.

[0363] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0364] In this invention, the server includes means for collecting visual information from a video device in real time, means for denoising and improving the image quality of the collected visual information to convert it into an analyzable state, and means for extracting feature quantities from the analyzable visual information and audio information. This makes it possible to identify abnormal behavior more quickly and accurately, taking into account the user's emotional state.

[0365] "Real-time" refers to a state where processing and responses occur the instant information is generated, resulting in extremely low latency.

[0366] "Visual information" refers to image and video data acquired by devices such as cameras.

[0367] "Noise reduction" refers to the process of removing unwanted errors and distortions contained in video and audio data to improve quality.

[0368] "Image quality enhancement" refers to the process of improving the visual quality of videos and images, specifically by adjusting clarity and contrast.

[0369] "Analyzable state" refers to a state where data has been properly formatted and is available for analysis and interpretation.

[0370] "Features" refer to important indicators or patterns extracted from data, which are elements used to identify information through analysis.

[0371] "Abnormal behavior" refers to actions or behaviors that deviate from normal patterns or standards, and is identified through analysis.

[0372] "Generating notifications" refers to the process of automatically creating alerts and information and communicating them to users.

[0373] An "information terminal" refers to an electronic device used to display data and for users to receive and manipulate information.

[0374] "Arithmetic method" refers to a technical method that defines a set of procedures and rules for performing calculations and processing.

[0375] The system according to this invention is a monitoring system designed to quickly and accurately identify abnormal behavior. This system mainly consists of a server, a user terminal, and an emotion analysis engine.

[0376] The server acquires video data from multiple cameras via the network. This video data is preprocessed in the initial stages through noise reduction and image quality improvement processes. During this process, algorithms are applied to improve the clarity and sharpness of the image.

[0377] Next, the server fuses the pre-processed video data with the audio and emotion data obtained from the emotion analysis engine. The emotion analysis engine analyzes the user's facial expressions and voice to identify their current emotional state. This process is performed in real time and fed back to the server.

[0378] The collected data is analyzed in detail through a computational method performed on the server. This method is designed to identify abnormal behavior, extracting features from the data and evaluating the degree of anomaly. Based on the identified abnormal behavior, the server promptly sends a notification to the user's terminal. This notification is customized according to the nature of the anomaly and the emotional state of the user.

[0379] The user's terminal receives this notification and immediately issues a warning to the user. This allows the user to respond quickly and take appropriate measures if necessary.

[0380] As a concrete example, in elderly care facilities, cameras constantly monitor activities in residents' rooms, and if any abnormality is detected, a notification is immediately sent to facility staff. This notification is particularly emphasized if there is a potential risk of falls or injuries. In schools, if stress or anxiety in students is detected, this information is notified to teachers and counselors, and necessary support is provided promptly.

[0381] Examples of prompt statements include the following:

[0382] "The emotional engine detects signs that suggest high stress levels among students."

[0383] "We will analyze changes in voice that may indicate elderly people are experiencing anxiety."

[0384] This system allows users to enjoy multi-dimensional security and use the environment with peace of mind.

[0385] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0386] Step 1:

[0387] The server receives visual information in real time from the video equipment. This information includes raw data from various viewpoints within the monitoring area. The input is video data from the camera, and the output is raw video frames that are temporarily stored in memory for preprocessing.

[0388] Step 2:

[0389] The server performs noise reduction and image quality enhancement on the received visual information. Noise reduction uses techniques to filter out errors and interference within the video. Image quality enhancement involves adjusting brightness and contrast to ensure quality suitable for analysis. The input is the raw video frame saved in step 1, and the output is a high-quality video frame suitable for analysis.

[0390] Step 3:

[0391] The server extracts features from analyzable visual information. This process utilizes a generative AI model to identify specific patterns that indicate abnormal behavior. The input is high-quality video frames, and the output is feature data related to position and motion.

[0392] Step 4:

[0393] The server receives user voice and facial expression information from the emotion analysis engine and analyzes the emotional state in real time. This is achieved by extracting features of voice tone and facial expressions. The input is voice and facial expression data obtained from the microphone and camera, and the output is metrics indicating the emotional state.

[0394] Step 5:

[0395] The server integrates the obtained features and emotional states to identify abnormal behavior. Here, it evaluates the likelihood of abnormality using comparisons with past data and computational methods. The input is the features and emotional metrics obtained in steps 3 and 4, and the output is the evaluation result of the abnormal behavior.

[0396] Step 6:

[0397] The server generates a notification on the user's terminal when abnormal behavior is detected. This notification includes detailed information such as the type, time, and location of the abnormality. The input is the evaluation result of the abnormal behavior, and the output is a customized notification message.

[0398] Step 7:

[0399] The user's terminal receives notifications from the server and immediately warns the user. Notifications are presented in various formats, such as screen pop-ups and audio alerts. The input is the notification message from the server, and the output is a real-time warning display to the user.

[0400] (Application Example 2)

[0401] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0402] In recent years, abnormal behavior detection systems have become increasingly important from the perspective of public safety and privacy. However, these systems are primarily focused on detecting physical abnormal behavior and are insufficient in understanding potential threats based on emotions. Therefore, it is necessary to incorporate more advanced emotion recognition to enable early detection of abnormal behavior and appropriate responses.

[0403] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0404] In this invention, the server includes means for acquiring visual information in real time, means for preprocessing the acquired visual information and converting it into an analyzable state, and means for extracting features from the preprocessed visual information. This makes it possible to analyze emotional states and improve the accuracy of anomaly detection.

[0405] "Visual information" refers to images and video data acquired by cameras and video equipment.

[0406] "Preprocessing" refers to processes such as noise reduction and image quality improvement performed to convert acquired data into a state that can be analyzed.

[0407] "Features" are statistical or morphological information extracted from data, and are the attributes that form the basis for algorithms to identify anomalies.

[0408] "Abnormal behavior" refers to actions or behaviors that deviate from normal patterns or expected behaviors, and is particularly identified through detection systems.

[0409] "Emotional state" refers to a psychological state that is analyzed from an individual's facial expressions, tone of voice, etc., and indicates emotions such as anxiety and anger.

[0410] "Recorded information" refers to data used to store abnormal events detected by the system and their details, which are used for later analysis and reference.

[0411] A "user" refers to an external individual or organization that receives notifications from the system and takes appropriate action.

[0412] A "method" is a set of algorithms and processes used within a system to achieve a specific objective.

[0413] The system for realizing this application works as follows: The server acquires visual information in real time and performs preprocessing such as noise reduction and sharpening on the data. Next, it extracts features from the preprocessed visual information and uses a machine learning algorithm to detect abnormal behavior. The acquired data is analyzed using an emotion recognition library (e.g., EmotionAPI) to improve the accuracy of abnormality detection based on changes in emotional state. This makes it possible to notify of abnormalities not only based on physical abnormal behavior but also on emotional state.

[0414] The device immediately notifies the user when abnormal behavior or emotional changes are detected. This notification includes a risk assessment based on specific emotional states, enabling the user to take quick and appropriate action. As a result, early warning and response to abnormal situations are improved.

[0415] As a concrete example, in public transportation, a server can use facial recognition technology to analyze passengers' expressions and detect signs of tension or anxiety at an early stage. Based on these detection results, an alert is sent to the administrator, and countermeasures are taken as necessary.

[0416] An example of a prompt to input into the generating AI model is, "Develop an application that analyzes camera video and audio data in real time to detect emotions such as anger and anxiety with high accuracy." This prompt helps the AI ​​model automatically generate code to execute the analysis process based on the appropriate instructions.

[0417] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0418] Step 1:

[0419] The server acquires visual information in real time through the camera. It continuously captures frames from the camera video stream as input, thereby obtaining current visual data.

[0420] Step 2:

[0421] The server performs preprocessing on the acquired visual information, including noise reduction and image sharpening. The input is the visual data obtained in step 1, and an image filter is applied to this data to reduce noise. The output is the visual data prepared for analysis.

[0422] Step 3:

[0423] The server extracts features from pre-processed visual data. The input is formatted visual data, and an image analysis algorithm is used to identify specific patterns related to shape and movement. The output is a list of detected features.

[0424] Step 4:

[0425] The server detects abnormal behavior using the extracted features. The feature list from step 3 is used as input, and a machine learning model performs pattern matching based on this. The output is a determination indicating whether or not abnormal behavior has occurred.

[0426] Step 5:

[0427] The server analyzes emotional states based on additional data from the camera and microphone. It takes facial expressions and voice tone as input and analyzes emotional characteristics using an emotion recognition library. The output is an evaluation result indicating the current emotional state.

[0428] Step 6:

[0429] The device notifies the user when abnormal behavior or emotional changes are detected. The input is the judgment results from steps 4 and 5, and an immediate warning is sent using various notification methods (text, alert sound, etc.). The output is the notification information displayed on the user's screen or device.

[0430] Step 7:

[0431] The server logs the events that occur as recorded information. As input, it integrates the results of all processing steps to generate a dataset containing information about the anomalies and emotional changes that occurred. The output is time-series data that can be used for future analysis and review.

[0432] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0433] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0434] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0435] [Third Embodiment]

[0436] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0437] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0438] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0439] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0440] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0441] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0442] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0443] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0444] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0445] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0446] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0447] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0448] This invention specifically describes the construction and operation of a monitoring system that acquires video data in real time, detects abnormal behavior, and immediately provides notifications. The system consists of a server, cameras, and user terminals.

[0449] Server Role

[0450] The server functions as a central processing unit that centrally manages and analyzes video data acquired from multiple cameras in real time. The server preprocesses the data and extracts features using machine learning algorithms. This allows the server to quickly detect abnormal behavior and send necessary alerts to the appropriate users and stakeholders.

[0451] The server has a database that stores log information and historical detection data, enabling later analysis and system improvements. To improve the system's capabilities, the server periodically updates its machine learning models with new data.

[0452] The role of the camera

[0453] The cameras are installed in the monitored environment and continuously acquire video data, which is then transmitted to a server. The cameras have the capability to capture high-resolution video and are equipped with infrared capabilities to provide appropriate images even at night or in low-light conditions.

[0454] The role of the user terminal

[0455] User devices (e.g., smartphones, tablets, computers) are devices that receive notifications from the server. Users can check notifications, monitor the system's real-time status through applications or web interfaces, and take quick action as needed.

[0456] Specific example

[0457] For example, in a facility for the elderly, a server could be used to detect falls at night, and if an abnormality is detected, a system could be provided that immediately notifies the care staff's smartphones. This would allow staff to quickly rush to the scene and take appropriate action.

[0458] In a traffic safety scenario, a server detects unusual pedestrian movements from cameras at intersections and sends this information as a warning to nearby drivers. This allows drivers to slow down quickly and prevent accidents.

[0459] In this way, an integrated system combining servers, cameras, and user terminals can address a wide range of societal safety needs. This system provides advanced monitoring capabilities that overcome the limitations of human resources and other factors, creating a safe and secure environment.

[0460] The following describes the processing flow.

[0461] Step 1:

[0462] The server monitors signals from network-connected cameras and acquires video data in real time. The video is immediately reflected in the monitored environment, and processing begins to minimize background noise and other effects.

[0463] Step 2:

[0464] The server performs preprocessing on the acquired video data. Preprocessing includes noise reduction, frame correction, and resolution adjustment, which together create a dataset suitable for analysis.

[0465] Step 3:

[0466] The server extracts features from the pre-processed video data. Using a machine learning model, it analyzes the characteristics of people and vehicles, such as their movement, speed, and direction, for each frame and extracts them as numerical values.

[0467] Step 4:

[0468] The server inputs the extracted features into the abnormal behavior detection algorithm. The algorithm uses past training data to evaluate whether the detected features match any abnormal behavior patterns.

[0469] Step 5:

[0470] When abnormal behavior is detected, the server immediately sends an alert to the user's terminal according to a pre-configured notification protocol. This notification is sent as a text message, push notification, or email containing the necessary information.

[0471] Step 6:

[0472] Users check notifications received through their devices to understand the situation in real time. They take appropriate action as needed, such as rushing to the scene or issuing instructions remotely.

[0473] Step 7:

[0474] The server logs anomaly detection events and stores them as data points for analysis and adjustment. The accumulated logs can be used to improve the system and enhance the accuracy of predictive models.

[0475] (Example 1)

[0476] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0477] Real-time detection of abnormal behavior requires rapid and accurate data analysis, but current technology makes it difficult to respond effectively in all situations. Furthermore, a notification system to appropriately respond to detected abnormal behavior and timely updates of models according to the situation are also necessary. This is required to enhance the security of the monitored entities.

[0478] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0479] In this invention, the server includes means for receiving visual information in real time from a video receiving device, means for preprocessing the acquired visual information to convert it into an analyzable state, and means for extracting specific features from the preprocessed visual information. This enables real-time, highly accurate detection of abnormal behavior and rapid notification.

[0480] A "video receiving device" is a device used to acquire visual information in real time from devices such as cameras.

[0481] "Visual information" refers to image data or video data acquired from cameras or other sensors.

[0482] "Preprocessing" refers to initial processing such as data manipulation and noise reduction performed to convert acquired visual information into a state that can be analyzed.

[0483] "Specific features" refer to elements that are important patterns or shapes for identifying abnormal behavior from visual information.

[0484] "Abnormal behavior" refers to movements or conditions in a subject that differ from normal behavior and require vigilance.

[0485] A "warning" refers to a message or alert sent to external users when abnormal behavior is detected.

[0486] An "event" refers to abnormal behavior or other related occurrences detected by the system.

[0487] A "learning model" refers to a process that includes machine learning algorithms and datasets used to analyze visual information and detect abnormal behavior.

[0488] "Updating" refers to the periodic adjustment process performed to improve the performance of machine learning models by incorporating new data.

[0489] This invention provides a specific method for constructing and operating a system that acquires visual information in real time and detects abnormal behavior. This system mainly consists of a server, a video receiving device, and a user terminal.

[0490] The server plays a central role in the system, aggregating and analyzing visual information in real time from multiple video receiving devices. The server incorporates a high-performance central processing unit (CPC), which performs preprocessing such as noise reduction and data normalization after receiving the visual information. This transforms the visual information into an analyzable state. Next, machine learning algorithms are used to extract specific features and rapidly detect abnormal behavior. For this purpose, specialized data analysis platforms and libraries are used.

[0491] Furthermore, the server periodically updates its learning model with new information to improve the system's accuracy and reliability. The server also has the ability to quickly generate appropriate warnings about detected abnormal behavior and send notifications to the relevant user terminals.

[0492] User terminals are devices that receive notifications sent from the server, and include smartphones, tablets, and computers. Users can use these terminals to check notifications through applications or web interfaces and understand the details of abnormal behavior in real time.

[0493] As a concrete example, a scenario in which this system can be used in elderly care facilities can be cited. The server analyzes information from video receiving devices within the facility, and if it detects abnormal behavior such as a fall by an elderly person, a notification is immediately sent to the terminal of the care staff. This allows staff to quickly rush to the scene and take appropriate action.

[0494] A concrete example of a prompt to a generative AI model is the question, "Please tell me the details of how the warning system processes when a camera installed at an intersection detects abnormal pedestrian behavior at night." By using such prompts, the AI ​​can provide answers detailing the system's operation.

[0495] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0496] Step 1:

[0497] The server acquires visual information from the video receiving device. This visual information is high-resolution video data and is transmitted to the server in real time. The visual information input is first broken down into frames, and then converted into a state where processing can proceed on an individual image basis.

[0498] Step 2:

[0499] The server preprocesses the acquired visual information. Specifically, it performs image noise reduction and normalization. This makes the visual information clear enough for analysis, improving accuracy in the next processing step. The input here is the decomposed frames, and the output is clear image data with noise removed.

[0500] Step 3:

[0501] The server extracts specific features from pre-processed image data. Here, machine learning algorithms are used to identify, for example, movement patterns or shapes representative of specific actions. Clear image data is taken as input, and the output is a dataset of the extracted features.

[0502] Step 4:

[0503] The server detects abnormal behavior based on extracted features. A machine learning model analyzes this feature data and identifies unusual behavioral patterns. The input is a feature dataset, and the output is an analysis report containing the results of the abnormal behavior.

[0504] Step 5:

[0505] The server immediately generates a warning and sends a notification to the relevant user terminal when abnormal behavior is detected. This notification is structured as an alert message, which the user receives in real time. The input is the result of the abnormal behavior detection, and the output is the warning notification message.

[0506] Step 6:

[0507] The server logs all detection events and processing data. This log includes the time and location of abnormal behavior, as well as more detailed analysis results. This builds a database that contributes to long-term analysis and future model improvements. Input is the overall processing results, and output is stored as log entries.

[0508] (Application Example 1)

[0509] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0510] In home and office surveillance, there are challenges in quickly detecting intruders or abnormal behavior and taking appropriate action promptly. Furthermore, conventional surveillance systems can only monitor limited areas, and real-time notifications and warnings may be delayed. This results in insufficient security for users to live and work with peace of mind.

[0511] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0512] In this invention, the server includes means for acquiring video information in real time, means for preprocessing and structuring the acquired video information based on its characteristics, and means for analyzing the extracted characteristics to detect abnormal behavior. This makes it possible to immediately detect intruders or abnormal behavior and quickly send warnings to the user's communication terminal.

[0513] "Video information" refers to visual data acquired in real time from surveillance cameras and other recording devices.

[0514] "Preprocessing" refers to the initial data processing step performed to convert acquired video information into an analyzable format.

[0515] "Characteristics" refer to important features or patterns extracted from video information using a specific algorithm.

[0516] "Abnormal behavior" refers to unexpected actions detected by characteristics that exhibit movements or behaviors different from the norm.

[0517] A "warning" is a notification sent to a user or administrator to alert them when abnormal behavior is detected.

[0518] A "communication terminal" refers to an electronic device used by a user, such as a smartphone or computer, that receives information from an external source.

[0519] This invention is a system that detects intruders and abnormal behavior and sends out a rapid warning. This system mainly consists of a server, cameras, and communication terminals. The following describes each of its main components in detail.

[0520] The server is responsible for acquiring real-time video information from multiple cameras installed in remote locations, and for centrally managing and analyzing it. The server preprocesses the video information using software such as Python and OpenCV, and then analyzes the information using machine learning algorithms such as TensorFlow to extract characteristics. Based on the extracted characteristics, if abnormal behavior is detected, the server sends a warning to a communication terminal using a communication API such as Twilio.

[0521] The cameras are installed in the area to be monitored and are designed to continuously acquire high-resolution video information day and night. This makes it possible to transmit clear images to the server even in low-light conditions.

[0522] Users can receive warnings sent from the server via their communication devices (smartphones or computers) and respond quickly to abnormal situations. For example, if a suspicious person crosses the front door in the middle of the night, a warning along with an image of the garden will be immediately sent to the user's device. This system allows users to manage their living space with peace of mind.

[0523] Example prompt: "Design an algorithm for detecting abnormal behavior in a home security system using a generative AI model. The model should process video footage of the garden and front door in real time and send a notification to the user's smartphone when suspicious behavior is detected."

[0524] In this way, by implementing this embodiment of the invention, it is possible to provide a system that allows users to effectively ensure safety.

[0525] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0526] Step 1:

[0527] The server acquires video information in real time from surveillance cameras installed in remote locations. The input is video information from the cameras, and the output is the raw video data sent to the server. In this step, the camera's video is transferred to the server in digital format and temporarily stored.

[0528] Step 2:

[0529] The server preprocesses the acquired video information. The input is raw video data, and the output is video data converted into a format suitable for analysis. In this step, OpenCV is used to convert the image to grayscale, remove noise, and resize it to a size suitable for analysis.

[0530] Step 3:

[0531] The server extracts features from pre-processed video data using a machine learning algorithm. The input is the pre-processed video data, and the output is data representing the extracted features. In this step, TensorFlow is used to identify important features related to people and movements in the video data and extract them as data necessary for detecting abnormal behavior.

[0532] Step 4:

[0533] The server detects abnormal behavior based on the extracted characteristics. The input is the extracted feature data, and the output is the result indicating whether or not abnormal behavior occurred. In this step, a generative AI model is used to determine abnormal behavior by comparing it with normal behavior patterns and to record the results.

[0534] Step 5:

[0535] If abnormal behavior is detected, the server uses an external communication API such as Twilio to send a warning to the terminal. The input is the result of the abnormal behavior detection, and the output is the warning notification sent to the user's terminal. In this step, the specific actions of constructing the warning message and quickly sending the notification to the user's smartphone are performed.

[0536] Step 6:

[0537] The user checks the warning notification received on their device and takes the necessary action. The input is the warning message displayed on the device, and the output is the user's response action. In this step, the user performs specific actions through the application, such as checking based on the warning and taking a quick response on-site.

[0538] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0539] This invention relates to an advanced abnormal behavior detection system that combines emotion recognition technology to enhance surveillance systems. The system consists of a server, cameras, a user terminal, and an emotion engine.

[0540] Server Role

[0541] The server is the central device responsible for processing and managing data across the entire system. The server acquires video data from cameras and first performs preprocessing on it, such as noise reduction and frame rate improvement. Next, it analyzes the content using machine learning algorithms to extract features that could indicate abnormal behavior. Furthermore, the server aggregates data from the emotion engine and analyzes changes in the user's facial expressions and voice to improve overall anomaly detection capabilities.

[0542] The role of the emotional engine

[0543] The emotion engine is a component for analyzing the user's emotional state. It analyzes facial and voice data acquired from the camera and microphone in real time to instantly determine the user's emotions. This information is fed back to the server and used to improve the accuracy of abnormal behavior detection and customize notification content.

[0544] The role of the user terminal

[0545] The user terminal is a device that receives notifications when abnormal behavior is detected. The notifications are adjusted according to the user's emotional state, as determined by the emotion engine, allowing the user to understand the information most effectively and take appropriate action.

[0546] Specific example

[0547] Specifically, in elderly care facilities, the server analyzes footage from in-room cameras to detect not only falls but also increases in stress and anxiety among residents through an emotional intelligence engine. In this way, the server can simultaneously notify of not only physical abnormalities but also emotional abnormalities. Facility staff receive these notifications and can respond quickly to the mental health needs of the elderly residents.

[0548] Furthermore, in school bullying prevention scenarios, the emotional engine detects students' stress levels and fears, which are then analyzed by a server and reported as abnormal behavior. Teachers and counselors can immediately take appropriate intervention measures based on this information.

[0549] By incorporating an emotion engine in this way, abnormal behavior detection systems can provide more multidimensional and comprehensive safety monitoring than before, achieving a higher level of reassurance and security.

[0550] The following describes the processing flow.

[0551] Step 1:

[0552] The server acquires video data of the monitored area in real time via cameras. The video data includes detailed information about the actions and background of the monitored area.

[0553] Step 2:

[0554] The server performs preprocessing on the acquired video data. During the preprocessing stage, frame adjustments and resolution settings are performed to remove noise and improve image quality.

[0555] Step 3:

[0556] The server extracts specific features from pre-processed video data. This process uses machine learning models to extract important data points such as human and object movements.

[0557] Step 4:

[0558] The server uses an emotion engine to analyze the user's emotional state from video and audio data. It analyzes changes in facial expressions and tone of voice to understand the user's emotional condition.

[0559] Step 5:

[0560] The server integrates extracted features and sentiment data to determine if abnormal behavior is occurring. Based on this combined data, if behavior deviating from the norm is detected, it is considered abnormal.

[0561] Step 6:

[0562] If an anomaly is detected, the server immediately sends an appropriate alert to the user's terminal. The notification is tailored based on sentiment data to help the user take the best possible action quickly.

[0563] Step 7:

[0564] Users can check notifications provided via their terminals and take on-site or remote action as needed. The terminals can instantly provide users with detailed system status information.

[0565] Step 8:

[0566] The server logs any abnormal events that occur and stores them in a database for analysis and future system improvements. The recorded data helps improve the accuracy of the system and refine the model.

[0567] (Example 2)

[0568] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0569] In recent years, improving safety and security in public facilities, elderly care facilities, and educational institutions has become crucial. However, conventional anomaly detection systems have focused only on physically abnormal behavior and have failed to take into account the emotional state of users. As a result, timely and appropriate responses have been difficult, and there is a need for systems that enable more detailed monitoring and prompt responses.

[0570] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0571] In this invention, the server includes means for collecting visual information from a video device in real time, means for denoising and improving the image quality of the collected visual information to convert it into an analyzable state, and means for extracting feature quantities from the analyzable visual information and audio information. This makes it possible to identify abnormal behavior more quickly and accurately, taking into account the user's emotional state.

[0572] "Real-time" refers to a state where processing and responses occur the instant information is generated, resulting in extremely low latency.

[0573] "Visual information" refers to image and video data acquired by devices such as cameras.

[0574] "Noise reduction" refers to the process of removing unwanted errors and distortions contained in video and audio data to improve quality.

[0575] "Image quality enhancement" refers to the process of improving the visual quality of videos and images, specifically by adjusting clarity and contrast.

[0576] "Analyzable state" refers to a state where data has been properly formatted and is available for analysis and interpretation.

[0577] "Features" refer to important indicators or patterns extracted from data, which are elements used to identify information through analysis.

[0578] "Abnormal behavior" refers to actions or behaviors that deviate from normal patterns or standards, and is identified through analysis.

[0579] "Generating notifications" refers to the process of automatically creating alerts and information and communicating them to users.

[0580] An "information terminal" refers to an electronic device used to display data and for users to receive and manipulate information.

[0581] "Arithmetic method" refers to a technical method that defines a set of procedures and rules for performing calculations and processing.

[0582] The system according to this invention is a monitoring system designed to quickly and accurately identify abnormal behavior. This system mainly consists of a server, a user terminal, and an emotion analysis engine.

[0583] The server acquires video data from multiple cameras via the network. This video data is preprocessed in the initial stages through noise reduction and image quality improvement processes. During this process, algorithms are applied to improve the clarity and sharpness of the image.

[0584] Next, the server fuses the pre-processed video data with the audio and emotion data obtained from the emotion analysis engine. The emotion analysis engine analyzes the user's facial expressions and voice to identify their current emotional state. This process is performed in real time and fed back to the server.

[0585] The collected data is analyzed in detail through a computational method performed on the server. This method is designed to identify abnormal behavior, extracting features from the data and evaluating the degree of anomaly. Based on the identified abnormal behavior, the server promptly sends a notification to the user's terminal. This notification is customized according to the nature of the anomaly and the emotional state of the user.

[0586] The user's terminal receives this notification and immediately issues a warning to the user. This allows the user to respond quickly and take appropriate measures if necessary.

[0587] As a concrete example, in elderly care facilities, cameras constantly monitor activities in residents' rooms, and if any abnormality is detected, a notification is immediately sent to facility staff. This notification is particularly emphasized if there is a potential risk of falls or injuries. In schools, if stress or anxiety in students is detected, this information is notified to teachers and counselors, and necessary support is provided promptly.

[0588] Examples of prompt statements include the following:

[0589] "The emotional engine detects signs that suggest high stress levels among students."

[0590] "We will analyze changes in voice that may indicate elderly people are experiencing anxiety."

[0591] This system allows users to enjoy multi-dimensional security and use the environment with peace of mind.

[0592] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0593] Step 1:

[0594] The server receives visual information in real time from the video equipment. This information includes raw data from various viewpoints within the monitoring area. The input is video data from the camera, and the output is raw video frames that are temporarily stored in memory for preprocessing.

[0595] Step 2:

[0596] The server performs noise reduction and image quality enhancement on the received visual information. Noise reduction uses techniques to filter out errors and interference within the video. Image quality enhancement involves adjusting brightness and contrast to ensure quality suitable for analysis. The input is the raw video frame saved in step 1, and the output is a high-quality video frame suitable for analysis.

[0597] Step 3:

[0598] The server extracts features from analyzable visual information. This process utilizes a generative AI model to identify specific patterns that indicate abnormal behavior. The input is high-quality video frames, and the output is feature data related to position and motion.

[0599] Step 4:

[0600] The server receives user voice and facial expression information from the emotion analysis engine and analyzes the emotional state in real time. This is achieved by extracting features of voice tone and facial expressions. The input is voice and facial expression data obtained from the microphone and camera, and the output is metrics indicating the emotional state.

[0601] Step 5:

[0602] The server integrates the obtained features and emotional states to identify abnormal behavior. Here, it evaluates the likelihood of abnormality using comparisons with past data and computational methods. The input is the features and emotional metrics obtained in steps 3 and 4, and the output is the evaluation result of the abnormal behavior.

[0603] Step 6:

[0604] The server generates a notification on the user's terminal when abnormal behavior is detected. This notification includes detailed information such as the type, time, and location of the abnormality. The input is the evaluation result of the abnormal behavior, and the output is a customized notification message.

[0605] Step 7:

[0606] The user's terminal receives notifications from the server and immediately warns the user. Notifications are presented in various formats, such as screen pop-ups and audio alerts. The input is the notification message from the server, and the output is a real-time warning display to the user.

[0607] (Application Example 2)

[0608] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0609] In recent years, abnormal behavior detection systems have become increasingly important from the perspective of public safety and privacy. However, these systems are primarily focused on detecting physical abnormal behavior and are insufficient in understanding potential threats based on emotions. Therefore, it is necessary to incorporate more advanced emotion recognition to enable early detection of abnormal behavior and appropriate responses.

[0610] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0611] In this invention, the server includes means for acquiring visual information in real time, means for preprocessing the acquired visual information and converting it into an analyzable state, and means for extracting features from the preprocessed visual information. This makes it possible to analyze emotional states and improve the accuracy of anomaly detection.

[0612] "Visual information" refers to images and video data acquired by cameras and video equipment.

[0613] "Preprocessing" refers to processes such as noise reduction and image quality improvement performed to convert acquired data into a state that can be analyzed.

[0614] "Features" are statistical or morphological information extracted from data, and are the attributes that form the basis for algorithms to identify anomalies.

[0615] "Abnormal behavior" refers to actions or behaviors that deviate from normal patterns or expected behaviors, and is particularly identified through detection systems.

[0616] "Emotional state" refers to a psychological state that is analyzed from an individual's facial expressions, tone of voice, etc., and indicates emotions such as anxiety and anger.

[0617] "Recorded information" refers to data used to store abnormal events detected by the system and their details, which are used for later analysis and reference.

[0618] A "user" refers to an external individual or organization that receives notifications from the system and takes appropriate action.

[0619] A "method" is a set of algorithms and processes used within a system to achieve a specific objective.

[0620] The system for realizing this application works as follows: The server acquires visual information in real time and performs preprocessing such as noise reduction and sharpening on the data. Next, it extracts features from the preprocessed visual information and uses a machine learning algorithm to detect abnormal behavior. The acquired data is analyzed using an emotion recognition library (e.g., EmotionAPI) to improve the accuracy of abnormality detection based on changes in emotional state. This makes it possible to notify of abnormalities not only based on physical abnormal behavior but also on emotional state.

[0621] The device immediately notifies the user when abnormal behavior or emotional changes are detected. This notification includes a risk assessment based on specific emotional states, enabling the user to take quick and appropriate action. As a result, early warning and response to abnormal situations are improved.

[0622] As a concrete example, in public transportation, a server can use facial recognition technology to analyze passengers' expressions and detect signs of tension or anxiety at an early stage. Based on these detection results, an alert is sent to the administrator, and countermeasures are taken as necessary.

[0623] An example of a prompt to input into the generating AI model is, "Develop an application that analyzes camera video and audio data in real time to detect emotions such as anger and anxiety with high accuracy." This prompt helps the AI ​​model automatically generate code to execute the analysis process based on the appropriate instructions.

[0624] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0625] Step 1:

[0626] The server acquires visual information in real time through the camera. It continuously captures frames from the camera video stream as input, thereby obtaining current visual data.

[0627] Step 2:

[0628] The server performs preprocessing on the acquired visual information, including noise reduction and image sharpening. The input is the visual data obtained in step 1, and an image filter is applied to this data to reduce noise. The output is the visual data prepared for analysis.

[0629] Step 3:

[0630] The server extracts features from pre-processed visual data. The input is formatted visual data, and an image analysis algorithm is used to identify specific patterns related to shape and movement. The output is a list of detected features.

[0631] Step 4:

[0632] The server detects abnormal behavior using the extracted features. The feature list from step 3 is used as input, and a machine learning model performs pattern matching based on this. The output is a determination indicating whether or not abnormal behavior has occurred.

[0633] Step 5:

[0634] The server analyzes emotional states based on additional data from the camera and microphone. It takes facial expressions and voice tone as input and analyzes emotional characteristics using an emotion recognition library. The output is an evaluation result indicating the current emotional state.

[0635] Step 6:

[0636] The device notifies the user when abnormal behavior or emotional changes are detected. The input is the judgment results from steps 4 and 5, and an immediate warning is sent using various notification methods (text, alert sound, etc.). The output is the notification information displayed on the user's screen or device.

[0637] Step 7:

[0638] The server logs the events that occur as recorded information. As input, it integrates the results of all processing steps to generate a dataset containing information about the anomalies and emotional changes that occurred. The output is time-series data that can be used for future analysis and review.

[0639] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0640] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0641] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0642] [Fourth Embodiment]

[0643] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0644] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0645] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0646] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0647] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0648] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0649] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0650] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0651] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0652] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0653] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0654] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0655] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0656] This invention specifically describes the construction and operation of a monitoring system that acquires video data in real time, detects abnormal behavior, and immediately provides notifications. The system consists of a server, cameras, and user terminals.

[0657] Server Role

[0658] The server functions as a central processing unit that centrally manages and analyzes video data acquired from multiple cameras in real time. The server preprocesses the data and extracts features using machine learning algorithms. This allows the server to quickly detect abnormal behavior and send necessary alerts to the appropriate users and stakeholders.

[0659] The server has a database that stores log information and historical detection data, enabling later analysis and system improvements. To improve the system's capabilities, the server periodically updates its machine learning models with new data.

[0660] The role of the camera

[0661] The cameras are installed in the monitored environment and continuously acquire video data, which is then transmitted to a server. The cameras have the capability to capture high-resolution video and are equipped with infrared capabilities to provide appropriate images even at night or in low-light conditions.

[0662] The role of the user terminal

[0663] User devices (e.g., smartphones, tablets, computers) are devices that receive notifications from the server. Users can check notifications, monitor the system's real-time status through applications or web interfaces, and take quick action as needed.

[0664] Specific example

[0665] For example, in a facility for the elderly, a server could be used to detect falls at night, and if an abnormality is detected, a system could be provided that immediately notifies the care staff's smartphones. This would allow staff to quickly rush to the scene and take appropriate action.

[0666] In a traffic safety scenario, a server detects unusual pedestrian movements from cameras at intersections and sends this information as a warning to nearby drivers. This allows drivers to slow down quickly and prevent accidents.

[0667] In this way, an integrated system combining servers, cameras, and user terminals can address a wide range of societal safety needs. This system provides advanced monitoring capabilities that overcome the limitations of human resources and other factors, creating a safe and secure environment.

[0668] The following describes the processing flow.

[0669] Step 1:

[0670] The server monitors signals from network-connected cameras and acquires video data in real time. The video is immediately reflected in the monitored environment, and processing begins to minimize background noise and other effects.

[0671] Step 2:

[0672] The server performs preprocessing on the acquired video data. Preprocessing includes noise reduction, frame correction, and resolution adjustment, which together create a dataset suitable for analysis.

[0673] Step 3:

[0674] The server extracts features from the pre-processed video data. Using a machine learning model, it analyzes the characteristics of people and vehicles, such as their movement, speed, and direction, for each frame and extracts them as numerical values.

[0675] Step 4:

[0676] The server inputs the extracted features into the abnormal behavior detection algorithm. The algorithm uses past training data to evaluate whether the detected features match any abnormal behavior patterns.

[0677] Step 5:

[0678] When abnormal behavior is detected, the server immediately sends an alert to the user's terminal according to a pre-configured notification protocol. This notification is sent as a text message, push notification, or email containing the necessary information.

[0679] Step 6:

[0680] Users check notifications received through their devices to understand the situation in real time. They take appropriate action as needed, such as rushing to the scene or issuing instructions remotely.

[0681] Step 7:

[0682] The server logs anomaly detection events and stores them as data points for analysis and adjustment. The accumulated logs can be used to improve the system and enhance the accuracy of predictive models.

[0683] (Example 1)

[0684] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0685] Real-time detection of abnormal behavior requires rapid and accurate data analysis, but current technology makes it difficult to respond effectively in all situations. Furthermore, a notification system to appropriately respond to detected abnormal behavior and timely updates of models according to the situation are also necessary. This is required to enhance the security of the monitored entities.

[0686] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0687] In this invention, the server includes means for receiving visual information in real time from a video receiving device, means for preprocessing the acquired visual information to convert it into an analyzable state, and means for extracting specific features from the preprocessed visual information. This enables real-time, highly accurate detection of abnormal behavior and rapid notification.

[0688] A "video receiving device" is a device used to acquire visual information in real time from devices such as cameras.

[0689] "Visual information" refers to image data or video data acquired from cameras or other sensors.

[0690] "Preprocessing" refers to initial processing such as data manipulation and noise reduction performed to convert acquired visual information into a state that can be analyzed.

[0691] "Specific features" refer to elements that are important patterns or shapes for identifying abnormal behavior from visual information.

[0692] "Abnormal behavior" refers to movements or conditions in a subject that differ from normal behavior and require vigilance.

[0693] A "warning" refers to a message or alert sent to external users when abnormal behavior is detected.

[0694] An "event" refers to abnormal behavior or other related occurrences detected by the system.

[0695] A "learning model" refers to a process that includes machine learning algorithms and datasets used to analyze visual information and detect abnormal behavior.

[0696] "Updating" refers to the periodic adjustment process performed to improve the performance of machine learning models by incorporating new data.

[0697] This invention provides a specific method for constructing and operating a system that acquires visual information in real time and detects abnormal behavior. This system mainly consists of a server, a video receiving device, and a user terminal.

[0698] The server plays a central role in the system, aggregating and analyzing visual information in real time from multiple video receiving devices. The server incorporates a high-performance central processing unit (CPC), which performs preprocessing such as noise reduction and data normalization after receiving the visual information. This transforms the visual information into an analyzable state. Next, machine learning algorithms are used to extract specific features and rapidly detect abnormal behavior. For this purpose, specialized data analysis platforms and libraries are used.

[0699] Furthermore, the server periodically updates its learning model with new information to improve the system's accuracy and reliability. The server also has the ability to quickly generate appropriate warnings about detected abnormal behavior and send notifications to the relevant user terminals.

[0700] User terminals are devices that receive notifications sent from the server, and include smartphones, tablets, and computers. Users can use these terminals to check notifications through applications or web interfaces and understand the details of abnormal behavior in real time.

[0701] As a concrete example, a scenario in which this system can be used in elderly care facilities can be cited. The server analyzes information from video receiving devices within the facility, and if it detects abnormal behavior such as a fall by an elderly person, a notification is immediately sent to the terminal of the care staff. This allows staff to quickly rush to the scene and take appropriate action.

[0702] A concrete example of a prompt to a generative AI model is the question, "Please tell me the details of how the warning system processes when a camera installed at an intersection detects abnormal pedestrian behavior at night." By using such prompts, the AI ​​can provide answers detailing the system's operation.

[0703] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0704] Step 1:

[0705] The server acquires visual information from the video receiving device. This visual information is high-resolution video data and is transmitted to the server in real time. The visual information input is first broken down into frames, and then converted into a state where processing can proceed on an individual image basis.

[0706] Step 2:

[0707] The server preprocesses the acquired visual information. Specifically, it performs image noise reduction and normalization. This makes the visual information clear enough for analysis, improving accuracy in the next processing step. The input here is the decomposed frames, and the output is clear image data with noise removed.

[0708] Step 3:

[0709] The server extracts specific features from pre-processed image data. Here, machine learning algorithms are used to identify, for example, movement patterns or shapes representative of specific actions. Clear image data is taken as input, and the output is a dataset of the extracted features.

[0710] Step 4:

[0711] The server detects abnormal behavior based on extracted features. A machine learning model analyzes this feature data and identifies unusual behavioral patterns. The input is a feature dataset, and the output is an analysis report containing the results of the abnormal behavior.

[0712] Step 5:

[0713] The server immediately generates a warning and sends a notification to the relevant user terminal when abnormal behavior is detected. This notification is structured as an alert message, which the user receives in real time. The input is the result of the abnormal behavior detection, and the output is the warning notification message.

[0714] Step 6:

[0715] The server logs all detection events and processing data. This log includes the time and location of abnormal behavior, as well as more detailed analysis results. This builds a database that contributes to long-term analysis and future model improvements. Input is the overall processing results, and output is stored as log entries.

[0716] (Application Example 1)

[0717] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0718] In home and office surveillance, there are challenges in quickly detecting intruders or abnormal behavior and taking appropriate action promptly. Furthermore, conventional surveillance systems can only monitor limited areas, and real-time notifications and warnings may be delayed. This results in insufficient security for users to live and work with peace of mind.

[0719] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0720] In this invention, the server includes means for acquiring video information in real time, means for preprocessing and structuring the acquired video information based on its characteristics, and means for analyzing the extracted characteristics to detect abnormal behavior. This makes it possible to immediately detect intruders or abnormal behavior and quickly send warnings to the user's communication terminal.

[0721] "Video information" refers to visual data acquired in real time from surveillance cameras and other recording devices.

[0722] "Preprocessing" refers to the initial data processing step performed to convert acquired video information into an analyzable format.

[0723] "Characteristics" refer to important features or patterns extracted from video information using a specific algorithm.

[0724] "Abnormal behavior" refers to unexpected actions detected by characteristics that exhibit movements or behaviors different from the norm.

[0725] A "warning" is a notification sent to a user or administrator to alert them when abnormal behavior is detected.

[0726] A "communication terminal" refers to an electronic device used by a user, such as a smartphone or computer, that receives information from an external source.

[0727] This invention is a system that detects intruders and abnormal behavior and sends out a rapid warning. This system mainly consists of a server, cameras, and communication terminals. The following describes each of its main components in detail.

[0728] The server is responsible for acquiring real-time video information from multiple cameras installed in remote locations, and for centrally managing and analyzing it. The server preprocesses the video information using software such as Python and OpenCV, and then analyzes the information using machine learning algorithms such as TensorFlow to extract characteristics. Based on the extracted characteristics, if abnormal behavior is detected, the server sends a warning to a communication terminal using a communication API such as Twilio.

[0729] The cameras are installed in the area to be monitored and are designed to continuously acquire high-resolution video information day and night. This makes it possible to transmit clear images to the server even in low-light conditions.

[0730] Users can receive warnings sent from the server via their communication devices (smartphones or computers) and respond quickly to abnormal situations. For example, if a suspicious person crosses the front door in the middle of the night, a warning along with an image of the garden will be immediately sent to the user's device. This system allows users to manage their living space with peace of mind.

[0731] Example prompt: "Design an algorithm for detecting abnormal behavior in a home security system using a generative AI model. The model should process video footage of the garden and front door in real time and send a notification to the user's smartphone when suspicious behavior is detected."

[0732] In this way, by implementing this embodiment of the invention, it is possible to provide a system that allows users to effectively ensure safety.

[0733] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0734] Step 1:

[0735] The server acquires video information in real time from surveillance cameras installed in remote locations. The input is video information from the cameras, and the output is the raw video data sent to the server. In this step, the camera's video is transferred to the server in digital format and temporarily stored.

[0736] Step 2:

[0737] The server preprocesses the acquired video information. The input is raw video data, and the output is video data converted into a format suitable for analysis. In this step, OpenCV is used to convert the image to grayscale, remove noise, and resize it to a size suitable for analysis.

[0738] Step 3:

[0739] The server extracts features from pre-processed video data using a machine learning algorithm. The input is the pre-processed video data, and the output is data representing the extracted features. In this step, TensorFlow is used to identify important features related to people and movements in the video data and extract them as data necessary for detecting abnormal behavior.

[0740] Step 4:

[0741] The server detects abnormal behavior based on the extracted characteristics. The input is the extracted feature data, and the output is the result indicating whether or not abnormal behavior occurred. In this step, a generative AI model is used to determine abnormal behavior by comparing it with normal behavior patterns and to record the results.

[0742] Step 5:

[0743] If abnormal behavior is detected, the server uses an external communication API such as Twilio to send a warning to the terminal. The input is the result of the abnormal behavior detection, and the output is the warning notification sent to the user's terminal. In this step, the specific actions of constructing the warning message and quickly sending the notification to the user's smartphone are performed.

[0744] Step 6:

[0745] The user checks the warning notification received on their device and takes the necessary action. The input is the warning message displayed on the device, and the output is the user's response action. In this step, the user performs specific actions through the application, such as checking based on the warning and taking a quick response on-site.

[0746] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0747] This invention relates to an advanced abnormal behavior detection system that combines emotion recognition technology to enhance surveillance systems. The system consists of a server, cameras, a user terminal, and an emotion engine.

[0748] Server Role

[0749] The server is the central device responsible for processing and managing data across the entire system. The server acquires video data from cameras and first performs preprocessing on it, such as noise reduction and frame rate improvement. Next, it analyzes the content using machine learning algorithms to extract features that could indicate abnormal behavior. Furthermore, the server aggregates data from the emotion engine and analyzes changes in the user's facial expressions and voice to improve overall anomaly detection capabilities.

[0750] The role of the emotional engine

[0751] The emotion engine is a component for analyzing the user's emotional state. It analyzes facial and voice data acquired from the camera and microphone in real time to instantly determine the user's emotions. This information is fed back to the server and used to improve the accuracy of abnormal behavior detection and customize notification content.

[0752] The role of the user terminal

[0753] The user terminal is a device that receives notifications when abnormal behavior is detected. The notifications are adjusted according to the user's emotional state, as determined by the emotion engine, allowing the user to understand the information most effectively and take appropriate action.

[0754] Specific example

[0755] Specifically, in elderly care facilities, the server analyzes footage from in-room cameras to detect not only falls but also increases in stress and anxiety among residents through an emotional intelligence engine. In this way, the server can simultaneously notify of not only physical abnormalities but also emotional abnormalities. Facility staff receive these notifications and can respond quickly to the mental health needs of the elderly residents.

[0756] Furthermore, in school bullying prevention scenarios, the emotional engine detects students' stress levels and fears, which are then analyzed by a server and reported as abnormal behavior. Teachers and counselors can immediately take appropriate intervention measures based on this information.

[0757] By incorporating an emotion engine in this way, abnormal behavior detection systems can provide more multidimensional and comprehensive safety monitoring than before, achieving a higher level of reassurance and security.

[0758] The following describes the processing flow.

[0759] Step 1:

[0760] The server acquires video data of the monitored area in real time via cameras. The video data includes detailed information about the actions and background of the monitored area.

[0761] Step 2:

[0762] The server performs preprocessing on the acquired video data. During the preprocessing stage, frame adjustments and resolution settings are performed to remove noise and improve image quality.

[0763] Step 3:

[0764] The server extracts specific features from pre-processed video data. This process uses machine learning models to extract important data points such as human and object movements.

[0765] Step 4:

[0766] The server uses an emotion engine to analyze the user's emotional state from video and audio data. It analyzes changes in facial expressions and tone of voice to understand the user's emotional condition.

[0767] Step 5:

[0768] The server integrates extracted features and sentiment data to determine if abnormal behavior is occurring. Based on this combined data, if behavior deviating from the norm is detected, it is considered abnormal.

[0769] Step 6:

[0770] If an anomaly is detected, the server immediately sends an appropriate alert to the user's terminal. The notification is tailored based on sentiment data to help the user take the best possible action quickly.

[0771] Step 7:

[0772] Users can check notifications provided via their terminals and take on-site or remote action as needed. The terminals can instantly provide users with detailed system status information.

[0773] Step 8:

[0774] The server logs any abnormal events that occur and stores them in a database for analysis and future system improvements. The recorded data helps improve the accuracy of the system and refine the model.

[0775] (Example 2)

[0776] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0777] In recent years, improving safety and security in public facilities, elderly care facilities, and educational institutions has become crucial. However, conventional anomaly detection systems have focused only on physically abnormal behavior and have failed to take into account the emotional state of users. As a result, timely and appropriate responses have been difficult, and there is a need for systems that enable more detailed monitoring and prompt responses.

[0778] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0779] In this invention, the server includes means for collecting visual information from a video device in real time, means for denoising and improving the image quality of the collected visual information to convert it into an analyzable state, and means for extracting feature quantities from the analyzable visual information and audio information. This makes it possible to identify abnormal behavior more quickly and accurately, taking into account the user's emotional state.

[0780] "Real-time" refers to a state where processing and responses occur the instant information is generated, resulting in extremely low latency.

[0781] "Visual information" refers to image and video data acquired by devices such as cameras.

[0782] "Noise reduction" refers to the process of removing unwanted errors and distortions contained in video and audio data to improve quality.

[0783] "Image quality enhancement" refers to the process of improving the visual quality of videos and images, specifically by adjusting clarity and contrast.

[0784] "Analyzable state" refers to a state where data has been properly formatted and is available for analysis and interpretation.

[0785] "Features" refer to important indicators or patterns extracted from data, which are elements used to identify information through analysis.

[0786] "Abnormal behavior" refers to actions or behaviors that deviate from normal patterns or standards, and is identified through analysis.

[0787] "Generating notifications" refers to the process of automatically creating alerts and information and communicating them to users.

[0788] An "information terminal" refers to an electronic device used to display data and for users to receive and manipulate information.

[0789] "Arithmetic method" refers to a technical method that defines a set of procedures and rules for performing calculations and processing.

[0790] The system according to this invention is a monitoring system designed to quickly and accurately identify abnormal behavior. This system mainly consists of a server, a user terminal, and an emotion analysis engine.

[0791] The server acquires video data from multiple cameras via the network. This video data is preprocessed in the initial stages through noise reduction and image quality improvement processes. During this process, algorithms are applied to improve the clarity and sharpness of the image.

[0792] Next, the server fuses the pre-processed video data with the audio and emotion data obtained from the emotion analysis engine. The emotion analysis engine analyzes the user's facial expressions and voice to identify their current emotional state. This process is performed in real time and fed back to the server.

[0793] The collected data is analyzed in detail through a computational method performed on the server. This method is designed to identify abnormal behavior, extracting features from the data and evaluating the degree of anomaly. Based on the identified abnormal behavior, the server promptly sends a notification to the user's terminal. This notification is customized according to the nature of the anomaly and the emotional state of the user.

[0794] The user's terminal receives this notification and immediately issues a warning to the user. This allows the user to respond quickly and take appropriate measures if necessary.

[0795] As a concrete example, in elderly care facilities, cameras constantly monitor activities in residents' rooms, and if any abnormality is detected, a notification is immediately sent to facility staff. This notification is particularly emphasized if there is a potential risk of falls or injuries. In schools, if stress or anxiety in students is detected, this information is notified to teachers and counselors, and necessary support is provided promptly.

[0796] Examples of prompt statements include the following:

[0797] "The emotional engine detects signs that suggest high stress levels among students."

[0798] "We will analyze changes in voice that may indicate elderly people are experiencing anxiety."

[0799] This system allows users to enjoy multi-dimensional security and use the environment with peace of mind.

[0800] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0801] Step 1:

[0802] The server receives visual information in real time from the video equipment. This information includes raw data from various viewpoints within the monitoring area. The input is video data from the camera, and the output is raw video frames that are temporarily stored in memory for preprocessing.

[0803] Step 2:

[0804] The server performs noise reduction and image quality enhancement on the received visual information. Noise reduction uses techniques to filter out errors and interference within the video. Image quality enhancement involves adjusting brightness and contrast to ensure quality suitable for analysis. The input is the raw video frame saved in step 1, and the output is a high-quality video frame suitable for analysis.

[0805] Step 3:

[0806] The server extracts features from analyzable visual information. This process utilizes a generative AI model to identify specific patterns that indicate abnormal behavior. The input is high-quality video frames, and the output is feature data related to position and motion.

[0807] Step 4:

[0808] The server receives user voice and facial expression information from the emotion analysis engine and analyzes the emotional state in real time. This is achieved by extracting features of voice tone and facial expressions. The input is voice and facial expression data obtained from the microphone and camera, and the output is metrics indicating the emotional state.

[0809] Step 5:

[0810] The server integrates the obtained features and emotional states to identify abnormal behavior. Here, it evaluates the likelihood of abnormality using comparisons with past data and computational methods. The input is the features and emotional metrics obtained in steps 3 and 4, and the output is the evaluation result of the abnormal behavior.

[0811] Step 6:

[0812] The server generates a notification on the user's terminal when abnormal behavior is detected. This notification includes detailed information such as the type, time, and location of the abnormality. The input is the evaluation result of the abnormal behavior, and the output is a customized notification message.

[0813] Step 7:

[0814] The user's terminal receives notifications from the server and immediately warns the user. Notifications are presented in various formats, such as screen pop-ups and audio alerts. The input is the notification message from the server, and the output is a real-time warning display to the user.

[0815] (Application Example 2)

[0816] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0817] In recent years, abnormal behavior detection systems have become increasingly important from the perspective of public safety and privacy. However, these systems are primarily focused on detecting physical abnormal behavior and are insufficient in understanding potential threats based on emotions. Therefore, it is necessary to incorporate more advanced emotion recognition to enable early detection of abnormal behavior and appropriate responses.

[0818] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0819] In this invention, the server includes means for acquiring visual information in real time, means for preprocessing the acquired visual information and converting it into an analyzable state, and means for extracting features from the preprocessed visual information. This makes it possible to analyze emotional states and improve the accuracy of anomaly detection.

[0820] "Visual information" refers to images and video data acquired by cameras and video equipment.

[0821] "Preprocessing" refers to processes such as noise reduction and image quality improvement performed to convert acquired data into a state that can be analyzed.

[0822] "Features" are statistical or morphological information extracted from data, and are the attributes that form the basis for algorithms to identify anomalies.

[0823] "Abnormal behavior" refers to actions or behaviors that deviate from normal patterns or expected behaviors, and is particularly identified through detection systems.

[0824] "Emotional state" refers to a psychological state that is analyzed from an individual's facial expressions, tone of voice, etc., and indicates emotions such as anxiety and anger.

[0825] "Recorded information" refers to data used to store abnormal events detected by the system and their details, which are used for later analysis and reference.

[0826] A "user" refers to an external individual or organization that receives notifications from the system and takes appropriate action.

[0827] A "method" is a set of algorithms and processes used within a system to achieve a specific objective.

[0828] The system for realizing this application works as follows: The server acquires visual information in real time and performs preprocessing such as noise reduction and sharpening on the data. Next, it extracts features from the preprocessed visual information and uses a machine learning algorithm to detect abnormal behavior. The acquired data is analyzed using an emotion recognition library (e.g., EmotionAPI) to improve the accuracy of abnormality detection based on changes in emotional state. This makes it possible to notify of abnormalities not only based on physical abnormal behavior but also on emotional state.

[0829] The device immediately notifies the user when abnormal behavior or emotional changes are detected. This notification includes a risk assessment based on specific emotional states, enabling the user to take quick and appropriate action. As a result, early warning and response to abnormal situations are improved.

[0830] As a concrete example, in public transportation, a server can use facial recognition technology to analyze passengers' expressions and detect signs of tension or anxiety at an early stage. Based on these detection results, an alert is sent to the administrator, and countermeasures are taken as necessary.

[0831] An example of a prompt to input into the generating AI model is, "Develop an application that analyzes camera video and audio data in real time to detect emotions such as anger and anxiety with high accuracy." This prompt helps the AI ​​model automatically generate code to execute the analysis process based on the appropriate instructions.

[0832] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0833] Step 1:

[0834] The server acquires visual information in real time through the camera. It continuously captures frames from the camera video stream as input, thereby obtaining current visual data.

[0835] Step 2:

[0836] The server performs preprocessing on the acquired visual information, including noise reduction and image sharpening. The input is the visual data obtained in step 1, and an image filter is applied to this data to reduce noise. The output is the visual data prepared for analysis.

[0837] Step 3:

[0838] The server extracts features from pre-processed visual data. The input is formatted visual data, and an image analysis algorithm is used to identify specific patterns related to shape and movement. The output is a list of detected features.

[0839] Step 4:

[0840] The server detects abnormal behavior using the extracted features. The feature list from step 3 is used as input, and a machine learning model performs pattern matching based on this. The output is a determination indicating whether or not abnormal behavior has occurred.

[0841] Step 5:

[0842] The server analyzes emotional states based on additional data from the camera and microphone. It takes facial expressions and voice tone as input and analyzes emotional characteristics using an emotion recognition library. The output is an evaluation result indicating the current emotional state.

[0843] Step 6:

[0844] The device notifies the user when abnormal behavior or emotional changes are detected. The input is the judgment results from steps 4 and 5, and an immediate warning is sent using various notification methods (text, alert sound, etc.). The output is the notification information displayed on the user's screen or device.

[0845] Step 7:

[0846] The server logs the events that occur as recorded information. As input, it integrates the results of all processing steps to generate a dataset containing information about the anomalies and emotional changes that occurred. The output is time-series data that can be used for future analysis and review.

[0847] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0848] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0849] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0850] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0851] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0852] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0853] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0854] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0855] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0856] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0857] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0858] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0859] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0860] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0861] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0862] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0863] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0864] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0865] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0866] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0867] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0868] The following is further disclosed regarding the embodiments described above.

[0869] (Claim 1)

[0870] The primary method for acquiring camera footage in real time,

[0871] A second method involves preprocessing the acquired video data to convert it into an analyzable state.

[0872] A third method for extracting features from pre-processed video data,

[0873] A fourth method for detecting abnormal behavior based on extracted features,

[0874] A fifth means of issuing a notification when abnormal behavior is detected,

[0875] A sixth method for recording events as logs,

[0876] A system that includes this.

[0877] (Claim 2)

[0878] The system according to claim 1, further comprising means for providing external users with notifications of abnormal behavior and enabling users to respond quickly.

[0879] (Claim 3)

[0880] The system according to claim 1, comprising an algorithm set up to identify abnormal behavior in traffic safety, school security, and elderly care.

[0881] "Example 1"

[0882] (Claim 1)

[0883] The primary means of receiving visual information in real time from a video receiving device,

[0884] A second method involves preprocessing the acquired visual information and converting it into an analyzable state.

[0885] A third method for extracting specific features from pre-processed visual information,

[0886] A fourth means for detecting abnormal behavior based on specific characteristics extracted,

[0887] A fifth method for issuing a warning when abnormal behavior is detected,

[0888] A sixth means of recording the events that occurred,

[0889] A seventh method for updating the learning model using new information,

[0890] A system that includes this.

[0891] (Claim 2)

[0892] The system according to claim 1, further comprising means for providing external users with notifications of abnormal behavior and enabling users to respond quickly.

[0893] (Claim 3)

[0894] The system according to claim 1, comprising an analytical method set up to identify abnormal behavior in traffic management, safety management of educational institutions, and support for the elderly.

[0895] "Application Example 1"

[0896] (Claim 1)

[0897] The primary means of acquiring video information in real time,

[0898] A second method involves preprocessing the acquired video information to convert it into an analyzable state,

[0899] A third method for extracting characteristics from pre-processed video information,

[0900] A fourth means for detecting abnormal behavior based on extracted characteristics,

[0901] A fifth method for issuing a warning when abnormal behavior is detected,

[0902] A sixth means of preserving events as records,

[0903] A seventh method for sending warnings to external personal devices,

[0904] A system that includes this.

[0905] (Claim 2)

[0906] The system according to claim 1, further comprising means for detecting suspicious persons, providing warnings to external communication terminals, and enabling users to respond quickly.

[0907] (Claim 3)

[0908] The system according to claim 1, which includes a response process that detects a suspicious person and sends a warning to an external communication terminal, and provides a support function for ensuring safety.

[0909] "Example 2 of combining an emotion engine"

[0910] (Claim 1)

[0911] A means of collecting visual information from a video device in real time,

[0912] A means for denoising and improving the image quality of collected visual information and converting it into an analyzable state,

[0913] A means for extracting features from analyzable visual and auditory information,

[0914] A means for identifying abnormal behavior based on extracted features and emotional information,

[0915] A means of generating and distributing notifications to information terminals when abnormal behavior is identified,

[0916] Means of preserving events as records,

[0917] A system that includes this.

[0918] (Claim 2)

[0919] The system according to claim 1, further comprising means for providing external users with notifications regarding abnormal behavior, enabling users to take prompt action.

[0920] (Claim 3)

[0921] The system according to claim 1, comprising a computational method for use in identifying abnormal behavior in public safety, monitoring of educational facilities and support for the elderly.

[0922] "Application example 2 when combining with an emotional engine"

[0923] (Claim 1)

[0924] The primary means of acquiring visual information in real time,

[0925] A second method involves preprocessing the acquired visual information and converting it into an analyzable state.

[0926] A third method for extracting features from preprocessed visual information,

[0927] A fourth method for detecting abnormal behavior based on extracted features,

[0928] A fifth method to improve the accuracy of abnormalities by analyzing emotional states,

[0929] A sixth method for issuing a notification when abnormal behavior is detected,

[0930] A seventh means of saving the events that occurred as recorded information,

[0931] A system that includes this.

[0932] (Claim 2)

[0933] The system according to claim 1, further comprising means for providing external users with notifications of abnormal behavior and emotional states, enabling users to respond quickly.

[0934] (Claim 3)

[0935] The system according to claim 1, comprising techniques set up to identify abnormal behavior and emotional changes in traffic safety, safety in educational facilities, and support for the elderly. [Explanation of symbols]

[0936] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. The primary method for acquiring camera footage in real time, A second method involves preprocessing the acquired video data to convert it into an analyzable state. A third method for extracting features from pre-processed video data, A fourth method for detecting abnormal behavior based on extracted features, A fifth means of issuing a notification when abnormal behavior is detected, A sixth method for recording events as logs, A system that includes this.

2. The system according to claim 1, further comprising means for providing external users with notifications of abnormal behavior so that users can respond quickly.

3. The system according to claim 1, comprising an algorithm set up to identify abnormal behavior in traffic safety, school security, and elderly care.

Images