System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving data in real time from a surveillance camera and an audio collection device; means for analyzing the received data to detect a pre-set abnormal sound; and means for generating an alert and notifying an interested party if an abnormal sound is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Early detection and prevention of crimes and accidents, as well as a decline in response capabilities due to labor shortages, are cited as serious problems in modern society. In particular, conventional surveillance systems have difficulty monitoring large amounts of data in real time, making it difficult to respond quickly even if an abnormality is detected. The purpose of this invention is to provide a system that solves these problems and realizes a society where people can live in peace of mind. [Means for solving the problem]

[0005] The present invention is a system equipped with means for receiving data in real time from a surveillance camera and an audio collection device and analyzing the received data. This system includes means for detecting pre-defined abnormal sounds and further includes means for generating an alert and notifying relevant parties when an abnormal sound is detected. Specifically, the system processes the received data frame by frame and performs video and audio analysis in parallel, thereby achieving highly accurate abnormality detection. In addition, the alert message includes detailed information about the abnormality (time, location, type), enabling a prompt and appropriate response. This enables early detection and prevention of crimes and accidents, improving the safety of society as a whole.

[0006] A "surveillance camera" is a device that captures video in real time and is used for surveillance purposes.

[0007] An "audio collection device" is a device that collects ambient audio in real time and is used for analysis purposes.

[0008] The "means for receiving data" is a system component that has the function of acquiring data output from the surveillance camera and the audio collecting device.

[0009] "Means for analyzing data" refers to a system component that has the function of analyzing received video and audio data and executing algorithms or programs to detect anomalies.

[0010] "Abnormal sound" refers to a specific sound that does not normally occur and meets the criteria preset by the system.

[0011] The "means for generating an alert" is a system component that has the function of generating a warning and notifying relevant parties when an abnormal sound is detected.

[0012] "Interested parties" refers to users or organizations (e.g., security companies, police, fire departments, etc.) that are expected to receive alerts from the system.

[0013] "Frame-by-frame processing" refers to a method of dividing video data into individual frames for analysis.

[0014] "Parallel processing" refers to a method of simultaneously executing multiple analysis processes to improve processing efficiency and speed.

[0015] An "alert message" is a warning message generated to notify that an abnormality has been detected, and includes detailed information (time, location, type).

[0016] "Detailed information" refers to specific information associated with anomaly detection (for example, the time when the anomaly occurred, the location, and the type of anomaly). [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] System Overview

[0039] The system based on this invention is a comprehensive monitoring system that receives data in real time from surveillance cameras and audio collection devices, detects abnormalities, and notifies relevant parties. The system is mainly composed of three components: a server, a terminal, and a user, which work together to perform effective monitoring and issue alerts.

[0040] In the embodiment of the present invention, the system is composed of the following three entities.

[0041] Server Processing

[0042] 1. Data Collection

[0043] The server receives and temporarily stores data streams from surveillance cameras and audio collection devices, and transcodes the data for real-time processing.

[0044] 2. Data Analysis

[0045] The server uses a pre-trained AI model to process the received data frame by frame, performing video and audio analysis in parallel, thereby achieving highly accurate anomaly detection.

[0046] 3. Anomaly Detection

[0047] When the server detects an abnormal sound (for example, the sound of breaking glass or a person screaming), it records the abnormality and identifies details (time, location, type).

[0048] 4. Alert generation and sending

[0049] If an anomaly is detected, the server generates an alert message and sends it to the relevant parties. The alert message contains detailed information about the anomaly.

[0050] Terminal handling

[0051] 1. Data Entry

[0052] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices, temporarily stores the data, and processes it for transfer to the server.

[0053] 2. Data Transfer

[0054] By sending the data received by the terminal to the server in real time, it supports rapid data analysis on the server side.

[0055] 3. Receive anomaly alerts

[0056] The terminal receives the anomaly detection alert sent from the server and prepares to notify the user.

[0057] 4. User Notices

[0058] The device will display the alert as a pop-up message or application notification, allowing the user to respond quickly.

[0059] User Action

[0060] 1. Initial Setup

[0061] Users can set the installation locations of surveillance cameras and audio collection devices, check the connection status, and configure notification settings (e.g., email, SMS, app notifications).

[0062] 2. System Check

[0063] Users can check the system status in real time and monitor whether it is working properly. Users can also manually test the anomaly detection function.

[0064] 3. Response to abnormality notifications

[0065] The user receives an abnormality notification from the device, checks the detailed information, and then rushes to the scene or reports it to the relevant authorities (police, fire department, etc.) as necessary.

[0066] Example scenario

[0067] For example, imagine there are surveillance cameras and audio collection devices installed in a shopping district at night. If the sound of glass breaking occurs in a store late at night, the sound is picked up by the sound-collecting microphone and the surveillance camera captures the video. This data is sent to a server in real time. The server analyzes the received data and detects the sound of glass breaking. After detection, the server generates an alert message and sends it to the security company. The device receives this alert and notifies the user (security guard) as a pop-up message. Upon seeing the notification, the user can immediately rush to the scene and prevent the damage from spreading. This system enables early detection of crimes and accidents and rapid response, improving social safety.

[0068] The embodiments of the present invention improve the efficiency and accuracy of monitoring operations, and make a significant contribution to preventing crimes and accidents.

[0069] The processing flow will be explained below.

[0070] System Program Processing

[0071] Server Processing

[0072] Step 1:

[0073] A server receives data streams from surveillance cameras and audio collection devices.

[0074] The received video and audio data is temporarily stored and transcoded.

[0075] Step 2:

[0076] The server loads a pre-trained AI model.

[0077] The received data is processed frame by frame, and video and audio analysis is performed in parallel.

[0078] Step 3:

[0079] The server detects abnormal movements in the video data and filters out abnormal sounds (such as breaking glass or people screaming) from the audio data.

[0080] Apply anomaly detection algorithms to check if anomalies have occurred.

[0081] Step 4:

[0082] When the server detects an abnormal sound or behavior, it records the details (time, location, type).

[0083] Apply a secondary check algorithm to ensure the anomaly detection is correct.

[0084] Step 5:

[0085] The server generates an alert message if an abnormality is detected.

[0086] Include detailed information about the anomaly (time, location, type) in the alert message.

[0087] Step 6:

[0088] The server generates an alert message and sends it to the relevant parties (security company, police, fire department, etc.).

[0089] Checks whether the message was sent successfully and attempts to send it again if it fails.

[0090] Terminal handling

[0091] Step 1:

[0092] The terminal receives video and audio data in real time from the connected surveillance cameras and audio collection devices.

[0093] The received data is temporarily stored and formatted for transfer to the server.

[0094] Step 2:

[0095] The terminal transmits the received data to the server in real time.

[0096] Check the data for completeness and attempt to resend any missing data.

[0097] Step 3:

[0098] The terminal receives an anomaly detection alert sent from the server.

[0099] Check the accuracy of the received alerts and send feedback to the server if necessary.

[0100] Step 4:

[0101] The device displays the alert to the user as a pop-up message or application notification.

[0102] The alarm can be set to a screen display, audio alarm, vibration alarm, etc.

[0103] User Action

[0104] Step 1:

[0105] The user sets the installation locations of the surveillance cameras and audio collection devices and checks the connection status.

[0106] The user configures system notification settings (e.g., email, SMS, app notifications).

[0107] Step 2:

[0108] Users can check the status of the system and monitor whether it is working properly.

[0109] It is also possible to manually test the anomaly detection feature.

[0110] Step 3:

[0111] The user receives the abnormality notification from the terminal and checks the content of the notification.

[0112] Check detailed information about the abnormality (time, location, type) and, if necessary, rush to the scene or report it to the relevant authorities.

[0113] Example 1

[0114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] In recent years, there has been an increasing demand for more accurate and efficient monitoring systems. However, existing monitoring systems have problems such as low anomaly detection accuracy and insufficient real-time data processing. Furthermore, they are often unable to quickly notify relevant parties of detected anomalies, making it difficult to respond quickly. To address these issues, there is a demand for monitoring systems that can detect anomalies more accurately in real time and quickly notify relevant parties.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0117] In this invention, the server includes means for receiving data in real time from the surveillance cameras and audio collecting devices, means for temporarily storing the received data and transcoding it into an optimal format, means for processing the received data frame by frame using a pre-trained artificial intelligence model to perform video and audio analysis in parallel, means for detecting abnormal sounds and images, recording the abnormality and identifying details (time, location, type), and means for generating an alert message and notifying relevant parties when an abnormality is detected. This enables real-time abnormality detection and highly accurate data analysis, realizing early detection of abnormalities and rapid response.

[0118] A "surveillance camera" is a device that acquires video data in real time and is used for monitoring and anomaly detection.

[0119] An "audio collection device" is a device that acquires audio data in real time and is used for monitoring and anomaly detection.

[0120] A "data stream" is a continuous flow of data transmitted in real time from a surveillance camera or audio collection device.

[0121] "Transcoding" is the process of converting received data into a different format.

[0122] An "artificial intelligence model (AI model)" is an algorithm that has been trained in advance using large amounts of data, and is used to detect and analyze anomalies.

[0123] "Frame-based processing" is a method in which video data and audio data are divided into fixed time intervals (frames) and analyzed and processed for each unit.

[0124] "Parallel processing" is a technology that improves processing speed by executing multiple processes or threads simultaneously.

[0125] An "abnormal sound" is a specific sound (such as the sound of breaking glass or screaming) that does not occur in a normal monitoring environment and indicates an abnormal situation.

[0126] "Abnormal footage" refers to specific footage (for example, suspicious activity) that is not seen in a normal surveillance environment and indicates an abnormal situation.

[0127] An "alert message" is a notification message that is generated when an abnormality is detected, and provides detailed information about the abnormality to relevant parties.

[0128] "Relevant parties" are people who manage and operate the monitoring system and people who are responsible for responding to abnormalities (e.g., security guards and administrators).

[0129] The monitoring system based on this invention is mainly composed of three entities: a server, a terminal, and a user. It is a comprehensive monitoring system that receives data in real time from monitoring cameras and audio collection devices, detects abnormalities, and notifies relevant parties. The role of each entity will be specifically explained below.

[0130] Server Processing

[0131] The server receives real-time data streams from surveillance cameras (e.g., general network cameras) and audio collection devices (e.g., standard microphones). The data is temporarily stored and transcoded into the optimal format for real-time processing. For example, video data is converted to H.264 format, and audio data is converted to WAV format. The transcoding process uses high-speed processing using NVIDIA CUDA.

[0132] The server then analyzes the data using pre-trained AI models. It uses TensorFlow to load an anomaly detection model, a YOLO model for video analysis, and a CNN model for audio analysis. Data is processed frame by frame and analyzed in parallel. Video data is analyzed using OpenCV, and audio data undergoes spectrogram analysis using Librosa.

[0133] When an abnormal sound or video is detected, the server records the anomaly and identifies the time, location, and type of the anomaly. Detailed information about the anomaly is recorded in a MySQL database. If an anomaly is detected, an alert message is generated and sent to relevant parties using Amazon SNS (Simple Notification Service).

[0134] Terminal handling

[0135] The terminal uses a Raspberry Pi or a standard edge device to receive data in real time from surveillance cameras and audio collection devices. The received data is temporarily stored in a buffer and processed. The terminal then transfers the received data to the server in real time using the MQTT protocol.

[0136] When an anomaly detection alert is received from the server, the device prepares a notification to the user. Firebase Cloud Messaging is used to notify the user of the alert message as a pop-up display or app notification, allowing the user to respond to the anomaly quickly.

[0137] User Action

[0138] Users can use a web-based management screen to set the installation locations of surveillance cameras and audio collection devices and check the connection status. They can also set notifications (e.g., email, SMS, app notifications). Users can check the system status from the management screen and monitor operation in real time. If an abnormality is detected, users will receive an alert message and check detailed information (e.g., date, time, location, and nature of the abnormality). They can then rush to the scene or notify the relevant authorities as necessary.

[0139] Example scenario

[0140] For example, consider the case where a surveillance camera and audio collection device installed in a shopping district detect the sound of glass breaking in a store late at night. The received data is sent to a server, where an AI model is used to detect the sound of glass breaking. The detected abnormality is recorded and an alert message is sent to the security company using Amazon SNS. The device receives this alert and notifies the security guard via a pop-up message. The security guard can then immediately rush to the scene and prevent further damage.

[0141] Example prompts to input to the generative AI model

[0142] "I'm in charge of the surveillance system for a shopping mall. Can you explain the outline of the AI ​​model that detects suspicious sounds and images at night and immediately notifies security guards?"

[0143] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0144] Step 1:

[0145] Receiving and temporarily storing data (server processing)

[0146] Input: Real-time data streams from surveillance cameras and audio collection devices

[0147] Specific operation: The server receives data streams from surveillance cameras and audio collection devices using RTSP (Real-Time Streaming Protocol). The received data is temporarily saved and stored in a buffer for subsequent processing.

[0148] Output: Temporarily saved data (video data and audio data)

[0149] Step 2:

[0150] Data transcoding (server processing)

[0151] Input: Temporarily saved data (video data and audio data)

[0152] Specific operation: The server transcodes the received video data into H.264 format and the audio data into WAV format. The transcoding process is performed at high speed using NVIDIA CUDA.

[0153] Output: Transcoded data (video data in H.264 format, audio data in WAV format)

[0154] Step 3:

[0155] Preparing for data analysis (server processing)

[0156] Input: Transcoded data (video data in H.264 format, audio data in WAV format)

[0157] How it works: The server loads pre-trained AI models using TensorFlow. The YOLO model is used for video analysis, and the CNN model is used for audio analysis.

[0158] Output: A ready-to-use AI model

[0159] Step 4:

[0160] Frame-by-frame data analysis (server processing)

[0161] Input: AI model ready, transcoded data

[0162] Specific operation: The server divides the video and audio data into frames and processes them in parallel. The video data is analyzed using OpenCV, and the audio data undergoes spectrogram analysis using Librosa.

[0163] Output: Parsed frame data

[0164] Step 5:

[0165] Anomaly detection (server processing)

[0166] Input: Parsed frame data

[0167] How it works: The server uses AI models to detect abnormal sounds and images. For example, if an abnormality such as the sound of broken glass, screams, or a suspicious person is detected, the server records the abnormality and identifies the time, location, and type of occurrence. Detailed information about the abnormality is then recorded in a MySQL database.

[0168] Output: Anomaly detection results and detailed information (time, location, type)

[0169] Step 6:

[0170] Alert generation and notification (server processing)

[0171] Input: Anomaly detection results and detailed information

[0172] Specific operation: If an anomaly is detected, the server generates an alert message. The message contains detailed information about the anomaly (e.g., date, time, location, and content of the anomaly). The alert message is sent to relevant parties using Amazon SNS.

[0173] Output: Alert message sent

[0174] Step 7:

[0175] Data entry (terminal processing)

[0176] Input: Data acquired from surveillance cameras and audio collection devices

[0177] How it works: The device uses a Raspberry Pi or a standard edge device to receive and temporarily store data from surveillance cameras and audio capture devices in real time, check for missing or abnormal data, and add a timestamp.

[0178] Output: Formatted data

[0179] Step 8:

[0180] Data transfer (terminal processing)

[0181] Input: Formatted data

[0182] Specific operation: The terminal uses the MQTT protocol to send the received data to the server in real time.

[0183] Output: Data transferred to the server

[0184] Step 9:

[0185] Receive an abnormality alert (terminal processing)

[0186] Input: The alert message sent from the server

[0187] Specific operation: The terminal receives the alert message sent from the server and prepares to notify the user.

[0188] Output: Prepared alert message

[0189] Step 10:

[0190] User notification (terminal processing)

[0191] Input: Prepared alert message

[0192] Specific operation: The device uses Firebase Cloud Messaging to notify the user of the received alert message as a pop-up or app notification.

[0193] Output: Alert message sent to the user

[0194] Step 11:

[0195] Initial settings (user processing)

[0196] Input: Setting information by accessing the administration screen

[0197] Specific operation: Using a web-based management interface, users can configure the locations of surveillance cameras and audio collection devices, check connection status, and configure notifications (e.g., email, SMS, app notifications).

[0198] Output: Management screen with saved settings

[0199] Step 12:

[0200] System confirmation (user processing)

[0201] Input:Administrator screen information

[0202] Specific operation: Users can check the system status from the management screen, monitor the operation status in real time, and manually test the anomaly detection function.

[0203] Output: Confirmation of system status, working properly

[0204] Step 13:

[0205] Handling abnormality notifications (user actions)

[0206] Input: The alert message that was sent to the user

[0207] Specific operations: The user checks the detailed information of the alert message (e.g., date and time, location, and content of the abnormality), and if necessary, rushes to the scene or notifies the relevant authorities.

[0208] Output: Notification information for which the action has been completed

[0209] (Application example 1)

[0210] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0211] While conventional surveillance systems analyze real-time data from surveillance cameras and audio collection devices and detect abnormalities, they lack the ability to quickly and effectively notify relevant parties of the abnormality. In particular, they lack the ability to provide optimal route information for rushing to the scene, and there is a need for a rapid response when an abnormality is notified. The present invention aims to solve these problems and provide a more efficient and responsive surveillance system.

[0212] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0213] In this invention, the server includes means for receiving data in real time from the surveillance camera and the audio collecting device, means for analyzing the received data and detecting a preset abnormal sound, means for generating an alert and notifying relevant parties when an abnormal sound is detected, means for displaying the abnormality notification as a pop-up message or an application notification using a smart device, and means for displaying the shortest route to the location of the abnormality. This not only enables rapid abnormality detection and notification, but also improves the efficiency of on-site response.

[0214] A "surveillance camera" is a device used to continuously monitor a specific area or location and collect video data.

[0215] An "audio collection device" is a device that collects ambient sounds and records and transmits them as analog or digital data.

[0216] "Real-time" refers to the ability to process and transmit data and information instantly, without delay.

[0217] The "means for receiving data" refers to a function or device for receiving data transmitted from the surveillance camera and the audio collecting device.

[0218] The "means for analyzing data" is a function or device that processes the received data and performs analysis to detect anomalies.

[0219] An "abnormal sound" is a sound that indicates a specific abnormal situation, unlike normal background sounds such as breaking glass or screams.

[0220] "Means for detecting abnormalities" refers to a function or device that identifies abnormal sounds or behaviors as a result of data analysis.

[0221] An "alert generating means" is a function or device that generates a warning message or notification when an abnormality is detected.

[0222] A "means for notifying interested parties" is a function or device that generates an alert and then sends it to a designated recipient.

[0223] A "smart device" is a mobile terminal with Internet connectivity that can run applications.

[0224] A "pop-up message" is a notification message that suddenly appears on the device screen.

[0225] "Application notification" is a notification function that notifies the user of information through a specific application.

[0226] The "means for displaying the shortest route" is a function or device for calculating and displaying the most efficient route from the current location to the location where the abnormality has occurred.

[0227] In order to implement the present invention, a system is constructed that receives data in real time from a surveillance camera and an audio collecting device, analyzes the data, and detects abnormalities. The system configuration and processing content are described in detail below.

[0228] System Overview

[0229] This system is mainly composed of three entities: a server, a terminal, and a user, which cooperate to monitor and detect anomalies.

[0230] Hardware and Software Environment

[0231] Hardware:

[0232] Surveillance cameras (IoT cameras)

[0233] Audio collection device (microphone)

[0234] Smart devices (smart glasses, smartphones)

[0235] Server (cloud platform)

[0236] software:

[0237] Server side: AWS Lambda, EC2, Amazon Rekognition, Amazon Transcribe

[0238] AI models: TensorFlow, PyTorch

[0239] Device applications: Android Studio, Xcode, Google Glass SDK

[0240] Server Processing

[0241] 1. Data Collection:

[0242] The server receives and temporarily stores data streams from surveillance cameras and audio capture devices, and transcodes the data for real-time processing.

[0243] 2. Data Analysis:

[0244] The server uses pre-trained AI models (TensorFlow or PyTorch) to process the received data frame by frame and perform video and audio analysis in parallel. Amazon Rekognition is used for video analysis, and Amazon Transcribe is used for audio analysis.

[0245] 3. Anomaly detection:

[0246] When the server detects an abnormal sound (such as the sound of breaking glass or a person screaming), it records the abnormality and identifies details (time, location, type).

[0247] 4. Alert generation and sending:

[0248] If an anomaly is detected, the server generates an alert message and sends it to the relevant parties. The alert message contains detailed information about the anomaly (time, location, type).

[0249] Terminal handling

[0250] 1. Data Entry:

[0251] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices, temporarily stores the data, and processes it for transfer to the server.

[0252] 2. Data Transfer:

[0253] The terminal transmits the received data to the server in real time, supporting rapid data analysis.

[0254] 3. Receive anomaly alerts:

[0255] The terminal receives the anomaly detection alert sent from the server and prepares to notify the user.

[0256] 4. User Notice:

[0257] The device will notify users of any abnormalities as a pop-up message or application notification, allowing them to take prompt action. It will also work with map apps to display the shortest route to the location of the abnormality.

[0258] User Roles

[0259] 1. Initial Setup:

[0260] Users can set the locations of surveillance cameras and audio collection devices, check the connection status, and configure notification settings (e.g., email, SMS, app notifications).

[0261] 2. System Check:

[0262] Users can check the system status in real time to monitor whether it is working properly, and can also manually test the anomaly detection function.

[0263] 3. Response to abnormality notification:

[0264] The user receives an abnormality notification from the device, checks the detailed information, and then quickly moves to the location of the abnormality and reports it to the relevant authorities (police, fire department, etc.) if necessary.

[0265] Example scenario

[0266] For example, imagine there are surveillance cameras and audio collection devices installed in a shopping district at night. If the sound of glass breaking occurs in a store late at night, the sound is picked up by the sound-collecting microphone and the surveillance camera captures the video. This data is sent to a server in real time, which analyzes the received data and detects the sound of glass breaking. After detection, the server generates an alert message and sends it to the security company. The device receives this alert and notifies the security guard as a pop-up message. Upon seeing the notification, the security guard immediately rushes to the scene and prevents the damage from spreading. This system enables early detection of crimes and accidents and rapid response, improving social safety.

[0267] Prompt Sentence Examples

[0268] scenario:

[0269] We use a system that receives data in real time from surveillance cameras and microphones installed in shopping districts at night and detects abnormalities. Please explain in detail how it handles the situation when the sound of glass being broken is detected.

[0270] Input data:

[0271] Camera video data

[0272] Microphone audio data

[0273] GPS location information

[0274] Process flow:

[0275] 1. Data collection and transmission

[0276] 2. Data analysis and anomaly detection using AI models

[0277] 3. Notification and response when an abnormality occurs

[0278] The above is a specific embodiment for carrying out the invention.

[0279] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0280] Step 1: Data collection

[0281] The server receives video and audio data in real time from surveillance cameras and audio collection devices. Specifically, the server acquires stream data from IoT cameras and microphones and temporarily stores it. This data is transcoded for later analysis. The input data is camera video data and microphone audio data, and the output is the transcoded data.

[0282] Step 2: Data analysis

[0283] The server processes the received data frame by frame using a pre-trained AI model (TensorFlow or PyTorch). It uses Amazon Rekognition to analyze the video and Amazon Transcribe to analyze the audio data. The input data is transcoded video and audio data, and the output after analysis is a detailed analysis result that includes any abnormal sounds or behavior.

[0284] Step 3: Anomaly detection

[0285] When the server detects an abnormal sound (such as the sound of breaking glass or a person screaming) or abnormal behavior (such as suspicious activity), it records the abnormality and identifies detailed information (time, location, type). During this process, if it recognizes that an abnormality has occurred based on the analysis results, it sets an anomaly detection flag. The input is the analysis result, and the output is detailed information about the abnormality.

[0286] Step 4: Alert Generation

[0287] The server generates an alert message when an anomaly is detected. The alert contains detailed information about the anomaly (time, location, type) and also determines which parties to notify. The input is the detailed information about the anomaly, and the output is the generated alert message.

[0288] Step 5: Sending an alert

[0289] The server generates an alert message and sends it to the appropriate parties via email, SMS, app notification, etc. The input is the generated alert message and the output is the sent alert.

[0290] Step 6: Data entry

[0291] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices. The received data is temporarily stored and then formatted before being transferred to the server. The input is data from the surveillance cameras and audio collection devices, and the output is formatted data.

[0292] Step 7: Data Transfer

[0293] The terminal transmits the received data to the server in real time, which supports fast data analysis on the server side. The input is the formatted data, and the output is the transmitted data.

[0294] Step 8: Receive anomaly alerts

[0295] The terminal receives an anomaly detection alert sent from the server. This alert contains detailed information about the anomaly and is prepared for notification to the user. The input is the alert message from the server, and the output is the preparation status for notification.

[0296] Step 9: User Notification

[0297] The device displays the abnormality notification as a pop-up message or application notification. It also works with a map application to display the shortest route to the location of the abnormality, allowing the user to respond quickly. The input is the prepared notification information, and the output is the displayed notification and route information.

[0298] Step 10: Initial Setup

[0299] The user sets the installation location of the surveillance camera and audio collection device, checks the connection status, and sets notification settings (e.g., email, SMS, app notifications). The input is the initial state of the device, and the output is the device state after the settings are complete.

[0300] Step 11: System Verification

[0301] Users can check the system status in real time to monitor whether it is operating normally. They can check the real-time status of the system and check whether there are any abnormalities. The input is the current system status and the output is the check result.

[0302] Step 12: Responding to abnormal notifications

[0303] The user receives an abnormality notification from the device and checks the detailed information. Then, they quickly move to the location of the abnormality and, if necessary, notify the relevant authorities (police, fire department, etc.). The input is the notification information from the device, and the output is the user's response action.

[0304] The above are the specific processing steps and their contents. This series of processes enables rapid and effective anomaly detection and response.

[0305] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0306] System Overview

[0307] The system based on this invention is a comprehensive monitoring system that receives data in real time from surveillance cameras and audio collection devices, detects abnormalities, and notifies relevant parties. This system, in particular, can improve the accuracy of anomaly detection and the speed of response by combining it with an emotion engine that recognizes user emotions. The system is mainly composed of three entities: a server, a terminal, and a user, which work together to perform effective monitoring and issue alerts.

[0308] In the embodiment of the present invention, the system is composed of the following three entities.

[0309] Server Processing

[0310] 1. Data Collection

[0311] The server receives and temporarily stores data streams from surveillance cameras and audio collection devices, and transcodes the data for real-time processing.

[0312] 2. Data Analysis

[0313] The server loads a pre-trained AI model, which includes algorithms for detecting abnormal sounds and movements and analyzing user emotions.

[0314] The received data is processed frame by frame, and video and audio analysis is performed in parallel, allowing for anomaly detection and emotion analysis to be performed simultaneously.

[0315] 3. Anomaly Detection

[0316] The server detects abnormal behavior and sounds (such as the sound of breaking glass or people screaming) from video and audio data.

[0317] The server uses an emotion engine to analyze the user's emotions (e.g., anger, anxiety, sadness, etc.) using voice tone and facial expression analysis.

[0318] 4. Alert generation and sending

[0319] If an anomaly or a specific emotion is detected, the server generates an alert message, which includes detailed information about the anomaly (time, location, type) and the result of emotion analysis.

[0320] Generate different levels of alerts depending on the type of anomaly and the degree of emotion.

[0321] The server generates an alert message and sends it to the relevant parties (security company, police, fire department, etc.).

[0322] Terminal handling

[0323] 1. Data Entry

[0324] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices.

[0325] The received data is temporarily stored and formatted for transfer to the server.

[0326] 2. Data Transfer

[0327] The terminal transmits the received data to the server in real time, supporting rapid data analysis on the server side.

[0328] 3. Receive anomaly alerts

[0329] The terminal receives an alert message for anomaly detection and emotion analysis sent from the server.

[0330] Verify the accuracy of received alerts and prepare to notify users.

[0331] 4. User Notices

[0332] The device displays alerts as pop-up messages or application notifications, allowing users to respond quickly. Different notification methods (screen display, audio alarm, vibration alarm, etc.) are provided depending on the level of the alert.

[0333] User Action

[0334] 1. Initial Setup

[0335] Users can set the installation locations of surveillance cameras and audio collection devices, check the connection status, and configure notification settings (e.g., email, SMS, app notifications).

[0336] 2. System Check

[0337] Users can check the system status in real time to monitor whether it is working properly, and can also manually test the anomaly detection and sentiment analysis functions.

[0338] 3. Response to abnormality notifications

[0339] The user receives an abnormality notification from the device, checks the content of the notification, checks detailed information about the abnormality and the results of emotion analysis, and, if necessary, rushes to the scene or reports the incident to the relevant authorities (police, fire department, etc.).

[0340] Example scenario

[0341] For example, suppose there are surveillance cameras and audio collection devices installed in a shopping district at night. If the sound of glass breaking occurs in a store late at night and a panicked user screams inside the store at the same time, the audio collection device picks up the sound and the surveillance camera captures the video. These data are sent to a server in real time, and the server analyzes the received data.

[0342] At the same time that the server detects the sound of breaking glass and a scream, the emotion engine recognizes the user's emotion as "anxiety." After detection, the server generates an alert message and sends it to the security company with the following content: "01:23 AM, Store A, sound of breaking glass detected, emotion of anxiety." The device receives this alert and notifies the security guard as a pop-up message. The user (security guard) who sees the notification immediately rushes to the scene and prevents the damage from spreading. This system enables early detection of crimes and accidents and rapid response, improving social safety.

[0343] According to the embodiment of the present invention, the efficiency and accuracy of monitoring work can be improved, and in particular, safety can be further enhanced by making it possible to take into account the emotions of users.

[0344] The processing flow will be explained below.

[0345] Detailed system programming

[0346] Server Processing

[0347] Step 1:

[0348] A server receives data streams from surveillance cameras and audio collection devices.

[0349] The received video and audio data is temporarily stored and transcoded.

[0350] Step 2:

[0351] The server loads a pre-trained AI model.

[0352] Prepare algorithms for anomaly detection and sentiment analysis.

[0353] Step 3:

[0354] The server processes the received data frame by frame.

[0355] Parallel processing is set up so that video data and audio data can be analyzed simultaneously.

[0356] Step 4:

[0357] The server detects abnormal behavior (e.g., violent acts or rapid movement) from the video data.

[0358] The audio data is filtered to detect abnormal sounds (such as glass breaking or screams).

[0359] Step 5:

[0360] If the server detects abnormal behavior or sound, it sends the frame from that point to the emotion analysis engine.

[0361] The sentiment analysis engine recognizes the user's emotions (e.g., anger, anxiety, sadness).

[0362] Step 6:

[0363] The server evaluates abnormal behavior, abnormal sounds, and emotion analysis results, and records details of the abnormality (time, location, type).

[0364] The reliability of the detected anomalies is confirmed by applying a secondary checking algorithm.

[0365] Step 7:

[0366] The server generates an alert message if an abnormality is detected.

[0367] The alert message includes detailed information about the anomaly (time, location, type) and the results of sentiment analysis.

[0368] Step 8:

[0369] The server generates an alert message and sends it to the relevant parties (security company, police, fire department, etc.).

[0370] Checks whether the message was sent successfully and attempts to send it again if it fails.

[0371] Terminal handling

[0372] Step 1:

[0373] The terminal receives video and audio data in real time from the connected surveillance cameras and audio collection devices.

[0374] The received data is temporarily stored and formatted for transfer to the server.

[0375] Step 2:

[0376] The terminal transmits the received data to the server in real time.

[0377] Check the data for completeness and attempt to resend any missing data.

[0378] Step 3:

[0379] The device receives anomaly detection and emotion analysis alert messages sent from the server.

[0380] Verify the accuracy of received alerts and prepare to notify users.

[0381] Step 4:

[0382] Your device will display the alert as a pop-up message or application notification.

[0383] Different notification methods (screen display, audio alarm, vibration alarm, etc.) are provided depending on the level of the alert.

[0384] User Action

[0385] Step 1:

[0386] The user sets the installation locations of the surveillance cameras and audio collection devices and checks the connection status.

[0387] The user configures system notification settings (e.g., email, SMS, app notifications).

[0388] Step 2:

[0389] Users can check the system status in real time and monitor whether it is operating normally.

[0390] It is also possible to manually test the anomaly detection and sentiment analysis features.

[0391] Step 3:

[0392] The user receives the abnormality notification from the terminal and checks the content of the notification.

[0393] Check the detailed information about the anomaly and the results of the emotion analysis, and if necessary, rush to the scene or notify the relevant authorities.

[0394] Example scenario

[0395] For example, suppose there are surveillance cameras and audio collection devices installed in a shopping district at night. If a store hears the sound of glass breaking late at night and a panicked user screams inside the store at the same time, the audio collection device picks up the sound and the surveillance camera captures the video. These data are sent to a server in real time.

[0396] The server analyzes the received data and detects the sound of glass breaking and a scream. At the same time, the emotion engine recognizes the user's emotion as "anxiety." After detection, the server generates an alert message and sends it to the security company with the following content: "01:23 AM, Store A, sound of glass breaking detected, emotion of anxiety."

[0397] The device receives this alert and notifies the security guard as a pop-up message. The user (security guard) who sees the notification can immediately rush to the scene to prevent the damage from spreading. This system enables early detection of crimes and accidents and a rapid response, improving social safety.

[0398] According to the embodiment of the present invention, the efficiency and accuracy of monitoring work can be improved, and in particular, safety can be further enhanced by making it possible to take into account the emotions of users.

[0399] Example 2

[0400] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0401] Modern surveillance systems require improved anomaly detection accuracy and rapid response, but conventional systems perform video and audio analysis separately, resulting in slow anomaly detection times and reduced accuracy. Furthermore, systems are unable to respond to user emotions, making it difficult to respond appropriately based on specific emotions. Furthermore, when an anomaly is detected, the alert content is limited and detailed information is lacking, making it difficult for relevant parties to respond quickly and appropriately.

[0402] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving data in real time from the monitoring camera and the audio collecting device, a data analysis means for processing the received data frame by frame and including an algorithm for performing video analysis and audio analysis in parallel, a means for detecting abnormal behavior or sound from the analyzed data, an emotion analysis means for recognizing the emotion of the user, and a means for generating an alert and notifying relevant parties when an abnormality or a specific emotion is detected. This improves the accuracy of anomaly detection and the speed of response, allowing relevant parties to respond quickly and appropriately based on detailed information.

[0403] A "surveillance camera" is a device that captures video data in real time and captures the situation at a monitored location.

[0404] An "audio collection device" is a device that collects surrounding audio data in real time and uses it for monitoring purposes.

[0405] An "algorithm that processes data frame by frame and performs video and audio analysis in parallel" is a computer program that divides video data into its constituent frames and performs analysis of the video and audio data simultaneously.

[0406] "Means for detecting abnormal behavior or sounds" refers to a system that has the ability to analyze collected data and identify pre-defined abnormal behavior or sound patterns.

[0407] "Emotion analysis means for recognizing user emotions" refers to an algorithm or model for determining a user's emotional state (e.g., anger, anxiety, sadness) from audio tone or video data.

[0408] "Means for generating alerts and notifying relevant parties" refers to a system that creates warning messages based on detected abnormalities and emotional information and sends them to relevant agencies (e.g., security companies, police, fire departments, etc.).

[0409] "Detailed information (time, location, type)" is specific information about the detected abnormal event, including the time of occurrence, the location of occurrence, and the type of abnormality.

[0410] "Means for checking the system status in real time" refers to an interface that allows a user to monitor the current operating status of the data collection device and analysis server in real time and check whether they are functioning normally.

[0411] The system based on this invention is a comprehensive monitoring system that receives data in real time from surveillance cameras and audio collection devices, detects abnormal events, and notifies relevant parties. In particular, by combining it with an emotion analysis engine that recognizes user emotions, the accuracy of abnormality detection and the speed of response can be improved. This system is composed of three entities: a server, a terminal, and a user.

[0412] Server Processing

[0413] The server first receives data in real time from surveillance cameras and audio collection devices. The received data is temporarily stored and transcoded for real-time processing. The server then loads a pre-trained AI model. This AI model includes algorithms for detecting abnormal sounds and behaviors, as well as algorithms for analyzing user emotions. The data is divided into frames, and video and audio analysis is performed in parallel. For example, frameworks such as TensorFlow and PyTorch are used. Abnormal behaviors and sounds are detected from the analyzed data, and an emotion analysis engine is used to identify the user's emotion (e.g., anger, anxiety, sadness, etc.). If an anomaly or emotion is detected, an alert message is generated and sent to relevant parties, along with detailed information about the anomaly (time, location, type) and the results of the emotion analysis.

[0414] Terminal handling

[0415] The device receives video and audio data from surveillance cameras and audio collection devices in real time. The received data is temporarily stored and formatted for transmission to the server. The formatted data is sent to the server in real time, supporting rapid data analysis on the server side. The device receives anomaly detection and emotion analysis alert messages sent from the server and checks their accuracy. The received alerts are displayed as pop-up messages or application notifications, allowing the user to respond quickly. Different notification methods (screen display, audio alarm, vibration alarm, etc.) are provided depending on the level of the alert.

[0416] User Action

[0417] First, the user sets the installation locations of the surveillance cameras and audio collection devices and checks the connection status. They also configure notification settings (e.g., email, SMS, app notifications). Users can check the system status in real time and monitor whether it is operating normally. They can also manually test the anomaly detection and emotion analysis functions. When a user receives an anomaly notification from their device, they can check the notification content and confirm the details of the anomaly and the results of the emotion analysis. If necessary, they can rush to the scene or notify the relevant authorities (police, fire department, etc.).

[0418] Example scenario

[0419] For example, consider a surveillance camera and audio collection device installed in a shopping district at night. Suppose a store hears the sound of glass breaking late at night, and simultaneously, a panicked user screams inside the store. The audio collection device picks up the sound, and the surveillance camera captures the video. These data are sent to a server in real time, which analyzes the received data. When the server detects the sound of glass breaking and the scream, the emotion engine recognizes the user's emotion as "anxiety." After detection, the server generates an alert message stating "1:23 AM, Store A, sound of glass breaking detected, emotion of anxiety," and sends it to the security company. The device receives this alert and notifies the security guard via a pop-up message. The user (security guard) immediately rushes to the scene after confirming the notification and prevents further damage. This system enables early detection and rapid response to crimes and accidents, improving social safety.

[0420] Example prompts for generative AI models

[0421] "Receive data in real time from surveillance cameras and audio collection devices, and generate an alert message if an anomaly is detected. For example, imagine a case where a glass breaks in a store at night, causing a panicked user to scream. Include relevant details (time, location, type) and the results of emotion analysis (anxiety)."

[0422] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0423] Step 1: Data collection

[0424] Terminal

[0425] 1. Receive video and audio data in real time from surveillance cameras and audio collection devices.

[0426] Input: Real-time data from surveillance cameras and audio collection devices.

[0427] Specific operation: The surveillance camera captures video data at 30 frames per second, and the audio collection device collects audio data at 44.1 kHz per second.

[0428] Output: Temporarily stored video and audio data.

[0429] Step 2: Data preparation and transfer

[0430] Terminal

[0431] 1. The received video and audio data is temporarily stored and formatted for transfer to the server.

[0432] Input: Temporarily stored video and audio data.

[0433] Specific operation: Frames of video data are stored in a buffer, and audio data is stored as chunks in certain ranges.

[0434] Output: The formatted data.

[0435] 2. Send the formatted data to the server in real time.

[0436] What it does: Streams data packets to a server using the TCP / IP protocol.

[0437] Step 3: Data analysis

[0438] server

[0439] 1. Split the data stream to analyze the received data frame by frame.

[0440] Input: Formatted data transferred to the server.

[0441] Specific operation: Video frames are extracted from the data stream, and the audio data is processed as is.

[0442] Output: Data divided into frames.

[0443] 2. Load the pre-trained AI model and prepare it for anomaly detection and sentiment analysis.

[0444] Specific operation: Load an anomaly detection model into memory using TensorFlow or PyTorch.

[0445] Step 4: Anomaly detection and sentiment analysis

[0446] server

[0447] 1. Video analysis: Analyze each frame of received video data to detect abnormal behavior.

[0448] Input: Video data divided into frames.

[0449] Specific operation: When the movement between frames changes drastically, an algorithm is applied that considers this to be abnormal behavior.

[0450] Output: Anomalous behavior detection results.

[0451] 2. Audio analysis: Spectral analysis of audio data is performed to detect abnormal sounds (such as the sound of glass breaking or a person screaming).

[0452] Input: Audio data divided into frames.

[0453] Specific behavior: Identify sudden changes in sound pressure level and specific frequency patterns.

[0454] Output: Abnormal sound detection result.

[0455] 3. Emotion analysis: Analyzes the tone of voice and facial expressions to identify the user's emotions.

[0456] Input: Received video and audio data.

[0457] Specific operation: Identify emotions by detecting changes in voice tone, pitch, and facial expressions in video.

[0458] Output: Emotion identification results.

[0459] Step 5: Generate and send an alert

[0460] server

[0461] 1. If an anomaly or a specific emotion is detected, an alert message is generated, containing detailed information about the anomaly (time, location, type) and the results of emotion analysis.

[0462] Input: Abnormal behavior, abnormal sound, and emotion analysis results.

[0463] Specific operations: Integrates and converts data, and generates alert messages.

[0464] Output: The alert message.

[0465] 2. Send the generated alert message to the relevant parties.

[0466] What it does: Deliver alerts via SMS, email, and app notifications.

[0467] Step 6: Receive alerts and notifications

[0468] Terminal

[0469] 1. Receive an alert message sent from the server.

[0470] Input: The alert message from the server.

[0471] Specific operation: Receive alert information via HTTPS encrypted communication.

[0472] Output: The received alert message.

[0473] 2. Review the accuracy of the alerts received and make arrangements to notify users.

[0474] Specific operation: Analyzes the message content and converts it into a display format.

[0475] 3. Display the alert to the user as a popup message or app notification.

[0476] Specific behavior: Use different notification methods such as popup, audio alarm, vibration, etc.

[0477] Step 7: Responding to abnormality notifications

[0478] User

[0479] 1. Receive an abnormality notification from the device and check the content of the notification.

[0480] Input: Alert notification from the terminal.

[0481] Specific operation: Display and confirm detailed information about anomalies and the results of sentiment analysis.

[0482] 2. If necessary, rush to the scene and notify the relevant authorities.

[0483] Specific actions: Decide on a response promptly based on the information confirmed.

[0484] (Application example 2)

[0485] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0486] Conventional monitoring systems monitor a huge amount of data and perform analysis to detect abnormalities, but are unable to analyze the user's emotional state when an abnormality occurs, making it difficult to respond quickly and appropriately.The present invention aims to enable faster and more appropriate responses by analyzing user emotional data in real time in addition to monitoring data and generating alerts including the results of emotional analysis when an abnormality occurs.

[0487] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0488] In this invention, the server includes means for receiving data in real time from the surveillance camera and the audio collecting device, means for analyzing the received data and detecting a predetermined abnormal sound, means for generating and reporting an alert when an abnormal sound is detected, means for analyzing the user's emotions, and means for generating and reporting an alert message including the emotion analysis result. This makes it possible to analyze the user's emotional state at the same time as detecting an abnormality and generate a detailed alert including emotional information.

[0489] A "surveillance camera" is a device that captures images of a location in real time and records or transmits the data.

[0490] An "audio collection device" is a device for collecting audio data in an environment in real time and recording or transmitting that data.

[0491] "Means for receiving data in real time" refers to technical means for receiving data almost simultaneously with its generation.

[0492] "Means for analyzing received data" refers to technical means for extracting information or detecting anomalies from data received in real time using specific algorithms.

[0493] "Means for detecting predetermined abnormal sounds" refers to technical means for detecting specific abnormal sounds (e.g., the sound of breaking glass, screams, etc.) that are defined in advance.

[0494] "Means for generating and reporting an alert when an abnormal sound is detected" refers to the technical means for generating a warning message or notification when an abnormal sound is detected and for promptly reporting it to designated parties.

[0495] "Means for analyzing user emotions" refers to technical means for analyzing the user's emotional state (e.g., anger, anxiety, sadness, etc.) from video and audio data.

[0496] "Means for generating and notifying an alert message including the result of emotion analysis" refers to a technical means for generating a detailed alert message including the result of the user's emotion analysis and promptly notifying relevant parties.

[0497] System configuration

[0498] The system embodying the present invention consists of three main components: a server, a terminal, and a user. The server receives data from the surveillance cameras and audio collection devices, analyzes the data, and detects abnormalities. The terminal transmits the collected data to the server and notifies the user of alert messages from the server. The user performs the initial system setup and responds appropriately when notified of an abnormality.

[0499] Hardware and software used

[0500] The server requires hardware with high-performance data analysis capabilities, and specifically, a server equipped with an NVIDIA GPU is suitable. Machine learning frameworks such as TensorFlow and PyTorch are used for data analysis. The terminals used are devices that users can carry, such as smartphones, tablets, and smart glasses. These devices are equipped with high-resolution cameras and highly sensitive microphones, making it possible to collect data in real time.

[0501] Specific examples

[0502] For example, if a nighttime surveillance system is installed in a shopping mall, surveillance cameras and audio collection devices are installed in each store. These devices collect video and audio data in real time and send it to a server via a terminal. The server analyzes the received data and detects abnormal sounds and behavior.

[0503] If an abnormality is detected, the server analyzes the user's emotions and generates an alert message containing the results of the emotion analysis. For example, a message may be generated stating, "The sound of glass breaking was detected at Store A at 2:00 AM, and an emotion of anxiety was recognized." This alert message is sent to the security guard's smart glasses in real time. The security guard can then check the notification and quickly rush to the scene.

[0504] Generative AI model and prompts

[0505] The system uses a generative AI model to analyze user sentiment. Specific examples of prompts include:

[0506] Example prompt sentence:

[0507] "Detect the sound of glass breaking or a person screaming, and analyze the user's emotions. For example, recognize emotions such as anxiety or anger. If an anomaly is detected, generate an alert containing the anomaly and the results of the emotion analysis, and notify the user."

[0508] This enables the monitoring system to detect abnormalities early and respond quickly and appropriately while taking into account the emotional state of the user. This invention is particularly effective in strengthening security in commercial facilities, public facilities, important facilities, etc.

[0509] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0510] Step 1:

[0511] Data collection (terminal)

[0512] The device (e.g., smartphone, smart glasses) collects data (video and audio) in real time from surveillance cameras and audio collectors. The cameras generate high-resolution video data, and the audio collectors collect audio data with high sensitivity. This data is temporarily stored on the device. The input includes video data from the surveillance cameras and audio data from the audio collectors, and the output is the temporary storage of these data.

[0513] Step 2:

[0514] Data transfer (terminal)

[0515] The device processes the temporarily stored video and audio data (for example, transcoding or compressing) and then transmits it to the server in real time. Here, the video data is encoded into JPEG or H.264 format, and the audio data is encoded into MP3 or AAC format. The temporarily stored raw data is the input, and the transcoded data is sent to the server as the output.

[0516] Step 3:

[0517] Data analysis (server)

[0518] The server receives the video and audio data sent from the device, loads a pre-trained generative AI model, and performs data analysis. The analysis includes detecting abnormal behavior from the video, detecting abnormal sounds from the audio, and analyzing the user's emotions. Specifically, it detects movement and shape changes from the video, and specific frequencies and patterns from the audio. The input is the received transcoded data, and the output is an analysis of the type of abnormality and the emotional state.

[0519] Step 4:

[0520] Anomaly detection and sentiment analysis (server)

[0521] If the server detects an anomaly from the data analysis results, it further analyzes the user's emotions (anger, anxiety, sadness, etc.) using a generative AI model. Specifically, emotions are recognized based on voice tone and facial expression analysis. The input is the preliminary results of the data analysis, and the output is the presence or absence of an anomaly and the user's emotional state.

[0522] Step 5:

[0523] Alert generation and transmission (server)

[0524] The server generates an alert message based on the anomaly and the results of sentiment analysis. The alert message contains detailed information such as the type of anomaly, time, location, and sentiment analysis results. The generated alert message is promptly sent to relevant parties (e.g., security guards, police, fire department, etc.). The inputs are the anomaly and the sentiment analysis results, and the generated alert message is the output.

[0525] Step 6:

[0526] User notification (terminal)

[0527] The terminal receives the alert message sent from the server and notifies the user (e.g., security guard). Notification methods include screen display, audio alarm, and vibration alarm. The alert message is the input, and the notification to the user is the output.

[0528] Step 7:

[0529] Response (user)

[0530] The user receives an alert notification from the device and checks the content of the notification. For example, if the alert content is "The sound of glass breaking was detected at Store A at 2:00 AM, and feelings of anxiety were recognized," the user will immediately rush to the scene and take appropriate action. The input is the alert notification, and the output is the user's response action.

[0531] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0532] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0533] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0534] [Second embodiment]

[0535] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0536] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0537] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0538] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0539] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0540] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0541] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0542] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0543] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0544] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0545] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0546] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0547] System Overview

[0548] The system based on this invention is a comprehensive monitoring system that receives data in real time from surveillance cameras and audio collection devices, detects abnormalities, and notifies relevant parties. The system is mainly composed of three components: a server, a terminal, and a user, which work together to perform effective monitoring and issue alerts.

[0549] In the embodiment of the present invention, the system is composed of the following three entities.

[0550] Server Processing

[0551] 1. Data Collection

[0552] The server receives and temporarily stores data streams from surveillance cameras and audio collection devices, and transcodes the data for real-time processing.

[0553] 2. Data Analysis

[0554] The server uses a pre-trained AI model to process the received data frame by frame, performing video and audio analysis in parallel, thereby achieving highly accurate anomaly detection.

[0555] 3. Anomaly Detection

[0556] When the server detects an abnormal sound (for example, the sound of breaking glass or a person screaming), it records the abnormality and identifies details (time, location, type).

[0557] 4. Alert generation and sending

[0558] If an anomaly is detected, the server generates an alert message and sends it to the relevant parties. The alert message contains detailed information about the anomaly.

[0559] Terminal handling

[0560] 1. Data Entry

[0561] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices, temporarily stores the data, and processes it for transfer to the server.

[0562] 2. Data Transfer

[0563] By sending the data received by the terminal to the server in real time, it supports rapid data analysis on the server side.

[0564] 3. Receive anomaly alerts

[0565] The terminal receives the anomaly detection alert sent from the server and prepares to notify the user.

[0566] 4. User Notices

[0567] The device will display the alert as a pop-up message or application notification, allowing the user to respond quickly.

[0568] User Action

[0569] 1. Initial Setup

[0570] Users can set the installation locations of surveillance cameras and audio collection devices, check the connection status, and configure notification settings (e.g., email, SMS, app notifications).

[0571] 2. System Check

[0572] Users can check the system status in real time and monitor whether it is working properly. Users can also manually test the anomaly detection function.

[0573] 3. Response to abnormality notifications

[0574] The user receives an abnormality notification from the device, checks the detailed information, and then rushes to the scene or reports it to the relevant authorities (police, fire department, etc.) as necessary.

[0575] Example scenario

[0576] For example, imagine there are surveillance cameras and audio collection devices installed in a shopping district at night. If the sound of glass breaking occurs in a store late at night, the sound is picked up by the sound-collecting microphone and the surveillance camera captures the video. This data is sent to a server in real time. The server analyzes the received data and detects the sound of glass breaking. After detection, the server generates an alert message and sends it to the security company. The device receives this alert and notifies the user (security guard) as a pop-up message. Upon seeing the notification, the user can immediately rush to the scene and prevent the damage from spreading. This system enables early detection of crimes and accidents and rapid response, improving social safety.

[0577] The embodiments of the present invention improve the efficiency and accuracy of monitoring operations, and make a significant contribution to preventing crimes and accidents.

[0578] The processing flow will be explained below.

[0579] System Program Processing

[0580] Server Processing

[0581] Step 1:

[0582] A server receives data streams from surveillance cameras and audio collection devices.

[0583] The received video and audio data is temporarily stored and transcoded.

[0584] Step 2:

[0585] The server loads a pre-trained AI model.

[0586] The received data is processed frame by frame, and video and audio analysis is performed in parallel.

[0587] Step 3:

[0588] The server detects abnormal movements in the video data and filters out abnormal sounds (such as breaking glass or people screaming) from the audio data.

[0589] Apply anomaly detection algorithms to check if anomalies have occurred.

[0590] Step 4:

[0591] When the server detects an abnormal sound or behavior, it records the details (time, location, type).

[0592] Apply a secondary check algorithm to ensure the anomaly detection is correct.

[0593] Step 5:

[0594] The server generates an alert message if an abnormality is detected.

[0595] Include detailed information about the anomaly (time, location, type) in the alert message.

[0596] Step 6:

[0597] The server generates an alert message and sends it to the relevant parties (security company, police, fire department, etc.).

[0598] Checks whether the message was sent successfully and attempts to send it again if it fails.

[0599] Terminal handling

[0600] Step 1:

[0601] The terminal receives video and audio data in real time from the connected surveillance cameras and audio collection devices.

[0602] The received data is temporarily stored and formatted for transfer to the server.

[0603] Step 2:

[0604] The terminal transmits the received data to the server in real time.

[0605] Check the data for completeness and attempt to resend any missing data.

[0606] Step 3:

[0607] The terminal receives an anomaly detection alert sent from the server.

[0608] Check the accuracy of the received alerts and send feedback to the server if necessary.

[0609] Step 4:

[0610] The device displays the alert to the user as a pop-up message or application notification.

[0611] The alarm can be set to a screen display, audio alarm, vibration alarm, etc.

[0612] User Action

[0613] Step 1:

[0614] The user sets the installation locations of the surveillance cameras and audio collection devices and checks the connection status.

[0615] The user configures system notification settings (e.g., email, SMS, app notifications).

[0616] Step 2:

[0617] Users can check the status of the system and monitor whether it is working properly.

[0618] It is also possible to manually test the anomaly detection feature.

[0619] Step 3:

[0620] The user receives the abnormality notification from the terminal and checks the content of the notification.

[0621] Check detailed information about the abnormality (time, location, type) and, if necessary, rush to the scene or report it to the relevant authorities.

[0622] Example 1

[0623] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0624] In recent years, there has been an increasing demand for more accurate and efficient monitoring systems. However, existing monitoring systems have problems such as low anomaly detection accuracy and insufficient real-time data processing. Furthermore, they are often unable to quickly notify relevant parties of detected anomalies, making it difficult to respond quickly. To address these issues, there is a demand for monitoring systems that can detect anomalies more accurately in real time and quickly notify relevant parties.

[0625] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0626] In this invention, the server includes means for receiving data in real time from the surveillance cameras and audio collecting devices, means for temporarily storing the received data and transcoding it into an optimal format, means for processing the received data frame by frame using a pre-trained artificial intelligence model to perform video and audio analysis in parallel, means for detecting abnormal sounds and images, recording the abnormality and identifying details (time, location, type), and means for generating an alert message and notifying relevant parties when an abnormality is detected. This enables real-time abnormality detection and highly accurate data analysis, realizing early detection of abnormalities and rapid response.

[0627] A "surveillance camera" is a device that acquires video data in real time and is used for monitoring and anomaly detection.

[0628] An "audio collection device" is a device that acquires audio data in real time and is used for monitoring and anomaly detection.

[0629] A "data stream" is a continuous flow of data transmitted in real time from a surveillance camera or audio collection device.

[0630] "Transcoding" is the process of converting received data into a different format.

[0631] An "artificial intelligence model (AI model)" is an algorithm that has been trained in advance using large amounts of data, and is used to detect and analyze anomalies.

[0632] "Frame-based processing" is a method in which video data and audio data are divided into fixed time intervals (frames) and analyzed and processed for each unit.

[0633] "Parallel processing" is a technology that improves processing speed by executing multiple processes or threads simultaneously.

[0634] An "abnormal sound" is a specific sound (such as the sound of breaking glass or screaming) that does not occur in a normal monitoring environment and indicates an abnormal situation.

[0635] "Abnormal footage" refers to specific footage (for example, suspicious activity) that is not seen in a normal surveillance environment and indicates an abnormal situation.

[0636] An "alert message" is a notification message that is generated when an abnormality is detected, and provides detailed information about the abnormality to relevant parties.

[0637] "Relevant parties" are people who manage and operate the monitoring system and people who are responsible for responding to abnormalities (e.g., security guards and administrators).

[0638] The monitoring system based on this invention is mainly composed of three entities: a server, a terminal, and a user. It is a comprehensive monitoring system that receives data in real time from monitoring cameras and audio collection devices, detects abnormalities, and notifies relevant parties. The role of each entity will be specifically explained below.

[0639] Server Processing

[0640] The server receives real-time data streams from surveillance cameras (e.g., general network cameras) and audio collection devices (e.g., standard microphones). The data is temporarily stored and transcoded into the optimal format for real-time processing. For example, video data is converted to H.264 format, and audio data is converted to WAV format. The transcoding process uses high-speed processing using NVIDIA CUDA.

[0641] The server then analyzes the data using pre-trained AI models. It uses TensorFlow to load an anomaly detection model, a YOLO model for video analysis, and a CNN model for audio analysis. Data is processed frame by frame and analyzed in parallel. Video data is analyzed using OpenCV, and audio data undergoes spectrogram analysis using Librosa.

[0642] When an abnormal sound or video is detected, the server records the anomaly and identifies the time, location, and type of the anomaly. Detailed information about the anomaly is recorded in a MySQL database. If an anomaly is detected, an alert message is generated and sent to relevant parties using Amazon SNS (Simple Notification Service).

[0643] Terminal handling

[0644] The terminal uses a Raspberry Pi or a standard edge device to receive data in real time from surveillance cameras and audio collection devices. The received data is temporarily stored in a buffer and processed. The terminal then transfers the received data to the server in real time using the MQTT protocol.

[0645] When an anomaly detection alert is received from the server, the device prepares a notification to the user. Firebase Cloud Messaging is used to notify the user of the alert message as a pop-up display or app notification, allowing the user to respond to the anomaly quickly.

[0646] User Action

[0647] Users can use a web-based management screen to set the installation locations of surveillance cameras and audio collection devices and check the connection status. They can also set notifications (e.g., email, SMS, app notifications). Users can check the system status from the management screen and monitor operation in real time. If an abnormality is detected, users will receive an alert message and check detailed information (e.g., date, time, location, and nature of the abnormality). They can then rush to the scene or notify the relevant authorities as necessary.

[0648] Example scenario

[0649] For example, consider the case where a surveillance camera and audio collection device installed in a shopping district detect the sound of glass breaking in a store late at night. The received data is sent to a server, where an AI model is used to detect the sound of glass breaking. The detected abnormality is recorded and an alert message is sent to the security company using Amazon SNS. The device receives this alert and notifies the security guard via a pop-up message. The security guard can then immediately rush to the scene and prevent further damage.

[0650] Example prompts to input to the generative AI model

[0651] "I'm in charge of the surveillance system for a shopping mall. Can you explain the outline of the AI ​​model that detects suspicious sounds and images at night and immediately notifies security guards?"

[0652] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0653] Step 1:

[0654] Receiving and temporarily storing data (server processing)

[0655] Input: Real-time data streams from surveillance cameras and audio collection devices

[0656] Specific operation: The server receives data streams from surveillance cameras and audio collection devices using RTSP (Real-Time Streaming Protocol). The received data is temporarily saved and stored in a buffer for subsequent processing.

[0657] Output: Temporarily saved data (video data and audio data)

[0658] Step 2:

[0659] Data transcoding (server processing)

[0660] Input: Temporarily saved data (video data and audio data)

[0661] Specific operation: The server transcodes the received video data into H.264 format and the audio data into WAV format. The transcoding process is performed at high speed using NVIDIA CUDA.

[0662] Output: Transcoded data (video data in H.264 format, audio data in WAV format)

[0663] Step 3:

[0664] Preparing for data analysis (server processing)

[0665] Input: Transcoded data (video data in H.264 format, audio data in WAV format)

[0666] How it works: The server loads pre-trained AI models using TensorFlow. The YOLO model is used for video analysis, and the CNN model is used for audio analysis.

[0667] Output: A ready-to-use AI model

[0668] Step 4:

[0669] Frame-by-frame data analysis (server processing)

[0670] Input: AI model ready, transcoded data

[0671] Specific operation: The server divides the video and audio data into frames and processes them in parallel. The video data is analyzed using OpenCV, and the audio data undergoes spectrogram analysis using Librosa.

[0672] Output: Parsed frame data

[0673] Step 5:

[0674] Anomaly detection (server processing)

[0675] Input: Parsed frame data

[0676] How it works: The server uses AI models to detect abnormal sounds and images. For example, if an abnormality such as the sound of broken glass, screams, or a suspicious person is detected, the server records the abnormality and identifies the time, location, and type of occurrence. Detailed information about the abnormality is then recorded in a MySQL database.

[0677] Output: Anomaly detection results and detailed information (time, location, type)

[0678] Step 6:

[0679] Alert generation and notification (server processing)

[0680] Input: Anomaly detection results and detailed information

[0681] Specific operation: If an anomaly is detected, the server generates an alert message. The message contains detailed information about the anomaly (e.g., date, time, location, and content of the anomaly). The alert message is sent to relevant parties using Amazon SNS.

[0682] Output: Alert message sent

[0683] Step 7:

[0684] Data entry (terminal processing)

[0685] Input: Data acquired from surveillance cameras and audio collection devices

[0686] How it works: The device uses a Raspberry Pi or a standard edge device to receive and temporarily store data from surveillance cameras and audio capture devices in real time, check for missing or abnormal data, and add a timestamp.

[0687] Output: Formatted data

[0688] Step 8:

[0689] Data transfer (terminal processing)

[0690] Input: Formatted data

[0691] Specific operation: The terminal uses the MQTT protocol to send the received data to the server in real time.

[0692] Output: Data transferred to the server

[0693] Step 9:

[0694] Receive an abnormality alert (terminal processing)

[0695] Input: The alert message sent from the server

[0696] Specific operation: The terminal receives the alert message sent from the server and prepares to notify the user.

[0697] Output: Prepared alert message

[0698] Step 10:

[0699] User notification (terminal processing)

[0700] Input: Prepared alert message

[0701] Specific operation: The device uses Firebase Cloud Messaging to notify the user of the received alert message as a pop-up or app notification.

[0702] Output: Alert message sent to the user

[0703] Step 11:

[0704] Initial settings (user processing)

[0705] Input: Setting information by accessing the administration screen

[0706] Specific operation: Using a web-based management interface, users can configure the locations of surveillance cameras and audio collection devices, check connection status, and configure notifications (e.g., email, SMS, app notifications).

[0707] Output: Management screen with saved settings

[0708] Step 12:

[0709] System confirmation (user processing)

[0710] Input:Administrator screen information

[0711] Specific operation: Users can check the system status from the management screen, monitor the operation status in real time, and manually test the anomaly detection function.

[0712] Output: Confirmation of system status, working properly

[0713] Step 13:

[0714] Handling abnormality notifications (user actions)

[0715] Input: The alert message that was sent to the user

[0716] Specific operations: The user checks the detailed information of the alert message (e.g., date and time, location, and content of the abnormality), and if necessary, rushes to the scene or notifies the relevant authorities.

[0717] Output: Notification information for which the action has been completed

[0718] (Application example 1)

[0719] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0720] While conventional surveillance systems analyze real-time data from surveillance cameras and audio collection devices and detect abnormalities, they lack the ability to quickly and effectively notify relevant parties of the abnormality. In particular, they lack the ability to provide optimal route information for rushing to the scene, and there is a need for a rapid response when an abnormality is notified. The present invention aims to solve these problems and provide a more efficient and responsive surveillance system.

[0721] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0722] In this invention, the server includes means for receiving data in real time from the surveillance camera and the audio collecting device, means for analyzing the received data and detecting a preset abnormal sound, means for generating an alert and notifying relevant parties when an abnormal sound is detected, means for displaying the abnormality notification as a pop-up message or an application notification using a smart device, and means for displaying the shortest route to the location of the abnormality. This not only enables rapid abnormality detection and notification, but also improves the efficiency of on-site response.

[0723] A "surveillance camera" is a device used to continuously monitor a specific area or location and collect video data.

[0724] An "audio collection device" is a device that collects ambient sounds and records and transmits them as analog or digital data.

[0725] "Real-time" refers to the ability to process and transmit data and information instantly, without delay.

[0726] The "means for receiving data" refers to a function or device for receiving data transmitted from the surveillance camera and the audio collecting device.

[0727] The "means for analyzing data" is a function or device that processes the received data and performs analysis to detect anomalies.

[0728] An "abnormal sound" is a sound that indicates a specific abnormal situation, unlike normal background sounds such as breaking glass or screams.

[0729] "Means for detecting abnormalities" refers to a function or device that identifies abnormal sounds or behaviors as a result of data analysis.

[0730] An "alert generating means" is a function or device that generates a warning message or notification when an abnormality is detected.

[0731] A "means for notifying interested parties" is a function or device that generates an alert and then sends it to a designated recipient.

[0732] A "smart device" is a mobile terminal with Internet connectivity that can run applications.

[0733] A "pop-up message" is a notification message that suddenly appears on the device screen.

[0734] "Application notification" is a notification function that notifies the user of information through a specific application.

[0735] The "means for displaying the shortest route" is a function or device for calculating and displaying the most efficient route from the current location to the location where the abnormality has occurred.

[0736] In order to implement the present invention, a system is constructed that receives data in real time from a surveillance camera and an audio collecting device, analyzes the data, and detects abnormalities. The system configuration and processing content are described in detail below.

[0737] System Overview

[0738] This system is mainly composed of three entities: a server, a terminal, and a user, which cooperate to monitor and detect anomalies.

[0739] Hardware and Software Environment

[0740] Hardware:

[0741] Surveillance cameras (IoT cameras)

[0742] Audio collection device (microphone)

[0743] Smart devices (smart glasses, smartphones)

[0744] Server (cloud platform)

[0745] software:

[0746] Server side: AWS Lambda, EC2, Amazon Rekognition, Amazon Transcribe

[0747] AI models: TensorFlow, PyTorch

[0748] Device applications: Android Studio, Xcode, Google Glass SDK

[0749] Server Processing

[0750] 1. Data Collection:

[0751] The server receives and temporarily stores data streams from surveillance cameras and audio capture devices, and transcodes the data for real-time processing.

[0752] 2. Data Analysis:

[0753] The server uses pre-trained AI models (TensorFlow or PyTorch) to process the received data frame by frame and perform video and audio analysis in parallel. Amazon Rekognition is used for video analysis, and Amazon Transcribe is used for audio analysis.

[0754] 3. Anomaly detection:

[0755] When the server detects an abnormal sound (such as the sound of breaking glass or a person screaming), it records the abnormality and identifies details (time, location, type).

[0756] 4. Alert generation and sending:

[0757] If an anomaly is detected, the server generates an alert message and sends it to the relevant parties. The alert message contains detailed information about the anomaly (time, location, type).

[0758] Terminal handling

[0759] 1. Data Entry:

[0760] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices, temporarily stores the data, and processes it for transfer to the server.

[0761] 2. Data Transfer:

[0762] The terminal transmits the received data to the server in real time, supporting rapid data analysis.

[0763] 3. Receive anomaly alerts:

[0764] The terminal receives the anomaly detection alert sent from the server and prepares to notify the user.

[0765] 4. User Notice:

[0766] The device will notify users of any abnormalities as a pop-up message or application notification, allowing them to take prompt action. It will also work with map apps to display the shortest route to the location of the abnormality.

[0767] User Roles

[0768] 1. Initial Setup:

[0769] Users can set the locations of surveillance cameras and audio collection devices, check the connection status, and configure notification settings (e.g., email, SMS, app notifications).

[0770] 2. System Check:

[0771] Users can check the system status in real time to monitor whether it is working properly, and can also manually test the anomaly detection function.

[0772] 3. Response to abnormality notification:

[0773] The user receives an abnormality notification from the device, checks the detailed information, and then quickly moves to the location of the abnormality and reports it to the relevant authorities (police, fire department, etc.) if necessary.

[0774] Example scenario

[0775] For example, imagine there are surveillance cameras and audio collection devices installed in a shopping district at night. If the sound of glass breaking occurs in a store late at night, the sound is picked up by the sound-collecting microphone and the surveillance camera captures the video. This data is sent to a server in real time, which analyzes the received data and detects the sound of glass breaking. After detection, the server generates an alert message and sends it to the security company. The device receives this alert and notifies the security guard as a pop-up message. Upon seeing the notification, the security guard immediately rushes to the scene and prevents the damage from spreading. This system enables early detection of crimes and accidents and rapid response, improving social safety.

[0776] Prompt Sentence Examples

[0777] scenario:

[0778] We use a system that receives data in real time from surveillance cameras and microphones installed in shopping districts at night and detects abnormalities. Please explain in detail how it handles the situation when the sound of glass being broken is detected.

[0779] Input data:

[0780] Camera video data

[0781] Microphone audio data

[0782] GPS location information

[0783] Process flow:

[0784] 1. Data collection and transmission

[0785] 2. Data analysis and anomaly detection using AI models

[0786] 3. Notification and response when an abnormality occurs

[0787] The above is a specific embodiment for carrying out the invention.

[0788] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0789] Step 1: Data collection

[0790] The server receives video and audio data in real time from surveillance cameras and audio collection devices. Specifically, the server acquires stream data from IoT cameras and microphones and temporarily stores it. This data is transcoded for later analysis. The input data is camera video data and microphone audio data, and the output is the transcoded data.

[0791] Step 2: Data analysis

[0792] The server processes the received data frame by frame using a pre-trained AI model (TensorFlow or PyTorch). It uses Amazon Rekognition to analyze the video and Amazon Transcribe to analyze the audio data. The input data is transcoded video and audio data, and the output after analysis is a detailed analysis result that includes any abnormal sounds or behavior.

[0793] Step 3: Anomaly detection

[0794] When the server detects an abnormal sound (such as the sound of breaking glass or a person screaming) or abnormal behavior (such as suspicious activity), it records the abnormality and identifies detailed information (time, location, type). During this process, if it recognizes that an abnormality has occurred based on the analysis results, it sets an anomaly detection flag. The input is the analysis result, and the output is detailed information about the abnormality.

[0795] Step 4: Alert Generation

[0796] The server generates an alert message when an anomaly is detected. The alert contains detailed information about the anomaly (time, location, type) and also determines which parties to notify. The input is the detailed information about the anomaly, and the output is the generated alert message.

[0797] Step 5: Sending an alert

[0798] The server generates an alert message and sends it to the appropriate parties via email, SMS, app notification, etc. The input is the generated alert message and the output is the sent alert.

[0799] Step 6: Data entry

[0800] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices. The received data is temporarily stored and then formatted before being transferred to the server. The input is data from the surveillance cameras and audio collection devices, and the output is formatted data.

[0801] Step 7: Data Transfer

[0802] The terminal transmits the received data to the server in real time, which supports fast data analysis on the server side. The input is the formatted data, and the output is the transmitted data.

[0803] Step 8: Receive anomaly alerts

[0804] The terminal receives an anomaly detection alert sent from the server. This alert contains detailed information about the anomaly and is prepared for notification to the user. The input is the alert message from the server, and the output is the preparation status for notification.

[0805] Step 9: User Notification

[0806] The device displays the abnormality notification as a pop-up message or application notification. It also works with a map application to display the shortest route to the location of the abnormality, allowing the user to respond quickly. The input is the prepared notification information, and the output is the displayed notification and route information.

[0807] Step 10: Initial Setup

[0808] The user sets the installation location of the surveillance camera and audio collection device, checks the connection status, and sets notification settings (e.g., email, SMS, app notifications). The input is the initial state of the device, and the output is the device state after the settings are complete.

[0809] Step 11: System Verification

[0810] Users can check the system status in real time to monitor whether it is operating normally. They can check the real-time status of the system and check whether there are any abnormalities. The input is the current system status and the output is the check result.

[0811] Step 12: Responding to abnormal notifications

[0812] The user receives an abnormality notification from the device and checks the detailed information. Then, they quickly move to the location of the abnormality and, if necessary, notify the relevant authorities (police, fire department, etc.). The input is the notification information from the device, and the output is the user's response action.

[0813] The above are the specific processing steps and their contents. This series of processes enables rapid and effective anomaly detection and response.

[0814] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0815] System Overview

[0816] The system based on this invention is a comprehensive monitoring system that receives data in real time from surveillance cameras and audio collection devices, detects abnormalities, and notifies relevant parties. This system, in particular, can improve the accuracy of anomaly detection and the speed of response by combining it with an emotion engine that recognizes user emotions. The system is mainly composed of three entities: a server, a terminal, and a user, which work together to perform effective monitoring and issue alerts.

[0817] In the embodiment of the present invention, the system is composed of the following three entities.

[0818] Server Processing

[0819] 1. Data Collection

[0820] The server receives and temporarily stores data streams from surveillance cameras and audio collection devices, and transcodes the data for real-time processing.

[0821] 2. Data Analysis

[0822] The server loads a pre-trained AI model, which includes algorithms for detecting abnormal sounds and movements and analyzing user emotions.

[0823] The received data is processed frame by frame, and video and audio analysis is performed in parallel, allowing for anomaly detection and emotion analysis to be performed simultaneously.

[0824] 3. Anomaly Detection

[0825] The server detects abnormal behavior and sounds (such as the sound of breaking glass or people screaming) from video and audio data.

[0826] The server uses an emotion engine to analyze the user's emotions (e.g., anger, anxiety, sadness, etc.) using voice tone and facial expression analysis.

[0827] 4. Alert generation and sending

[0828] If an anomaly or a specific emotion is detected, the server generates an alert message, which includes detailed information about the anomaly (time, location, type) and the result of emotion analysis.

[0829] Generate different levels of alerts depending on the type of anomaly and the degree of emotion.

[0830] The server generates an alert message and sends it to the relevant parties (security company, police, fire department, etc.).

[0831] Terminal handling

[0832] 1. Data Entry

[0833] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices.

[0834] The received data is temporarily stored and formatted for transfer to the server.

[0835] 2. Data Transfer

[0836] The terminal transmits the received data to the server in real time, supporting rapid data analysis on the server side.

[0837] 3. Receive anomaly alerts

[0838] The terminal receives an alert message for anomaly detection and emotion analysis sent from the server.

[0839] Verify the accuracy of received alerts and prepare to notify users.

[0840] 4. User Notices

[0841] The device displays alerts as pop-up messages or application notifications, allowing users to respond quickly. Different notification methods (screen display, audio alarm, vibration alarm, etc.) are provided depending on the level of the alert.

[0842] User Action

[0843] 1. Initial Setup

[0844] Users can set the installation locations of surveillance cameras and audio collection devices, check the connection status, and configure notification settings (e.g., email, SMS, app notifications).

[0845] 2. System Check

[0846] Users can check the system status in real time to monitor whether it is working properly, and can also manually test the anomaly detection and sentiment analysis functions.

[0847] 3. Response to abnormality notifications

[0848] The user receives an abnormality notification from the device, checks the content of the notification, checks detailed information about the abnormality and the results of emotion analysis, and, if necessary, rushes to the scene or reports the incident to the relevant authorities (police, fire department, etc.).

[0849] Example scenario

[0850] For example, suppose there are surveillance cameras and audio collection devices installed in a shopping district at night. If the sound of glass breaking occurs in a store late at night and a panicked user screams inside the store at the same time, the audio collection device picks up the sound and the surveillance camera captures the video. These data are sent to a server in real time, and the server analyzes the received data.

[0851] At the same time that the server detects the sound of breaking glass and a scream, the emotion engine recognizes the user's emotion as "anxiety." After detection, the server generates an alert message and sends it to the security company with the following content: "01:23 AM, Store A, sound of breaking glass detected, emotion of anxiety." The device receives this alert and notifies the security guard as a pop-up message. The user (security guard) who sees the notification immediately rushes to the scene and prevents the damage from spreading. This system enables early detection of crimes and accidents and rapid response, improving social safety.

[0852] According to the embodiment of the present invention, the efficiency and accuracy of monitoring work can be improved, and in particular, safety can be further enhanced by making it possible to take into account the emotions of users.

[0853] The processing flow will be explained below.

[0854] Detailed system programming

[0855] Server Processing

[0856] Step 1:

[0857] A server receives data streams from surveillance cameras and audio collection devices.

[0858] The received video and audio data is temporarily stored and transcoded.

[0859] Step 2:

[0860] The server loads a pre-trained AI model.

[0861] Prepare algorithms for anomaly detection and sentiment analysis.

[0862] Step 3:

[0863] The server processes the received data frame by frame.

[0864] Parallel processing is set up so that video data and audio data can be analyzed simultaneously.

[0865] Step 4:

[0866] The server detects abnormal behavior (e.g., violent acts or rapid movement) from the video data.

[0867] The audio data is filtered to detect abnormal sounds (such as glass breaking or screams).

[0868] Step 5:

[0869] If the server detects abnormal behavior or sound, it sends the frame from that point to the emotion analysis engine.

[0870] The sentiment analysis engine recognizes the user's emotions (e.g., anger, anxiety, sadness).

[0871] Step 6:

[0872] The server evaluates abnormal behavior, abnormal sounds, and emotion analysis results, and records details of the abnormality (time, location, type).

[0873] The reliability of the detected anomalies is confirmed by applying a secondary checking algorithm.

[0874] Step 7:

[0875] The server generates an alert message if an abnormality is detected.

[0876] The alert message includes detailed information about the anomaly (time, location, type) and the results of sentiment analysis.

[0877] Step 8:

[0878] The server generates an alert message and sends it to the relevant parties (security company, police, fire department, etc.).

[0879] Checks whether the message was sent successfully and attempts to send it again if it fails.

[0880] Terminal handling

[0881] Step 1:

[0882] The terminal receives video and audio data in real time from the connected surveillance cameras and audio collection devices.

[0883] The received data is temporarily stored and formatted for transfer to the server.

[0884] Step 2:

[0885] The terminal transmits the received data to the server in real time.

[0886] Check the data for completeness and attempt to resend any missing data.

[0887] Step 3:

[0888] The device receives anomaly detection and emotion analysis alert messages sent from the server.

[0889] Verify the accuracy of received alerts and prepare to notify users.

[0890] Step 4:

[0891] Your device will display the alert as a pop-up message or application notification.

[0892] Different notification methods (screen display, audio alarm, vibration alarm, etc.) are provided depending on the level of the alert.

[0893] User Action

[0894] Step 1:

[0895] The user sets the installation locations of the surveillance cameras and audio collection devices and checks the connection status.

[0896] The user configures system notification settings (e.g., email, SMS, app notifications).

[0897] Step 2:

[0898] Users can check the system status in real time and monitor whether it is operating normally.

[0899] It is also possible to manually test the anomaly detection and sentiment analysis features.

[0900] Step 3:

[0901] The user receives the abnormality notification from the terminal and checks the content of the notification.

[0902] Check the detailed information about the anomaly and the results of the emotion analysis, and if necessary, rush to the scene or notify the relevant authorities.

[0903] Example scenario

[0904] For example, suppose there are surveillance cameras and audio collection devices installed in a shopping district at night. If a store hears the sound of glass breaking late at night and a panicked user screams inside the store at the same time, the audio collection device picks up the sound and the surveillance camera captures the video. These data are sent to a server in real time.

[0905] The server analyzes the received data and detects the sound of glass breaking and a scream. At the same time, the emotion engine recognizes the user's emotion as "anxiety." After detection, the server generates an alert message and sends it to the security company with the following content: "01:23 AM, Store A, sound of glass breaking detected, emotion of anxiety."

[0906] The device receives this alert and notifies the security guard as a pop-up message. The user (security guard) who sees the notification can immediately rush to the scene to prevent the damage from spreading. This system enables early detection of crimes and accidents and a rapid response, improving social safety.

[0907] According to the embodiment of the present invention, the efficiency and accuracy of monitoring work can be improved, and in particular, safety can be further enhanced by making it possible to take into account the emotions of users.

[0908] Example 2

[0909] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0910] Modern surveillance systems require improved anomaly detection accuracy and rapid response, but conventional systems perform video and audio analysis separately, resulting in slow anomaly detection times and reduced accuracy. Furthermore, systems are unable to respond to user emotions, making it difficult to respond appropriately based on specific emotions. Furthermore, when an anomaly is detected, the alert content is limited and detailed information is lacking, making it difficult for relevant parties to respond quickly and appropriately.

[0911] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving data in real time from the monitoring camera and the audio collecting device, a data analysis means for processing the received data frame by frame and including an algorithm for performing video analysis and audio analysis in parallel, a means for detecting abnormal behavior or sound from the analyzed data, an emotion analysis means for recognizing the emotion of the user, and a means for generating an alert and notifying relevant parties when an abnormality or a specific emotion is detected. This improves the accuracy of anomaly detection and the speed of response, allowing relevant parties to respond quickly and appropriately based on detailed information.

[0912] A "surveillance camera" is a device that captures video data in real time and captures the situation at a monitored location.

[0913] An "audio collection device" is a device that collects surrounding audio data in real time and uses it for monitoring purposes.

[0914] An "algorithm that processes data frame by frame and performs video and audio analysis in parallel" is a computer program that divides video data into its constituent frames and performs analysis of the video and audio data simultaneously.

[0915] "Means for detecting abnormal behavior or sounds" refers to a system that has the ability to analyze collected data and identify pre-defined abnormal behavior or sound patterns.

[0916] "Emotion analysis means for recognizing user emotions" refers to an algorithm or model for determining a user's emotional state (e.g., anger, anxiety, sadness) from audio tone or video data.

[0917] "Means for generating alerts and notifying relevant parties" refers to a system that creates warning messages based on detected abnormalities and emotional information and sends them to relevant agencies (e.g., security companies, police, fire departments, etc.).

[0918] "Detailed information (time, location, type)" is specific information about the detected abnormal event, including the time of occurrence, the location of occurrence, and the type of abnormality.

[0919] "Means for checking the system status in real time" refers to an interface that allows a user to monitor the current operating status of the data collection device and analysis server in real time and check whether they are functioning normally.

[0920] The system based on this invention is a comprehensive monitoring system that receives data in real time from surveillance cameras and audio collection devices, detects abnormal events, and notifies relevant parties. In particular, by combining it with an emotion analysis engine that recognizes user emotions, the accuracy of abnormality detection and the speed of response can be improved. This system is composed of three entities: a server, a terminal, and a user.

[0921] Server Processing

[0922] The server first receives data in real time from surveillance cameras and audio collection devices. The received data is temporarily stored and transcoded for real-time processing. The server then loads a pre-trained AI model. This AI model includes algorithms for detecting abnormal sounds and behaviors, as well as algorithms for analyzing user emotions. The data is divided into frames, and video and audio analysis is performed in parallel. For example, frameworks such as TensorFlow and PyTorch are used. Abnormal behaviors and sounds are detected from the analyzed data, and an emotion analysis engine is used to identify the user's emotion (e.g., anger, anxiety, sadness, etc.). If an anomaly or emotion is detected, an alert message is generated and sent to relevant parties, along with detailed information about the anomaly (time, location, type) and the results of the emotion analysis.

[0923] Terminal handling

[0924] The device receives video and audio data from surveillance cameras and audio collection devices in real time. The received data is temporarily stored and formatted for transmission to the server. The formatted data is sent to the server in real time, supporting rapid data analysis on the server side. The device receives anomaly detection and emotion analysis alert messages sent from the server and checks their accuracy. The received alerts are displayed as pop-up messages or application notifications, allowing the user to respond quickly. Different notification methods (screen display, audio alarm, vibration alarm, etc.) are provided depending on the level of the alert.

[0925] User Action

[0926] First, the user sets the installation locations of the surveillance cameras and audio collection devices and checks the connection status. They also configure notification settings (e.g., email, SMS, app notifications). Users can check the system status in real time and monitor whether it is operating normally. They can also manually test the anomaly detection and emotion analysis functions. When a user receives an anomaly notification from their device, they can check the notification content and confirm the details of the anomaly and the results of the emotion analysis. If necessary, they can rush to the scene or notify the relevant authorities (police, fire department, etc.).

[0927] Example scenario

[0928] For example, consider a surveillance camera and audio collection device installed in a shopping district at night. Suppose a store hears the sound of glass breaking late at night, and simultaneously, a panicked user screams inside the store. The audio collection device picks up the sound, and the surveillance camera captures the video. These data are sent to a server in real time, which analyzes the received data. When the server detects the sound of glass breaking and the scream, the emotion engine recognizes the user's emotion as "anxiety." After detection, the server generates an alert message stating "1:23 AM, Store A, sound of glass breaking detected, emotion of anxiety," and sends it to the security company. The device receives this alert and notifies the security guard via a pop-up message. The user (security guard) immediately rushes to the scene after confirming the notification and prevents further damage. This system enables early detection and rapid response to crimes and accidents, improving social safety.

[0929] Example prompts for generative AI models

[0930] "Receive data in real time from surveillance cameras and audio collection devices, and generate an alert message if an anomaly is detected. For example, imagine a case where a glass breaks in a store at night, causing a panicked user to scream. Include relevant details (time, location, type) and the results of emotion analysis (anxiety)."

[0931] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0932] Step 1: Data collection

[0933] Terminal

[0934] 1. Receive video and audio data in real time from surveillance cameras and audio collection devices.

[0935] Input: Real-time data from surveillance cameras and audio collection devices.

[0936] Specific operation: The surveillance camera captures video data at 30 frames per second, and the audio collection device collects audio data at 44.1 kHz per second.

[0937] Output: Temporarily stored video and audio data.

[0938] Step 2: Data preparation and transfer

[0939] Terminal

[0940] 1. The received video and audio data is temporarily stored and formatted for transfer to the server.

[0941] Input: Temporarily stored video and audio data.

[0942] Specific operation: Frames of video data are stored in a buffer, and audio data is stored as chunks in certain ranges.

[0943] Output: The formatted data.

[0944] 2. Send the formatted data to the server in real time.

[0945] What it does: Streams data packets to a server using the TCP / IP protocol.

[0946] Step 3: Data analysis

[0947] server

[0948] 1. Split the data stream to analyze the received data frame by frame.

[0949] Input: Formatted data transferred to the server.

[0950] Specific operation: Video frames are extracted from the data stream, and the audio data is processed as is.

[0951] Output: Data divided into frames.

[0952] 2. Load the pre-trained AI model and prepare it for anomaly detection and sentiment analysis.

[0953] Specific operation: Load an anomaly detection model into memory using TensorFlow or PyTorch.

[0954] Step 4: Anomaly detection and sentiment analysis

[0955] server

[0956] 1. Video analysis: Analyze each frame of received video data to detect abnormal behavior.

[0957] Input: Video data divided into frames.

[0958] Specific operation: When the movement between frames changes drastically, an algorithm is applied that considers this to be abnormal behavior.

[0959] Output: Anomalous behavior detection results.

[0960] 2. Audio analysis: Spectral analysis of audio data is performed to detect abnormal sounds (such as the sound of glass breaking or a person screaming).

[0961] Input: Audio data divided into frames.

[0962] Specific behavior: Identify sudden changes in sound pressure level and specific frequency patterns.

[0963] Output: Abnormal sound detection result.

[0964] 3. Emotion analysis: Analyzes the tone of voice and facial expressions to identify the user's emotions.

[0965] Input: Received video and audio data.

[0966] Specific operation: Identify emotions by detecting changes in voice tone, pitch, and facial expressions in video.

[0967] Output: Emotion identification results.

[0968] Step 5: Generate and send an alert

[0969] server

[0970] 1. If an anomaly or a specific emotion is detected, an alert message is generated, containing detailed information about the anomaly (time, location, type) and the results of emotion analysis.

[0971] Input: Abnormal behavior, abnormal sound, and emotion analysis results.

[0972] Specific operations: Integrates and converts data, and generates alert messages.

[0973] Output: The alert message.

[0974] 2. Send the generated alert message to the relevant parties.

[0975] What it does: Deliver alerts via SMS, email, and app notifications.

[0976] Step 6: Receive alerts and notifications

[0977] Terminal

[0978] 1. Receive an alert message sent from the server.

[0979] Input: The alert message from the server.

[0980] Specific operation: Receive alert information via HTTPS encrypted communication.

[0981] Output: The received alert message.

[0982] 2. Review the accuracy of the alerts received and make arrangements to notify users.

[0983] Specific operation: Analyzes the message content and converts it into a display format.

[0984] 3. Display the alert to the user as a popup message or app notification.

[0985] Specific behavior: Use different notification methods such as popup, audio alarm, vibration, etc.

[0986] Step 7: Responding to abnormality notifications

[0987] User

[0988] 1. Receive an abnormality notification from the device and check the content of the notification.

[0989] Input: Alert notification from the terminal.

[0990] Specific operation: Display and confirm detailed information about anomalies and the results of sentiment analysis.

[0991] 2. If necessary, rush to the scene and notify the relevant authorities.

[0992] Specific actions: Decide on a response promptly based on the information confirmed.

[0993] (Application example 2)

[0994] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0995] Conventional monitoring systems monitor a huge amount of data and perform analysis to detect abnormalities, but are unable to analyze the user's emotional state when an abnormality occurs, making it difficult to respond quickly and appropriately.The present invention aims to enable faster and more appropriate responses by analyzing user emotional data in real time in addition to monitoring data and generating alerts including the results of emotional analysis when an abnormality occurs.

[0996] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0997] In this invention, the server includes means for receiving data in real time from the surveillance camera and the audio collecting device, means for analyzing the received data and detecting a predetermined abnormal sound, means for generating and reporting an alert when an abnormal sound is detected, means for analyzing the user's emotions, and means for generating and reporting an alert message including the emotion analysis result. This makes it possible to analyze the user's emotional state at the same time as detecting an abnormality and generate a detailed alert including emotional information.

[0998] A "surveillance camera" is a device that captures images of a location in real time and records or transmits the data.

[0999] An "audio collection device" is a device for collecting audio data in an environment in real time and recording or transmitting that data.

[1000] "Means for receiving data in real time" refers to technical means for receiving data almost simultaneously with its generation.

[1001] "Means for analyzing received data" refers to technical means for extracting information or detecting anomalies from data received in real time using specific algorithms.

[1002] "Means for detecting predetermined abnormal sounds" refers to technical means for detecting specific abnormal sounds (e.g., the sound of breaking glass, screams, etc.) that are defined in advance.

[1003] "Means for generating and reporting an alert when an abnormal sound is detected" refers to the technical means for generating a warning message or notification when an abnormal sound is detected and for promptly reporting it to designated parties.

[1004] "Means for analyzing user emotions" refers to technical means for analyzing the user's emotional state (e.g., anger, anxiety, sadness, etc.) from video and audio data.

[1005] "Means for generating and notifying an alert message including the result of emotion analysis" refers to a technical means for generating a detailed alert message including the result of the user's emotion analysis and promptly notifying relevant parties.

[1006] System configuration

[1007] The system embodying the present invention consists of three main components: a server, a terminal, and a user. The server receives data from the surveillance cameras and audio collection devices, analyzes the data, and detects abnormalities. The terminal transmits the collected data to the server and notifies the user of alert messages from the server. The user performs the initial system setup and responds appropriately when notified of an abnormality.

[1008] Hardware and software used

[1009] The server requires hardware with high-performance data analysis capabilities, and specifically, a server equipped with an NVIDIA GPU is suitable. Machine learning frameworks such as TensorFlow and PyTorch are used for data analysis. The terminals used are devices that users can carry, such as smartphones, tablets, and smart glasses. These devices are equipped with high-resolution cameras and highly sensitive microphones, making it possible to collect data in real time.

[1010] Specific examples

[1011] For example, if a nighttime surveillance system is installed in a shopping mall, surveillance cameras and audio collection devices are installed in each store. These devices collect video and audio data in real time and send it to a server via a terminal. The server analyzes the received data and detects abnormal sounds and behavior.

[1012] If an abnormality is detected, the server analyzes the user's emotions and generates an alert message containing the results of the emotion analysis. For example, a message may be generated stating, "The sound of glass breaking was detected at Store A at 2:00 AM, and an emotion of anxiety was recognized." This alert message is sent to the security guard's smart glasses in real time. The security guard can then check the notification and quickly rush to the scene.

[1013] Generative AI model and prompts

[1014] The system uses a generative AI model to analyze user sentiment. Specific examples of prompts include:

[1015] Example prompt sentence:

[1016] "Detect the sound of glass breaking or a person screaming, and analyze the user's emotions. For example, recognize emotions such as anxiety or anger. If an anomaly is detected, generate an alert containing the anomaly and the results of the emotion analysis, and notify the user."

[1017] This enables the monitoring system to detect abnormalities early and respond quickly and appropriately while taking into account the emotional state of the user. This invention is particularly effective in strengthening security in commercial facilities, public facilities, important facilities, etc.

[1018] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1019] Step 1:

[1020] Data collection (terminal)

[1021] The device (e.g., smartphone, smart glasses) collects data (video and audio) in real time from surveillance cameras and audio collectors. The cameras generate high-resolution video data, and the audio collectors collect audio data with high sensitivity. This data is temporarily stored on the device. The input includes video data from the surveillance cameras and audio data from the audio collectors, and the output is the temporary storage of these data.

[1022] Step 2:

[1023] Data transfer (terminal)

[1024] The device processes the temporarily stored video and audio data (for example, transcoding or compressing) and then transmits it to the server in real time. Here, the video data is encoded into JPEG or H.264 format, and the audio data is encoded into MP3 or AAC format. The temporarily stored raw data is the input, and the transcoded data is sent to the server as the output.

[1025] Step 3:

[1026] Data analysis (server)

[1027] The server receives the video and audio data sent from the device, loads a pre-trained generative AI model, and performs data analysis. The analysis includes detecting abnormal behavior from the video, detecting abnormal sounds from the audio, and analyzing the user's emotions. Specifically, it detects movement and shape changes from the video, and specific frequencies and patterns from the audio. The input is the received transcoded data, and the output is an analysis of the type of abnormality and the emotional state.

[1028] Step 4:

[1029] Anomaly detection and sentiment analysis (server)

[1030] If the server detects an anomaly from the data analysis results, it further analyzes the user's emotions (anger, anxiety, sadness, etc.) using a generative AI model. Specifically, emotions are recognized based on voice tone and facial expression analysis. The input is the preliminary results of the data analysis, and the output is the presence or absence of an anomaly and the user's emotional state.

[1031] Step 5:

[1032] Alert generation and transmission (server)

[1033] The server generates an alert message based on the anomaly and the results of sentiment analysis. The alert message contains detailed information such as the type of anomaly, time, location, and sentiment analysis results. The generated alert message is promptly sent to relevant parties (e.g., security guards, police, fire department, etc.). The inputs are the anomaly and the sentiment analysis results, and the generated alert message is the output.

[1034] Step 6:

[1035] User notification (terminal)

[1036] The terminal receives the alert message sent from the server and notifies the user (e.g., security guard). Notification methods include screen display, audio alarm, and vibration alarm. The alert message is the input, and the notification to the user is the output.

[1037] Step 7:

[1038] Response (user)

[1039] The user receives an alert notification from the device and checks the content of the notification. For example, if the alert content is "The sound of glass breaking was detected at Store A at 2:00 AM, and feelings of anxiety were recognized," the user will immediately rush to the scene and take appropriate action. The input is the alert notification, and the output is the user's response action.

[1040] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1041] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1042] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1043] [Third embodiment]

[1044] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1045] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1046] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1047] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1048] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1049] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1050] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1051] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1052] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1053] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1054] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1055] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1056] System Overview

[1057] The system based on this invention is a comprehensive monitoring system that receives data in real time from surveillance cameras and audio collection devices, detects abnormalities, and notifies relevant parties. The system is mainly composed of three components: a server, a terminal, and a user, which work together to perform effective monitoring and issue alerts.

[1058] In the embodiment of the present invention, the system is composed of the following three entities.

[1059] Server Processing

[1060] 1. Data Collection

[1061] The server receives and temporarily stores data streams from surveillance cameras and audio collection devices, and transcodes the data for real-time processing.

[1062] 2. Data Analysis

[1063] The server uses a pre-trained AI model to process the received data frame by frame, performing video and audio analysis in parallel, thereby achieving highly accurate anomaly detection.

[1064] 3. Anomaly Detection

[1065] When the server detects an abnormal sound (for example, the sound of breaking glass or a person screaming), it records the abnormality and identifies details (time, location, type).

[1066] 4. Alert generation and sending

[1067] If an anomaly is detected, the server generates an alert message and sends it to the relevant parties. The alert message contains detailed information about the anomaly.

[1068] Terminal handling

[1069] 1. Data Entry

[1070] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices, temporarily stores the data, and processes it for transfer to the server.

[1071] 2. Data Transfer

[1072] By sending the data received by the terminal to the server in real time, it supports rapid data analysis on the server side.

[1073] 3. Receive anomaly alerts

[1074] The terminal receives the anomaly detection alert sent from the server and prepares to notify the user.

[1075] 4. User Notices

[1076] The device will display the alert as a pop-up message or application notification, allowing the user to respond quickly.

[1077] User Action

[1078] 1. Initial Setup

[1079] Users can set the installation locations of surveillance cameras and audio collection devices, check the connection status, and configure notification settings (e.g., email, SMS, app notifications).

[1080] 2. System Check

[1081] Users can check the system status in real time and monitor whether it is working properly. Users can also manually test the anomaly detection function.

[1082] 3. Response to abnormality notifications

[1083] The user receives an abnormality notification from the device, checks the detailed information, and then rushes to the scene or reports it to the relevant authorities (police, fire department, etc.) as necessary.

[1084] Example scenario

[1085] For example, imagine there are surveillance cameras and audio collection devices installed in a shopping district at night. If the sound of glass breaking occurs in a store late at night, the sound is picked up by the sound-collecting microphone and the surveillance camera captures the video. This data is sent to a server in real time. The server analyzes the received data and detects the sound of glass breaking. After detection, the server generates an alert message and sends it to the security company. The device receives this alert and notifies the user (security guard) as a pop-up message. Upon seeing the notification, the user can immediately rush to the scene and prevent the damage from spreading. This system enables early detection of crimes and accidents and rapid response, improving social safety.

[1086] The embodiments of the present invention improve the efficiency and accuracy of monitoring operations, and make a significant contribution to preventing crimes and accidents.

[1087] The processing flow will be explained below.

[1088] System Program Processing

[1089] Server Processing

[1090] Step 1:

[1091] A server receives data streams from surveillance cameras and audio collection devices.

[1092] The received video and audio data is temporarily stored and transcoded.

[1093] Step 2:

[1094] The server loads a pre-trained AI model.

[1095] The received data is processed frame by frame, and video and audio analysis is performed in parallel.

[1096] Step 3:

[1097] The server detects abnormal movements in the video data and filters out abnormal sounds (such as breaking glass or people screaming) from the audio data.

[1098] Apply anomaly detection algorithms to check if anomalies have occurred.

[1099] Step 4:

[1100] When the server detects an abnormal sound or behavior, it records the details (time, location, type).

[1101] Apply a secondary check algorithm to ensure the anomaly detection is correct.

[1102] Step 5:

[1103] The server generates an alert message if an abnormality is detected.

[1104] Include detailed information about the anomaly (time, location, type) in the alert message.

[1105] Step 6:

[1106] The server generates an alert message and sends it to the relevant parties (security company, police, fire department, etc.).

[1107] Checks whether the message was sent successfully and attempts to send it again if it fails.

[1108] Terminal handling

[1109] Step 1:

[1110] The terminal receives video and audio data in real time from the connected surveillance cameras and audio collection devices.

[1111] The received data is temporarily stored and formatted for transfer to the server.

[1112] Step 2:

[1113] The terminal transmits the received data to the server in real time.

[1114] Check the data for completeness and attempt to resend any missing data.

[1115] Step 3:

[1116] The terminal receives an anomaly detection alert sent from the server.

[1117] Check the accuracy of the received alerts and send feedback to the server if necessary.

[1118] Step 4:

[1119] The device displays the alert to the user as a pop-up message or application notification.

[1120] The alarm can be set to a screen display, audio alarm, vibration alarm, etc.

[1121] User Action

[1122] Step 1:

[1123] The user sets the installation locations of the surveillance cameras and audio collection devices and checks the connection status.

[1124] The user configures system notification settings (e.g., email, SMS, app notifications).

[1125] Step 2:

[1126] Users can check the status of the system and monitor whether it is working properly.

[1127] It is also possible to manually test the anomaly detection feature.

[1128] Step 3:

[1129] The user receives the abnormality notification from the terminal and checks the content of the notification.

[1130] Check detailed information about the abnormality (time, location, type) and, if necessary, rush to the scene or report it to the relevant authorities.

[1131] Example 1

[1132] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1133] In recent years, there has been an increasing demand for more accurate and efficient monitoring systems. However, existing monitoring systems have problems such as low anomaly detection accuracy and insufficient real-time data processing. Furthermore, they are often unable to quickly notify relevant parties of detected anomalies, making it difficult to respond quickly. To address these issues, there is a demand for monitoring systems that can detect anomalies more accurately in real time and quickly notify relevant parties.

[1134] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1135] In this invention, the server includes means for receiving data in real time from the surveillance cameras and audio collecting devices, means for temporarily storing the received data and transcoding it into an optimal format, means for processing the received data frame by frame using a pre-trained artificial intelligence model to perform video and audio analysis in parallel, means for detecting abnormal sounds and images, recording the abnormality and identifying details (time, location, type), and means for generating an alert message and notifying relevant parties when an abnormality is detected. This enables real-time abnormality detection and highly accurate data analysis, realizing early detection of abnormalities and rapid response.

[1136] A "surveillance camera" is a device that acquires video data in real time and is used for monitoring and anomaly detection.

[1137] An "audio collection device" is a device that acquires audio data in real time and is used for monitoring and anomaly detection.

[1138] A "data stream" is a continuous flow of data transmitted in real time from a surveillance camera or audio collection device.

[1139] "Transcoding" is the process of converting received data into a different format.

[1140] An "artificial intelligence model (AI model)" is an algorithm that has been trained in advance using large amounts of data, and is used to detect and analyze anomalies.

[1141] "Frame-based processing" is a method in which video data and audio data are divided into fixed time intervals (frames) and analyzed and processed for each unit.

[1142] "Parallel processing" is a technology that improves processing speed by executing multiple processes or threads simultaneously.

[1143] An "abnormal sound" is a specific sound (such as the sound of breaking glass or screaming) that does not occur in a normal monitoring environment and indicates an abnormal situation.

[1144] "Abnormal footage" refers to specific footage (for example, suspicious activity) that is not seen in a normal surveillance environment and indicates an abnormal situation.

[1145] An "alert message" is a notification message that is generated when an abnormality is detected, and provides detailed information about the abnormality to relevant parties.

[1146] "Relevant parties" are people who manage and operate the monitoring system and people who are responsible for responding to abnormalities (e.g., security guards and administrators).

[1147] The monitoring system based on this invention is mainly composed of three entities: a server, a terminal, and a user. It is a comprehensive monitoring system that receives data in real time from monitoring cameras and audio collection devices, detects abnormalities, and notifies relevant parties. The role of each entity will be specifically explained below.

[1148] Server Processing

[1149] The server receives real-time data streams from surveillance cameras (e.g., general network cameras) and audio collection devices (e.g., standard microphones). The data is temporarily stored and transcoded into the optimal format for real-time processing. For example, video data is converted to H.264 format, and audio data is converted to WAV format. The transcoding process uses high-speed processing using NVIDIA CUDA.

[1150] The server then analyzes the data using pre-trained AI models. It uses TensorFlow to load an anomaly detection model, a YOLO model for video analysis, and a CNN model for audio analysis. Data is processed frame by frame and analyzed in parallel. Video data is analyzed using OpenCV, and audio data undergoes spectrogram analysis using Librosa.

[1151] When an abnormal sound or video is detected, the server records the anomaly and identifies the time, location, and type of the anomaly. Detailed information about the anomaly is recorded in a MySQL database. If an anomaly is detected, an alert message is generated and sent to relevant parties using Amazon SNS (Simple Notification Service).

[1152] Terminal handling

[1153] The terminal uses a Raspberry Pi or a standard edge device to receive data in real time from surveillance cameras and audio collection devices. The received data is temporarily stored in a buffer and processed. The terminal then transfers the received data to the server in real time using the MQTT protocol.

[1154] When an anomaly detection alert is received from the server, the device prepares a notification to the user. Firebase Cloud Messaging is used to notify the user of the alert message as a pop-up display or app notification, allowing the user to respond to the anomaly quickly.

[1155] User Action

[1156] Users can use a web-based management screen to set the installation locations of surveillance cameras and audio collection devices and check the connection status. They can also set notifications (e.g., email, SMS, app notifications). Users can check the system status from the management screen and monitor operation in real time. If an abnormality is detected, users will receive an alert message and check detailed information (e.g., date, time, location, and nature of the abnormality). They can then rush to the scene or notify the relevant authorities as necessary.

[1157] Example scenario

[1158] For example, consider the case where a surveillance camera and audio collection device installed in a shopping district detect the sound of glass breaking in a store late at night. The received data is sent to a server, where an AI model is used to detect the sound of glass breaking. The detected abnormality is recorded and an alert message is sent to the security company using Amazon SNS. The device receives this alert and notifies the security guard via a pop-up message. The security guard can then immediately rush to the scene and prevent further damage.

[1159] Example prompts to input to the generative AI model

[1160] "I'm in charge of the surveillance system for a shopping mall. Can you explain the outline of the AI ​​model that detects suspicious sounds and images at night and immediately notifies security guards?"

[1161] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1162] Step 1:

[1163] Receiving and temporarily storing data (server processing)

[1164] Input: Real-time data streams from surveillance cameras and audio collection devices

[1165] Specific operation: The server receives data streams from surveillance cameras and audio collection devices using RTSP (Real-Time Streaming Protocol). The received data is temporarily saved and stored in a buffer for subsequent processing.

[1166] Output: Temporarily saved data (video data and audio data)

[1167] Step 2:

[1168] Data transcoding (server processing)

[1169] Input: Temporarily saved data (video data and audio data)

[1170] Specific operation: The server transcodes the received video data into H.264 format and the audio data into WAV format. The transcoding process is performed at high speed using NVIDIA CUDA.

[1171] Output: Transcoded data (video data in H.264 format, audio data in WAV format)

[1172] Step 3:

[1173] Preparing for data analysis (server processing)

[1174] Input: Transcoded data (video data in H.264 format, audio data in WAV format)

[1175] How it works: The server loads pre-trained AI models using TensorFlow. The YOLO model is used for video analysis, and the CNN model is used for audio analysis.

[1176] Output: A ready-to-use AI model

[1177] Step 4:

[1178] Frame-by-frame data analysis (server processing)

[1179] Input: AI model ready, transcoded data

[1180] Specific operation: The server divides the video and audio data into frames and processes them in parallel. The video data is analyzed using OpenCV, and the audio data undergoes spectrogram analysis using Librosa.

[1181] Output: Parsed frame data

[1182] Step 5:

[1183] Anomaly detection (server processing)

[1184] Input: Parsed frame data

[1185] How it works: The server uses AI models to detect abnormal sounds and images. For example, if an abnormality such as the sound of broken glass, screams, or a suspicious person is detected, the server records the abnormality and identifies the time, location, and type of occurrence. Detailed information about the abnormality is then recorded in a MySQL database.

[1186] Output: Anomaly detection results and detailed information (time, location, type)

[1187] Step 6:

[1188] Alert generation and notification (server processing)

[1189] Input: Anomaly detection results and detailed information

[1190] Specific operation: If an anomaly is detected, the server generates an alert message. The message contains detailed information about the anomaly (e.g., date, time, location, and content of the anomaly). The alert message is sent to relevant parties using Amazon SNS.

[1191] Output: Alert message sent

[1192] Step 7:

[1193] Data entry (terminal processing)

[1194] Input: Data acquired from surveillance cameras and audio collection devices

[1195] How it works: The device uses a Raspberry Pi or a standard edge device to receive and temporarily store data from surveillance cameras and audio capture devices in real time, check for missing or abnormal data, and add a timestamp.

[1196] Output: Formatted data

[1197] Step 8:

[1198] Data transfer (terminal processing)

[1199] Input: Formatted data

[1200] Specific operation: The terminal uses the MQTT protocol to send the received data to the server in real time.

[1201] Output: Data transferred to the server

[1202] Step 9:

[1203] Receive an abnormality alert (terminal processing)

[1204] Input: The alert message sent from the server

[1205] Specific operation: The terminal receives the alert message sent from the server and prepares to notify the user.

[1206] Output: Prepared alert message

[1207] Step 10:

[1208] User notification (terminal processing)

[1209] Input: Prepared alert message

[1210] Specific operation: The device uses Firebase Cloud Messaging to notify the user of the received alert message as a pop-up or app notification.

[1211] Output: Alert message sent to the user

[1212] Step 11:

[1213] Initial settings (user processing)

[1214] Input: Setting information by accessing the administration screen

[1215] Specific operation: Using a web-based management interface, users can configure the locations of surveillance cameras and audio collection devices, check connection status, and configure notifications (e.g., email, SMS, app notifications).

[1216] Output: Management screen with saved settings

[1217] Step 12:

[1218] System confirmation (user processing)

[1219] Input:Administrator screen information

[1220] Specific operation: Users can check the system status from the management screen, monitor the operation status in real time, and manually test the anomaly detection function.

[1221] Output: Confirmation of system status, working properly

[1222] Step 13:

[1223] Handling abnormality notifications (user actions)

[1224] Input: The alert message that was sent to the user

[1225] Specific operations: The user checks the detailed information of the alert message (e.g., date and time, location, and content of the abnormality), and if necessary, rushes to the scene or notifies the relevant authorities.

[1226] Output: Notification information for which the action has been completed

[1227] (Application example 1)

[1228] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1229] While conventional surveillance systems analyze real-time data from surveillance cameras and audio collection devices and detect abnormalities, they lack the ability to quickly and effectively notify relevant parties of the abnormality. In particular, they lack the ability to provide optimal route information for rushing to the scene, and there is a need for a rapid response when an abnormality is notified. The present invention aims to solve these problems and provide a more efficient and responsive surveillance system.

[1230] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1231] In this invention, the server includes means for receiving data in real time from the surveillance camera and the audio collecting device, means for analyzing the received data and detecting a preset abnormal sound, means for generating an alert and notifying relevant parties when an abnormal sound is detected, means for displaying the abnormality notification as a pop-up message or an application notification using a smart device, and means for displaying the shortest route to the location of the abnormality. This not only enables rapid abnormality detection and notification, but also improves the efficiency of on-site response.

[1232] A "surveillance camera" is a device used to continuously monitor a specific area or location and collect video data.

[1233] An "audio collection device" is a device that collects ambient sounds and records and transmits them as analog or digital data.

[1234] "Real-time" refers to the ability to process and transmit data and information instantly, without delay.

[1235] The "means for receiving data" refers to a function or device for receiving data transmitted from the surveillance camera and the audio collecting device.

[1236] The "means for analyzing data" is a function or device that processes the received data and performs analysis to detect anomalies.

[1237] An "abnormal sound" is a sound that indicates a specific abnormal situation, unlike normal background sounds such as breaking glass or screams.

[1238] "Means for detecting abnormalities" refers to a function or device that identifies abnormal sounds or behaviors as a result of data analysis.

[1239] An "alert generating means" is a function or device that generates a warning message or notification when an abnormality is detected.

[1240] A "means for notifying interested parties" is a function or device that generates an alert and then sends it to a designated recipient.

[1241] A "smart device" is a mobile terminal with Internet connectivity that can run applications.

[1242] A "pop-up message" is a notification message that suddenly appears on the device screen.

[1243] "Application notification" is a notification function that notifies the user of information through a specific application.

[1244] The "means for displaying the shortest route" is a function or device for calculating and displaying the most efficient route from the current location to the location where the abnormality has occurred.

[1245] In order to implement the present invention, a system is constructed that receives data in real time from a surveillance camera and an audio collecting device, analyzes the data, and detects abnormalities. The system configuration and processing content are described in detail below.

[1246] System Overview

[1247] This system is mainly composed of three entities: a server, a terminal, and a user, which cooperate to monitor and detect anomalies.

[1248] Hardware and Software Environment

[1249] Hardware:

[1250] Surveillance cameras (IoT cameras)

[1251] Audio collection device (microphone)

[1252] Smart devices (smart glasses, smartphones)

[1253] Server (cloud platform)

[1254] software:

[1255] Server side: AWS Lambda, EC2, Amazon Rekognition, Amazon Transcribe

[1256] AI models: TensorFlow, PyTorch

[1257] Device applications: Android Studio, Xcode, Google Glass SDK

[1258] Server Processing

[1259] 1. Data Collection:

[1260] The server receives and temporarily stores data streams from surveillance cameras and audio capture devices, and transcodes the data for real-time processing.

[1261] 2. Data Analysis:

[1262] The server uses pre-trained AI models (TensorFlow or PyTorch) to process the received data frame by frame and perform video and audio analysis in parallel. Amazon Rekognition is used for video analysis, and Amazon Transcribe is used for audio analysis.

[1263] 3. Anomaly detection:

[1264] When the server detects an abnormal sound (such as the sound of breaking glass or a person screaming), it records the abnormality and identifies details (time, location, type).

[1265] 4. Alert generation and sending:

[1266] If an anomaly is detected, the server generates an alert message and sends it to the relevant parties. The alert message contains detailed information about the anomaly (time, location, type).

[1267] Terminal handling

[1268] 1. Data Entry:

[1269] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices, temporarily stores the data, and processes it for transfer to the server.

[1270] 2. Data Transfer:

[1271] The terminal transmits the received data to the server in real time, supporting rapid data analysis.

[1272] 3. Receive anomaly alerts:

[1273] The terminal receives the anomaly detection alert sent from the server and prepares to notify the user.

[1274] 4. User Notice:

[1275] The device will notify users of any abnormalities as a pop-up message or application notification, allowing them to take prompt action. It will also work with map apps to display the shortest route to the location of the abnormality.

[1276] User Roles

[1277] 1. Initial Setup:

[1278] Users can set the locations of surveillance cameras and audio collection devices, check the connection status, and configure notification settings (e.g., email, SMS, app notifications).

[1279] 2. System Check:

[1280] Users can check the system status in real time to monitor whether it is working properly, and can also manually test the anomaly detection function.

[1281] 3. Response to abnormality notification:

[1282] The user receives an abnormality notification from the device, checks the detailed information, and then quickly moves to the location of the abnormality and reports it to the relevant authorities (police, fire department, etc.) if necessary.

[1283] Example scenario

[1284] For example, imagine there are surveillance cameras and audio collection devices installed in a shopping district at night. If the sound of glass breaking occurs in a store late at night, the sound is picked up by the sound-collecting microphone and the surveillance camera captures the video. This data is sent to a server in real time, which analyzes the received data and detects the sound of glass breaking. After detection, the server generates an alert message and sends it to the security company. The device receives this alert and notifies the security guard as a pop-up message. Upon seeing the notification, the security guard immediately rushes to the scene and prevents the damage from spreading. This system enables early detection of crimes and accidents and rapid response, improving social safety.

[1285] Prompt Sentence Examples

[1286] scenario:

[1287] We use a system that receives data in real time from surveillance cameras and microphones installed in shopping districts at night and detects abnormalities. Please explain in detail how it handles the situation when the sound of glass being broken is detected.

[1288] Input data:

[1289] Camera video data

[1290] Microphone audio data

[1291] GPS location information

[1292] Process flow:

[1293] 1. Data collection and transmission

[1294] 2. Data analysis and anomaly detection using AI models

[1295] 3. Notification and response when an abnormality occurs

[1296] The above is a specific embodiment for carrying out the invention.

[1297] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1298] Step 1: Data collection

[1299] The server receives video and audio data in real time from surveillance cameras and audio collection devices. Specifically, the server acquires stream data from IoT cameras and microphones and temporarily stores it. This data is transcoded for later analysis. The input data is camera video data and microphone audio data, and the output is the transcoded data.

[1300] Step 2: Data analysis

[1301] The server processes the received data frame by frame using a pre-trained AI model (TensorFlow or PyTorch). It uses Amazon Rekognition to analyze the video and Amazon Transcribe to analyze the audio data. The input data is transcoded video and audio data, and the output after analysis is a detailed analysis result that includes any abnormal sounds or behavior.

[1302] Step 3: Anomaly detection

[1303] When the server detects an abnormal sound (such as the sound of breaking glass or a person screaming) or abnormal behavior (such as suspicious activity), it records the abnormality and identifies detailed information (time, location, type). During this process, if it recognizes that an abnormality has occurred based on the analysis results, it sets an anomaly detection flag. The input is the analysis result, and the output is detailed information about the abnormality.

[1304] Step 4: Alert Generation

[1305] The server generates an alert message when an anomaly is detected. The alert contains detailed information about the anomaly (time, location, type) and also determines which parties to notify. The input is the detailed information about the anomaly, and the output is the generated alert message.

[1306] Step 5: Sending an alert

[1307] The server generates an alert message and sends it to the appropriate parties via email, SMS, app notification, etc. The input is the generated alert message and the output is the sent alert.

[1308] Step 6: Data entry

[1309] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices. The received data is temporarily stored and then formatted before being transferred to the server. The input is data from the surveillance cameras and audio collection devices, and the output is formatted data.

[1310] Step 7: Data Transfer

[1311] The terminal transmits the received data to the server in real time, which supports fast data analysis on the server side. The input is the formatted data, and the output is the transmitted data.

[1312] Step 8: Receive anomaly alerts

[1313] The terminal receives an anomaly detection alert sent from the server. This alert contains detailed information about the anomaly and is prepared for notification to the user. The input is the alert message from the server, and the output is the preparation status for notification.

[1314] Step 9: User Notification

[1315] The device displays the abnormality notification as a pop-up message or application notification. It also works with a map application to display the shortest route to the location of the abnormality, allowing the user to respond quickly. The input is the prepared notification information, and the output is the displayed notification and route information.

[1316] Step 10: Initial Setup

[1317] The user sets the installation location of the surveillance camera and audio collection device, checks the connection status, and sets notification settings (e.g., email, SMS, app notifications). The input is the initial state of the device, and the output is the device state after the settings are complete.

[1318] Step 11: System Verification

[1319] Users can check the system status in real time to monitor whether it is operating normally. They can check the real-time status of the system and check whether there are any abnormalities. The input is the current system status and the output is the check result.

[1320] Step 12: Responding to abnormal notifications

[1321] The user receives an abnormality notification from the device and checks the detailed information. Then, they quickly move to the location of the abnormality and, if necessary, notify the relevant authorities (police, fire department, etc.). The input is the notification information from the device, and the output is the user's response action.

[1322] The above are the specific processing steps and their contents. This series of processes enables rapid and effective anomaly detection and response.

[1323] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1324] System Overview

[1325] The system based on this invention is a comprehensive monitoring system that receives data in real time from surveillance cameras and audio collection devices, detects abnormalities, and notifies relevant parties. This system, in particular, can improve the accuracy of anomaly detection and the speed of response by combining it with an emotion engine that recognizes user emotions. The system is mainly composed of three entities: a server, a terminal, and a user, which work together to perform effective monitoring and issue alerts.

[1326] In the embodiment of the present invention, the system is composed of the following three entities.

[1327] Server Processing

[1328] 1. Data Collection

[1329] The server receives and temporarily stores data streams from surveillance cameras and audio collection devices, and transcodes the data for real-time processing.

[1330] 2. Data Analysis

[1331] The server loads a pre-trained AI model, which includes algorithms for detecting abnormal sounds and movements and analyzing user emotions.

[1332] The received data is processed frame by frame, and video and audio analysis is performed in parallel, allowing for anomaly detection and emotion analysis to be performed simultaneously.

[1333] 3. Anomaly Detection

[1334] The server detects abnormal behavior and sounds (such as the sound of breaking glass or people screaming) from video and audio data.

[1335] The server uses an emotion engine to analyze the user's emotions (e.g., anger, anxiety, sadness, etc.) using voice tone and facial expression analysis.

[1336] 4. Alert generation and sending

[1337] If an anomaly or a specific emotion is detected, the server generates an alert message, which includes detailed information about the anomaly (time, location, type) and the result of emotion analysis.

[1338] Generate different levels of alerts depending on the type of anomaly and the degree of emotion.

[1339] The server generates an alert message and sends it to the relevant parties (security company, police, fire department, etc.).

[1340] Terminal handling

[1341] 1. Data Entry

[1342] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices.

[1343] The received data is temporarily stored and formatted for transfer to the server.

[1344] 2. Data Transfer

[1345] The terminal transmits the received data to the server in real time, supporting rapid data analysis on the server side.

[1346] 3. Receive anomaly alerts

[1347] The terminal receives an alert message for anomaly detection and emotion analysis sent from the server.

[1348] Verify the accuracy of received alerts and prepare to notify users.

[1349] 4. User Notices

[1350] The device displays alerts as pop-up messages or application notifications, allowing users to respond quickly. Different notification methods (screen display, audio alarm, vibration alarm, etc.) are provided depending on the level of the alert.

[1351] User Action

[1352] 1. Initial Setup

[1353] Users can set the installation locations of surveillance cameras and audio collection devices, check the connection status, and configure notification settings (e.g., email, SMS, app notifications).

[1354] 2. System Check

[1355] Users can check the system status in real time to monitor whether it is working properly, and can also manually test the anomaly detection and sentiment analysis functions.

[1356] 3. Response to abnormality notifications

[1357] The user receives an abnormality notification from the device, checks the content of the notification, checks detailed information about the abnormality and the results of emotion analysis, and, if necessary, rushes to the scene or reports the incident to the relevant authorities (police, fire department, etc.).

[1358] Example scenario

[1359] For example, suppose there are surveillance cameras and audio collection devices installed in a shopping district at night. If the sound of glass breaking occurs in a store late at night and a panicked user screams inside the store at the same time, the audio collection device picks up the sound and the surveillance camera captures the video. These data are sent to a server in real time, and the server analyzes the received data.

[1360] At the same time that the server detects the sound of breaking glass and a scream, the emotion engine recognizes the user's emotion as "anxiety." After detection, the server generates an alert message and sends it to the security company with the following content: "01:23 AM, Store A, sound of breaking glass detected, emotion of anxiety." The device receives this alert and notifies the security guard as a pop-up message. The user (security guard) who sees the notification immediately rushes to the scene and prevents the damage from spreading. This system enables early detection of crimes and accidents and rapid response, improving social safety.

[1361] According to the embodiment of the present invention, the efficiency and accuracy of monitoring work can be improved, and in particular, safety can be further enhanced by making it possible to take into account the emotions of users.

[1362] The processing flow will be explained below.

[1363] Detailed system programming

[1364] Server Processing

[1365] Step 1:

[1366] A server receives data streams from surveillance cameras and audio collection devices.

[1367] The received video and audio data is temporarily stored and transcoded.

[1368] Step 2:

[1369] The server loads a pre-trained AI model.

[1370] Prepare algorithms for anomaly detection and sentiment analysis.

[1371] Step 3:

[1372] The server processes the received data frame by frame.

[1373] Parallel processing is set up so that video data and audio data can be analyzed simultaneously.

[1374] Step 4:

[1375] The server detects abnormal behavior (e.g., violent acts or rapid movement) from the video data.

[1376] The audio data is filtered to detect abnormal sounds (such as glass breaking or screams).

[1377] Step 5:

[1378] If the server detects abnormal behavior or sound, it sends the frame from that point to the emotion analysis engine.

[1379] The sentiment analysis engine recognizes the user's emotions (e.g., anger, anxiety, sadness).

[1380] Step 6:

[1381] The server evaluates abnormal behavior, abnormal sounds, and emotion analysis results, and records details of the abnormality (time, location, type).

[1382] The reliability of the detected anomalies is confirmed by applying a secondary checking algorithm.

[1383] Step 7:

[1384] The server generates an alert message if an abnormality is detected.

[1385] The alert message includes detailed information about the anomaly (time, location, type) and the results of sentiment analysis.

[1386] Step 8:

[1387] The server generates an alert message and sends it to the relevant parties (security company, police, fire department, etc.).

[1388] Checks whether the message was sent successfully and attempts to send it again if it fails.

[1389] Terminal handling

[1390] Step 1:

[1391] The terminal receives video and audio data in real time from the connected surveillance cameras and audio collection devices.

[1392] The received data is temporarily stored and formatted for transfer to the server.

[1393] Step 2:

[1394] The terminal transmits the received data to the server in real time.

[1395] Check the data for completeness and attempt to resend any missing data.

[1396] Step 3:

[1397] The device receives anomaly detection and emotion analysis alert messages sent from the server.

[1398] Verify the accuracy of received alerts and prepare to notify users.

[1399] Step 4:

[1400] Your device will display the alert as a pop-up message or application notification.

[1401] Different notification methods (screen display, audio alarm, vibration alarm, etc.) are provided depending on the level of the alert.

[1402] User Action

[1403] Step 1:

[1404] The user sets the installation locations of the surveillance cameras and audio collection devices and checks the connection status.

[1405] The user configures system notification settings (e.g., email, SMS, app notifications).

[1406] Step 2:

[1407] Users can check the system status in real time and monitor whether it is operating normally.

[1408] It is also possible to manually test the anomaly detection and sentiment analysis features.

[1409] Step 3:

[1410] The user receives the abnormality notification from the terminal and checks the content of the notification.

[1411] Check the detailed information about the anomaly and the results of the emotion analysis, and if necessary, rush to the scene or notify the relevant authorities.

[1412] Example scenario

[1413] For example, suppose there are surveillance cameras and audio collection devices installed in a shopping district at night. If a store hears the sound of glass breaking late at night and a panicked user screams inside the store at the same time, the audio collection device picks up the sound and the surveillance camera captures the video. These data are sent to a server in real time.

[1414] The server analyzes the received data and detects the sound of glass breaking and a scream. At the same time, the emotion engine recognizes the user's emotion as "anxiety." After detection, the server generates an alert message and sends it to the security company with the following content: "01:23 AM, Store A, sound of glass breaking detected, emotion of anxiety."

[1415] The device receives this alert and notifies the security guard as a pop-up message. The user (security guard) who sees the notification can immediately rush to the scene to prevent the damage from spreading. This system enables early detection of crimes and accidents and a rapid response, improving social safety.

[1416] According to the embodiment of the present invention, the efficiency and accuracy of monitoring work can be improved, and in particular, safety can be further enhanced by making it possible to take into account the emotions of users.

[1417] Example 2

[1418] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1419] Modern surveillance systems require improved anomaly detection accuracy and rapid response, but conventional systems perform video and audio analysis separately, resulting in slow anomaly detection times and reduced accuracy. Furthermore, systems are unable to respond to user emotions, making it difficult to respond appropriately based on specific emotions. Furthermore, when an anomaly is detected, the alert content is limited and detailed information is lacking, making it difficult for relevant parties to respond quickly and appropriately.

[1420] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving data in real time from the monitoring camera and the audio collecting device, a data analysis means for processing the received data frame by frame and including an algorithm for performing video analysis and audio analysis in parallel, a means for detecting abnormal behavior or sound from the analyzed data, an emotion analysis means for recognizing the emotion of the user, and a means for generating an alert and notifying relevant parties when an abnormality or a specific emotion is detected. This improves the accuracy of anomaly detection and the speed of response, allowing relevant parties to respond quickly and appropriately based on detailed information.

[1421] A "surveillance camera" is a device that captures video data in real time and captures the situation at a monitored location.

[1422] An "audio collection device" is a device that collects surrounding audio data in real time and uses it for monitoring purposes.

[1423] An "algorithm that processes data frame by frame and performs video and audio analysis in parallel" is a computer program that divides video data into its constituent frames and performs analysis of the video and audio data simultaneously.

[1424] "Means for detecting abnormal behavior or sounds" refers to a system that has the ability to analyze collected data and identify pre-defined abnormal behavior or sound patterns.

[1425] "Emotion analysis means for recognizing user emotions" refers to an algorithm or model for determining a user's emotional state (e.g., anger, anxiety, sadness) from audio tone or video data.

[1426] "Means for generating alerts and notifying relevant parties" refers to a system that creates warning messages based on detected abnormalities and emotional information and sends them to relevant agencies (e.g., security companies, police, fire departments, etc.).

[1427] "Detailed information (time, location, type)" is specific information about the detected abnormal event, including the time of occurrence, the location of occurrence, and the type of abnormality.

[1428] "Means for checking the system status in real time" refers to an interface that allows a user to monitor the current operating status of the data collection device and analysis server in real time and check whether they are functioning normally.

[1429] The system based on this invention is a comprehensive monitoring system that receives data in real time from surveillance cameras and audio collection devices, detects abnormal events, and notifies relevant parties. In particular, by combining it with an emotion analysis engine that recognizes user emotions, the accuracy of abnormality detection and the speed of response can be improved. This system is composed of three entities: a server, a terminal, and a user.

[1430] Server Processing

[1431] The server first receives data in real time from surveillance cameras and audio collection devices. The received data is temporarily stored and transcoded for real-time processing. The server then loads a pre-trained AI model. This AI model includes algorithms for detecting abnormal sounds and behaviors, as well as algorithms for analyzing user emotions. The data is divided into frames, and video and audio analysis is performed in parallel. For example, frameworks such as TensorFlow and PyTorch are used. Abnormal behaviors and sounds are detected from the analyzed data, and an emotion analysis engine is used to identify the user's emotion (e.g., anger, anxiety, sadness, etc.). If an anomaly or emotion is detected, an alert message is generated and sent to relevant parties, along with detailed information about the anomaly (time, location, type) and the results of the emotion analysis.

[1432] Terminal handling

[1433] The device receives video and audio data from surveillance cameras and audio collection devices in real time. The received data is temporarily stored and formatted for transmission to the server. The formatted data is sent to the server in real time, supporting rapid data analysis on the server side. The device receives anomaly detection and emotion analysis alert messages sent from the server and checks their accuracy. The received alerts are displayed as pop-up messages or application notifications, allowing the user to respond quickly. Different notification methods (screen display, audio alarm, vibration alarm, etc.) are provided depending on the level of the alert.

[1434] User Action

[1435] First, the user sets the installation locations of the surveillance cameras and audio collection devices and checks the connection status. They also configure notification settings (e.g., email, SMS, app notifications). Users can check the system status in real time and monitor whether it is operating normally. They can also manually test the anomaly detection and emotion analysis functions. When a user receives an anomaly notification from their device, they can check the notification content and confirm the details of the anomaly and the results of the emotion analysis. If necessary, they can rush to the scene or notify the relevant authorities (police, fire department, etc.).

[1436] Example scenario

[1437] For example, consider a surveillance camera and audio collection device installed in a shopping district at night. Suppose a store hears the sound of glass breaking late at night, and simultaneously, a panicked user screams inside the store. The audio collection device picks up the sound, and the surveillance camera captures the video. These data are sent to a server in real time, which analyzes the received data. When the server detects the sound of glass breaking and the scream, the emotion engine recognizes the user's emotion as "anxiety." After detection, the server generates an alert message stating "1:23 AM, Store A, sound of glass breaking detected, emotion of anxiety," and sends it to the security company. The device receives this alert and notifies the security guard via a pop-up message. The user (security guard) immediately rushes to the scene after confirming the notification and prevents further damage. This system enables early detection and rapid response to crimes and accidents, improving social safety.

[1438] Example prompts for generative AI models

[1439] "Receive data in real time from surveillance cameras and audio collection devices, and generate an alert message if an anomaly is detected. For example, imagine a case where a glass breaks in a store at night, causing a panicked user to scream. Include relevant details (time, location, type) and the results of emotion analysis (anxiety)."

[1440] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1441] Step 1: Data collection

[1442] Terminal

[1443] 1. Receive video and audio data in real time from surveillance cameras and audio collection devices.

[1444] Input: Real-time data from surveillance cameras and audio collection devices.

[1445] Specific operation: The surveillance camera captures video data at 30 frames per second, and the audio collection device collects audio data at 44.1 kHz per second.

[1446] Output: Temporarily stored video and audio data.

[1447] Step 2: Data preparation and transfer

[1448] Terminal

[1449] 1. The received video and audio data is temporarily stored and formatted for transfer to the server.

[1450] Input: Temporarily stored video and audio data.

[1451] Specific operation: Frames of video data are stored in a buffer, and audio data is stored as chunks in certain ranges.

[1452] Output: The formatted data.

[1453] 2. Send the formatted data to the server in real time.

[1454] What it does: Streams data packets to a server using the TCP / IP protocol.

[1455] Step 3: Data analysis

[1456] server

[1457] 1. Split the data stream to analyze the received data frame by frame.

[1458] Input: Formatted data transferred to the server.

[1459] Specific operation: Video frames are extracted from the data stream, and the audio data is processed as is.

[1460] Output: Data divided into frames.

[1461] 2. Load the pre-trained AI model and prepare it for anomaly detection and sentiment analysis.

[1462] Specific operation: Load an anomaly detection model into memory using TensorFlow or PyTorch.

[1463] Step 4: Anomaly detection and sentiment analysis

[1464] server

[1465] 1. Video analysis: Analyze each frame of received video data to detect abnormal behavior.

[1466] Input: Video data divided into frames.

[1467] Specific operation: When the movement between frames changes drastically, an algorithm is applied that considers this to be abnormal behavior.

[1468] Output: Anomalous behavior detection results.

[1469] 2. Audio analysis: Spectral analysis of audio data is performed to detect abnormal sounds (such as the sound of glass breaking or a person screaming).

[1470] Input: Audio data divided into frames.

[1471] Specific behavior: Identify sudden changes in sound pressure level and specific frequency patterns.

[1472] Output: Abnormal sound detection result.

[1473] 3. Emotion analysis: Analyzes the tone of voice and facial expressions to identify the user's emotions.

[1474] Input: Received video and audio data.

[1475] Specific operation: Identify emotions by detecting changes in voice tone, pitch, and facial expressions in video.

[1476] Output: Emotion identification results.

[1477] Step 5: Generate and send an alert

[1478] server

[1479] 1. If an anomaly or a specific emotion is detected, an alert message is generated, containing detailed information about the anomaly (time, location, type) and the results of emotion analysis.

[1480] Input: Abnormal behavior, abnormal sound, and emotion analysis results.

[1481] Specific operations: Integrates and converts data, and generates alert messages.

[1482] Output: The alert message.

[1483] 2. Send the generated alert message to the relevant parties.

[1484] What it does: Deliver alerts via SMS, email, and app notifications.

[1485] Step 6: Receive alerts and notifications

[1486] Terminal

[1487] 1. Receive an alert message sent from the server.

[1488] Input: The alert message from the server.

[1489] Specific operation: Receive alert information via HTTPS encrypted communication.

[1490] Output: The received alert message.

[1491] 2. Review the accuracy of the alerts received and make arrangements to notify users.

[1492] Specific operation: Analyzes the message content and converts it into a display format.

[1493] 3. Display the alert to the user as a popup message or app notification.

[1494] Specific behavior: Use different notification methods such as popup, audio alarm, vibration, etc.

[1495] Step 7: Responding to abnormality notifications

[1496] User

[1497] 1. Receive an abnormality notification from the device and check the content of the notification.

[1498] Input: Alert notification from the terminal.

[1499] Specific operation: Display and confirm detailed information about anomalies and the results of sentiment analysis.

[1500] 2. If necessary, rush to the scene and notify the relevant authorities.

[1501] Specific actions: Decide on a response promptly based on the information confirmed.

[1502] (Application example 2)

[1503] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1504] Conventional monitoring systems monitor a huge amount of data and perform analysis to detect abnormalities, but are unable to analyze the user's emotional state when an abnormality occurs, making it difficult to respond quickly and appropriately.The present invention aims to enable faster and more appropriate responses by analyzing user emotional data in real time in addition to monitoring data and generating alerts including the results of emotional analysis when an abnormality occurs.

[1505] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1506] In this invention, the server includes means for receiving data in real time from the surveillance camera and the audio collecting device, means for analyzing the received data and detecting a predetermined abnormal sound, means for generating and reporting an alert when an abnormal sound is detected, means for analyzing the user's emotions, and means for generating and reporting an alert message including the emotion analysis result. This makes it possible to analyze the user's emotional state at the same time as detecting an abnormality and generate a detailed alert including emotional information.

[1507] A "surveillance camera" is a device that captures images of a location in real time and records or transmits the data.

[1508] An "audio collection device" is a device for collecting audio data in an environment in real time and recording or transmitting that data.

[1509] "Means for receiving data in real time" refers to technical means for receiving data almost simultaneously with its generation.

[1510] "Means for analyzing received data" refers to technical means for extracting information or detecting anomalies from data received in real time using specific algorithms.

[1511] "Means for detecting predetermined abnormal sounds" refers to technical means for detecting specific abnormal sounds (e.g., the sound of breaking glass, screams, etc.) that are defined in advance.

[1512] "Means for generating and reporting an alert when an abnormal sound is detected" refers to the technical means for generating a warning message or notification when an abnormal sound is detected and for promptly reporting it to designated parties.

[1513] "Means for analyzing user emotions" refers to technical means for analyzing the user's emotional state (e.g., anger, anxiety, sadness, etc.) from video and audio data.

[1514] "Means for generating and notifying an alert message including the result of emotion analysis" refers to a technical means for generating a detailed alert message including the result of the user's emotion analysis and promptly notifying relevant parties.

[1515] System configuration

[1516] The system embodying the present invention consists of three main components: a server, a terminal, and a user. The server receives data from the surveillance cameras and audio collection devices, analyzes the data, and detects abnormalities. The terminal transmits the collected data to the server and notifies the user of alert messages from the server. The user performs the initial system setup and responds appropriately when notified of an abnormality.

[1517] Hardware and software used

[1518] The server requires hardware with high-performance data analysis capabilities, and specifically, a server equipped with an NVIDIA GPU is suitable. Machine learning frameworks such as TensorFlow and PyTorch are used for data analysis. The terminals used are devices that users can carry, such as smartphones, tablets, and smart glasses. These devices are equipped with high-resolution cameras and highly sensitive microphones, making it possible to collect data in real time.

[1519] Specific examples

[1520] For example, if a nighttime surveillance system is installed in a shopping mall, surveillance cameras and audio collection devices are installed in each store. These devices collect video and audio data in real time and send it to a server via a terminal. The server analyzes the received data and detects abnormal sounds and behavior.

[1521] If an abnormality is detected, the server analyzes the user's emotions and generates an alert message containing the results of the emotion analysis. For example, a message may be generated stating, "The sound of glass breaking was detected at Store A at 2:00 AM, and an emotion of anxiety was recognized." This alert message is sent to the security guard's smart glasses in real time. The security guard can then check the notification and quickly rush to the scene.

[1522] Generative AI model and prompts

[1523] The system uses a generative AI model to analyze user sentiment. Specific examples of prompts include:

[1524] Example prompt sentence:

[1525] "Detect the sound of glass breaking or a person screaming, and analyze the user's emotions. For example, recognize emotions such as anxiety or anger. If an anomaly is detected, generate an alert containing the anomaly and the results of the emotion analysis, and notify the user."

[1526] This enables the monitoring system to detect abnormalities early and respond quickly and appropriately while taking into account the emotional state of the user. This invention is particularly effective in strengthening security in commercial facilities, public facilities, important facilities, etc.

[1527] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1528] Step 1:

[1529] Data collection (terminal)

[1530] The device (e.g., smartphone, smart glasses) collects data (video and audio) in real time from surveillance cameras and audio collectors. The cameras generate high-resolution video data, and the audio collectors collect audio data with high sensitivity. This data is temporarily stored on the device. The input includes video data from the surveillance cameras and audio data from the audio collectors, and the output is the temporary storage of these data.

[1531] Step 2:

[1532] Data transfer (terminal)

[1533] The device processes the temporarily stored video and audio data (for example, transcoding or compressing) and then transmits it to the server in real time. Here, the video data is encoded into JPEG or H.264 format, and the audio data is encoded into MP3 or AAC format. The temporarily stored raw data is the input, and the transcoded data is sent to the server as the output.

[1534] Step 3:

[1535] Data analysis (server)

[1536] The server receives the video and audio data sent from the device, loads a pre-trained generative AI model, and performs data analysis. The analysis includes detecting abnormal behavior from the video, detecting abnormal sounds from the audio, and analyzing the user's emotions. Specifically, it detects movement and shape changes from the video, and specific frequencies and patterns from the audio. The input is the received transcoded data, and the output is an analysis of the type of abnormality and the emotional state.

[1537] Step 4:

[1538] Anomaly detection and sentiment analysis (server)

[1539] If the server detects an anomaly from the data analysis results, it further analyzes the user's emotions (anger, anxiety, sadness, etc.) using a generative AI model. Specifically, emotions are recognized based on voice tone and facial expression analysis. The input is the preliminary results of the data analysis, and the output is the presence or absence of an anomaly and the user's emotional state.

[1540] Step 5:

[1541] Alert generation and transmission (server)

[1542] The server generates an alert message based on the anomaly and the results of sentiment analysis. The alert message contains detailed information such as the type of anomaly, time, location, and sentiment analysis results. The generated alert message is promptly sent to relevant parties (e.g., security guards, police, fire department, etc.). The inputs are the anomaly and the sentiment analysis results, and the generated alert message is the output.

[1543] Step 6:

[1544] User notification (terminal)

[1545] The terminal receives the alert message sent from the server and notifies the user (e.g., security guard). Notification methods include screen display, audio alarm, and vibration alarm. The alert message is the input, and the notification to the user is the output.

[1546] Step 7:

[1547] Response (user)

[1548] The user receives an alert notification from the device and checks the content of the notification. For example, if the alert content is "The sound of glass breaking was detected at Store A at 2:00 AM, and feelings of anxiety were recognized," the user will immediately rush to the scene and take appropriate action. The input is the alert notification, and the output is the user's response action.

[1549] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1550] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1551] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1552] [Fourth embodiment]

[1553] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1554] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1555] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1556] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1557] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1558] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1559] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1560] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1561] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1562] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1563] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1564] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1565] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1566] System Overview

[1567] The system based on this invention is a comprehensive monitoring system that receives data in real time from surveillance cameras and audio collection devices, detects abnormalities, and notifies relevant parties. The system is mainly composed of three components: a server, a terminal, and a user, which work together to perform effective monitoring and issue alerts.

[1568] In the embodiment of the present invention, the system is composed of the following three entities.

[1569] Server Processing

[1570] 1. Data Collection

[1571] The server receives and temporarily stores data streams from surveillance cameras and audio collection devices, and transcodes the data for real-time processing.

[1572] 2. Data Analysis

[1573] The server uses a pre-trained AI model to process the received data frame by frame, performing video and audio analysis in parallel, thereby achieving highly accurate anomaly detection.

[1574] 3. Anomaly Detection

[1575] When the server detects an abnormal sound (for example, the sound of breaking glass or a person screaming), it records the abnormality and identifies details (time, location, type).

[1576] 4. Alert generation and sending

[1577] If an anomaly is detected, the server generates an alert message and sends it to the relevant parties. The alert message contains detailed information about the anomaly.

[1578] Terminal handling

[1579] 1. Data Entry

[1580] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices, temporarily stores the data, and processes it for transfer to the server.

[1581] 2. Data Transfer

[1582] By sending the data received by the terminal to the server in real time, it supports rapid data analysis on the server side.

[1583] 3. Receive anomaly alerts

[1584] The terminal receives the anomaly detection alert sent from the server and prepares to notify the user.

[1585] 4. User Notices

[1586] The device will display the alert as a pop-up message or application notification, allowing the user to respond quickly.

[1587] User Action

[1588] 1. Initial Setup

[1589] Users can set the installation locations of surveillance cameras and audio collection devices, check the connection status, and configure notification settings (e.g., email, SMS, app notifications).

[1590] 2. System Check

[1591] Users can check the system status in real time and monitor whether it is working properly. Users can also manually test the anomaly detection function.

[1592] 3. Response to abnormality notifications

[1593] The user receives an abnormality notification from the device, checks the detailed information, and then rushes to the scene or reports it to the relevant authorities (police, fire department, etc.) as necessary.

[1594] Example scenario

[1595] For example, imagine there are surveillance cameras and audio collection devices installed in a shopping district at night. If the sound of glass breaking occurs in a store late at night, the sound is picked up by the sound-collecting microphone and the surveillance camera captures the video. This data is sent to a server in real time. The server analyzes the received data and detects the sound of glass breaking. After detection, the server generates an alert message and sends it to the security company. The device receives this alert and notifies the user (security guard) as a pop-up message. Upon seeing the notification, the user can immediately rush to the scene and prevent the damage from spreading. This system enables early detection of crimes and accidents and rapid response, improving social safety.

[1596] The embodiments of the present invention improve the efficiency and accuracy of monitoring operations, and make a significant contribution to preventing crimes and accidents.

[1597] The processing flow will be explained below.

[1598] System Program Processing

[1599] Server Processing

[1600] Step 1:

[1601] A server receives data streams from surveillance cameras and audio collection devices.

[1602] The received video and audio data is temporarily stored and transcoded.

[1603] Step 2:

[1604] The server loads a pre-trained AI model.

[1605] The received data is processed frame by frame, and video and audio analysis is performed in parallel.

[1606] Step 3:

[1607] The server detects abnormal movements in the video data and filters out abnormal sounds (such as breaking glass or people screaming) from the audio data.

[1608] Apply anomaly detection algorithms to check if anomalies have occurred.

[1609] Step 4:

[1610] When the server detects an abnormal sound or behavior, it records the details (time, location, type).

[1611] Apply a secondary check algorithm to ensure the anomaly detection is correct.

[1612] Step 5:

[1613] The server generates an alert message if an abnormality is detected.

[1614] Include detailed information about the anomaly (time, location, type) in the alert message.

[1615] Step 6:

[1616] The server generates an alert message and sends it to the relevant parties (security company, police, fire department, etc.).

[1617] Checks whether the message was sent successfully and attempts to send it again if it fails.

[1618] Terminal handling

[1619] Step 1:

[1620] The terminal receives video and audio data in real time from the connected surveillance cameras and audio collection devices.

[1621] The received data is temporarily stored and formatted for transfer to the server.

[1622] Step 2:

[1623] The terminal transmits the received data to the server in real time.

[1624] Check the data for completeness and attempt to resend any missing data.

[1625] Step 3:

[1626] The terminal receives an anomaly detection alert sent from the server.

[1627] Check the accuracy of the received alerts and send feedback to the server if necessary.

[1628] Step 4:

[1629] The device displays the alert to the user as a pop-up message or application notification.

[1630] The alarm can be set to a screen display, audio alarm, vibration alarm, etc.

[1631] User Action

[1632] Step 1:

[1633] The user sets the installation locations of the surveillance cameras and audio collection devices and checks the connection status.

[1634] The user configures system notification settings (e.g., email, SMS, app notifications).

[1635] Step 2:

[1636] Users can check the status of the system and monitor whether it is working properly.

[1637] It is also possible to manually test the anomaly detection feature.

[1638] Step 3:

[1639] The user receives the abnormality notification from the terminal and checks the content of the notification.

[1640] Check detailed information about the abnormality (time, location, type) and, if necessary, rush to the scene or report it to the relevant authorities.

[1641] Example 1

[1642] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1643] In recent years, there has been an increasing demand for more accurate and efficient monitoring systems. However, existing monitoring systems have problems such as low anomaly detection accuracy and insufficient real-time data processing. Furthermore, they are often unable to quickly notify relevant parties of detected anomalies, making it difficult to respond quickly. To address these issues, there is a demand for monitoring systems that can detect anomalies more accurately in real time and quickly notify relevant parties.

[1644] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1645] In this invention, the server includes means for receiving data in real time from the surveillance cameras and audio collecting devices, means for temporarily storing the received data and transcoding it into an optimal format, means for processing the received data frame by frame using a pre-trained artificial intelligence model to perform video and audio analysis in parallel, means for detecting abnormal sounds and images, recording the abnormality and identifying details (time, location, type), and means for generating an alert message and notifying relevant parties when an abnormality is detected. This enables real-time abnormality detection and highly accurate data analysis, realizing early detection of abnormalities and rapid response.

[1646] A "surveillance camera" is a device that acquires video data in real time and is used for monitoring and anomaly detection.

[1647] An "audio collection device" is a device that acquires audio data in real time and is used for monitoring and anomaly detection.

[1648] A "data stream" is a continuous flow of data transmitted in real time from a surveillance camera or audio collection device.

[1649] "Transcoding" is the process of converting received data into a different format.

[1650] An "artificial intelligence model (AI model)" is an algorithm that has been trained in advance using large amounts of data, and is used to detect and analyze anomalies.

[1651] "Frame-based processing" is a method in which video data and audio data are divided into fixed time intervals (frames) and analyzed and processed for each unit.

[1652] "Parallel processing" is a technology that improves processing speed by executing multiple processes or threads simultaneously.

[1653] An "abnormal sound" is a specific sound (such as the sound of breaking glass or screaming) that does not occur in a normal monitoring environment and indicates an abnormal situation.

[1654] "Abnormal footage" refers to specific footage (for example, suspicious activity) that is not seen in a normal surveillance environment and indicates an abnormal situation.

[1655] An "alert message" is a notification message that is generated when an abnormality is detected, and provides detailed information about the abnormality to relevant parties.

[1656] "Relevant parties" are people who manage and operate the monitoring system and people who are responsible for responding to abnormalities (e.g., security guards and administrators).

[1657] The monitoring system based on this invention is mainly composed of three entities: a server, a terminal, and a user. It is a comprehensive monitoring system that receives data in real time from monitoring cameras and audio collection devices, detects abnormalities, and notifies relevant parties. The role of each entity will be specifically explained below.

[1658] Server Processing

[1659] The server receives real-time data streams from surveillance cameras (e.g., general network cameras) and audio collection devices (e.g., standard microphones). The data is temporarily stored and transcoded into the optimal format for real-time processing. For example, video data is converted to H.264 format, and audio data is converted to WAV format. The transcoding process uses high-speed processing using NVIDIA CUDA.

[1660] The server then analyzes the data using pre-trained AI models. It uses TensorFlow to load an anomaly detection model, a YOLO model for video analysis, and a CNN model for audio analysis. Data is processed frame by frame and analyzed in parallel. Video data is analyzed using OpenCV, and audio data undergoes spectrogram analysis using Librosa.

[1661] When an abnormal sound or video is detected, the server records the anomaly and identifies the time, location, and type of the anomaly. Detailed information about the anomaly is recorded in a MySQL database. If an anomaly is detected, an alert message is generated and sent to relevant parties using Amazon SNS (Simple Notification Service).

[1662] Terminal handling

[1663] The terminal uses a Raspberry Pi or a standard edge device to receive data in real time from surveillance cameras and audio collection devices. The received data is temporarily stored in a buffer and processed. The terminal then transfers the received data to the server in real time using the MQTT protocol.

[1664] When an anomaly detection alert is received from the server, the device prepares a notification to the user. Firebase Cloud Messaging is used to notify the user of the alert message as a pop-up display or app notification, allowing the user to respond to the anomaly quickly.

[1665] User Action

[1666] Users can use a web-based management screen to set the installation locations of surveillance cameras and audio collection devices and check the connection status. They can also set notifications (e.g., email, SMS, app notifications). Users can check the system status from the management screen and monitor operation in real time. If an abnormality is detected, users will receive an alert message and check detailed information (e.g., date, time, location, and nature of the abnormality). They can then rush to the scene or notify the relevant authorities as necessary.

[1667] Example scenario

[1668] For example, consider the case where a surveillance camera and audio collection device installed in a shopping district detect the sound of glass breaking in a store late at night. The received data is sent to a server, where an AI model is used to detect the sound of glass breaking. The detected abnormality is recorded and an alert message is sent to the security company using Amazon SNS. The device receives this alert and notifies the security guard via a pop-up message. The security guard can then immediately rush to the scene and prevent further damage.

[1669] Example prompts to input to the generative AI model

[1670] "I'm in charge of the surveillance system for a shopping mall. Can you explain the outline of the AI ​​model that detects suspicious sounds and images at night and immediately notifies security guards?"

[1671] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1672] Step 1:

[1673] Receiving and temporarily storing data (server processing)

[1674] Input: Real-time data streams from surveillance cameras and audio collection devices

[1675] Specific operation: The server receives data streams from surveillance cameras and audio collection devices using RTSP (Real-Time Streaming Protocol). The received data is temporarily saved and stored in a buffer for subsequent processing.

[1676] Output: Temporarily saved data (video data and audio data)

[1677] Step 2:

[1678] Data transcoding (server processing)

[1679] Input: Temporarily saved data (video data and audio data)

[1680] Specific operation: The server transcodes the received video data into H.264 format and the audio data into WAV format. The transcoding process is performed at high speed using NVIDIA CUDA.

[1681] Output: Transcoded data (video data in H.264 format, audio data in WAV format)

[1682] Step 3:

[1683] Preparing for data analysis (server processing)

[1684] Input: Transcoded data (video data in H.264 format, audio data in WAV format)

[1685] How it works: The server loads pre-trained AI models using TensorFlow. The YOLO model is used for video analysis, and the CNN model is used for audio analysis.

[1686] Output: A ready-to-use AI model

[1687] Step 4:

[1688] Frame-by-frame data analysis (server processing)

[1689] Input: AI model ready, transcoded data

[1690] Specific operation: The server divides the video and audio data into frames and processes them in parallel. The video data is analyzed using OpenCV, and the audio data undergoes spectrogram analysis using Librosa.

[1691] Output: Parsed frame data

[1692] Step 5:

[1693] Anomaly detection (server processing)

[1694] Input: Parsed frame data

[1695] How it works: The server uses AI models to detect abnormal sounds and images. For example, if an abnormality such as the sound of broken glass, screams, or a suspicious person is detected, the server records the abnormality and identifies the time, location, and type of occurrence. Detailed information about the abnormality is then recorded in a MySQL database.

[1696] Output: Anomaly detection results and detailed information (time, location, type)

[1697] Step 6:

[1698] Alert generation and notification (server processing)

[1699] Input: Anomaly detection results and detailed information

[1700] Specific operation: If an anomaly is detected, the server generates an alert message. The message contains detailed information about the anomaly (e.g., date, time, location, and content of the anomaly). The alert message is sent to relevant parties using Amazon SNS.

[1701] Output: Alert message sent

[1702] Step 7:

[1703] Data entry (terminal processing)

[1704] Input: Data acquired from surveillance cameras and audio collection devices

[1705] How it works: The device uses a Raspberry Pi or a standard edge device to receive and temporarily store data from surveillance cameras and audio capture devices in real time, check for missing or abnormal data, and add a timestamp.

[1706] Output: Formatted data

[1707] Step 8:

[1708] Data transfer (terminal processing)

[1709] Input: Formatted data

[1710] Specific operation: The terminal uses the MQTT protocol to send the received data to the server in real time.

[1711] Output: Data transferred to the server

[1712] Step 9:

[1713] Receive an abnormality alert (terminal processing)

[1714] Input: The alert message sent from the server

[1715] Specific operation: The terminal receives the alert message sent from the server and prepares to notify the user.

[1716] Output: Prepared alert message

[1717] Step 10:

[1718] User notification (terminal processing)

[1719] Input: Prepared alert message

[1720] Specific operation: The device uses Firebase Cloud Messaging to notify the user of the received alert message as a pop-up or app notification.

[1721] Output: Alert message sent to the user

[1722] Step 11:

[1723] Initial settings (user processing)

[1724] Input: Setting information by accessing the administration screen

[1725] Specific operation: Using a web-based management interface, users can configure the locations of surveillance cameras and audio collection devices, check connection status, and configure notifications (e.g., email, SMS, app notifications).

[1726] Output: Management screen with saved settings

[1727] Step 12:

[1728] System confirmation (user processing)

[1729] Input:Administrator screen information

[1730] Specific operation: Users can check the system status from the management screen, monitor the operation status in real time, and manually test the anomaly detection function.

[1731] Output: Confirmation of system status, working properly

[1732] Step 13:

[1733] Handling abnormality notifications (user actions)

[1734] Input: The alert message that was sent to the user

[1735] Specific operations: The user checks the detailed information of the alert message (e.g., date and time, location, and content of the abnormality), and if necessary, rushes to the scene or notifies the relevant authorities.

[1736] Output: Notification information for which the action has been completed

[1737] (Application example 1)

[1738] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1739] While conventional surveillance systems analyze real-time data from surveillance cameras and audio collection devices and detect abnormalities, they lack the ability to quickly and effectively notify relevant parties of the abnormality. In particular, they lack the ability to provide optimal route information for rushing to the scene, and there is a need for a rapid response when an abnormality is notified. The present invention aims to solve these problems and provide a more efficient and responsive surveillance system.

[1740] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1741] In this invention, the server includes means for receiving data in real time from the surveillance camera and the audio collecting device, means for analyzing the received data and detecting a preset abnormal sound, means for generating an alert and notifying relevant parties when an abnormal sound is detected, means for displaying the abnormality notification as a pop-up message or an application notification using a smart device, and means for displaying the shortest route to the location of the abnormality. This not only enables rapid abnormality detection and notification, but also improves the efficiency of on-site response.

[1742] A "surveillance camera" is a device used to continuously monitor a specific area or location and collect video data.

[1743] An "audio collection device" is a device that collects ambient sounds and records and transmits them as analog or digital data.

[1744] "Real-time" refers to the ability to process and transmit data and information instantly, without delay.

[1745] The "means for receiving data" refers to a function or device for receiving data transmitted from the surveillance camera and the audio collecting device.

[1746] The "means for analyzing data" is a function or device that processes the received data and performs analysis to detect anomalies.

[1747] An "abnormal sound" is a sound that indicates a specific abnormal situation, unlike normal background sounds such as breaking glass or screams.

[1748] "Means for detecting abnormalities" refers to a function or device that identifies abnormal sounds or behaviors as a result of data analysis.

[1749] An "alert generating means" is a function or device that generates a warning message or notification when an abnormality is detected.

[1750] A "means for notifying interested parties" is a function or device that generates an alert and then sends it to a designated recipient.

[1751] A "smart device" is a mobile terminal with Internet connectivity that can run applications.

[1752] A "pop-up message" is a notification message that suddenly appears on the device screen.

[1753] "Application notification" is a notification function that notifies the user of information through a specific application.

[1754] The "means for displaying the shortest route" is a function or device for calculating and displaying the most efficient route from the current location to the location where the abnormality has occurred.

[1755] In order to implement the present invention, a system is constructed that receives data in real time from a surveillance camera and an audio collecting device, analyzes the data, and detects abnormalities. The system configuration and processing content are described in detail below.

[1756] System Overview

[1757] This system is mainly composed of three entities: a server, a terminal, and a user, which cooperate to monitor and detect anomalies.

[1758] Hardware and Software Environment

[1759] Hardware:

[1760] Surveillance cameras (IoT cameras)

[1761] Audio collection device (microphone)

[1762] Smart devices (smart glasses, smartphones)

[1763] Server (cloud platform)

[1764] software:

[1765] Server side: AWS Lambda, EC2, Amazon Rekognition, Amazon Transcribe

[1766] AI models: TensorFlow, PyTorch

[1767] Device applications: Android Studio, Xcode, Google Glass SDK

[1768] Server Processing

[1769] 1. Data Collection:

[1770] The server receives and temporarily stores data streams from surveillance cameras and audio capture devices, and transcodes the data for real-time processing.

[1771] 2. Data Analysis:

[1772] The server uses pre-trained AI models (TensorFlow or PyTorch) to process the received data frame by frame and perform video and audio analysis in parallel. Amazon Rekognition is used for video analysis, and Amazon Transcribe is used for audio analysis.

[1773] 3. Anomaly detection:

[1774] When the server detects an abnormal sound (such as the sound of breaking glass or a person screaming), it records the abnormality and identifies details (time, location, type).

[1775] 4. Alert generation and sending:

[1776] If an anomaly is detected, the server generates an alert message and sends it to the relevant parties. The alert message contains detailed information about the anomaly (time, location, type).

[1777] Terminal handling

[1778] 1. Data Entry:

[1779] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices, temporarily stores the data, and processes it for transfer to the server.

[1780] 2. Data Transfer:

[1781] The terminal transmits the received data to the server in real time, supporting rapid data analysis.

[1782] 3. Receive anomaly alerts:

[1783] The terminal receives the anomaly detection alert sent from the server and prepares to notify the user.

[1784] 4. User Notice:

[1785] The device will notify users of any abnormalities as a pop-up message or application notification, allowing them to take prompt action. It will also work with map apps to display the shortest route to the location of the abnormality.

[1786] User Roles

[1787] 1. Initial Setup:

[1788] Users can set the locations of surveillance cameras and audio collection devices, check the connection status, and configure notification settings (e.g., email, SMS, app notifications).

[1789] 2. System Check:

[1790] Users can check the system status in real time to monitor whether it is working properly, and can also manually test the anomaly detection function.

[1791] 3. Response to abnormality notification:

[1792] The user receives an abnormality notification from the device, checks the detailed information, and then quickly moves to the location of the abnormality and reports it to the relevant authorities (police, fire department, etc.) if necessary.

[1793] Example scenario

[1794] For example, imagine there are surveillance cameras and audio collection devices installed in a shopping district at night. If the sound of glass breaking occurs in a store late at night, the sound is picked up by the sound-collecting microphone and the surveillance camera captures the video. This data is sent to a server in real time, which analyzes the received data and detects the sound of glass breaking. After detection, the server generates an alert message and sends it to the security company. The device receives this alert and notifies the security guard as a pop-up message. Upon seeing the notification, the security guard immediately rushes to the scene and prevents the damage from spreading. This system enables early detection of crimes and accidents and rapid response, improving social safety.

[1795] Prompt Sentence Examples

[1796] scenario:

[1797] We use a system that receives data in real time from surveillance cameras and microphones installed in shopping districts at night and detects abnormalities. Please explain in detail how it handles the situation when the sound of glass being broken is detected.

[1798] Input data:

[1799] Camera video data

[1800] Microphone audio data

[1801] GPS location information

[1802] Process flow:

[1803] 1. Data collection and transmission

[1804] 2. Data analysis and anomaly detection using AI models

[1805] 3. Notification and response when an abnormality occurs

[1806] The above is a specific embodiment for carrying out the invention.

[1807] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1808] Step 1: Data collection

[1809] The server receives video and audio data in real time from surveillance cameras and audio collection devices. Specifically, the server acquires stream data from IoT cameras and microphones and temporarily stores it. This data is transcoded for later analysis. The input data is camera video data and microphone audio data, and the output is the transcoded data.

[1810] Step 2: Data analysis

[1811] The server processes the received data frame by frame using a pre-trained AI model (TensorFlow or PyTorch). It uses Amazon Rekognition to analyze the video and Amazon Transcribe to analyze the audio data. The input data is transcoded video and audio data, and the output after analysis is a detailed analysis result that includes any abnormal sounds or behavior.

[1812] Step 3: Anomaly detection

[1813] When the server detects an abnormal sound (such as the sound of breaking glass or a person screaming) or abnormal behavior (such as suspicious activity), it records the abnormality and identifies detailed information (time, location, type). During this process, if it recognizes that an abnormality has occurred based on the analysis results, it sets an anomaly detection flag. The input is the analysis result, and the output is detailed information about the abnormality.

[1814] Step 4: Alert Generation

[1815] The server generates an alert message when an anomaly is detected. The alert contains detailed information about the anomaly (time, location, type) and also determines which parties to notify. The input is the detailed information about the anomaly, and the output is the generated alert message.

[1816] Step 5: Sending an alert

[1817] The server generates an alert message and sends it to the appropriate parties via email, SMS, app notification, etc. The input is the generated alert message and the output is the sent alert.

[1818] Step 6: Data entry

[1819] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices. The received data is temporarily stored and then formatted before being transferred to the server. The input is data from the surveillance cameras and audio collection devices, and the output is formatted data.

[1820] Step 7: Data Transfer

[1821] The terminal transmits the received data to the server in real time, which supports fast data analysis on the server side. The input is the formatted data, and the output is the transmitted data.

[1822] Step 8: Receive anomaly alerts

[1823] The terminal receives an anomaly detection alert sent from the server. This alert contains detailed information about the anomaly and is prepared for notification to the user. The input is the alert message from the server, and the output is the preparation status for notification.

[1824] Step 9: User Notification

[1825] The device displays the abnormality notification as a pop-up message or application notification. It also works with a map application to display the shortest route to the location of the abnormality, allowing the user to respond quickly. The input is the prepared notification information, and the output is the displayed notification and route information.

[1826] Step 10: Initial Setup

[1827] The user sets the installation location of the surveillance camera and audio collection device, checks the connection status, and sets notification settings (e.g., email, SMS, app notifications). The input is the initial state of the device, and the output is the device state after the settings are complete.

[1828] Step 11: System Verification

[1829] Users can check the system status in real time to monitor whether it is operating normally. They can check the real-time status of the system and check whether there are any abnormalities. The input is the current system status and the output is the check result.

[1830] Step 12: Responding to abnormal notifications

[1831] The user receives an abnormality notification from the device and checks the detailed information. Then, they quickly move to the location of the abnormality and, if necessary, notify the relevant authorities (police, fire department, etc.). The input is the notification information from the device, and the output is the user's response action.

[1832] The above are the specific processing steps and their contents. This series of processes enables rapid and effective anomaly detection and response.

[1833] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1834] System Overview

[1835] The system based on this invention is a comprehensive monitoring system that receives data in real time from surveillance cameras and audio collection devices, detects abnormalities, and notifies relevant parties. This system, in particular, can improve the accuracy of anomaly detection and the speed of response by combining it with an emotion engine that recognizes user emotions. The system is mainly composed of three entities: a server, a terminal, and a user, which work together to perform effective monitoring and issue alerts.

[1836] In the embodiment of the present invention, the system is composed of the following three entities.

[1837] Server Processing

[1838] 1. Data Collection

[1839] The server receives and temporarily stores data streams from surveillance cameras and audio collection devices, and transcodes the data for real-time processing.

[1840] 2. Data Analysis

[1841] The server loads a pre-trained AI model, which includes algorithms for detecting abnormal sounds and movements and analyzing user emotions.

[1842] The received data is processed frame by frame, and video and audio analysis is performed in parallel, allowing for anomaly detection and emotion analysis to be performed simultaneously.

[1843] 3. Anomaly Detection

[1844] The server detects abnormal behavior and sounds (such as the sound of breaking glass or people screaming) from video and audio data.

[1845] The server uses an emotion engine to analyze the user's emotions (e.g., anger, anxiety, sadness, etc.) using voice tone and facial expression analysis.

[1846] 4. Alert generation and sending

[1847] If an anomaly or a specific emotion is detected, the server generates an alert message, which includes detailed information about the anomaly (time, location, type) and the result of emotion analysis.

[1848] Generate different levels of alerts depending on the type of anomaly and the degree of emotion.

[1849] The server generates an alert message and sends it to the relevant parties (security company, police, fire department, etc.).

[1850] Terminal handling

[1851] 1. Data Entry

[1852] The terminal receives video and audio data in real time from connected surveillance cameras and audio collection devices.

[1853] The received data is temporarily stored and formatted for transfer to the server.

[1854] 2. Data Transfer

[1855] The terminal transmits the received data to the server in real time, supporting rapid data analysis on the server side.

[1856] 3. Receive anomaly alerts

[1857] The terminal receives an alert message for anomaly detection and emotion analysis sent from the server.

[1858] Verify the accuracy of received alerts and prepare to notify users.

[1859] 4. User Notices

[1860] The device displays alerts as pop-up messages or application notifications, allowing users to respond quickly. Different notification methods (screen display, audio alarm, vibration alarm, etc.) are provided depending on the level of the alert.

[1861] User Action

[1862] 1. Initial Setup

[1863] Users can set the installation locations of surveillance cameras and audio collection devices, check the connection status, and configure notification settings (e.g., email, SMS, app notifications).

[1864] 2. System Check

[1865] Users can check the system status in real time to monitor whether it is working properly, and can also manually test the anomaly detection and sentiment analysis functions.

[1866] 3. Response to abnormality notifications

[1867] The user receives an abnormality notification from the device, checks the content of the notification, checks detailed information about the abnormality and the results of emotion analysis, and, if necessary, rushes to the scene or reports the incident to the relevant authorities (police, fire department, etc.).

[1868] Example scenario

[1869] For example, suppose there are surveillance cameras and audio collection devices installed in a shopping district at night. If the sound of glass breaking occurs in a store late at night and a panicked user screams inside the store at the same time, the audio collection device picks up the sound and the surveillance camera captures the video. These data are sent to a server in real time, and the server analyzes the received data.

[1870] At the same time that the server detects the sound of breaking glass and a scream, the emotion engine recognizes the user's emotion as "anxiety." After detection, the server generates an alert message and sends it to the security company with the following content: "01:23 AM, Store A, sound of breaking glass detected, emotion of anxiety." The device receives this alert and notifies the security guard as a pop-up message. The user (security guard) who sees the notification immediately rushes to the scene and prevents the damage from spreading. This system enables early detection of crimes and accidents and rapid response, improving social safety.

[1871] According to the embodiment of the present invention, the efficiency and accuracy of monitoring work can be improved, and in particular, safety can be further enhanced by making it possible to take into account the emotions of users.

[1872] The processing flow will be explained below.

[1873] Detailed system programming

[1874] Server Processing

[1875] Step 1:

[1876] A server receives data streams from surveillance cameras and audio collection devices.

[1877] The received video and audio data is temporarily stored and transcoded.

[1878] Step 2:

[1879] The server loads a pre-trained AI model.

[1880] Prepare algorithms for anomaly detection and sentiment analysis.

[1881] Step 3:

[1882] The server processes the received data frame by frame.

[1883] Parallel processing is set up so that video data and audio data can be analyzed simultaneously.

[1884] Step 4:

[1885] The server detects abnormal behavior (e.g., violent acts or rapid movement) from the video data.

[1886] The audio data is filtered to detect abnormal sounds (such as glass breaking or screams).

[1887] Step 5:

[1888] If the server detects abnormal behavior or sound, it sends the frame from that point to the emotion analysis engine.

[1889] The sentiment analysis engine recognizes the user's emotions (e.g., anger, anxiety, sadness).

[1890] Step 6:

[1891] The server evaluates abnormal behavior, abnormal sounds, and emotion analysis results, and records details of the abnormality (time, location, type).

[1892] The reliability of the detected anomalies is confirmed by applying a secondary checking algorithm.

[1893] Step 7:

[1894] The server generates an alert message if an abnormality is detected.

[1895] The alert message includes detailed information about the anomaly (time, location, type) and the results of sentiment analysis.

[1896] Step 8:

[1897] The server generates an alert message and sends it to the relevant parties (security company, police, fire department, etc.).

[1898] Checks whether the message was sent successfully and attempts to send it again if it fails.

[1899] Terminal handling

[1900] Step 1:

[1901] The terminal receives video and audio data in real time from the connected surveillance cameras and audio collection devices.

[1902] The received data is temporarily stored and formatted for transfer to the server.

[1903] Step 2:

[1904] The terminal transmits the received data to the server in real time.

[1905] Check the data for completeness and attempt to resend any missing data.

[1906] Step 3:

[1907] The device receives anomaly detection and emotion analysis alert messages sent from the server.

[1908] Verify the accuracy of received alerts and prepare to notify users.

[1909] Step 4:

[1910] Your device will display the alert as a pop-up message or application notification.

[1911] Different notification methods (screen display, audio alarm, vibration alarm, etc.) are provided depending on the level of the alert.

[1912] User Action

[1913] Step 1:

[1914] The user sets the installation locations of the surveillance cameras and audio collection devices and checks the connection status.

[1915] The user configures system notification settings (e.g., email, SMS, app notifications).

[1916] Step 2:

[1917] Users can check the system status in real time and monitor whether it is operating normally.

[1918] It is also possible to manually test the anomaly detection and sentiment analysis features.

[1919] Step 3:

[1920] The user receives the abnormality notification from the terminal and checks the content of the notification.

[1921] Check the detailed information about the anomaly and the results of the emotion analysis, and if necessary, rush to the scene or notify the relevant authorities.

[1922] Example scenario

[1923] For example, suppose there are surveillance cameras and audio collection devices installed in a shopping district at night. If a store hears the sound of glass breaking late at night and a panicked user screams inside the store at the same time, the audio collection device picks up the sound and the surveillance camera captures the video. These data are sent to a server in real time.

[1924] The server analyzes the received data and detects the sound of glass breaking and a scream. At the same time, the emotion engine recognizes the user's emotion as "anxiety." After detection, the server generates an alert message and sends it to the security company with the following content: "01:23 AM, Store A, sound of glass breaking detected, emotion of anxiety."

[1925] The device receives this alert and notifies the security guard as a pop-up message. The user (security guard) who sees the notification can immediately rush to the scene to prevent the damage from spreading. This system enables early detection of crimes and accidents and a rapid response, improving social safety.

[1926] According to the embodiment of the present invention, the efficiency and accuracy of monitoring work can be improved, and in particular, safety can be further enhanced by making it possible to take into account the emotions of users.

[1927] Example 2

[1928] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1929] Modern surveillance systems require improved anomaly detection accuracy and rapid response, but conventional systems perform video and audio analysis separately, resulting in slow anomaly detection times and reduced accuracy. Furthermore, systems are unable to respond to user emotions, making it difficult to respond appropriately based on specific emotions. Furthermore, when an anomaly is detected, the alert content is limited and detailed information is lacking, making it difficult for relevant parties to respond quickly and appropriately.

[1930] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving data in real time from the monitoring camera and the audio collecting device, a data analysis means for processing the received data frame by frame and including an algorithm for performing video analysis and audio analysis in parallel, a means for detecting abnormal behavior or sound from the analyzed data, an emotion analysis means for recognizing the emotion of the user, and a means for generating an alert and notifying relevant parties when an abnormality or a specific emotion is detected. This improves the accuracy of anomaly detection and the speed of response, allowing relevant parties to respond quickly and appropriately based on detailed information.

[1931] A "surveillance camera" is a device that captures video data in real time and captures the situation at a monitored location.

[1932] An "audio collection device" is a device that collects surrounding audio data in real time and uses it for monitoring purposes.

[1933] An "algorithm that processes data frame by frame and performs video and audio analysis in parallel" is a computer program that divides video data into its constituent frames and performs analysis of the video and audio data simultaneously.

[1934] "Means for detecting abnormal behavior or sounds" refers to a system that has the ability to analyze collected data and identify pre-defined abnormal behavior or sound patterns.

[1935] "Emotion analysis means for recognizing user emotions" refers to an algorithm or model for determining a user's emotional state (e.g., anger, anxiety, sadness) from audio tone or video data.

[1936] "Means for generating alerts and notifying relevant parties" refers to a system that creates warning messages based on detected abnormalities and emotional information and sends them to relevant agencies (e.g., security companies, police, fire departments, etc.).

[1937] "Detailed information (time, location, type)" is specific information about the detected abnormal event, including the time of occurrence, the location of occurrence, and the type of abnormality.

[1938] "Means for checking the system status in real time" refers to an interface that allows a user to monitor the current operating status of the data collection device and analysis server in real time and check whether they are functioning normally.

[1939] The system based on this invention is a comprehensive monitoring system that receives data in real time from surveillance cameras and audio collection devices, detects abnormal events, and notifies relevant parties. In particular, by combining it with an emotion analysis engine that recognizes user emotions, the accuracy of abnormality detection and the speed of response can be improved. This system is composed of three entities: a server, a terminal, and a user.

[1940] Server Processing

[1941] The server first receives data in real time from surveillance cameras and audio collection devices. The received data is temporarily stored and transcoded for real-time processing. The server then loads a pre-trained AI model. This AI model includes algorithms for detecting abnormal sounds and behaviors, as well as algorithms for analyzing user emotions. The data is divided into frames, and video and audio analysis is performed in parallel. For example, frameworks such as TensorFlow and PyTorch are used. Abnormal behaviors and sounds are detected from the analyzed data, and an emotion analysis engine is used to identify the user's emotion (e.g., anger, anxiety, sadness, etc.). If an anomaly or emotion is detected, an alert message is generated and sent to relevant parties, along with detailed information about the anomaly (time, location, type) and the results of the emotion analysis.

[1942] Terminal handling

[1943] The device receives video and audio data from surveillance cameras and audio collection devices in real time. The received data is temporarily stored and formatted for transmission to the server. The formatted data is sent to the server in real time, supporting rapid data analysis on the server side. The device receives anomaly detection and emotion analysis alert messages sent from the server and checks their accuracy. The received alerts are displayed as pop-up messages or application notifications, allowing the user to respond quickly. Different notification methods (screen display, audio alarm, vibration alarm, etc.) are provided depending on the level of the alert.

[1944] User Action

[1945] First, the user sets the installation locations of the surveillance cameras and audio collection devices and checks the connection status. They also configure notification settings (e.g., email, SMS, app notifications). Users can check the system status in real time and monitor whether it is operating normally. They can also manually test the anomaly detection and emotion analysis functions. When a user receives an anomaly notification from their device, they can check the notification content and confirm the details of the anomaly and the results of the emotion analysis. If necessary, they can rush to the scene or notify the relevant authorities (police, fire department, etc.).

[1946] Example scenario

[1947] For example, consider a surveillance camera and audio collection device installed in a shopping district at night. Suppose a store hears the sound of glass breaking late at night, and simultaneously, a panicked user screams inside the store. The audio collection device picks up the sound, and the surveillance camera captures the video. These data are sent to a server in real time, which analyzes the received data. When the server detects the sound of glass breaking and the scream, the emotion engine recognizes the user's emotion as "anxiety." After detection, the server generates an alert message stating "1:23 AM, Store A, sound of glass breaking detected, emotion of anxiety," and sends it to the security company. The device receives this alert and notifies the security guard via a pop-up message. The user (security guard) immediately rushes to the scene after confirming the notification and prevents further damage. This system enables early detection and rapid response to crimes and accidents, improving social safety.

[1948] Example prompts for generative AI models

[1949] "Receive data in real time from surveillance cameras and audio collection devices, and generate an alert message if an anomaly is detected. For example, imagine a case where a glass breaks in a store at night, causing a panicked user to scream. Include relevant details (time, location, type) and the results of emotion analysis (anxiety)."

[1950] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1951] Step 1: Data collection

[1952] Terminal

[1953] 1. Receive video and audio data in real time from surveillance cameras and audio collection devices.

[1954] Input: Real-time data from surveillance cameras and audio collection devices.

[1955] Specific operation: The surveillance camera captures video data at 30 frames per second, and the audio collection device collects audio data at 44.1 kHz per second.

[1956] Output: Temporarily stored video and audio data.

[1957] Step 2: Data preparation and transfer

[1958] Terminal

[1959] 1. The received video and audio data is temporarily stored and formatted for transfer to the server.

[1960] Input: Temporarily stored video and audio data.

[1961] Specific operation: Frames of video data are stored in a buffer, and audio data is stored as chunks in certain ranges.

[1962] Output: The formatted data.

[1963] 2. Send the formatted data to the server in real time.

[1964] What it does: Streams data packets to a server using the TCP / IP protocol.

[1965] Step 3: Data analysis

[1966] server

[1967] 1. Split the data stream to analyze the received data frame by frame.

[1968] Input: Formatted data transferred to the server.

[1969] Specific operation: Video frames are extracted from the data stream, and the audio data is processed as is.

[1970] Output: Data divided into frames.

[1971] 2. Load the pre-trained AI model and prepare it for anomaly detection and sentiment analysis.

[1972] Specific operation: Load an anomaly detection model into memory using TensorFlow or PyTorch.

[1973] Step 4: Anomaly detection and sentiment analysis

[1974] server

[1975] 1. Video analysis: Analyze each frame of received video data to detect abnormal behavior.

[1976] Input: Video data divided into frames.

[1977] Specific operation: When the movement between frames changes drastically, an algorithm is applied that considers this to be abnormal behavior.

[1978] Output: Anomalous behavior detection results.

[1979] 2. Audio analysis: Spectral analysis of audio data is performed to detect abnormal sounds (such as the sound of glass breaking or a person screaming).

[1980] Input: Audio data divided into frames.

[1981] Specific behavior: Identify sudden changes in sound pressure level and specific frequency patterns.

[1982] Output: Abnormal sound detection result.

[1983] 3. Emotion analysis: Analyzes the tone of voice and facial expressions to identify the user's emotions.

[1984] Input: Received video and audio data.

[1985] Specific operation: Identify emotions by detecting changes in voice tone, pitch, and facial expressions in video.

[1986] Output: Emotion identification results.

[1987] Step 5: Generate and send an alert

[1988] server

[1989] 1. If an anomaly or a specific emotion is detected, an alert message is generated, containing detailed information about the anomaly (time, location, type) and the results of emotion analysis.

[1990] Input: Abnormal behavior, abnormal sound, and emotion analysis results.

[1991] Specific operations: Integrates and converts data, and generates alert messages.

[1992] Output: The alert message.

[1993] 2. Send the generated alert message to the relevant parties.

[1994] What it does: Deliver alerts via SMS, email, and app notifications.

[1995] Step 6: Receive alerts and notifications

[1996] Terminal

[1997] 1. Receive an alert message sent from the server.

[1998] Input: The alert message from the server.

[1999] Specific operation: Receive alert information via HTTPS encrypted communication.

[2000] Output: The received alert message.

[2001] 2. Review the accuracy of the alerts received and make arrangements to notify users.

[2002] Specific operation: Analyzes the message content and converts it into a display format.

[2003] 3. Display the alert to the user as a popup message or app notification.

[2004] Specific behavior: Use different notification methods such as popup, audio alarm, vibration, etc.

[2005] Step 7: Responding to abnormality notifications

[2006] User

[2007] 1. Receive an abnormality notification from the device and check the content of the notification.

[2008] Input: Alert notification from the terminal.

[2009] Specific operation: Display and confirm detailed information about anomalies and the results of sentiment analysis.

[2010] 2. If necessary, rush to the scene and notify the relevant authorities.

[2011] Specific actions: Decide on a response promptly based on the information confirmed.

[2012] (Application example 2)

[2013] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2014] Conventional monitoring systems monitor a huge amount of data and perform analysis to detect abnormalities, but are unable to analyze the user's emotional state when an abnormality occurs, making it difficult to respond quickly and appropriately.The present invention aims to enable faster and more appropriate responses by analyzing user emotional data in real time in addition to monitoring data and generating alerts including the results of emotional analysis when an abnormality occurs.

[2015] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2016] In this invention, the server includes means for receiving data in real time from the surveillance camera and the audio collecting device, means for analyzing the received data and detecting a predetermined abnormal sound, means for generating and reporting an alert when an abnormal sound is detected, means for analyzing the user's emotions, and means for generating and reporting an alert message including the emotion analysis result. This makes it possible to analyze the user's emotional state at the same time as detecting an abnormality and generate a detailed alert including emotional information.

[2017] A "surveillance camera" is a device that captures images of a location in real time and records or transmits the data.

[2018] An "audio collection device" is a device for collecting audio data in an environment in real time and recording or transmitting that data.

[2019] "Means for receiving data in real time" refers to technical means for receiving data almost simultaneously with its generation.

[2020] "Means for analyzing received data" refers to technical means for extracting information or detecting anomalies from data received in real time using specific algorithms.

[2021] "Means for detecting predetermined abnormal sounds" refers to technical means for detecting specific abnormal sounds (e.g., the sound of breaking glass, screams, etc.) that are defined in advance.

[2022] "Means for generating and reporting an alert when an abnormal sound is detected" refers to the technical means for generating a warning message or notification when an abnormal sound is detected and for promptly reporting it to designated parties.

[2023] "Means for analyzing user emotions" refers to technical means for analyzing the user's emotional state (e.g., anger, anxiety, sadness, etc.) from video and audio data.

[2024] "Means for generating and notifying an alert message including the result of emotion analysis" refers to a technical means for generating a detailed alert message including the result of the user's emotion analysis and promptly notifying relevant parties.

[2025] System configuration

[2026] The system embodying the present invention consists of three main components: a server, a terminal, and a user. The server receives data from the surveillance cameras and audio collection devices, analyzes the data, and detects abnormalities. The terminal transmits the collected data to the server and notifies the user of alert messages from the server. The user performs the initial system setup and responds appropriately when notified of an abnormality.

[2027] Hardware and software used

[2028] The server requires hardware with high-performance data analysis capabilities, and specifically, a server equipped with an NVIDIA GPU is suitable. Machine learning frameworks such as TensorFlow and PyTorch are used for data analysis. The terminals used are devices that users can carry, such as smartphones, tablets, and smart glasses. These devices are equipped with high-resolution cameras and highly sensitive microphones, making it possible to collect data in real time.

[2029] Specific examples

[2030] For example, if a nighttime surveillance system is installed in a shopping mall, surveillance cameras and audio collection devices are installed in each store. These devices collect video and audio data in real time and send it to a server via a terminal. The server analyzes the received data and detects abnormal sounds and behavior.

[2031] If an abnormality is detected, the server analyzes the user's emotions and generates an alert message containing the results of the emotion analysis. For example, a message may be generated stating, "The sound of glass breaking was detected at Store A at 2:00 AM, and an emotion of anxiety was recognized." This alert message is sent to the security guard's smart glasses in real time. The security guard can then check the notification and quickly rush to the scene.

[2032] Generative AI model and prompts

[2033] The system uses a generative AI model to analyze user sentiment. Specific examples of prompts include:

[2034] Example prompt sentence:

[2035] "Detect the sound of glass breaking or a person screaming, and analyze the user's emotions. For example, recognize emotions such as anxiety or anger. If an anomaly is detected, generate an alert containing the anomaly and the results of the emotion analysis, and notify the user."

[2036] This enables the monitoring system to detect abnormalities early and respond quickly and appropriately while taking into account the emotional state of the user. This invention is particularly effective in strengthening security in commercial facilities, public facilities, important facilities, etc.

[2037] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2038] Step 1:

[2039] Data collection (terminal)

[2040] The device (e.g., smartphone, smart glasses) collects data (video and audio) in real time from surveillance cameras and audio collectors. The cameras generate high-resolution video data, and the audio collectors collect audio data with high sensitivity. This data is temporarily stored on the device. The input includes video data from the surveillance cameras and audio data from the audio collectors, and the output is the temporary storage of these data.

[2041] Step 2:

[2042] Data transfer (terminal)

[2043] The device processes the temporarily stored video and audio data (for example, transcoding or compressing) and then transmits it to the server in real time. Here, the video data is encoded into JPEG or H.264 format, and the audio data is encoded into MP3 or AAC format. The temporarily stored raw data is the input, and the transcoded data is sent to the server as the output.

[2044] Step 3:

[2045] Data analysis (server)

[2046] The server receives the video and audio data sent from the device, loads a pre-trained generative AI model, and performs data analysis. The analysis includes detecting abnormal behavior from the video, detecting abnormal sounds from the audio, and analyzing the user's emotions. Specifically, it detects movement and shape changes from the video, and specific frequencies and patterns from the audio. The input is the received transcoded data, and the output is an analysis of the type of abnormality and the emotional state.

[2047] Step 4:

[2048] Anomaly detection and sentiment analysis (server)

[2049] If the server detects an anomaly from the data analysis results, it further analyzes the user's emotions (anger, anxiety, sadness, etc.) using a generative AI model. Specifically, emotions are recognized based on voice tone and facial expression analysis. The input is the preliminary results of the data analysis, and the output is the presence or absence of an anomaly and the user's emotional state.

[2050] Step 5:

[2051] Alert generation and transmission (server)

[2052] The server generates an alert message based on the anomaly and the results of sentiment analysis. The alert message contains detailed information such as the type of anomaly, time, location, and sentiment analysis results. The generated alert message is promptly sent to relevant parties (e.g., security guards, police, fire department, etc.). The inputs are the anomaly and the sentiment analysis results, and the generated alert message is the output.

[2053] Step 6:

[2054] User notification (terminal)

[2055] The terminal receives the alert message sent from the server and notifies the user (e.g., security guard). Notification methods include screen display, audio alarm, and vibration alarm. The alert message is the input, and the notification to the user is the output.

[2056] Step 7:

[2057] Response (user)

[2058] The user receives an alert notification from the device and checks the content of the notification. For example, if the alert content is "The sound of glass breaking was detected at Store A at 2:00 AM, and feelings of anxiety were recognized," the user will immediately rush to the scene and take appropriate action. The input is the alert notification, and the output is the user's response action.

[2059] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2060] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2061] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2062] The e...

Claims

1. means for receiving data in real time from the surveillance cameras and audio collection devices; means for analyzing the received data and detecting a predetermined abnormal sound; A means for generating an alert and notifying relevant personnel when an abnormal sound is detected; A system including:

2. 10. The system of claim 1, The system further comprises means for processing the received data frame by frame and performing video analysis and audio analysis in parallel.

3. 10. The system of claim 1, The system further comprises means for including detailed information about the anomaly in the alert message.

Images