System

Abstract

A system is provided.SOLUTION: A system including means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a movement pattern of a person, means for evaluating the analyzed movement pattern by a generative artificial intelligence model and determining an abnormal behavior, means for notifying a security terminal when an abnormal behavior is detected, means for displaying information of the person by the security terminal that has received the notification, and means for recording data of the abnormal behavior and storing the data for relearning.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Groping is a serious social problem in Japan, often causing fear among women and making it difficult for them to seek help. There is also the risk of false accusations, which cause many people to suffer unjustly. Current surveillance systems and crime prevention measures lack the ability to detect groping incidents in real time, prevent them in advance, and respond appropriately afterward. The objective of this invention is to solve these problems, deter groping incidents, and respond quickly, while also preventing false accusations. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following: a system including: means for receiving video data acquired by a monitoring device in real time; means for analyzing the received video data and tracking a person's movement patterns; means for evaluating the analyzed movement patterns using a generative AI model to determine abnormal behavior; means for notifying a security terminal when abnormal behavior is detected; means for displaying information about the person on the security terminal that received the notification; and means for recording data on the abnormal behavior and saving it for re-learning. This system enables real-time detection and reporting of molestation acts, and further improves the accuracy of the model in the future, which is expected to contribute to deterring molestation acts, responding quickly, and preventing false accusations.

[0006] "Monitoring devices" are devices such as cameras and sensors installed at stations and public facilities to visually record and monitor the surrounding situation.

[0007] "Video data" refers to information in the form of images and videos acquired by a surveillance device, and is digital data that records people and their actions.

[0008] "Real time" refers to the timing in which processing is carried out almost simultaneously with the moment video data is acquired, and indicates a state in which delays are kept to a minimum.

[0009] "Analysis" is a series of processes in which acquired video data is processed using algorithms and artificial intelligence to understand and evaluate its content.

[0010] A "movement pattern" refers to a continuous change in a person's physical movements or behavior, and is a pattern that characterizes a specific behavior.

[0011] A "generative artificial intelligence model" is an artificial intelligence model that is trained based on massive amounts of data and is designed to be able to identify and evaluate human movements and behavior patterns.

[0012] "Evaluation" is the process of determining whether a behavior pattern is abnormal based on the analyzed data.

[0013] "Abnormal behavior" refers to actions or behaviors that are unusual or that pose a particular risk, such as sexual harassment, that may cause harm to people.

[0014] A "security terminal" is an information display device used by security guards to receive real-time notifications and alerts.

[0015] "Notification" refers to the act of transmitting alarms or alert information to security guards when abnormal behavior is detected.

[0016] "Recording" is the process of storing detected abnormal behavior and related data in a database or similar.

[0017] "Relearning" is the process of retraining a generative artificial intelligence model based on newly collected data to improve the model's accuracy. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention is a system for effectively detecting and deterring molestation, and is realized by including a monitoring device, a server, a terminal, and a user (security guard). Specific embodiments of the system are described in detail below.

[0040] System configuration and operation explanation

[0041] Data collection

[0042] The server receives video data in real time from surveillance cameras installed in stations and public facilities. This video data refers to information in the form of images and videos captured by the surveillance cameras. The server continuously receives and processes this data with minimal delay.

[0043] Examples:

[0044] 1. Multiple cameras installed on station platforms monitor passenger movements 24 hours a day.

[0045] 2. The video feed from each camera is sent over the network to a server.

[0046] Behavioral analysis and identification

[0047] The server analyzes the received video data and tracks the movement patterns of the individuals. This is done using an algorithm that sequentially detects and identifies multiple individuals. Specifically, image analysis technology (e.g., OpenPose or DeepSort) is used. The analyzed movement patterns are input into a generative artificial intelligence model, which evaluates whether there is any abnormal behavior. This model has learned the characteristics of molestation and determines suspicious behavior based on specific movement patterns.

[0048] Examples:

[0049] 1. The server detects that a specific person is moving their hands unnaturally in a video frame.

[0050] 2. The generative artificial intelligence model evaluates the behavior as matching the characteristics of molestation.

[0051] Reporting and monitoring

[0052] If any abnormal behavior is detected, the server notifies the security terminal. The security terminal is an information display device used by security guards to receive real-time notifications and warnings. This notification includes the relevant video frame and related information. The security guard (user) checks the information displayed on the terminal and rushes to the scene to understand the situation.

[0053] Examples:

[0054] 1. The server detects behavior that is suspected to be sexual harassment and sends an alert to the security terminal.

[0055] 2. The device will pop up a video of the suspicious activity and issue an audio alert to the security guard.

[0056] 3. Based on the information on the terminal, security guards rush to the area and check the situation.

[0057] Record and Relearn

[0058] The abnormal behavior data is recorded by the server and stored in a database for re-learning. This data is used to retrain the generative AI model, helping to improve its accuracy. Re-training the generative AI model with newly added data continuously improves the effectiveness of the system.

[0059] Examples:

[0060] 1. The server stores the video data of the detected abnormal behavior and the judgment results in a database.

[0061] 2. Periodically, retrain the generative AI model based on the stored data to improve the model's performance.

[0062] summary

[0063] This system analyzes video data from surveillance equipment in real time and uses a generative AI model to evaluate behaviors characteristic of molestation, thereby detecting and reporting suspicious behavior in real time. Furthermore, the generative AI model can be retrained based on recorded data, enabling continuous improvement in the effectiveness of the system. This is expected to deter molestation and enable rapid response, and also contribute to preventing false accusations.

[0064] The processing flow will be explained below.

[0065] Step 1:

[0066] The server receives video data from the surveillance equipment in real time. A communication protocol (e.g., RTSP) is used to establish continuous data streaming between the surveillance camera and the server. The received video data is temporarily stored in a buffer to minimize delays.

[0067] Step 2:

[0068] The server analyzes the received video data and uses video analysis algorithms (e.g., OpenPose or DeepSort) to identify people in each frame and track their movement patterns. This allows each person's movements to be continuously tracked and their location and movements recorded in a database.

[0069] Step 3:

[0070] The server inputs the analyzed behavior patterns into a generative AI model. The generative AI model evaluates the behavior patterns based on pre-learned characteristics of molestation and determines whether the behavior is abnormal. The evaluation results are calculated as a score, and if it exceeds a certain threshold, it is deemed suspicious.

[0071] Step 4:

[0072] If an abnormal behavior is detected, the server notifies the security terminal. The notification includes the relevant video frame, the detected behavior pattern, and an evaluation score. This information is sent to the security terminal in real time.

[0073] Step 5:

[0074] The terminal receives notifications from the server and displays them to the security guard. The terminal's user interface displays a pop-up with the video frame where the abnormal behavior was detected and related information, and an audio alert is issued, allowing the security guard to quickly identify the nature of the abnormality.

[0075] Step 6:

[0076] The user (security guard) rushes to the scene based on the information notified from the device. After arriving at the scene, the user checks the actual situation and takes appropriate action, such as intervening or guiding the guardian, if necessary. The user may also collect further evidence based on the situation.

[0077] Step 7:

[0078] The server records the detected anomalous behavior and the response results. This data is stored in a database and used to retrain the generative AI model at a later date. The recorded data includes details of the anomalous behavior, the date and time, the location, and the response results.

[0079] Step 8:

[0080] The server periodically uses the recorded data to retrain the generative AI model. The model is retrained based on newly accumulated case data, improving its accuracy. This retraining process makes it possible to continuously improve the effectiveness of the entire system.

[0081] Example 1

[0082] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0083] Molestation incidents that occur in crowded places such as public facilities and train stations cause great pain to victims and have become a social problem. Current surveillance systems have difficulty detecting molestation incidents quickly and accurately and taking appropriate action in a timely manner. Furthermore, reliable data analysis is necessary to prevent false accusations. Therefore, there is a need for the development of a system that can detect molestation incidents in real time and enable security guards to respond quickly.

[0084] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0085] In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for converting the received video data into an appropriate format, means for analyzing the received video data to track a person's movement patterns, means for evaluating the analyzed movement patterns using a generative artificial intelligence model to identify suspicious behavior, means for notifying a security terminal when suspicious behavior is detected, means for the security terminal that receives the notification to display information about the person in question, and means for recording data on the suspicious behavior and saving it for re-learning. This enables quick and accurate detection of molestation and enables security guards to respond promptly.

[0086] A "monitoring device" is a hardware device that acquires video data and supplies it to a monitoring system.

[0087] "Video data" refers to digital information of images and videos captured by a monitoring device.

[0088] A "server" is a computer that receives, analyzes, evaluates, and notifies video data, and manages the entire system.

[0089] The "appropriate format" refers to video data converted into a predetermined format and resolution to facilitate analysis and evaluation.

[0090] A "motion pattern" refers to a series of patterns of a person's movements or actions, including specific movements and positional transformations.

[0091] A "generative artificial intelligence model" is a model that includes algorithms that learn behavioral patterns and use them to identify suspicious behavior.

[0092] "Suspicious behavior" refers to specific abnormal behavior that differs from normal behavior, including acts of sexual harassment.

[0093] A "security terminal" is an information display device used by security guards to receive notifications from the system.

[0094] A "notification" is an alert or message sent from the server to a security terminal when suspicious behavior is detected.

[0095] "Information on the individual in question" refers to information about an individual who is suspected of engaging in suspicious behavior, including video footage and related data.

[0096] "Recording data" refers to saving detected suspicious behavior and related video data for later analysis and learning.

[0097] "Retraining" is the process of using recorded data to update a generative artificial intelligence model and improve its accuracy and effectiveness.

[0098] The present invention is a system for effectively detecting and deterring molestation, and is realized by including a monitoring device, a server, a terminal, and a user (security guard). Specific embodiments of the system are described in detail below.

[0099] Data collection

[0100] The server receives video data in real time from surveillance cameras installed in public facilities and stations. This video data includes image and video information captured by the surveillance cameras, and is continuously received and processed with minimal delay. A network communication protocol such as RTSP (Real-Time Streaming Protocol) is used to receive the video data. The server uses a video processing library such as FFmpeg to convert the received video data into a processable format.

[0101] Examples:

[0102] 1. The server receives the video sent from the camera installed on the station platform using the RTSP protocol and converts the frame rate to one per second using FFmpeg.

[0103] Behavioral analysis and identification

[0104] The server analyzes specific behavioral patterns from the received video data. This analysis uses an object detection algorithm (e.g., YOLOv4). It detects the position of the person in each received frame and records their coordinate information. The analyzed behavioral patterns of the person are then input into a generative AI model to identify suspicious behavior. The generative AI model is trained using machine learning platforms such as TensorFlow and learns the characteristics of molestation behavior.

[0105] Examples:

[0106] 1. The server detects people using YOLOv4 for each received video frame and records their coordinate information.

[0107] 2. The server uses a generative AI model (e.g., a TensorFlow model) to evaluate the behavioral patterns and calculate an anomaly score.

[0108] Reporting and monitoring

[0109] If abnormal behavior is detected, the server notifies the security terminal. The security terminal is an information display device used by security guards to receive real-time notifications and warnings. This notification includes the relevant video frame and related information. The terminal displays the notification content in a pop-up display and issues an audio alert to alert the security guard. The security guard then begins responding to the situation based on the displayed information.

[0110] Examples:

[0111] 1. The server sends the frame in which abnormal behavior was detected to the security terminal via Firebase Cloud Messaging.

[0112] 2. The device will display a pop-up message with the notification content and play an audio alert to alert the security guard.

[0113] Record and Relearn

[0114] The server records abnormal behavior data in a database (e.g., MySQL) for further analysis and retraining. This data is used to retrain the generative AI model, contributing to improving its accuracy. Retraining is performed periodically to improve the model's performance based on the latest data.

[0115] Examples:

[0116] 1. The server stores the video data of abnormal behavior and the judgment results in a database.

[0117] 2. The server periodically retrains the TensorFlow model using the recorded data.

[0118] Example prompts to input to the generative AI model

[0119] "Based on the video data obtained from the surveillance cameras, please evaluate whether certain patterns of behavior constitute molestation."

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Step 1:

[0122] The server receives video data from the surveillance camera in real time. The input here is streaming data sent from the surveillance camera using the RTSP protocol. The output is video data broken down into frames within the server. The server uses FFmpeg to convert this data into a format suitable for analysis, adjusting it to an appropriate frame rate.

[0123] Specific behavior:

[0124] The server receives the RTSP stream and uses FFmpeg to decompose the video frame by frame.

[0125] Adjust the frame rate to 1 frame per second.

[0126] Step 2:

[0127] The server analyzes the video data broken down into frames and tracks the movement patterns of people. The input is the transformed video frames, and the output is the coordinate information of the detected people. An object detection algorithm (e.g., YOLOv4) is used to identify people in each frame and record their positions.

[0128] Specific behavior:

[0129] The server calls YOLOv4 to detect people in each frame.

[0130] The coordinate information of the detected person is recorded in a database.

[0131] Step 3:

[0132] The server inputs the detected person's behavior data into a generative AI model to identify suspicious behavior. The input here is each person's behavior data, and the output is the evaluation result of the behavior pattern (anomaly score). The generative AI model (e.g., TensorFlow model) has learned the characteristics of molestation acts and makes an evaluation based on the input behavior pattern.

[0133] Specific behavior:

[0134] The server runs the TensorFlow model and analyzes the behavioral patterns.

[0135] An anomaly score is calculated for each movement pattern and the results are temporarily saved.

[0136] Step 4:

[0137] If the server detects suspicious behavior, it notifies the security terminal. The input is the anomaly score, and the output is the notification information sent to the security terminal. The notification includes the specific video frame and related data. A notification protocol (e.g., Firebase Cloud Messaging) is used.

[0138] Specific behavior:

[0139] The server evaluates the anomaly score and generates a notification if the threshold is exceeded.

[0140] Send notifications to security devices using Firebase Cloud Messaging.

[0141] Step 5:

[0142] The terminal receives the notification and displays the relevant video and related information to the security guard. The input here is the notification data to the security terminal, and the output is an alert display to the security guard. The terminal displays the notification content as a pop-up and issues an audio alert.

[0143] Specific behavior:

[0144] The device analyzes the notification received and displays it as a pop-up on the screen.

[0145] The specified audio file will be played to alert the security guard.

[0146] Step 6:

[0147] The server records suspicious behavior data in a database and stores it for retraining. The input is video data of abnormal behavior and the evaluation results, and the output is an updated retraining dataset. The data is periodically used for retraining.

[0148] Specific behavior:

[0149] The server stores the video frames of abnormal behavior and the evaluation results in a database.

[0150] Periodically extract data for re-learning and retrain the generative AI model.

[0151] Step 7:

[0152] The server uses the recorded data to retrain the generative AI model. The input here is the abnormal behavior data stored in the database, and the output is an updated generative AI model. Retraining is performed efficiently using batch processing.

[0153] Specific behavior:

[0154] The server retrieves previously stored data in batches and retrains the generative AI model.

[0155] Apply the updated model to improve the accuracy of the system.

[0156] (Application example 1)

[0157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0158] Molestation and other suspicious behavior in public places are serious problems that threaten the safety of users and cause anxiety throughout society. Current surveillance systems alone are insufficient to address this issue, and more effective surveillance and rapid response are needed. In particular, a system is needed that allows users on-site to detect and respond to situations in real time. The present invention aims to provide a system that combines real-time surveillance and instant reporting functions using smartphones to address these issues.

[0159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0160] In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a person's movement patterns, and means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior. This allows a security terminal to be notified when abnormal behavior is detected. The server also includes means for the security terminal that receives the notification to display information about the person in question and means for recording and saving the abnormal behavior data for re-learning. The server also includes means that operate as an application installed on a smartphone, monitor the safety status around the user, issue a real-time warning when suspicious behavior is detected, and means for transmitting image data of the corresponding frame to the server when suspicious behavior is confirmed and implementing security measures. This improves safety in public places and enables rapid detection and response to molestation and other suspicious behavior.

[0161] "Monitoring equipment" is a general term for cameras and sensor devices installed to acquire video data in real time.

[0162] "Video data" refers to information in the form of images and videos captured by a surveillance device.

[0163] "Real-time" means capturing and processing data immediately, with little or no delay.

[0164] "Analysis" refers to the process of examining the content of the received video data and tracking the movement patterns of people.

[0165] A "motion pattern" refers to a characteristic sequence of a person's movements or behavior.

[0166] A "generative artificial intelligence model" is an AI model trained using a large amount of learning data to perform specific pattern recognition and predictions.

[0167] "Abnormal behavior" refers to specific suspicious behavior that differs from normal behavior, and includes, in particular, sexual harassment.

[0168] A "security terminal" is an information display device used by security guards to receive real-time notifications and warnings.

[0169] "Notification" refers to the transmission of information to notify of the discovery of anomalous behavior.

[0170] "Information about a person in question" refers to information about a person who has been determined to be engaging in abnormal behavior through analysis.

[0171] "Recording" means storing data on abnormal behavior.

[0172] "Retraining" refers to the process of retraining an existing AI model using new data.

[0173] A "smartphone" is a mobile device that not only has telephone functions but also has advanced computing capabilities and can run a variety of applications.

[0174] "Application" refers to a software program that runs on a smartphone or other device.

[0175] A "server" refers to a computer system that provides data and services available to multiple users.

[0176] "Surrounding safety conditions" refers to dangerous or suspicious situations that may occur around the user.

[0177] "Warning" refers to information issued to the user to alert them when suspicious behavior is confirmed.

[0178] "Security response" refers to safety measures and response actions taken after abnormal behavior is detected.

[0179] The present invention provides a system for effectively detecting and quickly responding to acts of molestation and other suspicious behavior. The system includes a monitoring device, a server, a security terminal, and a user (security guard).

[0180] System Configuration

[0181] monitoring device

[0182] Surveillance equipment consists of cameras and sensor devices that capture video data. These devices are installed in public places, train stations, and other locations to monitor people's movements 24 hours a day.

[0183] server

[0184] The server receives video data from the monitoring devices in real time via the network. The received data is processed with minimal delay. The server analyzes the data and detects abnormal behavior in the following steps:

[0185] 1. Data Reception

[0186] Video data transmitted from a monitoring device is received in real time.

[0187] 2. Behavioral analysis and identification

[0188] The captured video data is analyzed to track a person's movement patterns using image analysis technologies such as OpenCV. The analyzed movement patterns are input into a generative artificial intelligence model, which evaluates abnormal behavior.

[0189] 3. Detecting Abnormal Behavior

[0190] The generative AI model determines whether a specific suspicious behavior is abnormal, including groping, and is pre-trained to identify the characteristics of groping.

[0191] 4. Notifications and Warnings

[0192] If any abnormal activity is detected, the server notifies the security terminal, which then alerts the security guard in real time and displays the relevant video frame and related information.

[0193] 5. Recording and Retraining Data

[0194] Data on detected anomalous behavior is recorded and stored in a database for retraining, which is used to retrain the generative AI model and help improve its accuracy.

[0195] Security terminal

[0196] The security terminal is a device that displays a pop-up video of the suspicious activity and issues an audio alert to security guards, who then rush to the area based on the information on the terminal and check the situation on the spot.

[0197] User (security guard)

[0198] Security guards receive notifications from security terminals, rush to the scene, quickly assess the situation, and respond accordingly.

[0199] Smartphone application

[0200] The system also operates as an application installed on a smartphone, which monitors the safety status of the user's surroundings and issues real-time alerts if suspicious behavior is detected.

[0201] 1. Real-time monitoring

[0202] Using the smartphone camera, video data of the surrounding area is collected and sent to a server.

[0203] 2. Notifications and Warnings

[0204] If any suspicious activity is detected, the application will alert the user in real time and send the image data of the relevant frame to the server.

[0205] 3. Security

[0206] The server receives the relevant data and immediately implements security measures.

[0207] Hardware / Software Used

[0208] Hardware: Smartphone (built-in camera, internet connection)

[0209] Software: OpenCV (image processing), Keras (generative AI model), Requests (server communication)

[0210] Specific examples

[0211] For example, when using a smartphone in a public place, the camera automatically analyzes the surrounding video data and if it detects any unusual activity, it displays a real-time warning saying "Suspicious activity detected." The smartphone then sends the relevant video frame to a server, requesting immediate security action. An example of a prompt is as follows:

[0212] "Detect if people in an image are moving unnaturally."

[0213] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0214] Step 1:

[0215] The server receives video data from the monitoring device in real time. In this process, the image and video data captured by the monitoring device are received in streaming format. The input is the video data from the monitoring device, and the output is the video data stored in the server's temporary storage device.

[0216] Step 2:

[0217] The server analyzes the received video data and tracks the person's movement patterns. This process uses OpenCV's image analysis technology. The input is the video data stored in temporary storage, and the movement and position information of the person is extracted using a movement analysis algorithm. The output is the analyzed movement pattern data.

[0218] Step 3:

[0219] The server inputs the analyzed behavior patterns into a generative AI model for evaluation. This generative AI model is trained with Keras and is designed to identify behavior patterns specific to molestation. The input is the analyzed behavior pattern data, and the AI ​​model determines whether abnormal behavior has been detected. The output is the presence or absence of abnormal behavior.

[0220] Step 4:

[0221] When abnormal behavior is detected, the server sends a notification to the security terminal. The notification includes the relevant video frame and related information (e.g., the date, time, and location of the detection). The input is the video data related to the abnormal behavior determination result, and the output is the notification data sent to the security terminal.

[0222] Step 5:

[0223] The security terminal displays the person's information based on the received notification. A pop-up display of the video data is also displayed, and an audio alert is also issued. The input is the notification data sent from the server, and the output is the video information and warning provided to the security guard.

[0224] Step 6:

[0225] The user (security guard) rushes to the relevant area based on the information from the security terminal and checks the situation on the scene. The input is the notification and video data from the security terminal, and the output is a prompt response at the scene.

[0226] Step 7:

[0227] The server records abnormal behavior data and stores it for re-learning. This data is used to retrain the generative AI model. The input is video data of abnormal behavior and its judgment results, and the output is data stored in the database for re-learning.

[0228] Step 8:

[0229] The smartphone application monitors the safety status of the user's surroundings and issues real-time alerts if suspicious behavior is detected. It uses a camera to collect video data and transmits it to a server. The input is the video data from the smartphone camera, and the output is an alert for detected suspicious behavior and transmission of the video data to the server.

[0230] Step 9:

[0231] The server receives the relevant data and immediately implements security measures. Security countermeasure instructions are sent to the relevant distribution network. The input is the video data sent from the smartphone application, and the output is notifications and response instructions to each security station.

[0232] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0233] The present invention provides a system that effectively detects molestation and recognizes the user's emotions to enable more appropriate responses. The system includes a monitoring device, a server, a terminal, an emotion engine, and a user (security guard).

[0234] System configuration and operation explanation

[0235] Data collection

[0236] The server receives video data in real time from surveillance cameras installed in stations and public facilities. Video data refers to information in the form of images and videos captured by surveillance cameras. The server continuously receives and processes this data with minimal delay.

[0237] Examples:

[0238] 1. Multiple cameras installed on station platforms monitor passenger movements in real time.

[0239] 2. The video feed from each camera is sent over the network to a server.

[0240] Behavioral analysis and identification

[0241] The server analyzes the received video data and tracks the person's movement patterns. Image analysis techniques (e.g., OpenPose and DeepSort) are used for video analysis. The analyzed movement patterns are input into a generative AI model that has previously learned the characteristics of molestation behavior, which then identifies abnormal behavior.

[0242] Examples:

[0243] 1. The server detects that a particular person is making unnatural hand movements.

[0244] 2. The generative artificial intelligence model evaluates whether the behavior matches the characteristics of molestation.

[0245] Emotion Analysis

[0246] Furthermore, the server uses an emotion engine to analyze the emotions of users (passengers) from the video data. The emotion engine has the following functions:

[0247] Facial expression analysis: Analyzes facial expressions from video data and recognizes specific emotions (e.g., fear, anxiety).

[0248] Vocal analysis: Analyzes the user's vocal changes from audio data to identify emotions.

[0249] Examples:

[0250] 1. The faces of people in the video are analyzed to detect expressions of fear.

[0251] 2. At the same time, the tense voice is analyzed from the audio data collected from the surrounding area.

[0252] Reporting and monitoring

[0253] If abnormal behavior is detected, the server notifies the security terminal. The notification includes the relevant video frame, the detected behavior pattern, and the emotion information identified by the emotion engine. The security terminal displays this information to the security guard in real time, helping them respond quickly.

[0254] Examples:

[0255] 1. The server notifies the terminal that suspected molestation behavior and feelings of fear have been detected.

[0256] 2. The device notifies the security guard of the relevant video and emotional information and issues an audio alert.

[0257] 3. Security personnel will promptly head to the area based on the information provided.

[0258] Record and Relearn

[0259] The server records the detected abnormal behaviors, emotional information, and the corresponding responses. This data is stored in a database and used to retrain the generative AI model. Based on the recorded data, the model is retrained to continuously improve the system's performance.

[0260] Examples:

[0261] 1. The server stores the detected abnormal behavior and emotional information in a database.

[0262] 2. Periodically use the saved data to retrain the generative AI model and update it with new detection algorithms.

[0263] summary

[0264] This system analyzes video data from surveillance equipment in real time to detect molestation. It also uses an emotion engine to analyze the user's facial expressions and vocal emotions, enabling more accurate judgment and response. Furthermore, the generative AI model can be retrained based on the recorded data, continuously improving the system's accuracy and effectiveness.

[0265] The processing flow will be explained below.

[0266] Step 1:

[0267] The server receives video data from the surveillance equipment in real time. A communication protocol (e.g., RTSP) is used to establish continuous data streaming between the surveillance camera and the server. The received video data is temporarily stored in a buffer and processed to minimize delays.

[0268] Step 2:

[0269] The server analyzes the received video data and uses video analysis algorithms (e.g., OpenPose or DeepSort) to detect people in each frame and track their movement patterns. This allows each person's movements to be continuously tracked and their location and movements recorded in detail.

[0270] Step 3:

[0271] The server inputs the analyzed behavior patterns into a generative AI model. The generative AI model evaluates the behavior patterns based on pre-learned characteristics of molestation. The model outputs a score indicating whether the behavior is abnormal, and if it exceeds a certain threshold, it is determined to be abnormal behavior.

[0272] Step 4:

[0273] If abnormal behavior is detected, the server activates the emotion engine. The emotion engine analyzes the facial expressions of the user (passenger) from the video data and detects specific emotions (e.g., fear, anxiety). If necessary, it also analyzes surrounding audio data and identifies emotions from changes in the vocal cords.

[0274] Step 5:

[0275] The server notifies the security terminal of the abnormal behavior determination result and the emotion information identified by the emotion engine. The notification includes the video frame of the abnormal behavior, the evaluation result of the movement pattern, and the emotion information. This information is sent to the security terminal in real time.

[0276] Step 6:

[0277] The terminal receives notifications from the server and displays them to the security guard. The terminal's user interface displays a pop-up message with information about the person's emotions along with any abnormal behavior, and an audio alert is issued. The security guard can immediately check the content of the notification.

[0278] Step 7:

[0279] The user (security guard) rushes to the scene based on the notification from the device. After arriving at the scene, they check the actual situation and take appropriate action, such as intervening or guiding the guardian, if necessary. They also collect additional evidence depending on the situation at the scene.

[0280] Step 8:

[0281] The server records the detected abnormal behaviors and emotional information, as well as the response results. This data is stored in a database and used to retrain the generative AI model at a later date. The recorded data includes details of the abnormal behavior, the date and time, the location, and the response results.

[0282] Step 9:

[0283] The server periodically uses the recorded data to retrain the generative AI model. The model is retrained based on newly accumulated case data to improve its accuracy. Retraining makes it possible to continuously improve the effectiveness of the entire system.

[0284] Example 2

[0285] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0286] Groping in public places is becoming an increasingly serious problem, but conventional surveillance systems have struggled to effectively detect such behavior and respond quickly. Furthermore, analyzing user emotions would improve the appropriateness of responses, but no such systems currently exist. Therefore, there is a need for a system that can quickly and effectively detect groping and respond more appropriately by analyzing user emotions.

[0287] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0288] In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a person's movement patterns, means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior, means for analyzing the user's emotions from the video data and audio data, means for notifying a security terminal when abnormal behavior is detected, means for the security terminal that receives the notification to display information about the person, and means for recording the abnormal behavior and emotion analysis data and saving it for re-learning. This makes it possible to quickly and effectively detect molestation in public places and improve the appropriateness of responses by analyzing the user's emotions.

[0289] "Monitoring equipment" refers to equipment such as cameras installed to acquire video data in real time.

[0290] "Video data" refers to information in the form of images and videos captured by a surveillance device.

[0291] "Server" refers to a computer system for processing and analyzing received video data.

[0292] "Movement patterns" refer to the characteristics detected over a series of frames of a particular person's movements or actions.

[0293] A "generative artificial intelligence model" refers to a machine learning model that is trained in advance using learning data to evaluate and judge specific patterns and behaviors.

[0294] "Emotion engine" refers to an algorithm or system for analyzing video and audio data to identify a user's emotions.

[0295] "Voice data" refers to acoustic information containing the user's voice and is used for emotion analysis.

[0296] "Security terminal" refers to a computer device used to receive notifications of abnormal behavior, emotional information, etc.

[0297] "Abnormal behavior" refers to a person's actions that the generative AI model identifies as groping or other suspicious behavior.

[0298] "Notification" refers to abnormal behavior and related information sent from the server to the security terminal.

[0299] "Recording" refers to the process of storing detected abnormal behavior and emotional information in a database or the like.

[0300] "Relearning" refers to the process of updating and improving a generative AI model based on recorded data.

[0301] "User" refers to a security guard or station user who uses the system.

[0302] The present invention provides a system that effectively detects molestation in public places and recognizes the user's emotions to provide an appropriate response. Specific embodiments of this system will be described below.

[0303] System configuration and operation explanation

[0304] Data collection

[0305] The server receives video data in real time from monitoring devices (cameras) installed in stations and public facilities. This video data includes information in the form of images and videos. For example, multiple cameras installed on a station platform monitor passenger movements in real time, and the video feed is sent to the server via a network.

[0306] Behavioral analysis and identification

[0307] The server analyzes the received video data and tracks the person's movement patterns. This analysis uses image analysis technologies such as OpenPose and DeepSort. The analyzed movement patterns are input into a generative AI model that has previously learned the characteristics of molestation, which then identifies abnormal behavior. As a specific example, the server detects an unnatural hand movement of a specific person, and the generative AI model evaluates that movement as molestation.

[0308] Emotion Analysis

[0309] Furthermore, the server uses an emotion engine to analyze the user's (passenger's) emotions from the video data. The emotion engine has two functions: facial expression analysis and vocal analysis, and recognizes specific emotions (e.g., fear, anxiety). As a specific example, the face of a person in the video is analyzed to detect a fearful expression, and at the same time, a tense voice is detected from the audio data collected from the surrounding area.

[0310] Reporting and monitoring

[0311] The server then sends a summary of the detected abnormal behavior and emotional information to the security terminal. The notification includes the relevant video frame, the detected behavior pattern, and the emotional information identified by the emotion engine. As a specific example, the server detects behavior that may be suggestive of molestation and the emotion of fear, and sends this information to the security terminal. The security terminal then displays the received information to the security guard in real time and issues an audio alert to encourage a prompt response.

[0312] Record and Relearn

[0313] The server stores the detected abnormal behavior, emotional information, and corresponding results in a database. This data is used to retrain the generative AI model. For example, the server stores the detected abnormal behavior and emotional information in a database and periodically uses the stored data to retrain the generative AI model and improve the accuracy of the system.

[0314] Specific examples

[0315] Here is an example of a prompt for this system:

[0316] "Write a program to detect groping on a train platform and identify the fear of the passengers around it."

[0317] "Analyze a person's movements and emotions in real time and generate system prompts to notify security personnel if anything unusual occurs."

[0318] Hardware and software used

[0319] This system uses the following hardware and software:

[0320] Hardware: Surveillance equipment (cameras), servers, security terminals

[0321] Software: OpenPose (image analysis), DeepSort (motion tracking), generative AI model (abnormal behavior detection), emotion engine (facial expression and vocal cord analysis)

[0322] As described above, the system of the present invention can quickly and effectively detect molestation in public places and improve the appropriateness of responses by analyzing the user's emotions.

[0323] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0324] Step 1:

[0325] Receiving video data

[0326] The server receives video data in real time from monitoring devices (cameras) installed at stations and public facilities.

[0327] Input: Video feed from camera

[0328] Output: Received video data

[0329] Specific operation: The server receives video feeds from multiple cameras simultaneously over the network.

[0330] Step 2:

[0331] Video data formatting

[0332] The server formats the received video data at a fixed frame rate and converts it into a format that can be applied to analysis processing.

[0333] Input: Received video data

[0334] Output: Video data formatted for each frame

[0335] Specific operation: The server divides the received data into frames and saves them in a format that can be applied to subsequent analysis processing.

[0336] Step 3:

[0337] Motion detection

[0338] The server detects the movement patterns of each person from the video data, using image analysis technologies such as OpenPose and DeepSort.

[0339] Input: Formatted video data

[0340] Output: Movement patterns for each person

[0341] Specific operation: The server uses OpenPose to identify the person's posture and joint positions for each frame, and uses DeepSort to track their movement patterns.

[0342] Step 4:

[0343] Identifying abnormal behavior

[0344] The server inputs the detected behavioral patterns into a generative AI model and evaluates whether they match the characteristics of molestation.

[0345] Input: Movement pattern

[0346] Output: Abnormal behavior determination result

[0347] Specific actions: The server inputs movement patterns into a pre-trained generative AI model and evaluates whether unnatural movements are judged to be acts of molestation.

[0348] Step 5:

[0349] Emotion Analysis

[0350] The server simultaneously analyzes the user's emotions from the video and audio data, and uses an emotion engine to analyze facial expressions and vocal chords to recognize specific emotions (e.g., fear, anxiety).

[0351] Input: Video and audio data

[0352] Output: User's emotional information

[0353] How it works: The server uses facial recognition technology to detect the faces of people in the video and analyze whether they show any fearful expressions. At the same time, it analyzes the audio data to determine whether the vocal cords are tense.

[0354] Step 6:

[0355] Abnormality notification

[0356] The server compiles the detected abnormal behavior and emotional information and notifies the security terminal.

[0357] Input: Abnormal behavior judgment results and emotional information

[0358] Output: Notification to security terminal

[0359] Specific operation: The server detects movements that are suspected to be molestation and emotions of fear, and sends them to the security terminal.

[0360] Step 7:

[0361] Sending alerts

[0362] The device displays the received information to the security guard and issues an audio alert if necessary.

[0363] Input: Notification from the server

[0364] Output: Display and audio alert to security guards

[0365] Specific operation: The device will sound an alert and display the relevant video and emotional information on the security guard's screen.

[0366] Step 8:

[0367] Data recording

[0368] The server records the detected abnormal behavior and emotion information in a database.

[0369] Input: Abnormal behavior judgment results and emotional information

[0370] Output: Records in the database

[0371] Specific operation: The server stores the detected abnormal behavior and the associated emotional information in a database.

[0372] Step 9:

[0373] Retraining the Model

[0374] The server uses the recorded data to retrain the generative AI model, continuously improving the accuracy of the system.

[0375] Input: Abnormal behavior and emotional information recorded in the database

[0376] Output: Improved generative AI model

[0377] How it works: Periodically, the server retrains the generative AI model with new data to improve its accuracy in identifying new patterns.

[0378] (Application example 2)

[0379] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0380] To ensure the safety of passengers in self-driving vehicles, a system is needed that can immediately detect abnormal behavior, such as sexual harassment or trouble between passengers, and respond as necessary. However, existing systems rely solely on the analysis of video data, and are unable to respond by taking into account changes in the user's emotions or voice data. This makes it difficult to respond effectively and quickly.

[0381] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a person's movement patterns, means for evaluating the analyzed movement patterns using a generative artificial intelligence model and determining abnormal behavior, means for analyzing the user's emotions from the video data and audio data, means for notifying a security terminal when abnormal behavior or emotions such as anxiety or fear are detected, means for the security terminal that receives the notification to display information about the person, and means for recording the abnormal behavior and emotion data and saving it for re-learning. This enables a more effective and rapid response that combines abnormal behavior detection and emotion analysis in an autonomous vehicle.

[0382] "Monitoring equipment" refers to equipment including cameras and microphones installed to capture user behavior patterns and environmental conditions in real time.

[0383] "Video data" refers to information in the form of moving images collected by a monitoring device, and is a record of the user's actions and the environment.

[0384] "Audio data" refers to audio information acquired by a microphone installed in a monitoring device, and includes environmental sounds and the user's voice.

[0385] "Movement pattern" refers to the repetition and characteristics of movements or actions by a particular person.

[0386] A "generative artificial intelligence model" is an artificial intelligence technology that has the ability to learn specific patterns and characteristics based on large amounts of data and then evaluate and judge them.

[0387] "Abnormal behavior" refers to unnatural actions or behavior that differ from the behavior of ordinary passengers, and specifically includes acts of molestation and trouble between passengers.

[0388] "Emotion analysis" is a technology that determines a user's emotions by recognizing facial expressions from video data and analyzing vocal cords from audio data.

[0389] The "security terminal" is a device that receives notifications from the server and displays abnormal behavior and emotion analysis results to security guards in real time.

[0390] "Retraining" is the process of retraining an existing artificial intelligence model based on recorded data to improve its accuracy and responsiveness.

[0391] An "autonomous vehicle" is a vehicle that can perform driving operations automatically using artificial intelligence and sensor technology.

[0392] The system for realizing this application example is based on technology that ensures passenger safety in autonomous vehicles and analyzes abnormal behavior and passenger emotions. The specific configuration and operation of the system are described below.

[0393] System Configuration

[0394] The system consists of the following main components:

[0395] 1. Surveillance equipment: Consists of multiple cameras and microphones installed inside the vehicle. This surveillance equipment collects video and audio data in real time.

[0396] 2. Server: A server equipped with a high-performance NVIDIA GPU. This server processes and analyzes the received data to determine abnormal behavior and emotions.

[0397] 3. Security terminal: A device that receives notifications from the server and displays information to security guards in real time. This can be a tablet or a dedicated mobile device.

[0398] Data processing and analysis

[0399] The server performs the following steps in sequence:

[0400] 1. Data reception: Receives video and audio data sent from the monitoring device in real time.

[0401] 2. Motion Analysis: Video data is analyzed and human movement patterns are identified using OpenPose and DeepSort technologies. These patterns are then fed into a generative AI model to identify abnormal behavior.

[0402] 3. Sentiment Analysis: Additionally, Google Cloud Vision API and Watson Tone Analyzer are used to analyze passenger emotions from video and audio data, thereby identifying emotions such as fear and anxiety.

[0403] 4. Notification and response: If abnormal behavior or specific emotions are detected, the server will send real-time notifications to the security terminal and display the information to the security guards. If necessary, the autonomous vehicle can be controlled via the MQTT protocol.

[0404] Specific examples

[0405] To illustrate, consider the following scenario:

[0406] Detecting conflict between passengers: An in-vehicle camera captures unnatural movements (e.g., raising an arm) between passengers A and B, and the generative AI model determines this behavior as a "fight." Voice data is also analyzed at the same time, and if emergency words such as "help" are detected, the server immediately notifies the security terminal and, if necessary, stops the vehicle.

[0407] Anomaly detection through emotion analysis: If the microphone detects that a passenger's voice is filled with anxiety and the video data identifies a frightened expression, the server will use this information to notify the security terminal and take immediate action to ensure the passenger's safety.

[0408] Prompt Sentence Examples

[0409] The following prompt is an example used to train and initialize a generative AI model:

[0410] "Please collect video and audio data when Passenger A raises his arm."

[0411] "Use the Google Cloud Vision API to recognize fearful facial expressions."

[0412] "Use Watson Tone Analyzer to detect conflicting voice patterns"

[0413] This enables the system to combine abnormal behavior detection and emotion analysis within autonomous vehicles to respond more effectively and quickly.

[0414] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0415] Step 1:

[0416] The server receives video and audio data sent from the monitoring device in real time. Specifically, it acquires the data stream from the camera and microphone and stores it in the server's input buffer. The input is video and audio data, and the output is a data stream for analysis.

[0417] Step 2:

[0418] The server analyzes the received video data using OpenPose and DeepSort technologies to identify the movement patterns of the people. Specifically, it divides the video data into frames and performs pose estimation for each frame. It then performs object tracking based on the estimated pose data. The input is the video data, and the output is the movement patterns of each person.

[0419] Step 3:

[0420] The server inputs the analyzed movement patterns into a generative AI model to determine whether the movement patterns are abnormal. In this step, a pre-trained model is used to evaluate whether the movement patterns are abnormal. Specific operations include converting the movement patterns into specific feature quantities, inputting them into the model, and calculating an abnormality score. The input is the movement pattern, and the output is the result of the abnormal behavior determination.

[0421] Step 4:

[0422] The server performs emotion analysis using video and audio data. The emotion analysis uses Google Cloud Vision API and Watson Tone Analyzer to determine emotions from the user's facial expressions. Specifically, the process involves detecting faces from video frames and recognizing facial expressions, while also inputting audio data into a tone analyzer to identify the emotion of the voice. The input is video and audio data, and the output is emotion data.

[0423] Step 5:

[0424] If the server detects abnormal behavior or a specific emotion, it notifies the security terminal of this data. The notification uses the MQTT protocol to send video frames, identified behavior patterns, and emotion data to the security terminal. Specifically, this involves a process of assembling various data into packets and transmitting them in real time. The input is the abnormal behavior judgment result and emotion data, and the output is notification data sent to the security terminal.

[0425] Step 6:

[0426] When the security terminal receives a notification, it displays the relevant person's information in real time. Specific operations include interpreting the received data and reflecting it on the UI. The terminal screen displays the person's image, movement patterns, and emotion data, allowing security guards to respond immediately. The input is notification data, and the output is display information.

[0427] Step 7:

[0428] The server records the detected abnormal behavior and emotion data and stores it in a database for re-learning. In this step, the collected data is properly formatted and written to a database for long-term storage. Specifically, the abnormal behavior features and emotion data are saved and made available for re-learning at a later date. The input is the abnormal behavior judgment result and emotion data, and the output is the record saved in the database.

[0429] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0430] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0431] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0432] [Second embodiment]

[0433] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0434] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0435] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0436] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0437] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0438] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0439] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0440] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0441] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0442] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0443] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0444] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0445] The present invention is a system for effectively detecting and deterring molestation, and is realized by including a monitoring device, a server, a terminal, and a user (security guard). Specific embodiments of the system are described in detail below.

[0446] System configuration and operation explanation

[0447] Data collection

[0448] The server receives video data in real time from surveillance cameras installed in stations and public facilities. This video data refers to information in the form of images and videos captured by the surveillance cameras. The server continuously receives and processes this data with minimal delay.

[0449] Examples:

[0450] 1. Multiple cameras installed on station platforms monitor passenger movements 24 hours a day.

[0451] 2. The video feed from each camera is sent over the network to a server.

[0452] Behavioral analysis and identification

[0453] The server analyzes the received video data and tracks the movement patterns of the individuals. This is done using an algorithm that sequentially detects and identifies multiple individuals. Specifically, image analysis technology (e.g., OpenPose or DeepSort) is used. The analyzed movement patterns are input into a generative artificial intelligence model, which evaluates whether there is any abnormal behavior. This model has learned the characteristics of molestation and determines suspicious behavior based on specific movement patterns.

[0454] Examples:

[0455] 1. The server detects that a specific person is moving their hands unnaturally in a video frame.

[0456] 2. The generative artificial intelligence model evaluates the behavior as matching the characteristics of molestation.

[0457] Reporting and monitoring

[0458] If any abnormal behavior is detected, the server notifies the security terminal. The security terminal is an information display device used by security guards to receive real-time notifications and warnings. This notification includes the relevant video frame and related information. The security guard (user) checks the information displayed on the terminal and rushes to the scene to understand the situation.

[0459] Examples:

[0460] 1. The server detects behavior that is suspected to be sexual harassment and sends an alert to the security terminal.

[0461] 2. The device will pop up a video of the suspicious activity and issue an audio alert to the security guard.

[0462] 3. Based on the information on the terminal, security guards rush to the area and check the situation.

[0463] Record and Relearn

[0464] The abnormal behavior data is recorded by the server and stored in a database for re-learning. This data is used to retrain the generative AI model, helping to improve its accuracy. Re-training the generative AI model with newly added data continuously improves the effectiveness of the system.

[0465] Examples:

[0466] 1. The server stores the video data of the detected abnormal behavior and the judgment results in a database.

[0467] 2. Periodically, retrain the generative AI model based on the stored data to improve the model's performance.

[0468] summary

[0469] This system analyzes video data from surveillance equipment in real time and uses a generative AI model to evaluate behaviors characteristic of molestation, thereby detecting and reporting suspicious behavior in real time. Furthermore, the generative AI model can be retrained based on recorded data, enabling continuous improvement in the effectiveness of the system. This is expected to deter molestation and enable rapid response, and also contribute to preventing false accusations.

[0470] The processing flow will be explained below.

[0471] Step 1:

[0472] The server receives video data from the surveillance equipment in real time. A communication protocol (e.g., RTSP) is used to establish continuous data streaming between the surveillance camera and the server. The received video data is temporarily stored in a buffer to minimize delays.

[0473] Step 2:

[0474] The server analyzes the received video data and uses video analysis algorithms (e.g., OpenPose or DeepSort) to identify people in each frame and track their movement patterns. This allows each person's movements to be continuously tracked and their location and movements recorded in a database.

[0475] Step 3:

[0476] The server inputs the analyzed behavior patterns into a generative AI model. The generative AI model evaluates the behavior patterns based on pre-learned characteristics of molestation and determines whether the behavior is abnormal. The evaluation results are calculated as a score, and if it exceeds a certain threshold, it is deemed suspicious.

[0477] Step 4:

[0478] If an abnormal behavior is detected, the server notifies the security terminal. The notification includes the relevant video frame, the detected behavior pattern, and an evaluation score. This information is sent to the security terminal in real time.

[0479] Step 5:

[0480] The terminal receives notifications from the server and displays them to the security guard. The terminal's user interface displays a pop-up with the video frame where the abnormal behavior was detected and related information, and an audio alert is issued, allowing the security guard to quickly identify the nature of the abnormality.

[0481] Step 6:

[0482] The user (security guard) rushes to the scene based on the information notified from the device. After arriving at the scene, the user checks the actual situation and takes appropriate action, such as intervening or guiding the guardian, if necessary. The user may also collect further evidence based on the situation.

[0483] Step 7:

[0484] The server records the detected anomalous behavior and the response results. This data is stored in a database and used to retrain the generative AI model at a later date. The recorded data includes details of the anomalous behavior, the date and time, the location, and the response results.

[0485] Step 8:

[0486] The server periodically uses the recorded data to retrain the generative AI model. The model is retrained based on newly accumulated case data, improving its accuracy. This retraining process makes it possible to continuously improve the effectiveness of the entire system.

[0487] Example 1

[0488] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0489] Molestation incidents that occur in crowded places such as public facilities and train stations cause great pain to victims and have become a social problem. Current surveillance systems have difficulty detecting molestation incidents quickly and accurately and taking appropriate action in a timely manner. Furthermore, reliable data analysis is necessary to prevent false accusations. Therefore, there is a need for the development of a system that can detect molestation incidents in real time and enable security guards to respond quickly.

[0490] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0491] In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for converting the received video data into an appropriate format, means for analyzing the received video data to track a person's movement patterns, means for evaluating the analyzed movement patterns using a generative artificial intelligence model to identify suspicious behavior, means for notifying a security terminal when suspicious behavior is detected, means for the security terminal that receives the notification to display information about the person in question, and means for recording data on the suspicious behavior and saving it for re-learning. This enables quick and accurate detection of molestation and enables security guards to respond promptly.

[0492] A "monitoring device" is a hardware device that acquires video data and supplies it to a monitoring system.

[0493] "Video data" refers to digital information of images and videos captured by a monitoring device.

[0494] A "server" is a computer that receives, analyzes, evaluates, and notifies video data, and manages the entire system.

[0495] The "appropriate format" refers to video data converted into a predetermined format and resolution to facilitate analysis and evaluation.

[0496] A "motion pattern" refers to a series of patterns of a person's movements or actions, including specific movements and positional transformations.

[0497] A "generative artificial intelligence model" is a model that includes algorithms that learn behavioral patterns and use them to identify suspicious behavior.

[0498] "Suspicious behavior" refers to specific abnormal behavior that differs from normal behavior, including acts of sexual harassment.

[0499] A "security terminal" is an information display device used by security guards to receive notifications from the system.

[0500] A "notification" is an alert or message sent from the server to a security terminal when suspicious behavior is detected.

[0501] "Information on the individual in question" refers to information about an individual who is suspected of engaging in suspicious behavior, including video footage and related data.

[0502] "Recording data" refers to saving detected suspicious behavior and related video data for later analysis and learning.

[0503] "Retraining" is the process of using recorded data to update a generative artificial intelligence model and improve its accuracy and effectiveness.

[0504] The present invention is a system for effectively detecting and deterring molestation, and is realized by including a monitoring device, a server, a terminal, and a user (security guard). Specific embodiments of the system are described in detail below.

[0505] Data collection

[0506] The server receives video data in real time from surveillance cameras installed in public facilities and stations. This video data includes image and video information captured by the surveillance cameras, and is continuously received and processed with minimal delay. A network communication protocol such as RTSP (Real-Time Streaming Protocol) is used to receive the video data. The server uses a video processing library such as FFmpeg to convert the received video data into a processable format.

[0507] Examples:

[0508] 1. The server receives the video sent from the camera installed on the station platform using the RTSP protocol and converts the frame rate to one per second using FFmpeg.

[0509] Behavioral analysis and identification

[0510] The server analyzes specific behavioral patterns from the received video data. This analysis uses an object detection algorithm (e.g., YOLOv4). It detects the position of the person in each received frame and records their coordinate information. The analyzed behavioral patterns of the person are then input into a generative AI model to identify suspicious behavior. The generative AI model is trained using machine learning platforms such as TensorFlow and learns the characteristics of molestation behavior.

[0511] Examples:

[0512] 1. The server detects people using YOLOv4 for each received video frame and records their coordinate information.

[0513] 2. The server uses a generative AI model (e.g., a TensorFlow model) to evaluate the behavioral patterns and calculate an anomaly score.

[0514] Reporting and monitoring

[0515] If abnormal behavior is detected, the server notifies the security terminal. The security terminal is an information display device used by security guards to receive real-time notifications and warnings. This notification includes the relevant video frame and related information. The terminal displays the notification content in a pop-up display and issues an audio alert to alert the security guard. The security guard then begins responding to the situation based on the displayed information.

[0516] Examples:

[0517] 1. The server sends the frame in which abnormal behavior was detected to the security terminal via Firebase Cloud Messaging.

[0518] 2. The device will display a pop-up message with the notification content and play an audio alert to alert the security guard.

[0519] Record and Relearn

[0520] The server records abnormal behavior data in a database (e.g., MySQL) for further analysis and retraining. This data is used to retrain the generative AI model, contributing to improving its accuracy. Retraining is performed periodically to improve the model's performance based on the latest data.

[0521] Examples:

[0522] 1. The server stores the video data of abnormal behavior and the judgment results in a database.

[0523] 2. The server periodically retrains the TensorFlow model using the recorded data.

[0524] Example prompts to input to the generative AI model

[0525] "Based on the video data obtained from the surveillance cameras, please evaluate whether certain patterns of behavior constitute molestation."

[0526] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0527] Step 1:

[0528] The server receives video data from the surveillance camera in real time. The input here is streaming data sent from the surveillance camera using the RTSP protocol. The output is video data broken down into frames within the server. The server uses FFmpeg to convert this data into a format suitable for analysis, adjusting it to an appropriate frame rate.

[0529] Specific behavior:

[0530] The server receives the RTSP stream and uses FFmpeg to decompose the video frame by frame.

[0531] Adjust the frame rate to 1 frame per second.

[0532] Step 2:

[0533] The server analyzes the video data broken down into frames and tracks the movement patterns of people. The input is the transformed video frames, and the output is the coordinate information of the detected people. An object detection algorithm (e.g., YOLOv4) is used to identify people in each frame and record their positions.

[0534] Specific behavior:

[0535] The server calls YOLOv4 to detect people in each frame.

[0536] The coordinate information of the detected person is recorded in a database.

[0537] Step 3:

[0538] The server inputs the detected person's behavior data into a generative AI model to identify suspicious behavior. The input here is each person's behavior data, and the output is the evaluation result of the behavior pattern (anomaly score). The generative AI model (e.g., TensorFlow model) has learned the characteristics of molestation acts and makes an evaluation based on the input behavior pattern.

[0539] Specific behavior:

[0540] The server runs the TensorFlow model and analyzes the behavioral patterns.

[0541] An anomaly score is calculated for each movement pattern and the results are temporarily saved.

[0542] Step 4:

[0543] If the server detects suspicious behavior, it notifies the security terminal. The input is the anomaly score, and the output is the notification information sent to the security terminal. The notification includes the specific video frame and related data. A notification protocol (e.g., Firebase Cloud Messaging) is used.

[0544] Specific behavior:

[0545] The server evaluates the anomaly score and generates a notification if the threshold is exceeded.

[0546] Send notifications to security devices using Firebase Cloud Messaging.

[0547] Step 5:

[0548] The terminal receives the notification and displays the relevant video and related information to the security guard. The input here is the notification data to the security terminal, and the output is an alert display to the security guard. The terminal displays the notification content as a pop-up and issues an audio alert.

[0549] Specific behavior:

[0550] The device analyzes the notification received and displays it as a pop-up on the screen.

[0551] The specified audio file will be played to alert the security guard.

[0552] Step 6:

[0553] The server records suspicious behavior data in a database and stores it for retraining. The input is video data of abnormal behavior and the evaluation results, and the output is an updated retraining dataset. The data is periodically used for retraining.

[0554] Specific behavior:

[0555] The server stores the video frames of abnormal behavior and the evaluation results in a database.

[0556] Periodically extract data for re-learning and retrain the generative AI model.

[0557] Step 7:

[0558] The server uses the recorded data to retrain the generative AI model. The input here is the abnormal behavior data stored in the database, and the output is an updated generative AI model. Retraining is performed efficiently using batch processing.

[0559] Specific behavior:

[0560] The server retrieves previously stored data in batches and retrains the generative AI model.

[0561] Apply the updated model to improve the accuracy of the system.

[0562] (Application example 1)

[0563] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0564] Molestation and other suspicious behavior in public places are serious problems that threaten the safety of users and cause anxiety throughout society. Current surveillance systems alone are insufficient to address this issue, and more effective surveillance and rapid response are needed. In particular, a system is needed that allows users on-site to detect and respond to situations in real time. The present invention aims to provide a system that combines real-time surveillance and instant reporting functions using smartphones to address these issues.

[0565] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0566] In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a person's movement patterns, and means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior. This allows a security terminal to be notified when abnormal behavior is detected. The server also includes means for the security terminal that receives the notification to display information about the person in question and means for recording and saving the abnormal behavior data for re-learning. The server also includes means that operate as an application installed on a smartphone, monitor the safety status around the user, issue a real-time warning when suspicious behavior is detected, and means for transmitting image data of the corresponding frame to the server when suspicious behavior is confirmed and implementing security measures. This improves safety in public places and enables rapid detection and response to molestation and other suspicious behavior.

[0567] "Monitoring equipment" is a general term for cameras and sensor devices installed to acquire video data in real time.

[0568] "Video data" refers to information in the form of images and videos captured by a surveillance device.

[0569] "Real-time" means capturing and processing data immediately, with little or no delay.

[0570] "Analysis" refers to the process of examining the content of the received video data and tracking the movement patterns of people.

[0571] A "motion pattern" refers to a characteristic sequence of a person's movements or behavior.

[0572] A "generative artificial intelligence model" is an AI model trained using a large amount of learning data to perform specific pattern recognition and predictions.

[0573] "Abnormal behavior" refers to specific suspicious behavior that differs from normal behavior, and includes, in particular, sexual harassment.

[0574] A "security terminal" is an information display device used by security guards to receive real-time notifications and warnings.

[0575] "Notification" refers to the transmission of information to notify of the discovery of anomalous behavior.

[0576] "Information about a person in question" refers to information about a person who has been determined to be engaging in abnormal behavior through analysis.

[0577] "Recording" means storing data on abnormal behavior.

[0578] "Retraining" refers to the process of retraining an existing AI model using new data.

[0579] A "smartphone" is a mobile device that not only has telephone functions but also has advanced computing capabilities and can run a variety of applications.

[0580] "Application" refers to a software program that runs on a smartphone or other device.

[0581] A "server" refers to a computer system that provides data and services available to multiple users.

[0582] "Surrounding safety conditions" refers to dangerous or suspicious situations that may occur around the user.

[0583] "Warning" refers to information issued to the user to alert them when suspicious behavior is confirmed.

[0584] "Security response" refers to safety measures and response actions taken after abnormal behavior is detected.

[0585] The present invention provides a system for effectively detecting and quickly responding to acts of molestation and other suspicious behavior. The system includes a monitoring device, a server, a security terminal, and a user (security guard).

[0586] System Configuration

[0587] monitoring device

[0588] Surveillance equipment consists of cameras and sensor devices that capture video data. These devices are installed in public places, train stations, and other locations to monitor people's movements 24 hours a day.

[0589] server

[0590] The server receives video data from the monitoring devices in real time via the network. The received data is processed with minimal delay. The server analyzes the data and detects abnormal behavior in the following steps:

[0591] 1. Data Reception

[0592] Video data transmitted from a monitoring device is received in real time.

[0593] 2. Behavioral analysis and identification

[0594] The captured video data is analyzed to track a person's movement patterns using image analysis technologies such as OpenCV. The analyzed movement patterns are input into a generative artificial intelligence model, which evaluates abnormal behavior.

[0595] 3. Detecting Abnormal Behavior

[0596] The generative AI model determines whether a specific suspicious behavior is abnormal, including groping, and is pre-trained to identify the characteristics of groping.

[0597] 4. Notifications and Warnings

[0598] If any abnormal activity is detected, the server notifies the security terminal, which then alerts the security guard in real time and displays the relevant video frame and related information.

[0599] 5. Recording and Retraining Data

[0600] Data on detected anomalous behavior is recorded and stored in a database for retraining, which is used to retrain the generative AI model and help improve its accuracy.

[0601] Security terminal

[0602] The security terminal is a device that displays a pop-up video of the suspicious activity and issues an audio alert to security guards, who then rush to the area based on the information on the terminal and check the situation on the spot.

[0603] User (security guard)

[0604] Security guards receive notifications from security terminals, rush to the scene, quickly assess the situation, and respond accordingly.

[0605] Smartphone application

[0606] The system also operates as an application installed on a smartphone, which monitors the safety status of the user's surroundings and issues real-time alerts if suspicious behavior is detected.

[0607] 1. Real-time monitoring

[0608] Using the smartphone camera, video data of the surrounding area is collected and sent to a server.

[0609] 2. Notifications and Warnings

[0610] If any suspicious activity is detected, the application will alert the user in real time and send the image data of the relevant frame to the server.

[0611] 3. Security

[0612] The server receives the relevant data and immediately implements security measures.

[0613] Hardware / Software Used

[0614] Hardware: Smartphone (built-in camera, internet connection)

[0615] Software: OpenCV (image processing), Keras (generative AI model), Requests (server communication)

[0616] Specific examples

[0617] For example, when using a smartphone in a public place, the camera automatically analyzes the surrounding video data and if it detects any unusual activity, it displays a real-time warning saying "Suspicious activity detected." The smartphone then sends the relevant video frame to a server, requesting immediate security action. An example of a prompt is as follows:

[0618] "Detect if people in an image are moving unnaturally."

[0619] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0620] Step 1:

[0621] The server receives video data from the monitoring device in real time. In this process, the image and video data captured by the monitoring device are received in streaming format. The input is the video data from the monitoring device, and the output is the video data stored in the server's temporary storage device.

[0622] Step 2:

[0623] The server analyzes the received video data and tracks the person's movement patterns. This process uses OpenCV's image analysis technology. The input is the video data stored in temporary storage, and the movement and position information of the person is extracted using a movement analysis algorithm. The output is the analyzed movement pattern data.

[0624] Step 3:

[0625] The server inputs the analyzed behavior patterns into a generative AI model for evaluation. This generative AI model is trained with Keras and is designed to identify behavior patterns specific to molestation. The input is the analyzed behavior pattern data, and the AI ​​model determines whether abnormal behavior has been detected. The output is the presence or absence of abnormal behavior.

[0626] Step 4:

[0627] When abnormal behavior is detected, the server sends a notification to the security terminal. The notification includes the relevant video frame and related information (e.g., the date, time, and location of the detection). The input is the video data related to the abnormal behavior determination result, and the output is the notification data sent to the security terminal.

[0628] Step 5:

[0629] The security terminal displays the person's information based on the received notification. A pop-up display of the video data is also displayed, and an audio alert is also issued. The input is the notification data sent from the server, and the output is the video information and warning provided to the security guard.

[0630] Step 6:

[0631] The user (security guard) rushes to the relevant area based on the information from the security terminal and checks the situation on the scene. The input is the notification and video data from the security terminal, and the output is a prompt response at the scene.

[0632] Step 7:

[0633] The server records abnormal behavior data and stores it for re-learning. This data is used to retrain the generative AI model. The input is video data of abnormal behavior and its judgment results, and the output is data stored in the database for re-learning.

[0634] Step 8:

[0635] The smartphone application monitors the safety status of the user's surroundings and issues real-time alerts if suspicious behavior is detected. It uses a camera to collect video data and transmits it to a server. The input is the video data from the smartphone camera, and the output is an alert for detected suspicious behavior and transmission of the video data to the server.

[0636] Step 9:

[0637] The server receives the relevant data and immediately implements security measures. Security countermeasure instructions are sent to the relevant distribution network. The input is the video data sent from the smartphone application, and the output is notifications and response instructions to each security station.

[0638] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0639] The present invention provides a system that effectively detects molestation and recognizes the user's emotions to enable more appropriate responses. The system includes a monitoring device, a server, a terminal, an emotion engine, and a user (security guard).

[0640] System configuration and operation explanation

[0641] Data collection

[0642] The server receives video data in real time from surveillance cameras installed in stations and public facilities. Video data refers to information in the form of images and videos captured by surveillance cameras. The server continuously receives and processes this data with minimal delay.

[0643] Examples:

[0644] 1. Multiple cameras installed on station platforms monitor passenger movements in real time.

[0645] 2. The video feed from each camera is sent over the network to a server.

[0646] Behavioral analysis and identification

[0647] The server analyzes the received video data and tracks the person's movement patterns. Image analysis techniques (e.g., OpenPose and DeepSort) are used for video analysis. The analyzed movement patterns are input into a generative AI model that has previously learned the characteristics of molestation behavior, which then identifies abnormal behavior.

[0648] Examples:

[0649] 1. The server detects that a particular person is making unnatural hand movements.

[0650] 2. The generative artificial intelligence model evaluates whether the behavior matches the characteristics of molestation.

[0651] Emotion Analysis

[0652] Furthermore, the server uses an emotion engine to analyze the emotions of users (passengers) from the video data. The emotion engine has the following functions:

[0653] Facial expression analysis: Analyzes facial expressions from video data and recognizes specific emotions (e.g., fear, anxiety).

[0654] Vocal analysis: Analyzes the user's vocal changes from audio data to identify emotions.

[0655] Examples:

[0656] 1. The faces of people in the video are analyzed to detect expressions of fear.

[0657] 2. At the same time, the tense voice is analyzed from the audio data collected from the surrounding area.

[0658] Reporting and monitoring

[0659] If abnormal behavior is detected, the server notifies the security terminal. The notification includes the relevant video frame, the detected behavior pattern, and the emotion information identified by the emotion engine. The security terminal displays this information to the security guard in real time, helping them respond quickly.

[0660] Examples:

[0661] 1. The server notifies the terminal that suspected molestation behavior and feelings of fear have been detected.

[0662] 2. The device notifies the security guard of the relevant video and emotional information and issues an audio alert.

[0663] 3. Security personnel will promptly head to the area based on the information provided.

[0664] Record and Relearn

[0665] The server records the detected abnormal behaviors, emotional information, and the corresponding responses. This data is stored in a database and used to retrain the generative AI model. Based on the recorded data, the model is retrained to continuously improve the system's performance.

[0666] Examples:

[0667] 1. The server stores the detected abnormal behavior and emotional information in a database.

[0668] 2. Periodically use the saved data to retrain the generative AI model and update it with new detection algorithms.

[0669] summary

[0670] This system analyzes video data from surveillance equipment in real time to detect molestation. It also uses an emotion engine to analyze the user's facial expressions and vocal emotions, enabling more accurate judgment and response. Furthermore, the generative AI model can be retrained based on the recorded data, continuously improving the system's accuracy and effectiveness.

[0671] The processing flow will be explained below.

[0672] Step 1:

[0673] The server receives video data from the surveillance equipment in real time. A communication protocol (e.g., RTSP) is used to establish continuous data streaming between the surveillance camera and the server. The received video data is temporarily stored in a buffer and processed to minimize delays.

[0674] Step 2:

[0675] The server analyzes the received video data and uses video analysis algorithms (e.g., OpenPose or DeepSort) to detect people in each frame and track their movement patterns. This allows each person's movements to be continuously tracked and their location and movements recorded in detail.

[0676] Step 3:

[0677] The server inputs the analyzed behavior patterns into a generative AI model. The generative AI model evaluates the behavior patterns based on pre-learned characteristics of molestation. The model outputs a score indicating whether the behavior is abnormal, and if it exceeds a certain threshold, it is determined to be abnormal behavior.

[0678] Step 4:

[0679] If abnormal behavior is detected, the server activates the emotion engine. The emotion engine analyzes the facial expressions of the user (passenger) from the video data and detects specific emotions (e.g., fear, anxiety). If necessary, it also analyzes surrounding audio data and identifies emotions from changes in the vocal cords.

[0680] Step 5:

[0681] The server notifies the security terminal of the abnormal behavior determination result and the emotion information identified by the emotion engine. The notification includes the video frame of the abnormal behavior, the evaluation result of the movement pattern, and the emotion information. This information is sent to the security terminal in real time.

[0682] Step 6:

[0683] The terminal receives notifications from the server and displays them to the security guard. The terminal's user interface displays a pop-up message with information about the person's emotions along with any abnormal behavior, and an audio alert is issued. The security guard can immediately check the content of the notification.

[0684] Step 7:

[0685] The user (security guard) rushes to the scene based on the notification from the device. After arriving at the scene, they check the actual situation and take appropriate action, such as intervening or guiding the guardian, if necessary. They also collect additional evidence depending on the situation at the scene.

[0686] Step 8:

[0687] The server records the detected abnormal behaviors and emotional information, as well as the response results. This data is stored in a database and used to retrain the generative AI model at a later date. The recorded data includes details of the abnormal behavior, the date and time, the location, and the response results.

[0688] Step 9:

[0689] The server periodically uses the recorded data to retrain the generative AI model. The model is retrained based on newly accumulated case data to improve its accuracy. Retraining makes it possible to continuously improve the effectiveness of the entire system.

[0690] Example 2

[0691] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0692] Groping in public places is becoming an increasingly serious problem, but conventional surveillance systems have struggled to effectively detect such behavior and respond quickly. Furthermore, analyzing user emotions would improve the appropriateness of responses, but no such systems currently exist. Therefore, there is a need for a system that can quickly and effectively detect groping and respond more appropriately by analyzing user emotions.

[0693] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0694] In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a person's movement patterns, means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior, means for analyzing the user's emotions from the video data and audio data, means for notifying a security terminal when abnormal behavior is detected, means for the security terminal that receives the notification to display information about the person, and means for recording the abnormal behavior and emotion analysis data and saving it for re-learning. This makes it possible to quickly and effectively detect molestation in public places and improve the appropriateness of responses by analyzing the user's emotions.

[0695] "Monitoring equipment" refers to equipment such as cameras installed to acquire video data in real time.

[0696] "Video data" refers to information in the form of images and videos captured by a surveillance device.

[0697] "Server" refers to a computer system for processing and analyzing received video data.

[0698] "Movement patterns" refer to the characteristics detected over a series of frames of a particular person's movements or actions.

[0699] A "generative artificial intelligence model" refers to a machine learning model that is trained in advance using learning data to evaluate and judge specific patterns and behaviors.

[0700] "Emotion engine" refers to an algorithm or system for analyzing video and audio data to identify a user's emotions.

[0701] "Voice data" refers to acoustic information containing the user's voice and is used for emotion analysis.

[0702] "Security terminal" refers to a computer device used to receive notifications of abnormal behavior, emotional information, etc.

[0703] "Abnormal behavior" refers to a person's actions that the generative AI model identifies as groping or other suspicious behavior.

[0704] "Notification" refers to abnormal behavior and related information sent from the server to the security terminal.

[0705] "Recording" refers to the process of storing detected abnormal behavior and emotional information in a database or the like.

[0706] "Relearning" refers to the process of updating and improving a generative AI model based on recorded data.

[0707] "User" refers to a security guard or station user who uses the system.

[0708] The present invention provides a system that effectively detects molestation in public places and recognizes the user's emotions to provide an appropriate response. Specific embodiments of this system will be described below.

[0709] System configuration and operation explanation

[0710] Data collection

[0711] The server receives video data in real time from monitoring devices (cameras) installed in stations and public facilities. This video data includes information in the form of images and videos. For example, multiple cameras installed on a station platform monitor passenger movements in real time, and the video feed is sent to the server via a network.

[0712] Behavioral analysis and identification

[0713] The server analyzes the received video data and tracks the person's movement patterns. This analysis uses image analysis technologies such as OpenPose and DeepSort. The analyzed movement patterns are input into a generative AI model that has previously learned the characteristics of molestation, which then identifies abnormal behavior. As a specific example, the server detects an unnatural hand movement of a specific person, and the generative AI model evaluates that movement as molestation.

[0714] Emotion Analysis

[0715] Furthermore, the server uses an emotion engine to analyze the user's (passenger's) emotions from the video data. The emotion engine has two functions: facial expression analysis and vocal analysis, and recognizes specific emotions (e.g., fear, anxiety). As a specific example, the face of a person in the video is analyzed to detect a fearful expression, and at the same time, a tense voice is detected from the audio data collected from the surrounding area.

[0716] Reporting and monitoring

[0717] The server then sends a summary of the detected abnormal behavior and emotional information to the security terminal. The notification includes the relevant video frame, the detected behavior pattern, and the emotional information identified by the emotion engine. As a specific example, the server detects behavior that may be suggestive of molestation and the emotion of fear, and sends this information to the security terminal. The security terminal then displays the received information to the security guard in real time and issues an audio alert to encourage a prompt response.

[0718] Record and Relearn

[0719] The server stores the detected abnormal behavior, emotional information, and corresponding results in a database. This data is used to retrain the generative AI model. For example, the server stores the detected abnormal behavior and emotional information in a database and periodically uses the stored data to retrain the generative AI model and improve the accuracy of the system.

[0720] Specific examples

[0721] Here is an example of a prompt for this system:

[0722] "Write a program to detect groping on a train platform and identify the fear of the passengers around it."

[0723] "Analyze a person's movements and emotions in real time and generate system prompts to notify security personnel if anything unusual occurs."

[0724] Hardware and software used

[0725] This system uses the following hardware and software:

[0726] Hardware: Surveillance equipment (cameras), servers, security terminals

[0727] Software: OpenPose (image analysis), DeepSort (motion tracking), generative AI model (abnormal behavior detection), emotion engine (facial expression and vocal cord analysis)

[0728] As described above, the system of the present invention can quickly and effectively detect molestation in public places and improve the appropriateness of responses by analyzing the user's emotions.

[0729] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0730] Step 1:

[0731] Receiving video data

[0732] The server receives video data in real time from monitoring devices (cameras) installed at stations and public facilities.

[0733] Input: Video feed from camera

[0734] Output: Received video data

[0735] Specific operation: The server receives video feeds from multiple cameras simultaneously over the network.

[0736] Step 2:

[0737] Video data formatting

[0738] The server formats the received video data at a fixed frame rate and converts it into a format that can be applied to analysis processing.

[0739] Input: Received video data

[0740] Output: Video data formatted for each frame

[0741] Specific operation: The server divides the received data into frames and saves them in a format that can be applied to subsequent analysis processing.

[0742] Step 3:

[0743] Motion detection

[0744] The server detects the movement patterns of each person from the video data, using image analysis technologies such as OpenPose and DeepSort.

[0745] Input: Formatted video data

[0746] Output: Movement patterns for each person

[0747] Specific operation: The server uses OpenPose to identify the person's posture and joint positions for each frame, and uses DeepSort to track their movement patterns.

[0748] Step 4:

[0749] Identifying abnormal behavior

[0750] The server inputs the detected behavioral patterns into a generative AI model and evaluates whether they match the characteristics of molestation.

[0751] Input: Movement pattern

[0752] Output: Abnormal behavior determination result

[0753] Specific actions: The server inputs movement patterns into a pre-trained generative AI model and evaluates whether unnatural movements are judged to be acts of molestation.

[0754] Step 5:

[0755] Emotion Analysis

[0756] The server simultaneously analyzes the user's emotions from the video and audio data, and uses an emotion engine to analyze facial expressions and vocal chords to recognize specific emotions (e.g., fear, anxiety).

[0757] Input: Video and audio data

[0758] Output: User's emotional information

[0759] How it works: The server uses facial recognition technology to detect the faces of people in the video and analyze whether they show any fearful expressions. At the same time, it analyzes the audio data to determine whether the vocal cords are tense.

[0760] Step 6:

[0761] Abnormality notification

[0762] The server compiles the detected abnormal behavior and emotional information and notifies the security terminal.

[0763] Input: Abnormal behavior judgment results and emotional information

[0764] Output: Notification to security terminal

[0765] Specific operation: The server detects movements that are suspected to be molestation and emotions of fear, and sends them to the security terminal.

[0766] Step 7:

[0767] Sending alerts

[0768] The device displays the received information to the security guard and issues an audio alert if necessary.

[0769] Input: Notification from the server

[0770] Output: Display and audio alert to security guards

[0771] Specific operation: The device will sound an alert and display the relevant video and emotional information on the security guard's screen.

[0772] Step 8:

[0773] Data recording

[0774] The server records the detected abnormal behavior and emotion information in a database.

[0775] Input: Abnormal behavior judgment results and emotional information

[0776] Output: Records in the database

[0777] Specific operation: The server stores the detected abnormal behavior and the associated emotional information in a database.

[0778] Step 9:

[0779] Retraining the Model

[0780] The server uses the recorded data to retrain the generative AI model, continuously improving the accuracy of the system.

[0781] Input: Abnormal behavior and emotional information recorded in the database

[0782] Output: Improved generative AI model

[0783] How it works: Periodically, the server retrains the generative AI model with new data to improve its accuracy in identifying new patterns.

[0784] (Application example 2)

[0785] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0786] To ensure the safety of passengers in self-driving vehicles, a system is needed that can immediately detect abnormal behavior, such as sexual harassment or trouble between passengers, and respond as necessary. However, existing systems rely solely on the analysis of video data, and are unable to respond by taking into account changes in the user's emotions or voice data. This makes it difficult to respond effectively and quickly.

[0787] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a person's movement patterns, means for evaluating the analyzed movement patterns using a generative artificial intelligence model and determining abnormal behavior, means for analyzing the user's emotions from the video data and audio data, means for notifying a security terminal when abnormal behavior or emotions such as anxiety or fear are detected, means for the security terminal that receives the notification to display information about the person, and means for recording the abnormal behavior and emotion data and saving it for re-learning. This enables a more effective and rapid response that combines abnormal behavior detection and emotion analysis in an autonomous vehicle.

[0788] "Monitoring equipment" refers to equipment including cameras and microphones installed to capture user behavior patterns and environmental conditions in real time.

[0789] "Video data" refers to information in the form of moving images collected by a monitoring device, and is a record of the user's actions and the environment.

[0790] "Audio data" refers to audio information acquired by a microphone installed in a monitoring device, and includes environmental sounds and the user's voice.

[0791] "Movement pattern" refers to the repetition and characteristics of movements or actions by a particular person.

[0792] A "generative artificial intelligence model" is an artificial intelligence technology that has the ability to learn specific patterns and characteristics based on large amounts of data and then evaluate and judge them.

[0793] "Abnormal behavior" refers to unnatural actions or behavior that differ from the behavior of ordinary passengers, and specifically includes acts of molestation and trouble between passengers.

[0794] "Emotion analysis" is a technology that determines a user's emotions by recognizing facial expressions from video data and analyzing vocal cords from audio data.

[0795] The "security terminal" is a device that receives notifications from the server and displays abnormal behavior and emotion analysis results to security guards in real time.

[0796] "Retraining" is the process of retraining an existing artificial intelligence model based on recorded data to improve its accuracy and responsiveness.

[0797] An "autonomous vehicle" is a vehicle that can perform driving operations automatically using artificial intelligence and sensor technology.

[0798] The system for realizing this application example is based on technology that ensures passenger safety in autonomous vehicles and analyzes abnormal behavior and passenger emotions. The specific configuration and operation of the system are described below.

[0799] System Configuration

[0800] The system consists of the following main components:

[0801] 1. Surveillance equipment: Consists of multiple cameras and microphones installed inside the vehicle. This surveillance equipment collects video and audio data in real time.

[0802] 2. Server: A server equipped with a high-performance NVIDIA GPU. This server processes and analyzes the received data to determine abnormal behavior and emotions.

[0803] 3. Security terminal: A device that receives notifications from the server and displays information to security guards in real time. This can be a tablet or a dedicated mobile device.

[0804] Data processing and analysis

[0805] The server performs the following steps in sequence:

[0806] 1. Data reception: Receives video and audio data sent from the monitoring device in real time.

[0807] 2. Motion Analysis: Video data is analyzed and human movement patterns are identified using OpenPose and DeepSort technologies. These patterns are then fed into a generative AI model to identify abnormal behavior.

[0808] 3. Sentiment Analysis: Additionally, Google Cloud Vision API and Watson Tone Analyzer are used to analyze passenger emotions from video and audio data, thereby identifying emotions such as fear and anxiety.

[0809] 4. Notification and response: If abnormal behavior or specific emotions are detected, the server will send real-time notifications to the security terminal and display the information to the security guards. If necessary, the autonomous vehicle can be controlled via the MQTT protocol.

[0810] Specific examples

[0811] To illustrate, consider the following scenario:

[0812] Detecting conflict between passengers: An in-vehicle camera captures unnatural movements (e.g., raising an arm) between passengers A and B, and the generative AI model determines this behavior as a "fight." Voice data is also analyzed at the same time, and if emergency words such as "help" are detected, the server immediately notifies the security terminal and, if necessary, stops the vehicle.

[0813] Anomaly detection through emotion analysis: If the microphone detects that a passenger's voice is filled with anxiety and the video data identifies a frightened expression, the server will use this information to notify the security terminal and take immediate action to ensure the passenger's safety.

[0814] Prompt Sentence Examples

[0815] The following prompt is an example used to train and initialize a generative AI model:

[0816] "Please collect video and audio data when Passenger A raises his arm."

[0817] "Use the Google Cloud Vision API to recognize fearful facial expressions."

[0818] "Use Watson Tone Analyzer to detect conflicting voice patterns"

[0819] This enables the system to combine abnormal behavior detection and emotion analysis within autonomous vehicles to respond more effectively and quickly.

[0820] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0821] Step 1:

[0822] The server receives video and audio data sent from the monitoring device in real time. Specifically, it acquires the data stream from the camera and microphone and stores it in the server's input buffer. The input is video and audio data, and the output is a data stream for analysis.

[0823] Step 2:

[0824] The server analyzes the received video data using OpenPose and DeepSort technologies to identify the movement patterns of the people. Specifically, it divides the video data into frames and performs pose estimation for each frame. It then performs object tracking based on the estimated pose data. The input is the video data, and the output is the movement patterns of each person.

[0825] Step 3:

[0826] The server inputs the analyzed movement patterns into a generative AI model to determine whether the movement patterns are abnormal. In this step, a pre-trained model is used to evaluate whether the movement patterns are abnormal. Specific operations include converting the movement patterns into specific feature quantities, inputting them into the model, and calculating an abnormality score. The input is the movement pattern, and the output is the result of the abnormal behavior determination.

[0827] Step 4:

[0828] The server performs emotion analysis using video and audio data. The emotion analysis uses Google Cloud Vision API and Watson Tone Analyzer to determine emotions from the user's facial expressions. Specifically, the process involves detecting faces from video frames and recognizing facial expressions, while also inputting audio data into a tone analyzer to identify the emotion of the voice. The input is video and audio data, and the output is emotion data.

[0829] Step 5:

[0830] If the server detects abnormal behavior or a specific emotion, it notifies the security terminal of this data. The notification uses the MQTT protocol to send video frames, identified behavior patterns, and emotion data to the security terminal. Specifically, this involves a process of assembling various data into packets and transmitting them in real time. The input is the abnormal behavior judgment result and emotion data, and the output is notification data sent to the security terminal.

[0831] Step 6:

[0832] When the security terminal receives a notification, it displays the relevant person's information in real time. Specific operations include interpreting the received data and reflecting it on the UI. The terminal screen displays the person's image, movement patterns, and emotion data, allowing security guards to respond immediately. The input is notification data, and the output is display information.

[0833] Step 7:

[0834] The server records the detected abnormal behavior and emotion data and stores it in a database for re-learning. In this step, the collected data is properly formatted and written to a database for long-term storage. Specifically, the abnormal behavior features and emotion data are saved and made available for re-learning at a later date. The input is the abnormal behavior judgment result and emotion data, and the output is the record saved in the database.

[0835] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0836] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0837] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0838] [Third embodiment]

[0839] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0840] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0841] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0842] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0843] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0844] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0845] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0846] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0847] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0848] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0849] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0850] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0851] The present invention is a system for effectively detecting and deterring molestation, and is realized by including a monitoring device, a server, a terminal, and a user (security guard). Specific embodiments of the system are described in detail below.

[0852] System configuration and operation explanation

[0853] Data collection

[0854] The server receives video data in real time from surveillance cameras installed in stations and public facilities. This video data refers to information in the form of images and videos captured by the surveillance cameras. The server continuously receives and processes this data with minimal delay.

[0855] Examples:

[0856] 1. Multiple cameras installed on station platforms monitor passenger movements 24 hours a day.

[0857] 2. The video feed from each camera is sent over the network to a server.

[0858] Behavioral analysis and identification

[0859] The server analyzes the received video data and tracks the movement patterns of the individuals. This is done using an algorithm that sequentially detects and identifies multiple individuals. Specifically, image analysis technology (e.g., OpenPose or DeepSort) is used. The analyzed movement patterns are input into a generative artificial intelligence model, which evaluates whether there is any abnormal behavior. This model has learned the characteristics of molestation and determines suspicious behavior based on specific movement patterns.

[0860] Examples:

[0861] 1. The server detects that a specific person is moving their hands unnaturally in a video frame.

[0862] 2. The generative artificial intelligence model evaluates the behavior as matching the characteristics of molestation.

[0863] Reporting and monitoring

[0864] If any abnormal behavior is detected, the server notifies the security terminal. The security terminal is an information display device used by security guards to receive real-time notifications and warnings. This notification includes the relevant video frame and related information. The security guard (user) checks the information displayed on the terminal and rushes to the scene to understand the situation.

[0865] Examples:

[0866] 1. The server detects behavior that is suspected to be sexual harassment and sends an alert to the security terminal.

[0867] 2. The device will pop up a video of the suspicious activity and issue an audio alert to the security guard.

[0868] 3. Based on the information on the terminal, security guards rush to the area and check the situation.

[0869] Record and Relearn

[0870] The abnormal behavior data is recorded by the server and stored in a database for re-learning. This data is used to retrain the generative AI model, helping to improve its accuracy. Re-training the generative AI model with newly added data continuously improves the effectiveness of the system.

[0871] Examples:

[0872] 1. The server stores the video data of the detected abnormal behavior and the judgment results in a database.

[0873] 2. Periodically, retrain the generative AI model based on the stored data to improve the model's performance.

[0874] summary

[0875] This system analyzes video data from surveillance equipment in real time and uses a generative AI model to evaluate behaviors characteristic of molestation, thereby detecting and reporting suspicious behavior in real time. Furthermore, the generative AI model can be retrained based on recorded data, enabling continuous improvement in the effectiveness of the system. This is expected to deter molestation and enable rapid response, and also contribute to preventing false accusations.

[0876] The processing flow will be explained below.

[0877] Step 1:

[0878] The server receives video data from the surveillance equipment in real time. A communication protocol (e.g., RTSP) is used to establish continuous data streaming between the surveillance camera and the server. The received video data is temporarily stored in a buffer to minimize delays.

[0879] Step 2:

[0880] The server analyzes the received video data and uses video analysis algorithms (e.g., OpenPose or DeepSort) to identify people in each frame and track their movement patterns. This allows each person's movements to be continuously tracked and their location and movements recorded in a database.

[0881] Step 3:

[0882] The server inputs the analyzed behavior patterns into a generative AI model. The generative AI model evaluates the behavior patterns based on pre-learned characteristics of molestation and determines whether the behavior is abnormal. The evaluation results are calculated as a score, and if it exceeds a certain threshold, it is deemed suspicious.

[0883] Step 4:

[0884] If an abnormal behavior is detected, the server notifies the security terminal. The notification includes the relevant video frame, the detected behavior pattern, and an evaluation score. This information is sent to the security terminal in real time.

[0885] Step 5:

[0886] The terminal receives notifications from the server and displays them to the security guard. The terminal's user interface displays a pop-up with the video frame where the abnormal behavior was detected and related information, and an audio alert is issued, allowing the security guard to quickly identify the nature of the abnormality.

[0887] Step 6:

[0888] The user (security guard) rushes to the scene based on the information notified from the device. After arriving at the scene, the user checks the actual situation and takes appropriate action, such as intervening or guiding the guardian, if necessary. The user may also collect further evidence based on the situation.

[0889] Step 7:

[0890] The server records the detected anomalous behavior and the response results. This data is stored in a database and used to retrain the generative AI model at a later date. The recorded data includes details of the anomalous behavior, the date and time, the location, and the response results.

[0891] Step 8:

[0892] The server periodically uses the recorded data to retrain the generative AI model. The model is retrained based on newly accumulated case data, improving its accuracy. This retraining process makes it possible to continuously improve the effectiveness of the entire system.

[0893] Example 1

[0894] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0895] Molestation incidents that occur in crowded places such as public facilities and train stations cause great pain to victims and have become a social problem. Current surveillance systems have difficulty detecting molestation incidents quickly and accurately and taking appropriate action in a timely manner. Furthermore, reliable data analysis is necessary to prevent false accusations. Therefore, there is a need for the development of a system that can detect molestation incidents in real time and enable security guards to respond quickly.

[0896] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0897] In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for converting the received video data into an appropriate format, means for analyzing the received video data to track a person's movement patterns, means for evaluating the analyzed movement patterns using a generative artificial intelligence model to identify suspicious behavior, means for notifying a security terminal when suspicious behavior is detected, means for the security terminal that receives the notification to display information about the person in question, and means for recording data on the suspicious behavior and saving it for re-learning. This enables quick and accurate detection of molestation and enables security guards to respond promptly.

[0898] A "monitoring device" is a hardware device that acquires video data and supplies it to a monitoring system.

[0899] "Video data" refers to digital information of images and videos captured by a monitoring device.

[0900] A "server" is a computer that receives, analyzes, evaluates, and notifies video data, and manages the entire system.

[0901] The "appropriate format" refers to video data converted into a predetermined format and resolution to facilitate analysis and evaluation.

[0902] A "motion pattern" refers to a series of patterns of a person's movements or actions, including specific movements and positional transformations.

[0903] A "generative artificial intelligence model" is a model that includes algorithms that learn behavioral patterns and use them to identify suspicious behavior.

[0904] "Suspicious behavior" refers to specific abnormal behavior that differs from normal behavior, including acts of sexual harassment.

[0905] A "security terminal" is an information display device used by security guards to receive notifications from the system.

[0906] A "notification" is an alert or message sent from the server to a security terminal when suspicious behavior is detected.

[0907] "Information on the individual in question" refers to information about an individual who is suspected of engaging in suspicious behavior, including video footage and related data.

[0908] "Recording data" refers to saving detected suspicious behavior and related video data for later analysis and learning.

[0909] "Retraining" is the process of using recorded data to update a generative artificial intelligence model and improve its accuracy and effectiveness.

[0910] The present invention is a system for effectively detecting and deterring molestation, and is realized by including a monitoring device, a server, a terminal, and a user (security guard). Specific embodiments of the system are described in detail below.

[0911] Data collection

[0912] The server receives video data in real time from surveillance cameras installed in public facilities and stations. This video data includes image and video information captured by the surveillance cameras, and is continuously received and processed with minimal delay. A network communication protocol such as RTSP (Real-Time Streaming Protocol) is used to receive the video data. The server uses a video processing library such as FFmpeg to convert the received video data into a processable format.

[0913] Examples:

[0914] 1. The server receives the video sent from the camera installed on the station platform using the RTSP protocol and converts the frame rate to one per second using FFmpeg.

[0915] Behavioral analysis and identification

[0916] The server analyzes specific behavioral patterns from the received video data. This analysis uses an object detection algorithm (e.g., YOLOv4). It detects the position of the person in each received frame and records their coordinate information. The analyzed behavioral patterns of the person are then input into a generative AI model to identify suspicious behavior. The generative AI model is trained using machine learning platforms such as TensorFlow and learns the characteristics of molestation behavior.

[0917] Examples:

[0918] 1. The server detects people using YOLOv4 for each received video frame and records their coordinate information.

[0919] 2. The server uses a generative AI model (e.g., a TensorFlow model) to evaluate the behavioral patterns and calculate an anomaly score.

[0920] Reporting and monitoring

[0921] If abnormal behavior is detected, the server notifies the security terminal. The security terminal is an information display device used by security guards to receive real-time notifications and warnings. This notification includes the relevant video frame and related information. The terminal displays the notification content in a pop-up display and issues an audio alert to alert the security guard. The security guard then begins responding to the situation based on the displayed information.

[0922] Examples:

[0923] 1. The server sends the frame in which abnormal behavior was detected to the security terminal via Firebase Cloud Messaging.

[0924] 2. The device will display a pop-up message with the notification content and play an audio alert to alert the security guard.

[0925] Record and Relearn

[0926] The server records abnormal behavior data in a database (e.g., MySQL) for further analysis and retraining. This data is used to retrain the generative AI model, contributing to improving its accuracy. Retraining is performed periodically to improve the model's performance based on the latest data.

[0927] Examples:

[0928] 1. The server stores the video data of abnormal behavior and the judgment results in a database.

[0929] 2. The server periodically retrains the TensorFlow model using the recorded data.

[0930] Example prompts to input to the generative AI model

[0931] "Based on the video data obtained from the surveillance cameras, please evaluate whether certain patterns of behavior constitute molestation."

[0932] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0933] Step 1:

[0934] The server receives video data from the surveillance camera in real time. The input here is streaming data sent from the surveillance camera using the RTSP protocol. The output is video data broken down into frames within the server. The server uses FFmpeg to convert this data into a format suitable for analysis, adjusting it to an appropriate frame rate.

[0935] Specific behavior:

[0936] The server receives the RTSP stream and uses FFmpeg to decompose the video frame by frame.

[0937] Adjust the frame rate to 1 frame per second.

[0938] Step 2:

[0939] The server analyzes the video data broken down into frames and tracks the movement patterns of people. The input is the transformed video frames, and the output is the coordinate information of the detected people. An object detection algorithm (e.g., YOLOv4) is used to identify people in each frame and record their positions.

[0940] Specific behavior:

[0941] The server calls YOLOv4 to detect people in each frame.

[0942] The coordinate information of the detected person is recorded in a database.

[0943] Step 3:

[0944] The server inputs the detected person's behavior data into a generative AI model to identify suspicious behavior. The input here is each person's behavior data, and the output is the evaluation result of the behavior pattern (anomaly score). The generative AI model (e.g., TensorFlow model) has learned the characteristics of molestation acts and makes an evaluation based on the input behavior pattern.

[0945] Specific behavior:

[0946] The server runs the TensorFlow model and analyzes the behavioral patterns.

[0947] An anomaly score is calculated for each movement pattern and the results are temporarily saved.

[0948] Step 4:

[0949] If the server detects suspicious behavior, it notifies the security terminal. The input is the anomaly score, and the output is the notification information sent to the security terminal. The notification includes the specific video frame and related data. A notification protocol (e.g., Firebase Cloud Messaging) is used.

[0950] Specific behavior:

[0951] The server evaluates the anomaly score and generates a notification if the threshold is exceeded.

[0952] Send notifications to security devices using Firebase Cloud Messaging.

[0953] Step 5:

[0954] The terminal receives the notification and displays the relevant video and related information to the security guard. The input here is the notification data to the security terminal, and the output is an alert display to the security guard. The terminal displays the notification content as a pop-up and issues an audio alert.

[0955] Specific behavior:

[0956] The device analyzes the notification received and displays it as a pop-up on the screen.

[0957] The specified audio file will be played to alert the security guard.

[0958] Step 6:

[0959] The server records suspicious behavior data in a database and stores it for retraining. The input is video data of abnormal behavior and the evaluation results, and the output is an updated retraining dataset. The data is periodically used for retraining.

[0960] Specific behavior:

[0961] The server stores the video frames of abnormal behavior and the evaluation results in a database.

[0962] Periodically extract data for re-learning and retrain the generative AI model.

[0963] Step 7:

[0964] The server uses the recorded data to retrain the generative AI model. The input here is the abnormal behavior data stored in the database, and the output is an updated generative AI model. Retraining is performed efficiently using batch processing.

[0965] Specific behavior:

[0966] The server retrieves previously stored data in batches and retrains the generative AI model.

[0967] Apply the updated model to improve the accuracy of the system.

[0968] (Application example 1)

[0969] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0970] Molestation and other suspicious behavior in public places are serious problems that threaten the safety of users and cause anxiety throughout society. Current surveillance systems alone are insufficient to address this issue, and more effective surveillance and rapid response are needed. In particular, a system is needed that allows users on-site to detect and respond to situations in real time. The present invention aims to provide a system that combines real-time surveillance and instant reporting functions using smartphones to address these issues.

[0971] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0972] In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a person's movement patterns, and means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior. This allows a security terminal to be notified when abnormal behavior is detected. The server also includes means for the security terminal that receives the notification to display information about the person in question and means for recording and saving the abnormal behavior data for re-learning. The server also includes means that operate as an application installed on a smartphone, monitor the safety status around the user, issue a real-time warning when suspicious behavior is detected, and means for transmitting image data of the corresponding frame to the server when suspicious behavior is confirmed and implementing security measures. This improves safety in public places and enables rapid detection and response to molestation and other suspicious behavior.

[0973] "Monitoring equipment" is a general term for cameras and sensor devices installed to acquire video data in real time.

[0974] "Video data" refers to information in the form of images and videos captured by a surveillance device.

[0975] "Real-time" means capturing and processing data immediately, with little or no delay.

[0976] "Analysis" refers to the process of examining the content of the received video data and tracking the movement patterns of people.

[0977] A "motion pattern" refers to a characteristic sequence of a person's movements or behavior.

[0978] A "generative artificial intelligence model" is an AI model trained using a large amount of learning data to perform specific pattern recognition and predictions.

[0979] "Abnormal behavior" refers to specific suspicious behavior that differs from normal behavior, and includes, in particular, sexual harassment.

[0980] A "security terminal" is an information display device used by security guards to receive real-time notifications and warnings.

[0981] "Notification" refers to the transmission of information to notify of the discovery of anomalous behavior.

[0982] "Information about a person in question" refers to information about a person who has been determined to be engaging in abnormal behavior through analysis.

[0983] "Recording" means storing data on abnormal behavior.

[0984] "Retraining" refers to the process of retraining an existing AI model using new data.

[0985] A "smartphone" is a mobile device that not only has telephone functions but also has advanced computing capabilities and can run a variety of applications.

[0986] "Application" refers to a software program that runs on a smartphone or other device.

[0987] A "server" refers to a computer system that provides data and services available to multiple users.

[0988] "Surrounding safety conditions" refers to dangerous or suspicious situations that may occur around the user.

[0989] "Warning" refers to information issued to the user to alert them when suspicious behavior is confirmed.

[0990] "Security response" refers to safety measures and response actions taken after abnormal behavior is detected.

[0991] The present invention provides a system for effectively detecting and quickly responding to acts of molestation and other suspicious behavior. The system includes a monitoring device, a server, a security terminal, and a user (security guard).

[0992] System Configuration

[0993] monitoring device

[0994] Surveillance equipment consists of cameras and sensor devices that capture video data. These devices are installed in public places, train stations, and other locations to monitor people's movements 24 hours a day.

[0995] server

[0996] The server receives video data from the monitoring devices in real time via the network. The received data is processed with minimal delay. The server analyzes the data and detects abnormal behavior in the following steps:

[0997] 1. Data Reception

[0998] Video data transmitted from a monitoring device is received in real time.

[0999] 2. Behavioral analysis and identification

[1000] The captured video data is analyzed to track a person's movement patterns using image analysis technologies such as OpenCV. The analyzed movement patterns are input into a generative artificial intelligence model, which evaluates abnormal behavior.

[1001] 3. Detecting Abnormal Behavior

[1002] The generative AI model determines whether a specific suspicious behavior is abnormal, including groping, and is pre-trained to identify the characteristics of groping.

[1003] 4. Notifications and Warnings

[1004] If any abnormal activity is detected, the server notifies the security terminal, which then alerts the security guard in real time and displays the relevant video frame and related information.

[1005] 5. Recording and Retraining Data

[1006] Data on detected anomalous behavior is recorded and stored in a database for retraining, which is used to retrain the generative AI model and help improve its accuracy.

[1007] Security terminal

[1008] The security terminal is a device that displays a pop-up video of the suspicious activity and issues an audio alert to security guards, who then rush to the area based on the information on the terminal and check the situation on the spot.

[1009] User (security guard)

[1010] Security guards receive notifications from security terminals, rush to the scene, quickly assess the situation, and respond accordingly.

[1011] Smartphone application

[1012] The system also operates as an application installed on a smartphone, which monitors the safety status of the user's surroundings and issues real-time alerts if suspicious behavior is detected.

[1013] 1. Real-time monitoring

[1014] Using the smartphone camera, video data of the surrounding area is collected and sent to a server.

[1015] 2. Notifications and Warnings

[1016] If any suspicious activity is detected, the application will alert the user in real time and send the image data of the relevant frame to the server.

[1017] 3. Security

[1018] The server receives the relevant data and immediately implements security measures.

[1019] Hardware / Software Used

[1020] Hardware: Smartphone (built-in camera, internet connection)

[1021] Software: OpenCV (image processing), Keras (generative AI model), Requests (server communication)

[1022] Specific examples

[1023] For example, when using a smartphone in a public place, the camera automatically analyzes the surrounding video data and if it detects any unusual activity, it displays a real-time warning saying "Suspicious activity detected." The smartphone then sends the relevant video frame to a server, requesting immediate security action. An example of a prompt is as follows:

[1024] "Detect if people in an image are moving unnaturally."

[1025] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1026] Step 1:

[1027] The server receives video data from the monitoring device in real time. In this process, the image and video data captured by the monitoring device are received in streaming format. The input is the video data from the monitoring device, and the output is the video data stored in the server's temporary storage device.

[1028] Step 2:

[1029] The server analyzes the received video data and tracks the person's movement patterns. This process uses OpenCV's image analysis technology. The input is the video data stored in temporary storage, and the movement and position information of the person is extracted using a movement analysis algorithm. The output is the analyzed movement pattern data.

[1030] Step 3:

[1031] The server inputs the analyzed behavior patterns into a generative AI model for evaluation. This generative AI model is trained with Keras and is designed to identify behavior patterns specific to molestation. The input is the analyzed behavior pattern data, and the AI ​​model determines whether abnormal behavior has been detected. The output is the presence or absence of abnormal behavior.

[1032] Step 4:

[1033] When abnormal behavior is detected, the server sends a notification to the security terminal. The notification includes the relevant video frame and related information (e.g., the date, time, and location of the detection). The input is the video data related to the abnormal behavior determination result, and the output is the notification data sent to the security terminal.

[1034] Step 5:

[1035] The security terminal displays the person's information based on the received notification. A pop-up display of the video data is also displayed, and an audio alert is also issued. The input is the notification data sent from the server, and the output is the video information and warning provided to the security guard.

[1036] Step 6:

[1037] The user (security guard) rushes to the relevant area based on the information from the security terminal and checks the situation on the scene. The input is the notification and video data from the security terminal, and the output is a prompt response at the scene.

[1038] Step 7:

[1039] The server records abnormal behavior data and stores it for re-learning. This data is used to retrain the generative AI model. The input is video data of abnormal behavior and its judgment results, and the output is data stored in the database for re-learning.

[1040] Step 8:

[1041] The smartphone application monitors the safety status of the user's surroundings and issues real-time alerts if suspicious behavior is detected. It uses a camera to collect video data and transmits it to a server. The input is the video data from the smartphone camera, and the output is an alert for detected suspicious behavior and transmission of the video data to the server.

[1042] Step 9:

[1043] The server receives the relevant data and immediately implements security measures. Security countermeasure instructions are sent to the relevant distribution network. The input is the video data sent from the smartphone application, and the output is notifications and response instructions to each security station.

[1044] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1045] The present invention provides a system that effectively detects molestation and recognizes the user's emotions to enable more appropriate responses. The system includes a monitoring device, a server, a terminal, an emotion engine, and a user (security guard).

[1046] System configuration and operation explanation

[1047] Data collection

[1048] The server receives video data in real time from surveillance cameras installed in stations and public facilities. Video data refers to information in the form of images and videos captured by surveillance cameras. The server continuously receives and processes this data with minimal delay.

[1049] Examples:

[1050] 1. Multiple cameras installed on station platforms monitor passenger movements in real time.

[1051] 2. The video feed from each camera is sent over the network to a server.

[1052] Behavioral analysis and identification

[1053] The server analyzes the received video data and tracks the person's movement patterns. Image analysis techniques (e.g., OpenPose and DeepSort) are used for video analysis. The analyzed movement patterns are input into a generative AI model that has previously learned the characteristics of molestation behavior, which then identifies abnormal behavior.

[1054] Examples:

[1055] 1. The server detects that a particular person is making unnatural hand movements.

[1056] 2. The generative artificial intelligence model evaluates whether the behavior matches the characteristics of molestation.

[1057] Emotion Analysis

[1058] Furthermore, the server uses an emotion engine to analyze the emotions of users (passengers) from the video data. The emotion engine has the following functions:

[1059] Facial expression analysis: Analyzes facial expressions from video data and recognizes specific emotions (e.g., fear, anxiety).

[1060] Vocal analysis: Analyzes the user's vocal changes from audio data to identify emotions.

[1061] Examples:

[1062] 1. The faces of people in the video are analyzed to detect expressions of fear.

[1063] 2. At the same time, the tense voice is analyzed from the audio data collected from the surrounding area.

[1064] Reporting and monitoring

[1065] If abnormal behavior is detected, the server notifies the security terminal. The notification includes the relevant video frame, the detected behavior pattern, and the emotion information identified by the emotion engine. The security terminal displays this information to the security guard in real time, helping them respond quickly.

[1066] Examples:

[1067] 1. The server notifies the terminal that suspected molestation behavior and feelings of fear have been detected.

[1068] 2. The device notifies the security guard of the relevant video and emotional information and issues an audio alert.

[1069] 3. Security personnel will promptly head to the area based on the information provided.

[1070] Record and Relearn

[1071] The server records the detected abnormal behaviors, emotional information, and the corresponding responses. This data is stored in a database and used to retrain the generative AI model. Based on the recorded data, the model is retrained to continuously improve the system's performance.

[1072] Examples:

[1073] 1. The server stores the detected abnormal behavior and emotional information in a database.

[1074] 2. Periodically use the saved data to retrain the generative AI model and update it with new detection algorithms.

[1075] summary

[1076] This system analyzes video data from surveillance equipment in real time to detect molestation. It also uses an emotion engine to analyze the user's facial expressions and vocal emotions, enabling more accurate judgment and response. Furthermore, the generative AI model can be retrained based on the recorded data, continuously improving the system's accuracy and effectiveness.

[1077] The processing flow will be explained below.

[1078] Step 1:

[1079] The server receives video data from the surveillance equipment in real time. A communication protocol (e.g., RTSP) is used to establish continuous data streaming between the surveillance camera and the server. The received video data is temporarily stored in a buffer and processed to minimize delays.

[1080] Step 2:

[1081] The server analyzes the received video data and uses video analysis algorithms (e.g., OpenPose or DeepSort) to detect people in each frame and track their movement patterns. This allows each person's movements to be continuously tracked and their location and movements recorded in detail.

[1082] Step 3:

[1083] The server inputs the analyzed behavior patterns into a generative AI model. The generative AI model evaluates the behavior patterns based on pre-learned characteristics of molestation. The model outputs a score indicating whether the behavior is abnormal, and if it exceeds a certain threshold, it is determined to be abnormal behavior.

[1084] Step 4:

[1085] If abnormal behavior is detected, the server activates the emotion engine. The emotion engine analyzes the facial expressions of the user (passenger) from the video data and detects specific emotions (e.g., fear, anxiety). If necessary, it also analyzes surrounding audio data and identifies emotions from changes in the vocal cords.

[1086] Step 5:

[1087] The server notifies the security terminal of the abnormal behavior determination result and the emotion information identified by the emotion engine. The notification includes the video frame of the abnormal behavior, the evaluation result of the movement pattern, and the emotion information. This information is sent to the security terminal in real time.

[1088] Step 6:

[1089] The terminal receives notifications from the server and displays them to the security guard. The terminal's user interface displays a pop-up message with information about the person's emotions along with any abnormal behavior, and an audio alert is issued. The security guard can immediately check the content of the notification.

[1090] Step 7:

[1091] The user (security guard) rushes to the scene based on the notification from the device. After arriving at the scene, they check the actual situation and take appropriate action, such as intervening or guiding the guardian, if necessary. They also collect additional evidence depending on the situation at the scene.

[1092] Step 8:

[1093] The server records the detected abnormal behaviors and emotional information, as well as the response results. This data is stored in a database and used to retrain the generative AI model at a later date. The recorded data includes details of the abnormal behavior, the date and time, the location, and the response results.

[1094] Step 9:

[1095] The server periodically uses the recorded data to retrain the generative AI model. The model is retrained based on newly accumulated case data to improve its accuracy. Retraining makes it possible to continuously improve the effectiveness of the entire system.

[1096] Example 2

[1097] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1098] Groping in public places is becoming an increasingly serious problem, but conventional surveillance systems have struggled to effectively detect such behavior and respond quickly. Furthermore, analyzing user emotions would improve the appropriateness of responses, but no such systems currently exist. Therefore, there is a need for a system that can quickly and effectively detect groping and respond more appropriately by analyzing user emotions.

[1099] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1100] In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a person's movement patterns, means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior, means for analyzing the user's emotions from the video data and audio data, means for notifying a security terminal when abnormal behavior is detected, means for the security terminal that receives the notification to display information about the person, and means for recording the abnormal behavior and emotion analysis data and saving it for re-learning. This makes it possible to quickly and effectively detect molestation in public places and improve the appropriateness of responses by analyzing the user's emotions.

[1101] "Monitoring equipment" refers to equipment such as cameras installed to acquire video data in real time.

[1102] "Video data" refers to information in the form of images and videos captured by a surveillance device.

[1103] "Server" refers to a computer system for processing and analyzing received video data.

[1104] "Movement patterns" refer to the characteristics detected over a series of frames of a particular person's movements or actions.

[1105] A "generative artificial intelligence model" refers to a machine learning model that is trained in advance using learning data to evaluate and judge specific patterns and behaviors.

[1106] "Emotion engine" refers to an algorithm or system for analyzing video and audio data to identify a user's emotions.

[1107] "Voice data" refers to acoustic information containing the user's voice and is used for emotion analysis.

[1108] "Security terminal" refers to a computer device used to receive notifications of abnormal behavior, emotional information, etc.

[1109] "Abnormal behavior" refers to a person's actions that the generative AI model identifies as groping or other suspicious behavior.

[1110] "Notification" refers to abnormal behavior and related information sent from the server to the security terminal.

[1111] "Recording" refers to the process of storing detected abnormal behavior and emotional information in a database or the like.

[1112] "Relearning" refers to the process of updating and improving a generative AI model based on recorded data.

[1113] "User" refers to a security guard or station user who uses the system.

[1114] The present invention provides a system that effectively detects molestation in public places and recognizes the user's emotions to provide an appropriate response. Specific embodiments of this system will be described below.

[1115] System configuration and operation explanation

[1116] Data collection

[1117] The server receives video data in real time from monitoring devices (cameras) installed in stations and public facilities. This video data includes information in the form of images and videos. For example, multiple cameras installed on a station platform monitor passenger movements in real time, and the video feed is sent to the server via a network.

[1118] Behavioral analysis and identification

[1119] The server analyzes the received video data and tracks the person's movement patterns. This analysis uses image analysis technologies such as OpenPose and DeepSort. The analyzed movement patterns are input into a generative AI model that has previously learned the characteristics of molestation, which then identifies abnormal behavior. As a specific example, the server detects an unnatural hand movement of a specific person, and the generative AI model evaluates that movement as molestation.

[1120] Emotion Analysis

[1121] Furthermore, the server uses an emotion engine to analyze the user's (passenger's) emotions from the video data. The emotion engine has two functions: facial expression analysis and vocal analysis, and recognizes specific emotions (e.g., fear, anxiety). As a specific example, the face of a person in the video is analyzed to detect a fearful expression, and at the same time, a tense voice is detected from the audio data collected from the surrounding area.

[1122] Reporting and monitoring

[1123] The server then sends a summary of the detected abnormal behavior and emotional information to the security terminal. The notification includes the relevant video frame, the detected behavior pattern, and the emotional information identified by the emotion engine. As a specific example, the server detects behavior that may be suggestive of molestation and the emotion of fear, and sends this information to the security terminal. The security terminal then displays the received information to the security guard in real time and issues an audio alert to encourage a prompt response.

[1124] Record and Relearn

[1125] The server stores the detected abnormal behavior, emotional information, and corresponding results in a database. This data is used to retrain the generative AI model. For example, the server stores the detected abnormal behavior and emotional information in a database and periodically uses the stored data to retrain the generative AI model and improve the accuracy of the system.

[1126] Specific examples

[1127] Here is an example of a prompt for this system:

[1128] "Write a program to detect groping on a train platform and identify the fear of the passengers around it."

[1129] "Analyze a person's movements and emotions in real time and generate system prompts to notify security personnel if anything unusual occurs."

[1130] Hardware and software used

[1131] This system uses the following hardware and software:

[1132] Hardware: Surveillance equipment (cameras), servers, security terminals

[1133] Software: OpenPose (image analysis), DeepSort (motion tracking), generative AI model (abnormal behavior detection), emotion engine (facial expression and vocal cord analysis)

[1134] As described above, the system of the present invention can quickly and effectively detect molestation in public places and improve the appropriateness of responses by analyzing the user's emotions.

[1135] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1136] Step 1:

[1137] Receiving video data

[1138] The server receives video data in real time from monitoring devices (cameras) installed at stations and public facilities.

[1139] Input: Video feed from camera

[1140] Output: Received video data

[1141] Specific operation: The server receives video feeds from multiple cameras simultaneously over the network.

[1142] Step 2:

[1143] Video data formatting

[1144] The server formats the received video data at a fixed frame rate and converts it into a format that can be applied to analysis processing.

[1145] Input: Received video data

[1146] Output: Video data formatted for each frame

[1147] Specific operation: The server divides the received data into frames and saves them in a format that can be applied to subsequent analysis processing.

[1148] Step 3:

[1149] Motion detection

[1150] The server detects the movement patterns of each person from the video data, using image analysis technologies such as OpenPose and DeepSort.

[1151] Input: Formatted video data

[1152] Output: Movement patterns for each person

[1153] Specific operation: The server uses OpenPose to identify the person's posture and joint positions for each frame, and uses DeepSort to track their movement patterns.

[1154] Step 4:

[1155] Identifying abnormal behavior

[1156] The server inputs the detected behavioral patterns into a generative AI model and evaluates whether they match the characteristics of molestation.

[1157] Input: Movement pattern

[1158] Output: Abnormal behavior determination result

[1159] Specific actions: The server inputs movement patterns into a pre-trained generative AI model and evaluates whether unnatural movements are judged to be acts of molestation.

[1160] Step 5:

[1161] Emotion Analysis

[1162] The server simultaneously analyzes the user's emotions from the video and audio data, and uses an emotion engine to analyze facial expressions and vocal chords to recognize specific emotions (e.g., fear, anxiety).

[1163] Input: Video and audio data

[1164] Output: User's emotional information

[1165] How it works: The server uses facial recognition technology to detect the faces of people in the video and analyze whether they show any fearful expressions. At the same time, it analyzes the audio data to determine whether the vocal cords are tense.

[1166] Step 6:

[1167] Abnormality notification

[1168] The server compiles the detected abnormal behavior and emotional information and notifies the security terminal.

[1169] Input: Abnormal behavior judgment results and emotional information

[1170] Output: Notification to security terminal

[1171] Specific operation: The server detects movements that are suspected to be molestation and emotions of fear, and sends them to the security terminal.

[1172] Step 7:

[1173] Sending alerts

[1174] The device displays the received information to the security guard and issues an audio alert if necessary.

[1175] Input: Notification from the server

[1176] Output: Display and audio alert to security guards

[1177] Specific operation: The device will sound an alert and display the relevant video and emotional information on the security guard's screen.

[1178] Step 8:

[1179] Data recording

[1180] The server records the detected abnormal behavior and emotion information in a database.

[1181] Input: Abnormal behavior judgment results and emotional information

[1182] Output: Records in the database

[1183] Specific operation: The server stores the detected abnormal behavior and the associated emotional information in a database.

[1184] Step 9:

[1185] Retraining the Model

[1186] The server uses the recorded data to retrain the generative AI model, continuously improving the accuracy of the system.

[1187] Input: Abnormal behavior and emotional information recorded in the database

[1188] Output: Improved generative AI model

[1189] How it works: Periodically, the server retrains the generative AI model with new data to improve its accuracy in identifying new patterns.

[1190] (Application example 2)

[1191] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1192] To ensure the safety of passengers in self-driving vehicles, a system is needed that can immediately detect abnormal behavior, such as sexual harassment or trouble between passengers, and respond as necessary. However, existing systems rely solely on the analysis of video data, and are unable to respond by taking into account changes in the user's emotions or voice data. This makes it difficult to respond effectively and quickly.

[1193] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a person's movement patterns, means for evaluating the analyzed movement patterns using a generative artificial intelligence model and determining abnormal behavior, means for analyzing the user's emotions from the video data and audio data, means for notifying a security terminal when abnormal behavior or emotions such as anxiety or fear are detected, means for the security terminal that receives the notification to display information about the person, and means for recording the abnormal behavior and emotion data and saving it for re-learning. This enables a more effective and rapid response that combines abnormal behavior detection and emotion analysis in an autonomous vehicle.

[1194] "Monitoring equipment" refers to equipment including cameras and microphones installed to capture user behavior patterns and environmental conditions in real time.

[1195] "Video data" refers to information in the form of moving images collected by a monitoring device, and is a record of the user's actions and the environment.

[1196] "Audio data" refers to audio information acquired by a microphone installed in a monitoring device, and includes environmental sounds and the user's voice.

[1197] "Movement pattern" refers to the repetition and characteristics of movements or actions by a particular person.

[1198] A "generative artificial intelligence model" is an artificial intelligence technology that has the ability to learn specific patterns and characteristics based on large amounts of data and then evaluate and judge them.

[1199] "Abnormal behavior" refers to unnatural actions or behavior that differ from the behavior of ordinary passengers, and specifically includes acts of molestation and trouble between passengers.

[1200] "Emotion analysis" is a technology that determines a user's emotions by recognizing facial expressions from video data and analyzing vocal cords from audio data.

[1201] The "security terminal" is a device that receives notifications from the server and displays abnormal behavior and emotion analysis results to security guards in real time.

[1202] "Retraining" is the process of retraining an existing artificial intelligence model based on recorded data to improve its accuracy and responsiveness.

[1203] An "autonomous vehicle" is a vehicle that can perform driving operations automatically using artificial intelligence and sensor technology.

[1204] The system for realizing this application example is based on technology that ensures passenger safety in autonomous vehicles and analyzes abnormal behavior and passenger emotions. The specific configuration and operation of the system are described below.

[1205] System Configuration

[1206] The system consists of the following main components:

[1207] 1. Surveillance equipment: Consists of multiple cameras and microphones installed inside the vehicle. This surveillance equipment collects video and audio data in real time.

[1208] 2. Server: A server equipped with a high-performance NVIDIA GPU. This server processes and analyzes the received data to determine abnormal behavior and emotions.

[1209] 3. Security terminal: A device that receives notifications from the server and displays information to security guards in real time. This can be a tablet or a dedicated mobile device.

[1210] Data processing and analysis

[1211] The server performs the following steps in sequence:

[1212] 1. Data reception: Receives video and audio data sent from the monitoring device in real time.

[1213] 2. Motion Analysis: Video data is analyzed and human movement patterns are identified using OpenPose and DeepSort technologies. These patterns are then fed into a generative AI model to identify abnormal behavior.

[1214] 3. Sentiment Analysis: Additionally, Google Cloud Vision API and Watson Tone Analyzer are used to analyze passenger emotions from video and audio data, thereby identifying emotions such as fear and anxiety.

[1215] 4. Notification and response: If abnormal behavior or specific emotions are detected, the server will send real-time notifications to the security terminal and display the information to the security guards. If necessary, the autonomous vehicle can be controlled via the MQTT protocol.

[1216] Specific examples

[1217] To illustrate, consider the following scenario:

[1218] Detecting conflict between passengers: An in-vehicle camera captures unnatural movements (e.g., raising an arm) between passengers A and B, and the generative AI model determines this behavior as a "fight." Voice data is also analyzed at the same time, and if emergency words such as "help" are detected, the server immediately notifies the security terminal and, if necessary, stops the vehicle.

[1219] Anomaly detection through emotion analysis: If the microphone detects that a passenger's voice is filled with anxiety and the video data identifies a frightened expression, the server will use this information to notify the security terminal and take immediate action to ensure the passenger's safety.

[1220] Prompt Sentence Examples

[1221] The following prompt is an example used to train and initialize a generative AI model:

[1222] "Please collect video and audio data when Passenger A raises his arm."

[1223] "Use the Google Cloud Vision API to recognize fearful facial expressions."

[1224] "Use Watson Tone Analyzer to detect conflicting voice patterns"

[1225] This enables the system to combine abnormal behavior detection and emotion analysis within autonomous vehicles to respond more effectively and quickly.

[1226] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1227] Step 1:

[1228] The server receives video and audio data sent from the monitoring device in real time. Specifically, it acquires the data stream from the camera and microphone and stores it in the server's input buffer. The input is video and audio data, and the output is a data stream for analysis.

[1229] Step 2:

[1230] The server analyzes the received video data using OpenPose and DeepSort technologies to identify the movement patterns of the people. Specifically, it divides the video data into frames and performs pose estimation for each frame. It then performs object tracking based on the estimated pose data. The input is the video data, and the output is the movement patterns of each person.

[1231] Step 3:

[1232] The server inputs the analyzed movement patterns into a generative AI model to determine whether the movement patterns are abnormal. In this step, a pre-trained model is used to evaluate whether the movement patterns are abnormal. Specific operations include converting the movement patterns into specific feature quantities, inputting them into the model, and calculating an abnormality score. The input is the movement pattern, and the output is the result of the abnormal behavior determination.

[1233] Step 4:

[1234] The server performs emotion analysis using video and audio data. The emotion analysis uses Google Cloud Vision API and Watson Tone Analyzer to determine emotions from the user's facial expressions. Specifically, the process involves detecting faces from video frames and recognizing facial expressions, while also inputting audio data into a tone analyzer to identify the emotion of the voice. The input is video and audio data, and the output is emotion data.

[1235] Step 5:

[1236] If the server detects abnormal behavior or a specific emotion, it notifies the security terminal of this data. The notification uses the MQTT protocol to send video frames, identified behavior patterns, and emotion data to the security terminal. Specifically, this involves a process of assembling various data into packets and transmitting them in real time. The input is the abnormal behavior judgment result and emotion data, and the output is notification data sent to the security terminal.

[1237] Step 6:

[1238] When the security terminal receives a notification, it displays the relevant person's information in real time. Specific operations include interpreting the received data and reflecting it on the UI. The terminal screen displays the person's image, movement patterns, and emotion data, allowing security guards to respond immediately. The input is notification data, and the output is display information.

[1239] Step 7:

[1240] The server records the detected abnormal behavior and emotion data and stores it in a database for re-learning. In this step, the collected data is properly formatted and written to a database for long-term storage. Specifically, the abnormal behavior features and emotion data are saved and made available for re-learning at a later date. The input is the abnormal behavior judgment result and emotion data, and the output is the record saved in the database.

[1241] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1242] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1243] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1244] [Fourth embodiment]

[1245] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1246] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1247] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1248] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1249] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1250] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1251] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1252] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1253] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1254] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1255] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1256] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1257] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1258] The present invention is a system for effectively detecting and deterring molestation, and is realized by including a monitoring device, a server, a terminal, and a user (security guard). Specific embodiments of the system are described in detail below.

[1259] System configuration and operation explanation

[1260] Data collection

[1261] The server receives video data in real time from surveillance cameras installed in stations and public facilities. This video data refers to information in the form of images and videos captured by the surveillance cameras. The server continuously receives and processes this data with minimal delay.

[1262] Examples:

[1263] 1. Multiple cameras installed on station platforms monitor passenger movements 24 hours a day.

[1264] 2. The video feed from each camera is sent over the network to a server.

[1265] Behavioral analysis and identification

[1266] The server analyzes the received video data and tracks the movement patterns of the individuals. This is done using an algorithm that sequentially detects and identifies multiple individuals. Specifically, image analysis technology (e.g., OpenPose or DeepSort) is used. The analyzed movement patterns are input into a generative artificial intelligence model, which evaluates whether there is any abnormal behavior. This model has learned the characteristics of molestation and determines suspicious behavior based on specific movement patterns.

[1267] Examples:

[1268] 1. The server detects that a specific person is moving their hands unnaturally in a video frame.

[1269] 2. The generative artificial intelligence model evaluates the behavior as matching the characteristics of molestation.

[1270] Reporting and monitoring

[1271] If any abnormal behavior is detected, the server notifies the security terminal. The security terminal is an information display device used by security guards to receive real-time notifications and warnings. This notification includes the relevant video frame and related information. The security guard (user) checks the information displayed on the terminal and rushes to the scene to understand the situation.

[1272] Examples:

[1273] 1. The server detects behavior that is suspected to be sexual harassment and sends an alert to the security terminal.

[1274] 2. The device will pop up a video of the suspicious activity and issue an audio alert to the security guard.

[1275] 3. Based on the information on the terminal, security guards rush to the area and check the situation.

[1276] Record and Relearn

[1277] The abnormal behavior data is recorded by the server and stored in a database for re-learning. This data is used to retrain the generative AI model, helping to improve its accuracy. Re-training the generative AI model with newly added data continuously improves the effectiveness of the system.

[1278] Examples:

[1279] 1. The server stores the video data of the detected abnormal behavior and the judgment results in a database.

[1280] 2. Periodically, retrain the generative AI model based on the stored data to improve the model's performance.

[1281] summary

[1282] This system analyzes video data from surveillance equipment in real time and uses a generative AI model to evaluate behaviors characteristic of molestation, thereby detecting and reporting suspicious behavior in real time. Furthermore, the generative AI model can be retrained based on recorded data, enabling continuous improvement in the effectiveness of the system. This is expected to deter molestation and enable rapid response, as well as contribute to preventing false accusations.

[1283] The processing flow will be explained below.

[1284] Step 1:

[1285] The server receives video data from the surveillance equipment in real time. A communication protocol (e.g., RTSP) is used to establish continuous data streaming between the surveillance camera and the server. The received video data is temporarily stored in a buffer to minimize delays.

[1286] Step 2:

[1287] The server analyzes the received video data and uses video analysis algorithms (e.g., OpenPose or DeepSort) to identify people in each frame and track their movement patterns. This allows each person's movements to be continuously tracked and their location and movements recorded in a database.

[1288] Step 3:

[1289] The server inputs the analyzed behavior patterns into a generative AI model. The generative AI model evaluates the behavior patterns based on pre-learned characteristics of molestation and determines whether the behavior is abnormal. The evaluation results are calculated as a score, and if it exceeds a certain threshold, it is deemed suspicious.

[1290] Step 4:

[1291] If an abnormal behavior is detected, the server notifies the security terminal. The notification includes the relevant video frame, the detected behavior pattern, and an evaluation score. This information is sent to the security terminal in real time.

[1292] Step 5:

[1293] The terminal receives notifications from the server and displays them to the security guard. The terminal's user interface displays a pop-up with the video frame where the abnormal behavior was detected and related information, and an audio alert is issued, allowing the security guard to quickly identify the nature of the anomaly.

[1294] Step 6:

[1295] The user (security guard) rushes to the scene based on the information notified from the device. After arriving at the scene, the user checks the actual situation and takes appropriate action, such as intervening or guiding the guardian, if necessary. The user may also collect further evidence based on the situation.

[1296] Step 7:

[1297] The server records the detected anomalous behavior and the response results. This data is stored in a database and used to retrain the generative AI model at a later date. The recorded data includes details of the anomalous behavior, the date and time, the location, and the response results.

[1298] Step 8:

[1299] The server periodically uses the recorded data to retrain the generative AI model. The model is retrained based on newly accumulated case data to improve its accuracy. Retraining makes it possible to continuously improve the effectiveness of the entire system.

[1300] Example 1

[1301] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1302] Molestation incidents that occur in crowded places such as public facilities and train stations cause great pain to victims and have become a social problem. Current surveillance systems have difficulty detecting molestation incidents quickly and accurately and taking appropriate action in a timely manner. Furthermore, reliable data analysis is necessary to prevent false accusations. Therefore, there is a need for the development of a system that can detect molestation incidents in real time and enable security guards to respond quickly.

[1303] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1304] In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for converting the received video data into an appropriate format, means for analyzing the received video data to track a person's movement patterns, means for evaluating the analyzed movement patterns using a generative artificial intelligence model to identify suspicious behavior, means for notifying a security terminal when suspicious behavior is detected, means for the security terminal that receives the notification to display information about the person in question, and means for recording data on the suspicious behavior and saving it for re-learning. This enables quick and accurate detection of molestation and enables security guards to respond promptly.

[1305] A "monitoring device" is a hardware device that acquires video data and supplies it to a monitoring system.

[1306] "Video data" refers to digital information of images and videos captured by a monitoring device.

[1307] A "server" is a computer that receives, analyzes, evaluates, and notifies video data, and manages the entire system.

[1308] The "appropriate format" refers to video data converted into a predetermined format and resolution to facilitate analysis and evaluation.

[1309] A "motion pattern" refers to a series of patterns of a person's movements or actions, including specific movements and positional transformations.

[1310] A "generative artificial intelligence model" is a model that includes algorithms that learn behavioral patterns and use them to identify suspicious behavior.

[1311] "Suspicious behavior" refers to specific abnormal behavior that differs from normal behavior, including acts of sexual harassment.

[1312] A "security terminal" is an information display device used by security guards to receive notifications from the system.

[1313] A "notification" is an alert or message sent from the server to a security terminal when suspicious behavior is detected.

[1314] "Information on the individual in question" refers to information about an individual who is suspected of engaging in suspicious behavior, including video footage and related data.

[1315] "Recording data" refers to saving detected suspicious behavior and related video data for later analysis and learning.

[1316] "Retraining" is the process of using recorded data to update a generative artificial intelligence model and improve its accuracy and effectiveness.

[1317] The present invention is a system for effectively detecting and deterring molestation, and is realized by including a monitoring device, a server, a terminal, and a user (security guard). Specific embodiments of the system are described in detail below.

[1318] Data collection

[1319] The server receives video data in real time from surveillance cameras installed in public facilities and stations. This video data includes image and video information captured by the surveillance cameras, and is continuously received and processed with minimal delay. A network communication protocol such as RTSP (Real-Time Streaming Protocol) is used to receive the video data. The server uses a video processing library such as FFmpeg to convert the received video data into a processable format.

[1320] Examples:

[1321] 1. The server receives the video sent from the camera installed on the station platform using the RTSP protocol and converts the frame rate to one per second using FFmpeg.

[1322] Behavioral analysis and identification

[1323] The server analyzes specific behavioral patterns from the received video data. This analysis uses an object detection algorithm (e.g., YOLOv4). It detects the position of the person in each received frame and records their coordinate information. The analyzed behavioral patterns of the person are then input into a generative AI model to identify suspicious behavior. The generative AI model is trained using machine learning platforms such as TensorFlow and learns the characteristics of molestation behavior.

[1324] Examples:

[1325] 1. The server detects people using YOLOv4 for each received video frame and records their coordinate information.

[1326] 2. The server uses a generative AI model (e.g., a TensorFlow model) to evaluate the behavioral patterns and calculate an anomaly score.

[1327] Reporting and monitoring

[1328] If abnormal behavior is detected, the server notifies the security terminal. The security terminal is an information display device used by security guards to receive real-time notifications and warnings. This notification includes the relevant video frame and related information. The terminal displays the notification content in a pop-up display and issues an audio alert to alert the security guard. The security guard then begins responding to the situation based on the displayed information.

[1329] Examples:

[1330] 1. The server sends the frame in which abnormal behavior was detected to the security terminal via Firebase Cloud Messaging.

[1331] 2. The device will display a pop-up message with the notification content and play an audio alert to alert the security guard.

[1332] Record and Relearn

[1333] The server records abnormal behavior data in a database (e.g., MySQL) for further analysis and retraining. This data is used to retrain the generative AI model, contributing to improving its accuracy. Retraining is performed periodically to improve the model's performance based on the latest data.

[1334] Examples:

[1335] 1. The server stores the video data of abnormal behavior and the judgment results in a database.

[1336] 2. The server periodically retrains the TensorFlow model using the recorded data.

[1337] Example prompts to be input to the generative AI model

[1338] "Based on the video data obtained from the surveillance cameras, please evaluate whether certain patterns of behavior constitute molestation."

[1339] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1340] Step 1:

[1341] The server receives video data from the surveillance camera in real time. The input here is streaming data sent from the surveillance camera using the RTSP protocol. The output is video data broken down into frames within the server. The server uses FFmpeg to convert this data into a format suitable for analysis, adjusting it to an appropriate frame rate.

[1342] Specific behavior:

[1343] The server receives the RTSP stream and uses FFmpeg to decompose the video frame by frame.

[1344] Adjust the frame rate to 1 frame per second.

[1345] Step 2:

[1346] The server analyzes the video data broken down into frames and tracks the movement patterns of people. The input is the transformed video frames, and the output is the coordinate information of the detected people. An object detection algorithm (e.g., YOLOv4) is used to identify people in each frame and record their positions.

[1347] Specific behavior:

[1348] The server calls YOLOv4 to detect people in each frame.

[1349] The coordinate information of the detected person is recorded in a database.

[1350] Step 3:

[1351] The server inputs the detected person's behavior data into a generative AI model to identify suspicious behavior. The input here is each person's behavior data, and the output is the evaluation result of the behavior pattern (anomaly score). The generative AI model (e.g., TensorFlow model) has learned the characteristics of molestation acts and makes an evaluation based on the input behavior pattern.

[1352] Specific behavior:

[1353] The server runs the TensorFlow model and analyzes the behavioral patterns.

[1354] An anomaly score is calculated for each movement pattern and the results are temporarily saved.

[1355] Step 4:

[1356] If the server detects suspicious behavior, it notifies the security terminal. The input is the anomaly score, and the output is the notification information sent to the security terminal. The notification includes the specific video frame and related data. A notification protocol (e.g., Firebase Cloud Messaging) is used.

[1357] Specific behavior:

[1358] The server evaluates the anomaly score and generates a notification if the threshold is exceeded.

[1359] Send notifications to security devices using Firebase Cloud Messaging.

[1360] Step 5:

[1361] The terminal receives the notification and displays the relevant video and related information to the security guard. The input here is the notification data to the security terminal, and the output is an alert display to the security guard. The terminal displays the notification content as a pop-up and issues an audio alert.

[1362] Specific behavior:

[1363] The device analyzes the notification received and displays it as a pop-up on the screen.

[1364] The specified audio file will be played to alert the security guard.

[1365] Step 6:

[1366] The server records suspicious behavior data in a database and stores it for retraining. The input is video data of abnormal behavior and the evaluation results, and the output is an updated retraining dataset. The data is periodically used for retraining.

[1367] Specific behavior:

[1368] The server stores the video frames of abnormal behavior and the evaluation results in a database.

[1369] Periodically extract data for re-learning and retrain the generative AI model.

[1370] Step 7:

[1371] The server uses the recorded data to retrain the generative AI model. The input here is the abnormal behavior data stored in the database, and the output is an updated generative AI model. Retraining is performed efficiently using batch processing.

[1372] Specific behavior:

[1373] The server retrieves previously stored data in batches and retrains the generative AI model.

[1374] Apply the updated model to improve the accuracy of the system.

[1375] (Application example 1)

[1376] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1377] Molestation and other suspicious behavior in public places are serious problems that threaten the safety of users and cause anxiety throughout society. Current surveillance systems alone are insufficient to address this issue, and more effective surveillance and rapid response are needed. In particular, a system is needed that allows users on-site to detect and respond to situations in real time. The present invention aims to provide a system that combines real-time surveillance and instant reporting functions using smartphones to address these issues.

[1378] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1379] In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a person's movement patterns, and means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior. This allows a security terminal to be notified when abnormal behavior is detected. The server also includes means for the security terminal that receives the notification to display information about the person in question and means for recording and saving the abnormal behavior data for re-learning. The server also includes means that operate as an application installed on a smartphone, monitor the safety status around the user, issue a real-time warning when suspicious behavior is detected, and means for transmitting image data of the corresponding frame to the server when suspicious behavior is confirmed and implementing security measures. This improves safety in public places and enables rapid detection and response to molestation and other suspicious behavior.

[1380] "Monitoring equipment" is a general term for cameras and sensor devices installed to acquire video data in real time.

[1381] "Video data" refers to information in the form of images and videos captured by a surveillance device.

[1382] "Real-time" means capturing and processing data immediately, with little or no delay.

[1383] "Analysis" refers to the process of examining the content of the received video data and tracking the movement patterns of people.

[1384] A "motion pattern" refers to a characteristic sequence of a person's movements or behavior.

[1385] A "generative artificial intelligence model" is an AI model trained using a large amount of learning data to perform specific pattern recognition and predictions.

[1386] "Abnormal behavior" refers to specific suspicious behavior that differs from normal behavior, and includes, in particular, sexual harassment.

[1387] A "security terminal" is an information display device used by security guards to receive real-time notifications and warnings.

[1388] "Notification" refers to the transmission of information to notify of the discovery of anomalous behavior.

[1389] "Information about a person in question" refers to information about a person who has been determined to be engaging in abnormal behavior through analysis.

[1390] "Recording" means storing data on abnormal behavior.

[1391] "Retraining" refers to the process of retraining an existing AI model using new data.

[1392] A "smartphone" is a mobile device that not only has telephone functions but also has advanced computing capabilities and can run a variety of applications.

[1393] "Application" refers to a software program that runs on a smartphone or other device.

[1394] A "server" refers to a computer system that provides data and services available to multiple users.

[1395] "Surrounding safety conditions" refers to dangerous or suspicious situations that may occur around the user.

[1396] "Warning" refers to information issued to the user to alert them when suspicious behavior is confirmed.

[1397] "Security response" refers to safety measures and response actions taken after abnormal behavior is detected.

[1398] The present invention provides a system for effectively detecting and quickly responding to acts of molestation and other suspicious behavior. The system includes a monitoring device, a server, a security terminal, and a user (security guard).

[1399] System Configuration

[1400] monitoring device

[1401] Surveillance equipment consists of cameras and sensor devices that capture video data. These devices are installed in public places, train stations, and other locations to monitor people's movements 24 hours a day.

[1402] server

[1403] The server receives video data from the monitoring devices in real time via the network. The received data is processed with minimal delay. The server analyzes the data and detects abnormal behavior in the following steps:

[1404] 1. Data Reception

[1405] Video data transmitted from a monitoring device is received in real time.

[1406] 2. Behavioral analysis and identification

[1407] The captured video data is analyzed to track a person's movement patterns using image analysis technologies such as OpenCV. The analyzed movement patterns are input into a generative artificial intelligence model, which evaluates abnormal behavior.

[1408] 3. Detecting Abnormal Behavior

[1409] The generative AI model determines whether a specific suspicious behavior is abnormal, including groping, and is pre-trained to identify the characteristics of groping.

[1410] 4. Notifications and Warnings

[1411] If any abnormal activity is detected, the server notifies the security terminal, which then alerts the security guard in real time and displays the relevant video frame and related information.

[1412] 5. Recording and Retraining Data

[1413] Data on detected anomalous behavior is recorded and stored in a database for retraining, which is used to retrain the generative AI model and help improve its accuracy.

[1414] Security terminal

[1415] The security terminal is a device that displays a pop-up video of the suspicious activity and issues an audio alert to security guards, who then rush to the area based on the information on the terminal and check the situation on the spot.

[1416] User (security guard)

[1417] Security guards receive notifications from security terminals, rush to the scene, quickly assess the situation, and respond accordingly.

[1418] Smartphone application

[1419] The system also operates as an application installed on a smartphone, which monitors the safety status of the user's surroundings and issues real-time alerts if suspicious behavior is detected.

[1420] 1. Real-time monitoring

[1421] Using the smartphone camera, video data of the surrounding area is collected and sent to a server.

[1422] 2. Notifications and Warnings

[1423] If any suspicious activity is detected, the application will alert the user in real time and send the image data of the relevant frame to the server.

[1424] 3. Security

[1425] The server receives the relevant data and immediately implements security measures.

[1426] Hardware / Software Used

[1427] Hardware: Smartphone (built-in camera, internet connection)

[1428] Software: OpenCV (image processing), Keras (generative AI model), Requests (server communication)

[1429] Specific examples

[1430] For example, when using a smartphone in a public place, the camera automatically analyzes the surrounding video data and if it detects any unusual activity, it displays a real-time warning saying "Suspicious activity detected." The smartphone then sends the relevant video frame to a server, requesting immediate security action. An example of a prompt is as follows:

[1431] "Detect if people in an image are moving unnaturally."

[1432] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1433] Step 1:

[1434] The server receives video data from the monitoring device in real time. In this process, the image and video data captured by the monitoring device are received in streaming format. The input is the video data from the monitoring device, and the output is the video data stored in the server's temporary storage device.

[1435] Step 2:

[1436] The server analyzes the received video data and tracks the person's movement patterns. This process uses OpenCV's image analysis technology. The input is the video data stored in temporary storage, and the movement and position information of the person is extracted using a movement analysis algorithm. The output is the analyzed movement pattern data.

[1437] Step 3:

[1438] The server inputs the analyzed behavior patterns into a generative AI model for evaluation. This generative AI model is trained with Keras and is designed to identify behavior patterns specific to molestation. The input is the analyzed behavior pattern data, and the AI ​​model determines whether abnormal behavior has been detected. The output is the presence or absence of abnormal behavior.

[1439] Step 4:

[1440] When abnormal behavior is detected, the server sends a notification to the security terminal. The notification includes the relevant video frame and related information (e.g., the date, time, and location of the detection). The input is the video data related to the abnormal behavior determination result, and the output is the notification data sent to the security terminal.

[1441] Step 5:

[1442] The security terminal displays the person's information based on the received notification. A pop-up display of the video data is also displayed, and an audio alert is also issued. The input is the notification data sent from the server, and the output is the video information and warning provided to the security guard.

[1443] Step 6:

[1444] The user (security guard) rushes to the relevant area based on the information from the security terminal and checks the situation on the scene. The input is the notification and video data from the security terminal, and the output is a prompt response at the scene.

[1445] Step 7:

[1446] The server records abnormal behavior data and stores it for re-learning. This data is used to retrain the generative AI model. The input is video data of abnormal behavior and its judgment results, and the output is data stored in the database for re-learning.

[1447] Step 8:

[1448] The smartphone application monitors the safety status of the user's surroundings and issues real-time alerts if suspicious behavior is detected. It uses a camera to collect video data and transmits it to a server. The input is the video data from the smartphone camera, and the output is an alert for detected suspicious behavior and transmission of the video data to the server.

[1449] Step 9:

[1450] The server receives the relevant data and immediately implements security measures. Security countermeasure instructions are sent to the relevant distribution network. The input is the video data sent from the smartphone application, and the output is notifications and response instructions to each security station.

[1451] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1452] The present invention provides a system that effectively detects molestation and recognizes the user's emotions to enable more appropriate responses. The system includes a monitoring device, a server, a terminal, an emotion engine, and a user (security guard).

[1453] System configuration and operation explanation

[1454] Data collection

[1455] The server receives video data in real time from surveillance cameras installed in stations and public facilities. Video data refers to information in the form of images and videos captured by surveillance cameras. The server continuously receives and processes this data with minimal delay.

[1456] Examples:

[1457] 1. Multiple cameras installed on station platforms monitor passenger movements in real time.

[1458] 2. The video feed from each camera is sent over the network to a server.

[1459] Behavioral analysis and identification

[1460] The server analyzes the received video data and tracks the person's movement patterns. Image analysis techniques (e.g., OpenPose and DeepSort) are used for video analysis. The analyzed movement patterns are input into a generative AI model that has previously learned the characteristics of molestation behavior, which then identifies abnormal behavior.

[1461] Examples:

[1462] 1. The server detects that a particular person is making unnatural hand movements.

[1463] 2. The generative artificial intelligence model evaluates whether the behavior matches the characteristics of molestation.

[1464] Emotion Analysis

[1465] Furthermore, the server uses an emotion engine to analyze the emotions of users (passengers) from the video data. The emotion engine has the following functions:

[1466] Facial expression analysis: Analyze facial expressions from video data to recognize specific emotions (e.g., fear, anxiety).

[1467] Vocal analysis: Analyzes the user's vocal changes from audio data to identify emotions.

[1468] Examples:

[1469] 1. The faces of people in the video are analyzed to detect expressions of fear.

[1470] 2. At the same time, the tense voice is analyzed from the audio data collected from the surrounding area.

[1471] Reporting and monitoring

[1472] If abnormal behavior is detected, the server notifies the security terminal. The notification includes the relevant video frame, the detected behavior pattern, and the emotion information identified by the emotion engine. The security terminal displays this information to the security guard in real time, helping them respond quickly.

[1473] Examples:

[1474] 1. The server notifies the terminal that suspected molestation behavior and feelings of fear have been detected.

[1475] 2. The device notifies the security guard of the relevant video and emotional information and issues an audio alert.

[1476] 3. Security personnel will promptly head to the area based on the information provided.

[1477] Record and Relearn

[1478] The server records the detected abnormal behaviors, emotional information, and the corresponding responses. This data is stored in a database and used to retrain the generative AI model. Based on the recorded data, the model is retrained to continuously improve the system's performance.

[1479] Examples:

[1480] 1. The server stores the detected abnormal behavior and emotional information in a database.

[1481] 2. Periodically use the saved data to retrain the generative AI model and update it with new detection algorithms.

[1482] summary

[1483] This system analyzes video data from surveillance equipment in real time to detect molestation. It also uses an emotion engine to analyze the user's facial expressions and vocal emotions, enabling more accurate judgment and response. Furthermore, the generative AI model can be retrained based on the recorded data, continuously improving the system's accuracy and effectiveness.

[1484] The processing flow will be explained below.

[1485] Step 1:

[1486] The server receives video data from the surveillance equipment in real time. A communication protocol (e.g., RTSP) is used to establish continuous data streaming between the surveillance camera and the server. The received video data is temporarily stored in a buffer and processed to minimize delays.

[1487] Step 2:

[1488] The server analyzes the received video data and uses video analysis algorithms (e.g., OpenPose or DeepSort) to detect people in each frame and track their movement patterns. This allows each person's movements to be continuously tracked and their location and movements recorded in detail.

[1489] Step 3:

[1490] The server inputs the analyzed behavior patterns into a generative AI model. The generative AI model evaluates the behavior patterns based on pre-learned characteristics of molestation. The model outputs a score indicating whether the behavior is abnormal, and if it exceeds a certain threshold, it is determined to be abnormal behavior.

[1491] Step 4:

[1492] If abnormal behavior is detected, the server activates the emotion engine. The emotion engine analyzes the facial expressions of the user (passenger) from the video data and detects specific emotions (e.g., fear, anxiety). If necessary, it also analyzes surrounding audio data and identifies emotions from changes in the vocal cords.

[1493] Step 5:

[1494] The server notifies the security terminal of the abnormal behavior determination result and the emotion information identified by the emotion engine. The notification includes the video frame of the abnormal behavior, the evaluation result of the movement pattern, and the emotion information. This information is sent to the security terminal in real time.

[1495] Step 6:

[1496] The terminal receives notifications from the server and displays them to the security guard. The terminal's user interface displays a pop-up message with information about the person's emotions along with any abnormal behavior, and an audio alert is issued. The security guard can immediately check the content of the notification.

[1497] Step 7:

[1498] The user (security guard) rushes to the scene based on the notification from the device. After arriving at the scene, they check the actual situation and take appropriate action, such as intervening or guiding the guardian, if necessary. They also collect additional evidence depending on the situation at the scene.

[1499] Step 8:

[1500] The server records the detected abnormal behaviors and emotional information, as well as the response results. This data is stored in a database and used to retrain the generative AI model at a later date. The recorded data includes details of the abnormal behavior, the date and time, the location, and the response results.

[1501] Step 9:

[1502] The server periodically uses the recorded data to retrain the generative AI model. The model is retrained based on newly accumulated case data to improve its accuracy. Retraining makes it possible to continuously improve the effectiveness of the entire system.

[1503] Example 2

[1504] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1505] Groping in public places is becoming an increasingly serious problem, but conventional surveillance systems have struggled to effectively detect such behavior and respond quickly. Furthermore, analyzing user emotions would improve the appropriateness of responses, but no such systems currently exist. Therefore, there is a need for a system that can quickly and effectively detect groping and respond more appropriately by analyzing user emotions.

[1506] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1507] In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a person's movement patterns, means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior, means for analyzing the user's emotions from the video data and audio data, means for notifying a security terminal when abnormal behavior is detected, means for the security terminal that receives the notification to display information about the person, and means for recording the abnormal behavior and emotion analysis data and saving it for re-learning. This makes it possible to quickly and effectively detect molestation in public places and improve the appropriateness of responses by analyzing the user's emotions.

[1508] "Monitoring equipment" refers to equipment such as cameras installed to acquire video data in real time.

[1509] "Video data" refers to information in the form of images and videos captured by a surveillance device.

[1510] "Server" refers to a computer system for processing and analyzing received video data.

[1511] "Movement patterns" refer to the characteristics detected over a series of frames of a particular person's movements or actions.

[1512] A "generative artificial intelligence model" refers to a machine learning model that is trained in advance using learning data to evaluate and judge specific patterns and behaviors.

[1513] "Emotion engine" refers to an algorithm or system for analyzing video and audio data to identify a user's emotions.

[1514] "Voice data" refers to acoustic information containing the user's voice and is used for emotion analysis.

[1515] "Security terminal" refers to a computer device used to receive notifications of abnormal behavior, emotional information, etc.

[1516] "Abnormal behavior" refers to a person's actions that the generative AI model identifies as groping or other suspicious behavior.

[1517] "Notification" refers to abnormal behavior and related information sent from the server to the security terminal.

[1518] "Recording" refers to the process of storing detected abnormal behavior and emotional information in a database or the like.

[1519] "Relearning" refers to the process of updating and improving a generative AI model based on recorded data.

[1520] "User" refers to a security guard or station user who uses the system.

[1521] The present invention provides a system that effectively detects molestation in public places and recognizes the user's emotions to provide an appropriate response. Specific embodiments of this system will be described below.

[1522] System configuration and operation explanation

[1523] Data collection

[1524] The server receives video data in real time from monitoring devices (cameras) installed in stations and public facilities. This video data includes information in the form of images and videos. For example, multiple cameras installed on a station platform monitor passenger movements in real time, and the video feed is sent to the server via a network.

[1525] Behavioral analysis and identification

[1526] The server analyzes the received video data and tracks the person's movement patterns. This analysis uses image analysis technologies such as OpenPose and DeepSort. The analyzed movement patterns are input into a generative AI model that has previously learned the characteristics of molestation, which then identifies abnormal behavior. As a specific example, the server detects an unnatural hand movement of a specific person, and the generative AI model evaluates that movement as molestation.

[1527] Emotion Analysis

[1528] Furthermore, the server uses an emotion engine to analyze the user's (passenger's) emotions from the video data. The emotion engine has two functions: facial expression analysis and vocal analysis, and recognizes specific emotions (e.g., fear, anxiety). As a specific example, the face of a person in the video is analyzed to detect a fearful expression, and at the same time, a tense voice is detected from the audio data collected from the surrounding area.

[1529] Reporting and monitoring

[1530] The server then sends a summary of the detected abnormal behavior and emotional information to the security terminal. The notification includes the relevant video frame, the detected behavior pattern, and the emotional information identified by the emotion engine. As a specific example, the server detects behavior that may be suggestive of molestation and the emotion of fear, and sends this information to the security terminal. The security terminal then displays the received information to the security guard in real time and issues an audio alert to encourage a prompt response.

[1531] Record and Relearn

[1532] The server stores the detected abnormal behavior, emotional information, and corresponding results in a database. This data is used to retrain the generative AI model. For example, the server stores the detected abnormal behavior and emotional information in a database and periodically uses the stored data to retrain the generative AI model and improve the accuracy of the system.

[1533] Specific examples

[1534] Here is an example of a prompt for this system:

[1535] "Write a program to detect groping on a train platform and identify the fear of the passengers around it."

[1536] "Analyze a person's movements and emotions in real time and generate system prompts to notify security personnel if anything unusual occurs."

[1537] Hardware and software used

[1538] This system uses the following hardware and software:

[1539] Hardware: Surveillance equipment (cameras), servers, security terminals

[1540] Software: OpenPose (image analysis), DeepSort (motion tracking), generative AI model (abnormal behavior detection), emotion engine (facial expression and vocal cord analysis)

[1541] As described above, the system of the present invention can quickly and effectively detect molestation in public places and improve the appropriateness of responses by analyzing the user's emotions.

[1542] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1543] Step 1:

[1544] Receiving video data

[1545] The server receives video data in real time from monitoring devices (cameras) installed at stations and public facilities.

[1546] Input: Video feed from camera

[1547] Output: Received video data

[1548] Specific operation: The server receives video feeds from multiple cameras simultaneously over the network.

[1549] Step 2:

[1550] Video data formatting

[1551] The server formats the received video data at a fixed frame rate and converts it into a format that can be applied to analysis processing.

[1552] Input: Received video data

[1553] Output: Video data formatted for each frame

[1554] Specific operation: The server divides the received data into frames and saves them in a format that can be applied to subsequent analysis processing.

[1555] Step 3:

[1556] Motion detection

[1557] The server detects the movement patterns of each person from the video data, using image analysis technologies such as OpenPose and DeepSort.

[1558] Input: Formatted video data

[1559] Output: Movement patterns for each person

[1560] Specific operation: The server uses OpenPose to identify the person's posture and joint positions for each frame, and uses DeepSort to track their movement patterns.

[1561] Step 4:

[1562] Identifying abnormal behavior

[1563] The server inputs the detected behavioral patterns into a generative AI model and evaluates whether they match the characteristics of molestation.

[1564] Input: Movement pattern

[1565] Output: Abnormal behavior determination result

[1566] Specific actions: The server inputs movement patterns into a pre-trained generative AI model and evaluates whether unnatural movements are judged to be acts of molestation.

[1567] Step 5:

[1568] Emotion Analysis

[1569] The server simultaneously analyzes the user's emotions from the video and audio data, and uses an emotion engine to analyze facial expressions and vocal chords to recognize specific emotions (e.g., fear, anxiety).

[1570] Input: Video and audio data

[1571] Output: User's emotional information

[1572] How it works: The server uses facial recognition technology to detect the faces of people in the video and analyze whether they show any fearful expressions. At the same time, it analyzes the audio data to determine whether the vocal cords are tense.

[1573] Step 6:

[1574] Abnormality notification

[1575] The server compiles the detected abnormal behavior and emotional information and notifies the security terminal.

[1576] Input: Abnormal behavior judgment results and emotional information

[1577] Output: Notification to security terminal

[1578] Specific operation: The server detects movements that are suspected to be molestation and emotions of fear, and sends them to the security terminal.

[1579] Step 7:

[1580] Sending alerts

[1581] The device displays the received information to the security guard and issues an audio alert if necessary.

[1582] Input: Notification from the server

[1583] Output: Display and audio alert to security guards

[1584] Specific operation: The device will sound an alert and display the relevant video and emotional information on the security guard's screen.

[1585] Step 8:

[1586] Data recording

[1587] The server records the detected abnormal behavior and emotion information in a database.

[1588] Input: Abnormal behavior judgment results and emotional information

[1589] Output: Records in the database

[1590] Specific operation: The server stores the detected abnormal behavior and the associated emotional information in a database.

[1591] Step 9:

[1592] Retraining the Model

[1593] The server uses the recorded data to retrain the generative AI model, continuously improving the accuracy of the system.

[1594] Input: Abnormal behavior and emotional information recorded in the database

[1595] Output: Improved generative AI model

[1596] How it works: Periodically, the server retrains the generative AI model with new data to improve its accuracy in identifying new patterns.

[1597] (Application example 2)

[1598] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1599] To ensure the safety of passengers in self-driving vehicles, a system is needed that can immediately detect abnormal behavior, such as sexual harassment or trouble between passengers, and respond as necessary. However, existing systems rely solely on the analysis of video data, and are unable to respond by taking into account changes in the user's emotions or voice data. This makes it difficult to respond effectively and quickly.

[1600] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving video data acquired by a monitoring device in real time, means for analyzing the received video data and tracking a person's movement patterns, means for evaluating the analyzed movement patterns using a generative artificial intelligence model and determining abnormal behavior, means for analyzing the user's emotions from the video data and audio data, means for notifying a security terminal when abnormal behavior or emotions such as anxiety or fear are detected, means for the security terminal that receives the notification to display information about the person, and means for recording the abnormal behavior and emotion data and saving it for re-learning. This enables a more effective and rapid response that combines abnormal behavior detection and emotion analysis in an autonomous vehicle.

[1601] "Monitoring equipment" refers to equipment including cameras and microphones installed to capture user behavior patterns and environmental conditions in real time.

[1602] "Video data" refers to information in the form of moving images collected by a monitoring device, and is a record of the user's actions and the environment.

[1603] "Audio data" refers to audio information acquired by a microphone installed in a monitoring device, and includes environmental sounds and the user's voice.

[1604] "Movement pattern" refers to the repetition and characteristics of movements or actions by a particular person.

[1605] A "generative artificial intelligence model" is an artificial intelligence technology that has the ability to learn specific patterns and characteristics based on large amounts of data and then evaluate and judge them.

[1606] "Abnormal behavior" refers to unnatural actions or behavior that differ from the behavior of ordinary passengers, and specifically includes acts of molestation and trouble between passengers.

[1607] "Emotion analysis" is a technology that determines a user's emotions by recognizing facial expressions from video data and analyzing vocal cords from audio data.

[1608] The "security terminal" is a device that receives notifications from the server and displays abnormal behavior and emotion analysis results to security guards in real time.

[1609] "Retraining" is the process of retraining an existing artificial intelligence model based on recorded data to improve its accuracy and responsiveness.

[1610] An "autonomous vehicle" is a vehicle that can perform driving operations automatically using artificial intelligence and sensor technology.

[1611] The system for realizing this application example is based on technology that ensures passenger safety in autonomous vehicles and analyzes abnormal behavior and passenger emotions. The specific configuration and operation of the system are described below.

[1612] System Configuration

[1613] The system consists of the following main components:

[1614] 1. Surveillance equipment: Consists of multiple cameras and microphones installed inside the vehicle. This surveillance equipment collects video and audio data in real time.

[1615] 2. Server: A server equipped with a high-performance NVIDIA GPU. This server processes and analyzes the received data to determine abnormal behavior and emotions.

[1616] 3. Security terminal: A device that receives notifications from the server and displays information to security guards in real time. This can be a tablet or a dedicated mobile device.

[1617] Data processing and analysis

[1618] The server performs the following steps in sequence:

[1619] 1. Data reception: Receives video and audio data sent from the monitoring device in real time.

[1620] 2. Motion Analysis: Video data is analyzed and human movement patterns are identified using OpenPose and DeepSort technologies. These patterns are then fed into a generative AI model to identify abnormal behavior.

[1621] 3. Sentiment Analysis: Additionally, Google Cloud Vision API and Watson Tone Analyzer are used to analyze passenger emotions from video and audio data, thereby identifying emotions such as fear and anxiety.

[1622] 4. Notification and response: If abnormal behavior or specific emotions are detected, the server will send real-time notifications to the security terminal and display the information to the security guards. If necessary, the autonomous vehicle can be controlled via the MQTT protocol.

[1623] Specific examples

[1624] To illustrate, consider the following scenario:

[1625] Detecting conflict between passengers: An in-vehicle camera captures unnatural movements (e.g., raising an arm) between passengers A and B, and the generative AI model determines this behavior as a "fight." Voice data is also analyzed at the same time, and if emergency words such as "help" are detected, the server immediately notifies the security terminal and, if necessary, stops the vehicle.

[1626] Anomaly detection through emotion analysis: If the microphone detects that a passenger's voice is filled with anxiety and the video data identifies a frightened expression, the server will use this information to notify the security terminal and take immediate action to ensure the passenger's safety.

[1627] Prompt Sentence Examples

[1628] The following prompt is an example used to train and initialize a generative AI model:

[1629] "Please collect video and audio data when Passenger A raises his arm."

[1630] "Use the Google Cloud Vision API to recognize fearful facial expressions."

[1631] "Use Watson Tone Analyzer to detect conflicting voice patterns"

[1632] This enables the system to combine abnormal behavior detection and emotion analysis within autonomous vehicles to respond more effectively and quickly.

[1633] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1634] Step 1:

[1635] The server receives video and audio data sent from the monitoring device in real time. Specifically, it acquires the data stream from the camera and microphone and stores it in the server's input buffer. The input is video and audio data, and the output is a data stream for analysis.

[1636] Step 2:

[1637] The server analyzes the received video data using OpenPose and DeepSort technologies to identify the movement patterns of the people. Specifically, it divides the video data into frames and performs pose estimation for each frame. It then performs object tracking based on the estimated pose data. The input is the video data, and the output is the movement patterns of each person.

[1638] Step 3:

[1639] The server inputs the analyzed movement patterns into a generative AI model to determine whether the movement patterns are abnormal. In this step, a pre-trained model is used to evaluate whether the movement patterns are abnormal. Specific operations include converting the movement patterns into specific feature quantities, inputting them into the model, and calculating an abnormality score. The input is the movement pattern, and the output is the result of the abnormal behavior determination.

[1640] Step 4:

[1641] The server performs emotion analysis using video and audio data. The emotion analysis uses Google Cloud Vision API and Watson Tone Analyzer to determine emotions from the user's facial expressions. Specifically, the process involves detecting faces from video frames and recognizing facial expressions, while also inputting audio data into a tone analyzer to identify the emotion of the voice. The input is video and audio data, and the output is emotion data.

[1642] Step 5:

[1643] If the server detects abnormal behavior or a specific emotion, it notifies the security terminal of this data. The notification uses the MQTT protocol to send video frames, identified behavior patterns, and emotion data to the security terminal. Specifically, this involves a process of assembling various data into packets and transmitting them in real time. The input is the abnormal behavior judgment result and emotion data, and the output is notification data sent to the security terminal.

[1644] Step 6:

[1645] When the security terminal receives a notification, it displays the relevant person's information in real time. Specific operations include interpreting the received data and reflecting it on the UI. The terminal screen displays the person's image, movement patterns, and emotion data, allowing security guards to respond immediately. The input is notification data, and the output is display information.

[1646] Step 7:

[1647] The server records the detected abnormal behavior and emotion data and stores it in a database for re-learning. In this step, the collected data is properly formatted and written to a database for long-term storage. Specifically, the abnormal behavior features and emotion data are saved and made available for re-learning at a later date. The input is the abnormal behavior judgment result and emotion data, and the output is the record saved in the database.

[1648] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1649] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1650] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1651] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1652] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1653] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1654] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1655] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1656] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1657] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1658] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1659] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1660] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1661] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1662] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1663] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1664] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1665] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1666] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1667] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1668] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1669] The following is further disclosed regarding the above embodiment.

[1670] (Claim 1)

[1671] means for receiving video data acquired by a monitoring device in real time;

[1672] means for analyzing the received video data and tracking the movement patterns of the person;

[1673] A means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior;

[1674] a means for notifying a security terminal when abnormal behavior is detected;

[1675] a means for displaying the information of the person in question by the security terminal that received the notification;

[1676] The system includes means for recording and storing said abnormal behavior data for retraining.

[1677] (Claim 2)

[1678] 2. The system according to claim 1, wherein the generative artificial intelligence model is a model that learns behavior patterns specific to molestation acts.

[1679] (Claim 3)

[1680] 2. The system according to claim 1, wherein the monitoring devices are a plurality of cameras installed at a station.

[1681] "Example 1"

[1682] (Claim 1)

[1683] means for receiving video data acquired by a monitoring device in real time;

[1684] means for converting the received video data into an appropriate format;

[1685] means for analyzing the received video data to track a person's movement pattern;

[1686] means for evaluating the analyzed behavioral patterns using a generative artificial intelligence model to identify suspicious behavior;

[1687] a means for notifying a security terminal when suspicious activity is detected;

[1688] a means for displaying the information of the person in question by the security terminal that received the notification;

[1689] The system includes a means for recording data of the suspicious activity and storing it for re-learning.

[1690] (Claim 2)

[1691] 2. The system according to claim 1, wherein the generative artificial intelligence model is a model that learns behavior patterns specific to molestation acts.

[1692] (Claim 3)

[1693] 2. The system according to claim 1, wherein the monitoring devices are a plurality of cameras installed in a public facility.

[1694] "Application Example 1"

[1695] (Claim 1)

[1696] means for receiving video data acquired by a monitoring device in real time;

[1697] means for analyzing the received video data and tracking the movement patterns of the person;

[1698] A means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior;

[1699] a means for notifying a security terminal when abnormal behavior is detected;

[1700] a means for displaying the information of the person in question by the security terminal that received the notification;

[1701] means for recording and storing said abnormal behavior data for re-learning;

[1702] It operates as an application installed on a smartphone, monitors the safety status of the user's surroundings, and issues a real-time warning if suspicious behavior is detected.

[1703] A system that includes a means for sending image data of the relevant frame to a server when suspicious behavior is confirmed and implementing security measures.

[1704] (Claim 2)

[1705] 2. The system according to claim 1, wherein the generative artificial intelligence model is a model that learns behavior patterns specific to molestation acts.

[1706] (Claim 3)

[1707] 2. The system of claim 1, wherein the surveillance devices are a plurality of cameras installed in a public place.

[1708] "Example 2: Combining Emotion Engines"

[1709] (Claim 1)

[1710] means for receiving video data acquired by a monitoring device in real time;

[1711] means for analyzing the received video data and tracking the movement patterns of the person;

[1712] A means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior;

[1713] means for analyzing user emotions from video data and audio data;

[1714] a means for notifying a security terminal when abnormal behavior is detected;

[1715] a means for displaying the information of the person in question by the security terminal that received the notification;

[1716] The system includes means for recording and storing said anomalous behavior and emotion analysis data for retraining.

[1717] (Claim 2)

[1718] 2. The system according to claim 1, wherein the generative artificial intelligence model is a model that learns behavior patterns specific to molestation acts.

[1719] (Claim 3)

[1720] 2. The system according to claim 1, wherein the monitoring devices are a plurality of cameras installed at a station.

[1721] "Application example 2 when combining emotion engines"

[1722] (Claim 1)

[1723] means for receiving video data acquired by a monitoring device in real time;

[1724] means for analyzing the received video data and tracking the movement patterns of the person;

[1725] A means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior;

[1726] means for analyzing user emotions from video data and audio data;

[1727] A means of notifying the security terminal when abnormal behavior or emotions such as anxiety or fear are detected;

[1728] a means for displaying the information of the person in question by the security terminal that received the notification;

[1729] The system includes means for recording and storing said anomalous behavioral and emotional data for retraining.

[1730] (Claim 2)

[1731] The system according to claim 1, characterized in that the generative artificial intelligence model is a model that learns behavior patterns specialized for molestation acts and trouble acts between passengers.

[1732] (Claim 3)

[1733] 10. The system of claim 1, wherein the monitoring device is a plurality of cameras and microphones installed in a public transport vehicle. [Explanation of symbols]

[1734] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving video data acquired by a monitoring device in real time; means for analyzing the received video data and tracking the movement patterns of the person; A means for evaluating the analyzed movement patterns using a generative artificial intelligence model to determine abnormal behavior; a means for notifying a security terminal when abnormal behavior is detected; a means for displaying the information of the person in question by the security terminal that received the notification; The system includes means for recording and storing said abnormal behavior data for retraining.

2. 2. The system according to claim 1, wherein the generative artificial intelligence model is a model that learns behavior patterns specific to molestation.

3. 2. The system of claim 1, wherein the monitoring devices are a plurality of cameras installed in a station.

Images