System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring video data from a video acquisition device; means for dividing the acquired video data into frames and performing preprocessing; means for analyzing the preprocessed video data and detecting a moving object; means for analyzing an action of the detected moving object and evaluating a possibility of a criminal action; means for transmitting data to a generative artificial intelligence to generate evidence information when a criminal action is detected; and means for organizing the generated evidence information and automatically transmitting the evidence information to police.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, criminal activity that threatens public safety has been increasing, creating a demand for surveillance systems that can detect it quickly. Conventional surveillance systems require manual monitoring of large amounts of video data, which is labor-intensive and time-consuming. In addition, it often takes time for a criminal activity to be reported to the police after it occurs, making it difficult to respond quickly. It is necessary to solve these issues and detect and respond to criminal activity efficiently and immediately. [Means for solving the problem]

[0005] The present invention provides a system for detecting moving objects and analyzing their behavior by acquiring video data from a video acquisition device and analyzing it in real time. Specifically, the system includes a means for dividing acquired video data into frames and performing preprocessing, and a means for detecting moving objects based on the preprocessed video data. The system also includes a means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior. Furthermore, if criminal behavior is detected, the system includes a means for transmitting the data to a generating artificial intelligence and generating evidentiary information, and a means for automatically organizing the generated evidentiary information and sending it to the police. This system enables more efficient surveillance and a faster response.

[0006] "Video acquisition device" refers to a device for acquiring video data, such as a surveillance camera or camera system.

[0007] "Video data" refers to data including video information acquired from a surveillance camera or the like.

[0008] "Frame unit" refers to the unit by which video data is divided into individual still images (frames).

[0009] "Preprocessing" refers to performing processes such as noise removal and resolution adjustment on video data prior to analysis.

[0010] "Moving object" refers to an object or person that moves within surveillance footage.

[0011] "Behavioral analysis" refers to analyzing the movement patterns and behavior of detected moving objects.

[0012] "Criminal behavior" refers to behavior that threatens public safety, such as theft or violence.

[0013] "Generative artificial intelligence" refers to an artificial intelligence system for generating evidential information based on received data.

[0014] "Evidence information" refers to detailed information (such as date, time, location, and details of actions) that proves the detection of criminal activity.

[0015] "Police" refers to law enforcement agencies that protect public safety. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The overall system configuration and program processing flow for implementing this invention are shown below. Specifically, it explains how the server, terminals, and users work together to operate the system, automatically detect criminal behavior, and generate and transmit evidence information.

[0038] System Configuration

[0039] The system mainly consists of the following components:

[0040] 1. Video capture devices - Various surveillance cameras in stores, public facilities, etc.

[0041] 2. Server - Receives data from the video capture device, analyzes it, and generates and transmits evidence information.

[0042] 3. Generative AI - An artificial intelligence system that generates detailed evidence information based on data on detected criminal behavior.

[0043] 4. Police system - receives and responds to the evidence generated.

[0044] What the program does

[0045] Image acquisition and preprocessing

[0046] Video acquisition

[0047] The server acquires video data from the surveillance camera in real time.

[0048] For example, a server uses the RTSP protocol to periodically connect to a video capture device and receive a video stream.

[0049] Pretreatment

[0050] The video data acquired by the server is divided into frames.

[0051] Preprocessing such as noise removal is performed on each frame to improve data quality.

[0052] For example, apply a denoising filter using the OpenCV library.

[0053] Motion detection and behavior analysis

[0054] Motion Detection

[0055] The server analyzes the pre-processed video data, calculates the differences between frames, and detects moving objects.

[0056] If a moving object is detected, its location is identified and recorded using a bounding box.

[0057] Behavioral analysis

[0058] The server analyzes the location, speed, direction, etc. of the detected moving object.

[0059] The Kaliman filter and tracking algorithms are used to track the target and analyze its behavioral patterns.

[0060] For example, identifying criminal behavior such as quickly removing an object within a certain area.

[0061] Automated detection of criminal behavior

[0062] Rule-based Decision

[0063] The server applies rules to assess the likelihood of criminal behavior based on the analyzed behavioral data.

[0064] Automatically detects suspicious behavior based on pre-defined rules.

[0065] Alarm Generation

[0066] If the server detects suspicious activity, it generates a real-time alarm and notifies the management terminal.

[0067] Generating evidence

[0068] Data transmission

[0069] The server sends details of the suspicious activity to the generating artificial intelligence.

[0070] For example, behavioral data is sent to the generation AI in JSON format.

[0071] Evidence generation

[0072] The generation AI generates detailed evidence information based on the data it receives.

[0073] The evidence information includes a detailed description of the action, date and time, location information, etc.

[0074] Automatic transmission to police

[0075] Data reduction and format conversion

[0076] The server receives the evidence information from the generation AI and converts it into the appropriate format.

[0077] Convert evidence into email format and attach links and relevant images.

[0078] send

[0079] The server automatically sends evidence information to the police system.

[0080] For security reasons, SMTP or other communication protocols are used.

[0081] Specific examples

[0082] Image acquisition and preprocessing

[0083] Example 1

[0084] The server acquires video data from the surveillance cameras in the store via RTSP.

[0085] The video data is divided into 30fps frames and noise is removed using an OpenCV filter.

[0086] Motion detection and behavior analysis

[0087] Example 2

[0088] The server uses background subtraction to detect customers moving around the store.

[0089] A Kaliman filter is used to analyze the customer's location and movement speed to detect unnatural behavior.

[0090] Automated detection of criminal behavior

[0091] Example 3

[0092] The server detects the act of quickly removing an item from a specific area as "theft."

[0093] If any suspicious activity is detected, a real-time notification is sent to the management terminal via WebSocket.

[0094] Generating and sending evidence

[0095] Example 4

[0096] The server sends behavioral data to the generation AI, which then generates evidentiary information including date, time, location, and behavioral details.

[0097] Evidence information is converted into email and automatically sent to the police system.

[0098] Through these processes, this system will improve the efficiency of surveillance work and enable rapid crime response.

[0099] The processing flow will be explained below.

[0100] Step 1:

[0101] The server connects to the surveillance camera and acquires real-time video data. Specifically, the server receives the video stream using the RTSP protocol. The server continues to acquire data from the camera at regular intervals.

[0102] Step 2:

[0103] The server divides the acquired video data into frames. Specifically, it divides the video data into 30 frames per second (fps) and converts each frame into an easy-to-handle format. Next, preprocessing is performed, such as noise removal and resolution unification. The server performs preprocessing using the OpenCV library.

[0104] Step 3:

[0105] The server analyzes the preprocessed video data to detect moving objects. Specifically, it uses background subtraction and optical flow to calculate the difference between consecutive frames and identify moving areas. The server then uses bounding boxes to surround moving objects.

[0106] Step 4:

[0107] The server tracks the detected moving object and analyzes its behavior. Using a Kaliman filter and local tracking algorithm, the server calculates the object's location, speed, and direction of movement. This allows the server to analyze the object's movement pattern and identify any unnatural behavior.

[0108] Step 5:

[0109] The server evaluates the likelihood of criminal activity based on the results of behavioral analysis. Specifically, it uses a pre-configured rule-based system to automatically detect suspicious behavior. For example, quickly retrieving an object in a specific area is detected as criminal activity.

[0110] Step 6:

[0111] If the server detects suspicious activity, it generates a real-time alarm. Specifically, it uses WebSocket to send a notification to the management terminal, prompting immediate action. The server notifies the management terminal with detailed data about the suspicious activity.

[0112] Step 7:

[0113] The server sends suspicious behavior data to the generation AI, which then formats the behavior data in JSON format and sends it to the generation AI's API endpoint, which then starts generating detailed evidence.

[0114] Step 8:

[0115] The generation AI generates detailed evidence information based on the received data. Specifically, it generates a detailed description of the behavior, the date and time, the location, and the identification information of related people and objects. The generated evidence information is returned to the server.

[0116] Step 9:

[0117] The server receives the generated evidence and converts it into the appropriate format. The evidence is formatted as an email and includes any relevant images or video links.

[0118] Step 10:

[0119] The server automatically sends evidence information to the police system. Specifically, it sends evidence information to a dedicated police email address using the SMTP protocol. This allows evidence information to be shared quickly and securely.

[0120] Example 1

[0121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] Conventional surveillance systems require manual analysis of video data obtained from surveillance cameras, making it difficult for limited personnel to review large volumes of data. Furthermore, the accuracy of motion detection and behavior analysis is low, making it difficult to detect suspicious activity. This has led to issues with early detection of criminal activity and rapid response.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0124] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data and detecting moving objects, means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior, means for transmitting data to a generation artificial intelligence to generate evidence information when criminal behavior is detected, means for organizing the generated evidence information and automatically transmitting it to the police, means for acquiring video data from a surveillance camera in real time using the RTSP protocol, means for applying a noise reduction filter to the acquired video data using the OpenCV library, means for detecting moving objects using background subtraction, means for analyzing the behavioral patterns of moving objects using a Kaliman filter or tracking algorithm, and means for notifying a management terminal of suspicious behavior in real time using WebSocket. This enables automatic analysis of large amounts of video data, highly accurate detection of suspicious behavior, and rapid detection and response to criminal behavior.

[0125] The "image acquisition device" is a device for acquiring image data such as a surveillance camera.

[0126] "Video data" refers to digital video information obtained from a video capture device such as a surveillance camera.

[0127] "Server" means a central system for analyzing video data received from video capture devices and detecting and responding to criminal activity.

[0128] "Preprocessing" is a process of improving the quality of acquired video data by performing processes such as noise removal and frame division.

[0129] A "frame unit" is a unit into which video data is divided into individual still images.

[0130] A "moving object" is a moving object or person that is detected by the difference between frames.

[0131] "Motion detection" is the process of analyzing pre-processed video data and calculating the differences between frames to find moving objects.

[0132] "Behavioral analysis" refers to analyzing the location information, movement speed, and movement direction of a detected moving object to understand its behavioral patterns.

[0133] "Assessing the possibility of criminal behavior" refers to applying pre-set rules to the analyzed behavioral data to determine whether or not there is a possibility of criminal behavior.

[0134] "Generative AI" is an AI system that automatically generates detailed evidence information based on data on criminal behavior.

[0135] "Evidence information" is data containing detailed information about criminal activity, including date, time, location, and activity details.

[0136] "Automatic transmission to police" refers to the process of converting the generated evidence information into an appropriate format and automatically transmitting it to the police system.

[0137] The "RTSP protocol" is a communication protocol for acquiring video streams in real time.

[0138] The "OpenCV library" is an open-source library for performing computer vision tasks.

[0139] A "noise reduction filter" is a tool used to remove unnecessary noise from video data and improve the quality of the data.

[0140] "Background subtraction" is a technique used for motion detection, which detects moving objects by comparing them with the background.

[0141] The "Kaliman filter" is an algorithm for tracking the location information of moving objects, and estimates and predicts their location.

[0142] A "tracking algorithm" is an algorithm for tracking the successive positions of a moving object.

[0143] "WebSocket" is a protocol for two-way communication between a server and a management terminal.

[0144] A "management terminal" is a terminal device for receiving notifications of detected criminal behavior, and is usually used by an administrator.

[0145] The overall system configuration and specific program processing for implementing this invention are described below. The system consists of a video capture device including a surveillance camera, a server that analyzes and generates and transmits evidence information, a generation AI, and a police system.

[0146] Video data acquisition and preprocessing

[0147] The server receives video data from surveillance cameras in real time using the RTSP protocol. For example, it receives a video stream from a surveillance camera in a store, divides it into frames, and applies a noise reduction filter to each frame using the OpenCV library. This improves the quality of the data and makes the subsequent analysis process smoother.

[0148] Motion detection and behavior analysis

[0149] The server analyzes the preprocessed video data and detects moving objects using background subtraction. If a moving object is detected, its location is identified using a bounding box and its location information is recorded. The server then analyzes the location, speed, and direction of the detected moving object using a Kaliman filter and tracking algorithm to understand its behavioral patterns.

[0150] Automated detection of criminal behavior

[0151] The server then applies pre-defined rules to assess the likelihood of criminal activity based on the analyzed behavioral data. For example, a rule could be applied that defines "taking an item from a shelf within five seconds" as theft, and if suspicious activity is detected, a real-time alarm is generated and a notification is sent to the management terminal via WebSocket.

[0152] Generating and sending evidence

[0153] The server sends the detected behavioral data in JSON format to the generation AI. The generation AI generates detailed evidence information based on the received data, creating content including date, time, location, and behavior details. The server then converts this evidence information into email format and automatically sends it to the police system using the SMTP protocol.

[0154] Specific examples

[0155] For example, the server acquires a 30fps video stream from a store's surveillance cameras using the RTSP protocol and removes noise using OpenCV's GaussianBlur filter. Next, it detects moving people using background subtraction and tracks their location using a Kaliman filter. If the server detects someone quickly removing an item in a specific area, it sends an alarm notification to the management terminal in real time via WebSocket. The server then sends the behavioral data to the generation AI, which creates detailed evidence information. The server then converts this evidence information into email format and sends it to the police system using the SMTP protocol.

[0156] Example prompts for generative AI models

[0157] For example, the following prompt could be entered into the generation AI: "Generate detailed evidence information based on the following data: Data: Date and time: October 10, 2023, Location: Store A, Action: A person quickly removed an item from a specific area."

[0158] Through the above process, this invention automatically analyzes large amounts of video data, detects suspicious activity with high accuracy, and enables rapid detection and response to criminal activity. Furthermore, by automatically transmitting the generated detailed evidence information to the police, it supports rapid legal response.

[0159] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0160] Step 1:

[0161] The server acquires video data from a surveillance camera. As input, it uses the RTSP stream URL of the surveillance camera. Specifically, the server connects to the surveillance camera using the RTSP protocol and receives the video stream in real time. As output, it obtains the stream data.

[0162] Step 2:

[0163] The server divides the acquired video data into frames. The input is real-time video stream data. Specifically, the server divides the video stream into 30 frames per second (30 fps). The output is individual frame data.

[0164] Step 3:

[0165] The server performs noise reduction on each frame. As input, it receives the segmented frame data. Specifically, it applies a Gaussian Blur filter using the OpenCV library to remove noise from the frame. As output, it obtains high-quality frame data.

[0166] Step 4:

[0167] The server analyzes the preprocessed frame data and detects moving objects. The input is noise-removed frame data. Specifically, it uses background subtraction to detect movement between frames and identifies the location of moving objects using bounding boxes. The output is the location information of the moving objects.

[0168] Step 5:

[0169] The server analyzes the behavior of the detected moving object. The input is the location information of the moving object. Specifically, it uses a Kaliman filter and tracking algorithm to analyze the object's location, speed, and direction of movement, and identifies its behavioral pattern. The output is the analyzed behavioral data.

[0170] Step 6:

[0171] The server evaluates the likelihood of criminal behavior based on the analyzed behavioral data. The input is the analyzed behavioral data. Specific actions are detected by applying pre-set rules. For example, "taking an item from a shelf within five seconds" is evaluated as theft. The output is a notification of suspicious behavior.

[0172] Step 7:

[0173] If the server detects suspicious behavior, it generates an alarm and notifies the management terminal. The input is a notification of the detected suspicious behavior. The specific operation is to send a real-time notification to the management terminal using the WebSocket protocol. The output is a notification displayed on the management terminal.

[0174] Step 8:

[0175] The server sends behavioral data to the generation AI to generate evidential information. The input is detailed data on suspicious behavior. Specifically, the behavioral data is sent to the generation AI in JSON format, and the AI ​​generates evidential information including date, time, location, and details of the behavior. The generated evidential information is obtained as output.

[0176] Step 9:

[0177] The server automatically sends the generated evidence information to the police. The input is the evidence information obtained from the generation AI. Specifically, the server converts the evidence information into email format, attaches links and related images, and sends it to the police system using the SMTP protocol. The output is the evidence information sent to the police system.

[0178] (Application example 1)

[0179] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0180] Automatic detection systems for criminal behavior using surveillance cameras are essential for responding quickly when a crime occurs. However, current systems not only need to analyze video data from surveillance cameras in real time, but also need the ability to quickly and accurately generate and transmit detailed evidence information when criminal behavior is detected. Real-time alert generation and notification are also required to enable police to respond immediately. To achieve this, an effective system is required that integrates motion detection, behavior analysis, the generation of evidence information using generative AI, and automatic transmission functions.

[0181] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0182] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data and detecting moving objects, means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior, means for transmitting data to a generating AI and generating evidential information when criminal behavior is detected, means for organizing the generated evidential information and automatically transmitting it to the police, and means for saving screenshots and transmitting images to the police by email when criminal behavior is detected. This makes it possible to realize a system that detects criminal behavior in real time, quickly and accurately generates and transmits evidence, and enables the police to respond immediately.

[0183] An "image capture device" is a device for capturing image data in real time, such as a surveillance camera or video camera.

[0184] "Video data" refers to digital information that is organized for each frame of a captured video.

[0185] "Preprocessing" refers to the process of processing the acquired video data, such as by removing noise and dividing the data into frames, in order to improve the quality.

[0186] "Motion detection" is the process of identifying moving objects in video using differences between frames.

[0187] "Behavioral analysis" is a method of analyzing the movements of moving objects based on their location, speed, direction, etc., to find specific behavioral patterns.

[0188] The "means for assessing the likelihood of criminal activity" is a process that includes a rule-based algorithm for assessing and detecting the likelihood of criminal activity based on the analyzed data.

[0189] "Generative artificial intelligence" is an AI system that generates detailed evidential information based on specified data.

[0190] "Evidential information" is information necessary to prove criminal activity, such as the date, time, location, and details of the action.

[0191] "Automatic transmission" is the process of quickly transmitting generated evidence information to the police using a pre-defined protocol.

[0192] "Real-time alert" is a warning system that immediately notifies you of any suspicious behavior when it is detected, prompting you to take action.

[0193] A "screenshot" is a technique for capturing video data at a specific point in time and saving it as a still image.

[0194] "Email transmission" is a communication method for transmitting information in the form of email via the Internet.

[0195] To implement this invention, the following specific system configuration and processing procedures are required. The entire system operates through the cooperation of three parties: a server, a terminal, and a user. Automatic detection of criminal behavior and generation and transmission of evidence information are carried out according to the following procedures.

[0196] System Configuration

[0197] 1. Video acquisition device: A device such as a surveillance camera installed in stores and public facilities that acquires video data in real time.

[0198] 2. Server: Plays the central role of receiving and analyzing the video data sent from the video capture device, and generating and sending evidence information.

[0199] 3. Generative AI model: An artificial intelligence system that generates detailed evidential information based on data sent from the server.

[0200] 4. Police system: A system for receiving and responding to the evidence information generated.

[0201] What the program does

[0202] Image acquisition and preprocessing

[0203] The server acquires video data in real time from a surveillance camera. A typical surveillance camera is used as the video acquisition device. The acquired video data is divided into frames and preprocessed using OpenCV filters, such as noise removal.

[0204] Motion detection and behavior analysis

[0205] The server analyzes the preprocessed video data and calculates the difference between frames to detect moving objects. Using a Kaliman filter and tracking algorithm, the server analyzes the location, speed, and direction of the detected moving object to identify its behavioral pattern. For example, the action of quickly removing an object within a certain area can be detected as "theft."

[0206] Automated detection of criminal behavior

[0207] The server uses rule-based decision-making to assess the likelihood of criminal activity based on the analyzed behavioral data. If suspicious activity is detected, it generates a real-time alarm and notifies the management terminal. It also has the ability to save screenshots and send images via email to the police if criminal activity is detected.

[0208] Generating and sending evidence

[0209] The server sends details of suspicious behavior to a generative AI model, which then generates evidence including the date, time, location, and details of the behavior. The server then organizes the evidence, converts it into an appropriate format, and automatically transmits it to the police system. A secure communication protocol, such as SMTP, is used for transmission.

[0210] Specific examples

[0211] 1. Example of image acquisition and pre-processing:

[0212] The server acquires video data from the surveillance cameras inside the store using the RTSP protocol, divides it into 30fps frames, and removes noise using an OpenCV filter.

[0213] 2. Examples of motion detection and behavior analysis:

[0214] The server uses background subtraction to detect customers moving around the store, and uses a Kaliman filter to analyze the customer's location and movement speed to detect any unnatural behavior.

[0215] 3. Examples of evidence generation and transmission to police:

[0216] The server sends the behavioral data to the generation AI, which then generates evidential information including the date, time, location, and details of the behavior. The generated evidential information is converted into an email and automatically sent to the police system.

[0217] Prompt Sentence Examples

[0218] "Write a script that grabs frames from an RTSP stream, detects suspicious activity, saves the image, and notifies the police."

[0219] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0220] Step 1:

[0221] The server acquires video data from the video acquisition device. Specifically, the server connects to the surveillance camera via the RTSP protocol and receives the video stream in real time. The input of this step is the video stream provided by the surveillance camera, and the output is the video data received by the server.

[0222] Step 2:

[0223] The server divides the acquired video data into frames and performs preprocessing. Specifically, the video data is divided into 30 fps frames and noise is removed using an OpenCV filter. The input to this step is the video data received in real time, and the output is the preprocessed frame data.

[0224] Step 3:

[0225] The server analyzes the preprocessed video data to detect moving objects. Specifically, it uses background subtraction to calculate the difference between frames and identify moving objects. The input to this step is the preprocessed frame data, and the output is data with identified moving objects.

[0226] Step 4:

[0227] The server analyzes the behavior of the detected moving object. Specifically, it analyzes the location information, movement speed, and movement direction of the moving object using a Kaliman filter or tracking algorithm. The input of this step is the data on the identified moving object, and the output is the analyzed behavior information of the moving object.

[0228] Step 5:

[0229] The server evaluates the likelihood of criminal behavior based on the analyzed behavioral data. Specifically, it uses rule-based judgment to identify suspicious behavior. The input to this step is the analyzed behavioral information of the moving object, and the output is data evaluating the likelihood of criminal behavior.

[0230] Step 6:

[0231] When criminal behavior is detected, the server sends the data to the generation AI to generate evidence. Specifically, the behavioral data is sent in JSON format to the generation AI model, which generates detailed evidence. The input to this step is the data determined to be criminal behavior, and the output is the generated evidence.

[0232] Step 7:

[0233] The server organizes the generated evidence information and automatically sends it to the police. Specifically, it converts the evidence information into an appropriate format and sends it to the police system using the SMTP protocol. The input to this step is the evidence information from the generative AI model, and the output is the evidence data sent to the police.

[0234] Step 8:

[0235] If the server detects a criminal activity, it saves a screenshot and sends the image to the police by email. Specifically, it sends a warning message along with the saved screenshot by email. The input of this step is the video data at the time when the suspicious activity was detected, and the output is the screenshot and warning message sent to the police.

[0236] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0237] This invention is a system that acquires video data from a video capture device, performs moving object detection and behavior analysis, and analyzes user emotions through an emotion engine, detects criminal behavior, and automatically transmits evidence of the behavior to the police. The system configuration, program processing flow, and specific examples are described below.

[0238] System Configuration

[0239] The system mainly consists of the following components:

[0240] 1. Video capture device - Surveillance cameras installed in stores, public facilities, etc.

[0241] 2. Server - Receives data from the video capture device, performs motion detection, behavior analysis, emotion analysis, and generates and transmits evidence information.

[0242] 3. Generative AI - Artificial intelligence that generates detailed evidence from data that detects criminal behavior.

[0243] 4. Emotion Engine - Analyzes the user's facial expressions and movements from video to recognize their emotional state.

[0244] 5. Police system - receives and responds to generated evidence information.

[0245] What the program does

[0246] Image acquisition and preprocessing

[0247] Video acquisition

[0248] The server obtains video data from the surveillance camera in real time, for example, by receiving the video stream using the RTSP protocol.

[0249] Pretreatment

[0250] The server performs preprocessing such as noise removal on the video data divided into frames, and improves the quality of the data using the OpenCV library.

[0251] Motion detection and behavior analysis

[0252] Motion Detection

[0253] The server analyzes the pre-processed video data and calculates the difference between consecutive frames to detect moving objects. It identifies and records moving objects using bounding boxes.

[0254] Behavioral analysis

[0255] The server analyzes the location information, movement speed, and movement direction of the moving object, and tracks the object and analyzes its behavior using a Kaliman filter and tracking algorithm.

[0256] Emotion analysis

[0257] emotion recognition

[0258] In addition to analyzing the behavior of moving objects, the server uses an emotion engine to analyze emotions from the user's facial expressions and movements. For example, it uses facial recognition technology to detect suspicious expressions or tension.

[0259] Emotional Data Evaluation

[0260] The server evaluates the likelihood of criminal behavior based on the recognized emotional data, and performs a comprehensive analysis of emotional changes and behavioral patterns according to a set algorithm.

[0261] Automated detection of criminal behavior

[0262] Rule-based Decision

[0263] The server uses a pre-configured rules-based system based on behavioral and emotional analysis to detect suspicious behavior, such as quickly picking up an object in a specific area or displaying a suspicious facial expression.

[0264] Alarm Generation

[0265] When the server detects suspicious activity, it generates a real-time alarm and sends a notification to the management terminal.

[0266] Generating evidence

[0267] Data transmission

[0268] The server sends suspicious behavior and emotion data to the generation AI, formats the data in JSON format, and sends it to the generation AI's API endpoint.

[0269] Evidence generation

[0270] The generation AI generates detailed evidence based on the received data, including a detailed description of the behavior, the date and time, the location, and the identity of the people and objects involved.

[0271] Automatic transmission to police

[0272] Data reduction and format conversion

[0273] The server receives the generated evidence and converts it into the appropriate format. The evidence is formatted as an email and includes any relevant images or video links.

[0274] send

[0275] The server automatically sends the evidence information to the police system. Specifically, it sends the evidence information to a dedicated email address for the police using the SMTP protocol.

[0276] Specific examples

[0277] Image acquisition and preprocessing

[0278] Example 1

[0279] The server receives video data from the surveillance cameras in the store via RTSP. The video data is split into 30fps frames and noise is removed using an OpenCV filter.

[0280] Motion detection and behavior analysis

[0281] Example 2

[0282] The server uses background subtraction to detect customers moving around the store, and a Kaliman filter is used to analyze their location and speed to detect unusual behavior.

[0283] Emotion analysis

[0284] Example 3

[0285] The server uses an emotion engine to analyze customer facial expressions in specific areas and detect nervous or suspicious expressions, using machine learning algorithms to recognize emotions in real time.

[0286] Automated detection of criminal behavior

[0287] Example 4

[0288] Based on the results of the behavioral and emotional analysis of the moving object, the server detects the behavior of quickly picking up an item in a specific area and showing a suspicious expression as "theft behavior."

[0289] Generating and sending evidence

[0290] Example 5

[0291] The server sends behavioral and emotional data to the AI, which then generates evidential information including date, time, location, and details of the behavior. The evidential information is then converted into an email and automatically sent to the police system.

[0292] This system will improve public safety by streamlining surveillance operations and enabling rapid crime response through analysis of motion and emotions.

[0293] The processing flow will be explained below.

[0294] Step 1:

[0295] The server connects to the surveillance camera and acquires real-time video data. Specifically, the server receives the video stream using the RTSP protocol. The server continues to acquire data from the camera at regular intervals.

[0296] Step 2:

[0297] The server divides the acquired video data into frames. Specifically, it divides the video data into 30 frames per second (fps) and converts each frame into an easy-to-handle format. Next, preprocessing is performed, such as noise removal and resolution unification. The server performs preprocessing using the OpenCV library.

[0298] Step 3:

[0299] The server analyzes the preprocessed video data to detect moving objects. Specifically, it uses background subtraction and optical flow to calculate the difference between consecutive frames and identify moving areas. The server then uses bounding boxes to surround moving objects.

[0300] Step 4:

[0301] The server tracks the detected moving object and analyzes its behavior. Using a Kaliman filter and local tracking algorithm, the server calculates the object's location, speed, and direction of movement. This allows the server to analyze the object's movement pattern and identify any unnatural behavior.

[0302] Step 5:

[0303] Based on the results of the motion analysis, the server uses an emotion engine to analyze the user's emotions from their facial expressions and movements. Specifically, it uses facial expression recognition technology to detect nervous or suspicious expressions, and generates emotion data based on this.

[0304] Step 6:

[0305] The server evaluates the possibility of criminal behavior based on emotion data and combines it with the results of motion analysis. For example, if a combination of a certain behavioral pattern and emotion is considered to be criminal behavior, the server evaluates the possibility.

[0306] Step 7:

[0307] Based on the results of behavioral and emotional analysis, the server automatically uses a rule-based system to determine the likelihood of criminal behavior. For example, if someone quickly picks up an object in a specific area and shows a suspicious expression, it will be determined to be "theft behavior."

[0308] Step 8:

[0309] If the server detects any criminal activity, it generates an alarm in real time on the management terminal, specifically by sending a notification using WebSocket to prompt immediate action.

[0310] Step 9:

[0311] The server sends suspicious behavior and emotion data to the AI ​​generator. The server formats the data in JSON format and sends it to the AI ​​generator's API endpoint. This process generates detailed evidence.

[0312] Step 10:

[0313] Based on the data received, the generation AI generates detailed evidence, including a detailed description of the behavior, the date and time, the location, and the identity of the people and objects involved.

[0314] Step 11:

[0315] The server receives the generated evidence, converts it into an appropriate format, and formats the evidence into an email with relevant images and video links attached.

[0316] Step 12:

[0317] The server automatically sends evidence information to the police system. Specifically, it sends evidence information to a dedicated email address for the police using the SMTP protocol. This allows evidence information to be shared quickly and securely.

[0318] Example 2

[0319] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0320] Conventional surveillance systems focus on detecting moving objects and analyzing behavior, but they do not analyze the user's emotional state, making it difficult to accurately detect criminal behavior. Furthermore, they lack the ability to quickly and accurately generate evidence of detected criminal behavior and automatically send it to the police. This can delay early response to criminal behavior and potentially reduce public safety.

[0321] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0322] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data and detecting a moving object, means for analyzing the behavior of the detected moving object and analyzing its behavioral pattern using its moving speed and direction, means for analyzing the emotional state of the user using an emotion engine based on the results of the behavioral analysis of the moving object and evaluating the possibility of criminal behavior, means for sending data to an artificial intelligence generated when criminal behavior is detected and generating evidential information, and means for organizing the generated evidential information and automatically transmitting it to the police. This enables accurate detection of criminal behavior and rapid response by comprehensively performing behavioral analysis and emotion analysis of the moving object.

[0323] The "image acquisition device" is a device for acquiring image data such as a surveillance camera.

[0324] "Video data" refers to data that expresses acquired video information in a digital format.

[0325] "Preprocessing" refers to performing processes such as noise removal and frame division to make video data easier to analyze.

[0326] A "moving object" refers to an object that changes position between successive video frames.

[0327] "Motion detection" refers to identifying moving objects in video data and recognizing their position and movement.

[0328] "Behavioral analysis" refers to analyzing the location information, movement speed, movement direction, etc. of detected moving objects to clarify their behavioral patterns.

[0329] An "emotion engine" refers to a system that analyzes the emotional state of a moving subject from its facial expressions and movements.

[0330] "User" refers to a person or other object that appears in the video data.

[0331] "Possible criminal behavior" refers to assessing whether a target's behavior constitutes a crime based on behavioral and emotional analysis.

[0332] "Generative AI" refers to AI that generates detailed evidentiary information using data evaluated as criminal behavior.

[0333] "Evidence information" refers to information that details criminal behavior and is generated based on analyzed behaviors and emotions.

[0334] "Police" refers to law enforcement agencies that investigate crimes to maintain public safety and order.

[0335] "Data transmission" refers to sending required data to other systems or devices via a network.

[0336] This invention is a system that acquires video data from a video capture device, performs motion detection and behavior analysis, and analyzes user emotions through an emotion engine, detects criminal behavior, and automatically transmits evidence to the police. This system is composed of multiple components, including a video capture device, a server, a generation AI, an emotion engine, and a police system.

[0337] System configuration and processing content

[0338] Image acquisition and preprocessing

[0339] Image acquisition device

[0340] The server acquires video data in real time from video acquisition devices such as surveillance cameras. Specifically, it uses RTSP (Real-Time Streaming Protocol) to receive video streams in the format "rtsp: / / username:password@cameraIP:554 / stream."

[0341] Pretreatment

[0342] The server divides the acquired video data into frames and performs preprocessing such as noise removal and color correction using the OpenCV library, thereby preparing the data for improved analysis accuracy.

[0343] Motion detection and behavior analysis

[0344] Motion Detection

[0345] The server detects motion by applying background subtraction to the pre-processed video data, which analyzes the differences between consecutive frames to identify moving objects.

[0346] Behavioral analysis

[0347] The server analyzes the behavioral patterns of the detected moving object using its location, speed, and direction. Specifically, it uses a Kaliman filter and tracking algorithm to track the object and perform detailed behavioral analysis.

[0348] Emotion analysis

[0349] emotion recognition

[0350] Based on the behavioral analysis results of the moving object, the server uses an emotion engine to analyze the user's emotions from their facial expressions and movements, and uses facial recognition technology to detect suspicious expressions and tension in real time.

[0351] Emotional Data Evaluation

[0352] The server evaluates the likelihood of criminal behavior based on the recognized emotional data, and performs a comprehensive analysis of emotional changes and behavioral patterns according to a specific algorithm.

[0353] Automated detection of criminal behavior

[0354] Rule-based Decision

[0355] The server uses a pre-configured rule-based system to detect suspicious behavior based on the results of behavioral and emotional analysis, such as quickly taking an object from a specific area or displaying a suspicious facial expression, as a "theft attempt."

[0356] Alarm Generation

[0357] If the server detects any suspicious activity, it generates an alarm in real time and sends a notification to the management terminal.

[0358] Evidence generation and transmission

[0359] Data transmission

[0360] The server sends suspicious behavior and emotion data to the generator AI, formatted in JSON and sent through an API endpoint.

[0361] Evidence generation

[0362] Based on the data it receives, the generation AI generates detailed evidence, including a detailed description of the behavior, the date and time, the location, and the identity of the people and objects involved.

[0363] Data reduction and format conversion

[0364] The server receives the generated evidence and converts it into the appropriate format, arranging it into an email format and including related images and video links.

[0365] send

[0366] The server automatically sends the organized evidence information to the police system, specifically to a dedicated email address for the police using the SMTP protocol.

[0367] Specific examples

[0368] Specific examples of image acquisition and preprocessing

[0369] The server connects to "rtsp: / / username:password@cameraIP:554 / stream", acquires video data at 30 frames per second, and performs noise reduction using an OpenCV filter.

[0370] Examples of motion detection and behavior analysis

[0371] The server uses background subtraction to detect customers moving around the store and analyzes their movement speed and location using a Kaliman filter.

[0372] Specific examples of sentiment analysis

[0373] The server uses an emotion engine to analyze customers' facial expressions in specific areas and detect nervous or suspicious expressions in real time.

[0374] Specific examples of automated detection of criminal behavior

[0375] Based on the results of behavioral and emotional analysis, the server quickly retrieves products from specific areas and detects any behavior showing suspicious facial expressions as "theft behavior."

[0376] Specific examples of evidence generation and transmission

[0377] The server sends behavioral and emotional data to the generating AI, which then organizes the generated evidence information in an appropriate format and automatically sends it to the police.

[0378] Prompt Sentence Examples

[0379] "Please explain in detail the processing steps of a system that uses video footage acquired from surveillance cameras to detect motion, analyze behavior and emotions, automatically detect criminal activity, and transmit evidence to the police."

[0380] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0381] Step 1:

[0382] The server acquires video data in real time from a video acquisition device. The input is streaming video from a surveillance camera, which the server receives using the RTSP protocol. Specifically, it connects to a URL in the format "rtsp: / / username:password@cameraIP:554 / stream" to acquire the video data. The output is raw video data.

[0383] Step 2:

[0384] The server divides the acquired video data into frames and performs preprocessing. The input is the video data acquired in step 1, and the server uses the OpenCV library to perform noise removal and color correction. Specifically, it removes noise using "cv2.GaussianBlur(image, (5, 5), 0)". The output is preprocessed, high-quality video data.

[0385] Step 3:

[0386] The server analyzes the preprocessed video data and detects moving objects. The input is the preprocessed video data obtained in step 2, and the server detects moving objects using background subtraction. Specifically, it calculates the difference between consecutive frames and marks moving objects with bounding boxes. The output is video data containing the position information of moving objects.

[0387] Step 4:

[0388] The server analyzes the behavior of the detected moving object. The input is the video data containing the location information of the moving object obtained in step 3, and the server analyzes the moving speed and direction of the moving object using a Kaliman filter or tracking algorithm. For example, a tracking algorithm is used to compare the moving object's current position with its past positions and analyze its behavioral patterns. The output is the data resulting from the behavioral analysis.

[0389] Step 5:

[0390] The server uses an emotion engine to analyze the user's emotional state based on the behavioral analysis results. The input is the behavioral analysis results obtained in step 4, and the server uses facial recognition technology to recognize emotions from facial expressions. Specifically, it uses a deep learning model to detect suspicious expressions and tension. The output is the analyzed emotional data.

[0391] Step 6:

[0392] The server comprehensively evaluates the emotional data and behavioral analysis data to determine the likelihood of criminal behavior. The input is the data obtained in steps 4 and 5, and the server determines the likelihood of criminal behavior based on set rules. As a specific example, if a person quickly picks up an item in a specific area and shows a suspicious expression, it will be determined to be "theft behavior." The output is an evaluation result indicating the likelihood of criminal behavior.

[0393] Step 7:

[0394] If criminal behavior is detected, the server sends data to the generation AI. The input is the evaluation result obtained in step 6, and the server sends the data in JSON format to the API endpoint of the generation AI. The output is the data sent to the generation AI.

[0395] Step 8:

[0396] The generation AI generates detailed evidential information based on the received data. The input is the data sent in step 7, and the generation AI generates evidential information including a detailed description of the action, date and time, location, and identification information of people and objects involved. The output is the generated evidential information.

[0397] Step 9:

[0398] The server organizes the generated evidence and converts it into the appropriate format. The input is the evidence obtained in step 8, and the server formats the evidence into an email and includes related image and video links. The output is the evidence converted into email format.

[0399] Step 10:

[0400] The server automatically sends the organized evidence information to the police. The input is the evidence information in email format obtained in step 9, which the server sends to the police's dedicated email address using the SMTP protocol. The output is the evidence information sent to the police system.

[0401] (Application example 2)

[0402] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0403] Conventional security systems require manual review of surveillance camera footage, making it difficult to detect criminal behavior in real time and respond quickly. Furthermore, simply reviewing surveillance footage makes it difficult to grasp the details of a criminal's behavior or the victim's emotional state, resulting in low crime detection accuracy. Furthermore, reporting to the police must be done manually, making it difficult to respond quickly.

[0404] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0405] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data to detect moving objects, means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior, means for sending a real-time alert to a smart device when criminal behavior is detected, means for sending data to a generating artificial intelligence when criminal behavior is detected to generate evidence information, and means for organizing the generated evidence information and automatically sending it to the police. This enables the detection of criminal behavior and rapid reporting, making it possible to improve the efficiency of surveillance work and public safety.

[0406] An "image capture device" is a device such as a surveillance camera that captures image data and transfers it to a processing device such as a server.

[0407] "Splitting into frames" refers to the process of splitting continuous video data into still images (frames) at regular intervals.

[0408] "Preprocessing" is the process of performing initial processing such as noise removal and image quality adjustment on acquired video data to make it easier to analyze.

[0409] "Motion detection" refers to the process of identifying moving objects in video data and identifying their position and range.

[0410] "Behavior analysis" is a process of analyzing the position information, movement speed, and movement direction of a moving object and extracting a specific behavior pattern.

[0411] "Evaluating the possibility of criminal behavior" refers to the process of analyzing the behavior of a detected moving object and determining whether or not that behavior constitutes a crime.

[0412] A "smart device" is a portable information terminal with advanced processing capabilities, such as a smartphone, tablet, or smart glasses.

[0413] "Real-time alerts" are notifications that are generated immediately and send a warning when a specific condition is detected.

[0414] "Generative AI" is an AI technology used to generate detailed evidential information based on acquired data.

[0415] "Evidence information" is information that includes a detailed description of the detected criminal activity, as well as the people involved, dates, times, and locations.

[0416] "Automatic transmission to police" is a process in which the generated evidence information is formatted into a specific format and automatically transmitted to the police system.

[0417] This invention provides a system that acquires video data from a video capture device, performs real-time motion detection and behavior analysis, captures signs of criminal behavior, sends alerts to smart devices, generates evidence information using artificial intelligence, and automatically notifies the police.

[0418] System Configuration

[0419] 1. Image acquisition device

[0420] In this system, surveillance cameras installed in stores and public facilities function as video capture devices, and the video data is sent to a server in real time.

[0421] 2. Server

[0422] The server performs the following steps:

[0423] Video Acquisition:

[0424] The server acquires video data in real time from the video acquisition device, and the video data is streamed using the RTSP protocol.

[0425] Pretreatment:

[0426] The server divides the acquired video data into frames and performs preprocessing such as noise removal using OpenCV.

[0427] Motion detection:

[0428] The preprocessed video data is analyzed and moving objects are detected using background subtraction and a Kaliman filter.

[0429] Behavior analysis:

[0430] The location, speed, and direction of the detected moving objects are analyzed to evaluate their behavioral patterns.Face detection is also performed using the dlib library to analyze their emotional states.

[0431] Criminal Behavior Assessment:

[0432] Based on the results of behavioral and sentiment analysis, a pre-defined rules-based algorithm is used to assess the likelihood of criminal behavior.

[0433] 3. Smart Devices

[0434] If criminal activity is detected, the server sends the information in real time to a smart device (smartphone, smart glasses, etc.) and notifies the user with an alert.

[0435] 4. Generative Artificial Intelligence

[0436] The server sends the behavioral data and emotional data of the moving object to the AI ​​generator, which then generates detailed evidence information, including the date, time, location, details of the behavior, and emotional data.

[0437] 5. Police System

[0438] The generated evidence information is automatically sent to the police. The server uses the SMTP protocol to send the evidence information to the police's dedicated email address.

[0439] Specific examples

[0440] Image acquisition and preprocessing

[0441] For example, video data is acquired from a surveillance camera installed in a store using the RTSP protocol, divided into 30 fps frames, and noise is removed through an OpenCV filter.

[0442] Motion detection and behavior analysis

[0443] It uses background subtraction to detect customers moving around the store, and a Kaliman filter to analyze their location and speed. It also uses the dlib library for face detection and emotion analysis.

[0444] Automatic detection and alerting of criminal activity

[0445] The system detects in real time any suspicious behavior such as quickly picking up an item in a specific area as "theft behavior" and sends an alert to a smart device.

[0446] Generate evidence and automatically send it to the police

[0447] The server sends behavioral and emotional data to the generating AI, which then converts the generated evidence information into an email and automatically sends it to the police system.

[0448] Prompt Sentence Examples

[0449] "Write a program that performs speed anomaly detection using bounding box averaging in a cluster setting. This program uses OpenCV to analyze video data acquired from a surveillance camera and detect objects moving at speeds above a certain threshold."

[0450] In this way, it is possible to improve public safety by making surveillance more efficient and enabling a quicker response to crime.

[0451] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0452] Step 1:

[0453] The server acquires video data in real time from a video acquisition device. The input is a video stream sent using the RTSP protocol, and the output is video data divided into frames. Specifically, the server receives the video captured by the surveillance camera as is and divides the video into frames.

[0454] Step 2:

[0455] The server performs preprocessing on the acquired video data. The input is video divided into frames, and the output is preprocessed frame data that has undergone noise removal and image quality adjustment. Specifically, OpenCV is used to remove noise from the video, and brightness and contrast are adjusted as necessary.

[0456] Step 3:

[0457] The server analyzes the preprocessed video data and performs moving object detection. The input is the preprocessed frame data, and the output is the position information and bounding box of the detected moving object. Specifically, moving objects are extracted using background subtraction, and their position information is obtained using a Kaliman filter.

[0458] Step 4:

[0459] The server analyzes the behavior of the detected moving object. The input is the object's location information, and the output is the object's behavior pattern, movement speed, and movement direction data. Specifically, an analysis algorithm is used to extract the object's movement pattern and detect abnormalities in its movement.

[0460] Step 5:

[0461] The server uses the dlib library to recognize the faces of detected moving objects and analyze their emotional states. The input is frame data of the moving objects, and the output is emotion-analyzed information. Specifically, it detects faces and infers emotions from their facial expressions.

[0462] Step 6:

[0463] The server evaluates the likelihood of criminal behavior based on the results of behavioral and emotional analysis. The input is behavioral patterns and emotional data, and the output is the evaluation result of criminal behavior. Specifically, it uses a pre-configured rule-based model to determine whether the detected behavior and emotion match the behavioral criteria.

[0464] Step 7:

[0465] If criminal behavior is detected, the server sends a real-time alert to the smart device. The input is the criminal behavior evaluation result, and the output is an alert notification to the smart device. Specifically, the detected information is sent to a smartphone or smart glasses via a notification application.

[0466] Step 8:

[0467] When criminal behavior is detected, the server sends data to the generation AI to generate evidence. The input is behavioral data and emotional data, and the output is the generated evidence. Specifically, the generation AI model uses prompt sentences to generate detailed evidence.

[0468] Step 9:

[0469] The server organizes the generated evidence information and automatically sends it to the police. The input is the generated evidence information, and the output is a notification of completion of transmission to the police system. Specifically, the evidence information is formatted appropriately and sent to a dedicated email address for the police using the SMTP protocol.

[0470] Prompt Sentence Examples

[0471] "Write a program that performs speed anomaly detection using bounding box averaging in a cluster setting. This program uses OpenCV to analyze video data acquired from a surveillance camera and detect objects moving at speeds above a certain threshold."

[0472] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0473] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0474] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0475] [Second embodiment]

[0476] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0477] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0478] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0479] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0480] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0481] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0482] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0483] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0484] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0485] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0486] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0487] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0488] The overall system configuration and program processing flow for implementing this invention are shown below. Specifically, it explains how the server, terminals, and users work together to operate the system, automatically detect criminal behavior, and generate and transmit evidence information.

[0489] System Configuration

[0490] The system mainly consists of the following components:

[0491] 1. Video capture devices - Various surveillance cameras in stores, public facilities, etc.

[0492] 2. Server - Receives data from the video capture device, analyzes it, and generates and transmits evidence information.

[0493] 3. Generative AI - An artificial intelligence system that generates detailed evidence information based on data on detected criminal behavior.

[0494] 4. Police system - receives and responds to the evidence generated.

[0495] What the program does

[0496] Image acquisition and preprocessing

[0497] Video acquisition

[0498] The server acquires video data from the surveillance camera in real time.

[0499] For example, a server uses the RTSP protocol to periodically connect to a video capture device and receive a video stream.

[0500] Pretreatment

[0501] The video data acquired by the server is divided into frames.

[0502] Preprocessing such as noise removal is performed on each frame to improve data quality.

[0503] For example, apply a denoising filter using the OpenCV library.

[0504] Motion detection and behavior analysis

[0505] Motion Detection

[0506] The server analyzes the pre-processed video data, calculates the differences between frames, and detects moving objects.

[0507] If a moving object is detected, its location is identified and recorded using a bounding box.

[0508] Behavioral analysis

[0509] The server analyzes the location, speed, direction, etc. of the detected moving object.

[0510] The Kaliman filter and tracking algorithms are used to track the target and analyze its behavioral patterns.

[0511] For example, identifying criminal behavior such as quickly removing an object within a certain area.

[0512] Automated detection of criminal behavior

[0513] Rule-based Decision

[0514] The server applies rules to assess the likelihood of criminal behavior based on the analyzed behavioral data.

[0515] Automatically detects suspicious behavior based on pre-defined rules.

[0516] Alarm Generation

[0517] If the server detects suspicious activity, it generates a real-time alarm and notifies the management terminal.

[0518] Generating evidence

[0519] Data transmission

[0520] The server sends details of the suspicious activity to the generating artificial intelligence.

[0521] For example, behavioral data is sent to the generation AI in JSON format.

[0522] Evidence generation

[0523] The generation AI generates detailed evidence information based on the data it receives.

[0524] The evidence information includes a detailed description of the action, date and time, location information, etc.

[0525] Automatic transmission to police

[0526] Data reduction and format conversion

[0527] The server receives the evidence information from the generation AI and converts it into the appropriate format.

[0528] Convert evidence into email format and attach links and relevant images.

[0529] send

[0530] The server automatically sends evidence information to the police system.

[0531] For security reasons, SMTP or other communication protocols are used.

[0532] Specific examples

[0533] Image acquisition and preprocessing

[0534] Example 1

[0535] The server acquires video data from the surveillance cameras in the store via RTSP.

[0536] The video data is divided into 30fps frames and noise is removed using an OpenCV filter.

[0537] Motion detection and behavior analysis

[0538] Example 2

[0539] The server uses background subtraction to detect customers moving around the store.

[0540] A Kaliman filter is used to analyze the customer's location and movement speed to detect unnatural behavior.

[0541] Automated detection of criminal behavior

[0542] Example 3

[0543] The server detects the act of quickly removing an item from a specific area as "theft."

[0544] If any suspicious activity is detected, a real-time notification is sent to the management terminal via WebSocket.

[0545] Generating and sending evidence

[0546] Example 4

[0547] The server sends behavioral data to the generation AI, which then generates evidentiary information including date, time, location, and behavioral details.

[0548] Evidence information is converted into email and automatically sent to the police system.

[0549] Through these processes, this system will improve the efficiency of surveillance work and enable rapid crime response.

[0550] The processing flow will be explained below.

[0551] Step 1:

[0552] The server connects to the surveillance camera and acquires real-time video data. Specifically, the server receives the video stream using the RTSP protocol. The server continues to acquire data from the camera at regular intervals.

[0553] Step 2:

[0554] The server divides the acquired video data into frames. Specifically, it divides the video data into 30 frames per second (fps) and converts each frame into an easy-to-handle format. Next, preprocessing is performed, such as noise removal and resolution unification. The server performs preprocessing using the OpenCV library.

[0555] Step 3:

[0556] The server analyzes the preprocessed video data to detect moving objects. Specifically, it uses background subtraction and optical flow to calculate the difference between consecutive frames and identify moving areas. The server then uses bounding boxes to surround moving objects.

[0557] Step 4:

[0558] The server tracks the detected moving object and analyzes its behavior. Using a Kaliman filter and local tracking algorithm, the server calculates the object's location, speed, and direction of movement. This allows the server to analyze the object's movement pattern and identify any unnatural behavior.

[0559] Step 5:

[0560] The server evaluates the likelihood of criminal activity based on the results of behavioral analysis. Specifically, it uses a pre-configured rule-based system to automatically detect suspicious behavior. For example, quickly retrieving an object in a specific area is detected as criminal activity.

[0561] Step 6:

[0562] If the server detects suspicious activity, it generates a real-time alarm. Specifically, it uses WebSocket to send a notification to the management terminal, prompting immediate action. The server notifies the management terminal with detailed data about the suspicious activity.

[0563] Step 7:

[0564] The server sends suspicious behavior data to the generation AI, which then formats the behavior data in JSON format and sends it to the generation AI's API endpoint, which then starts generating detailed evidence.

[0565] Step 8:

[0566] The generation AI generates detailed evidence information based on the received data. Specifically, it generates a detailed description of the behavior, the date and time, the location, and the identification information of related people and objects. The generated evidence information is returned to the server.

[0567] Step 9:

[0568] The server receives the generated evidence and converts it into the appropriate format. The evidence is formatted as an email and includes any relevant images or video links.

[0569] Step 10:

[0570] The server automatically sends evidence information to the police system. Specifically, it sends evidence information to a dedicated police email address using the SMTP protocol. This allows evidence information to be shared quickly and securely.

[0571] Example 1

[0572] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0573] Conventional surveillance systems require manual analysis of video data obtained from surveillance cameras, making it difficult for limited personnel to review large volumes of data. Furthermore, the accuracy of motion detection and behavior analysis is low, making it difficult to detect suspicious activity. This has led to issues with early detection of criminal activity and rapid response.

[0574] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0575] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data and detecting moving objects, means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior, means for transmitting data to a generation artificial intelligence to generate evidence information when criminal behavior is detected, means for organizing the generated evidence information and automatically transmitting it to the police, means for acquiring video data from a surveillance camera in real time using the RTSP protocol, means for applying a noise reduction filter to the acquired video data using the OpenCV library, means for detecting moving objects using background subtraction, means for analyzing the behavioral patterns of moving objects using a Kaliman filter or tracking algorithm, and means for notifying a management terminal of suspicious behavior in real time using WebSocket. This enables automatic analysis of large amounts of video data, highly accurate detection of suspicious behavior, and rapid detection and response to criminal behavior.

[0576] The "image acquisition device" is a device for acquiring image data such as a surveillance camera.

[0577] "Video data" refers to digital video information obtained from a video capture device such as a surveillance camera.

[0578] "Server" means a central system for analyzing video data received from video capture devices and detecting and responding to criminal activity.

[0579] "Preprocessing" is a process of improving the quality of acquired video data by performing processes such as noise removal and frame division.

[0580] A "frame unit" is a unit into which video data is divided into individual still images.

[0581] A "moving object" is a moving object or person that is detected by the difference between frames.

[0582] "Motion detection" is the process of analyzing pre-processed video data and calculating the differences between frames to find moving objects.

[0583] "Behavioral analysis" refers to analyzing the location information, movement speed, and movement direction of a detected moving object to understand its behavioral patterns.

[0584] "Assessing the possibility of criminal behavior" refers to applying pre-set rules to the analyzed behavioral data to determine whether or not there is a possibility of criminal behavior.

[0585] "Generative AI" is an AI system that automatically generates detailed evidence information based on data on criminal behavior.

[0586] "Evidence information" is data containing detailed information about criminal activity, including date, time, location, and activity details.

[0587] "Automatic transmission to police" refers to the process of converting the generated evidence information into an appropriate format and automatically transmitting it to the police system.

[0588] The "RTSP protocol" is a communication protocol for acquiring video streams in real time.

[0589] The "OpenCV library" is an open-source library for performing computer vision tasks.

[0590] A "noise reduction filter" is a tool used to remove unnecessary noise from video data and improve the quality of the data.

[0591] "Background subtraction" is a technique used for motion detection, which detects moving objects by comparing them with the background.

[0592] The "Kaliman filter" is an algorithm for tracking the location information of moving objects, and estimates and predicts their location.

[0593] A "tracking algorithm" is an algorithm for tracking the successive positions of a moving object.

[0594] "WebSocket" is a protocol for two-way communication between a server and a management terminal.

[0595] A "management terminal" is a terminal device for receiving notifications of detected criminal behavior, and is usually used by an administrator.

[0596] The overall system configuration and specific program processing for implementing this invention are described below. The system consists of a video capture device including a surveillance camera, a server that analyzes and generates and transmits evidence information, a generation AI, and a police system.

[0597] Video data acquisition and preprocessing

[0598] The server receives video data from surveillance cameras in real time using the RTSP protocol. For example, it receives a video stream from a surveillance camera in a store, divides it into frames, and applies a noise reduction filter to each frame using the OpenCV library. This improves the quality of the data and makes the subsequent analysis process smoother.

[0599] Motion detection and behavior analysis

[0600] The server analyzes the preprocessed video data and detects moving objects using background subtraction. If a moving object is detected, its location is identified using a bounding box and its location information is recorded. The server then analyzes the location, speed, and direction of the detected moving object using a Kaliman filter and tracking algorithm to understand its behavioral patterns.

[0601] Automated detection of criminal behavior

[0602] The server then applies pre-defined rules to assess the likelihood of criminal activity based on the analyzed behavioral data. For example, a rule could be applied that defines "taking an item from a shelf within five seconds" as theft, and if suspicious activity is detected, a real-time alarm is generated and a notification is sent to the management terminal via WebSocket.

[0603] Generating and sending evidence

[0604] The server sends the detected behavioral data in JSON format to the generation AI. The generation AI generates detailed evidence information based on the received data, creating content including date, time, location, and behavior details. The server then converts this evidence information into email format and automatically sends it to the police system using the SMTP protocol.

[0605] Specific examples

[0606] For example, the server acquires a 30fps video stream from a store's surveillance cameras using the RTSP protocol and removes noise using OpenCV's GaussianBlur filter. Next, it detects moving people using background subtraction and tracks their location using a Kaliman filter. If the server detects someone quickly removing an item in a specific area, it sends an alarm notification to the management terminal in real time via WebSocket. The server then sends the behavioral data to the generation AI, which creates detailed evidence information. The server then converts this evidence information into email format and sends it to the police system using the SMTP protocol.

[0607] Example prompts for generative AI models

[0608] For example, the following prompt could be entered into the generation AI: "Generate detailed evidence information based on the following data: Data: Date and time: October 10, 2023, Location: Store A, Action: A person quickly removed an item from a specific area."

[0609] Through the above process, this invention automatically analyzes large amounts of video data, detects suspicious activity with high accuracy, and enables rapid detection and response to criminal activity. Furthermore, by automatically transmitting the generated detailed evidence information to the police, it supports rapid legal response.

[0610] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0611] Step 1:

[0612] The server acquires video data from a surveillance camera. As input, it uses the RTSP stream URL of the surveillance camera. Specifically, the server connects to the surveillance camera using the RTSP protocol and receives the video stream in real time. As output, it obtains the stream data.

[0613] Step 2:

[0614] The server divides the acquired video data into frames. The input is real-time video stream data. Specifically, the server divides the video stream into 30 frames per second (30 fps). The output is individual frame data.

[0615] Step 3:

[0616] The server performs noise reduction on each frame. As input, it receives the segmented frame data. Specifically, it applies a Gaussian Blur filter using the OpenCV library to remove noise from the frame. As output, it obtains high-quality frame data.

[0617] Step 4:

[0618] The server analyzes the preprocessed frame data and detects moving objects. The input is noise-removed frame data. Specifically, it uses background subtraction to detect movement between frames and identifies the location of moving objects using bounding boxes. The output is the location information of the moving objects.

[0619] Step 5:

[0620] The server analyzes the behavior of the detected moving object. The input is the location information of the moving object. Specifically, it uses a Kaliman filter and tracking algorithm to analyze the object's location, speed, and direction of movement, and identifies its behavioral pattern. The output is the analyzed behavioral data.

[0621] Step 6:

[0622] The server evaluates the likelihood of criminal behavior based on the analyzed behavioral data. The input is the analyzed behavioral data. Specific actions are detected by applying pre-set rules. For example, "taking an item from a shelf within five seconds" is evaluated as theft. The output is a notification of suspicious behavior.

[0623] Step 7:

[0624] If the server detects suspicious behavior, it generates an alarm and notifies the management terminal. The input is a notification of the detected suspicious behavior. The specific operation is to send a real-time notification to the management terminal using the WebSocket protocol. The output is a notification displayed on the management terminal.

[0625] Step 8:

[0626] The server sends behavioral data to the generation AI to generate evidential information. The input is detailed data on suspicious behavior. Specifically, the behavioral data is sent to the generation AI in JSON format, and the AI ​​generates evidential information including date, time, location, and details of the behavior. The generated evidential information is obtained as output.

[0627] Step 9:

[0628] The server automatically sends the generated evidence information to the police. The input is the evidence information obtained from the generation AI. Specifically, the server converts the evidence information into email format, attaches links and related images, and sends it to the police system using the SMTP protocol. The output is the evidence information sent to the police system.

[0629] (Application example 1)

[0630] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0631] Automatic detection systems for criminal behavior using surveillance cameras are essential for responding quickly when a crime occurs. However, current systems not only need to analyze video data from surveillance cameras in real time, but also need the ability to quickly and accurately generate and transmit detailed evidence information when criminal behavior is detected. Real-time alert generation and notification are also required to enable police to respond immediately. To achieve this, an effective system is required that integrates motion detection, behavior analysis, the generation of evidence information using generative AI, and automatic transmission functions.

[0632] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0633] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data and detecting moving objects, means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior, means for transmitting data to a generating AI and generating evidential information when criminal behavior is detected, means for organizing the generated evidential information and automatically transmitting it to the police, and means for saving screenshots and transmitting images to the police by email when criminal behavior is detected. This makes it possible to realize a system that detects criminal behavior in real time, quickly and accurately generates and transmits evidence, and enables the police to respond immediately.

[0634] An "image capture device" is a device for capturing image data in real time, such as a surveillance camera or video camera.

[0635] "Video data" refers to digital information that is organized for each frame of a captured video.

[0636] "Preprocessing" refers to the process of processing the acquired video data, such as by removing noise and dividing the data into frames, in order to improve the quality.

[0637] "Motion detection" is the process of identifying moving objects in video using differences between frames.

[0638] "Behavioral analysis" is a method of analyzing the movements of moving objects based on their location, speed, direction, etc., to find specific behavioral patterns.

[0639] The "means for assessing the likelihood of criminal activity" is a process that includes a rule-based algorithm for assessing and detecting the likelihood of criminal activity based on the analyzed data.

[0640] "Generative artificial intelligence" is an AI system that generates detailed evidential information based on specified data.

[0641] "Evidential information" is information necessary to prove criminal activity, such as the date, time, location, and details of the action.

[0642] "Automatic transmission" is the process of quickly transmitting generated evidence information to the police using a pre-defined protocol.

[0643] "Real-time alert" is a warning system that immediately notifies you of any suspicious behavior when it is detected, prompting you to take action.

[0644] A "screenshot" is a technique for capturing video data at a specific point in time and saving it as a still image.

[0645] "Email transmission" is a communication method for transmitting information in the form of email via the Internet.

[0646] To implement this invention, the following specific system configuration and processing procedures are required. The entire system operates through the cooperation of three parties: a server, a terminal, and a user. Automatic detection of criminal behavior and generation and transmission of evidence information are carried out according to the following procedures.

[0647] System Configuration

[0648] 1. Video acquisition device: A device such as a surveillance camera installed in stores and public facilities that acquires video data in real time.

[0649] 2. Server: Plays the central role of receiving and analyzing the video data sent from the video capture device, and generating and sending evidence information.

[0650] 3. Generative AI model: An artificial intelligence system that generates detailed evidential information based on data sent from the server.

[0651] 4. Police system: A system for receiving and responding to the evidence information generated.

[0652] What the program does

[0653] Image acquisition and preprocessing

[0654] The server acquires video data in real time from a surveillance camera. A typical surveillance camera is used as the video acquisition device. The acquired video data is divided into frames and preprocessed using OpenCV filters, such as noise removal.

[0655] Motion detection and behavior analysis

[0656] The server analyzes the preprocessed video data and calculates the difference between frames to detect moving objects. Using a Kaliman filter and tracking algorithm, the server analyzes the location, speed, and direction of the detected moving object to identify its behavioral pattern. For example, the action of quickly removing an object within a certain area can be detected as "theft."

[0657] Automated detection of criminal behavior

[0658] The server uses rule-based decision-making to assess the likelihood of criminal activity based on the analyzed behavioral data. If suspicious activity is detected, it generates a real-time alarm and notifies the management terminal. It also has the ability to save screenshots and send images via email to the police if criminal activity is detected.

[0659] Generating and sending evidence

[0660] The server sends details of suspicious behavior to a generative AI model, which then generates evidence including the date, time, location, and details of the behavior. The server then organizes the evidence, converts it into an appropriate format, and automatically transmits it to the police system. A secure communication protocol, such as SMTP, is used for transmission.

[0661] Specific examples

[0662] 1. Example of image acquisition and pre-processing:

[0663] The server acquires video data from the surveillance cameras inside the store using the RTSP protocol, divides it into 30fps frames, and removes noise using an OpenCV filter.

[0664] 2. Examples of motion detection and behavior analysis:

[0665] The server uses background subtraction to detect customers moving around the store, and uses a Kaliman filter to analyze the customer's location and movement speed to detect any unnatural behavior.

[0666] 3. Examples of evidence generation and transmission to police:

[0667] The server sends the behavioral data to the generation AI, which then generates evidential information including the date, time, location, and details of the behavior. The generated evidential information is converted into an email and automatically sent to the police system.

[0668] Prompt Sentence Examples

[0669] "Write a script that grabs frames from an RTSP stream, detects suspicious activity, saves the image, and notifies the police."

[0670] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0671] Step 1:

[0672] The server acquires video data from the video acquisition device. Specifically, the server connects to the surveillance camera via the RTSP protocol and receives the video stream in real time. The input of this step is the video stream provided by the surveillance camera, and the output is the video data received by the server.

[0673] Step 2:

[0674] The server divides the acquired video data into frames and performs preprocessing. Specifically, the video data is divided into 30 fps frames and noise is removed using an OpenCV filter. The input to this step is the video data received in real time, and the output is the preprocessed frame data.

[0675] Step 3:

[0676] The server analyzes the preprocessed video data to detect moving objects. Specifically, it uses background subtraction to calculate the difference between frames and identify moving objects. The input to this step is the preprocessed frame data, and the output is data with identified moving objects.

[0677] Step 4:

[0678] The server analyzes the behavior of the detected moving object. Specifically, it analyzes the location information, movement speed, and movement direction of the moving object using a Kaliman filter or tracking algorithm. The input of this step is the data on the identified moving object, and the output is the analyzed behavior information of the moving object.

[0679] Step 5:

[0680] The server evaluates the likelihood of criminal behavior based on the analyzed behavioral data. Specifically, it uses rule-based judgment to identify suspicious behavior. The input to this step is the analyzed behavioral information of the moving object, and the output is data evaluating the likelihood of criminal behavior.

[0681] Step 6:

[0682] When criminal behavior is detected, the server sends the data to the generation AI to generate evidence. Specifically, the behavioral data is sent in JSON format to the generation AI model, which generates detailed evidence. The input to this step is the data determined to be criminal behavior, and the output is the generated evidence.

[0683] Step 7:

[0684] The server organizes the generated evidence information and automatically sends it to the police. Specifically, it converts the evidence information into an appropriate format and sends it to the police system using the SMTP protocol. The input to this step is the evidence information from the generative AI model, and the output is the evidence data sent to the police.

[0685] Step 8:

[0686] If the server detects a criminal activity, it saves a screenshot and sends the image to the police by email. Specifically, it sends a warning message along with the saved screenshot by email. The input of this step is the video data at the time when the suspicious activity was detected, and the output is the screenshot and warning message sent to the police.

[0687] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0688] This invention is a system that acquires video data from a video capture device, performs moving object detection and behavior analysis, and analyzes user emotions through an emotion engine, detects criminal behavior, and automatically transmits evidence of the behavior to the police. The system configuration, program processing flow, and specific examples are described below.

[0689] System Configuration

[0690] The system mainly consists of the following components:

[0691] 1. Video capture device - Surveillance cameras installed in stores, public facilities, etc.

[0692] 2. Server - Receives data from the video capture device, performs motion detection, behavior analysis, emotion analysis, and generates and transmits evidence information.

[0693] 3. Generative AI - Artificial intelligence that generates detailed evidence from data that detects criminal behavior.

[0694] 4. Emotion Engine - Analyzes the user's facial expressions and movements from video to recognize their emotional state.

[0695] 5. Police system - receives and responds to generated evidence information.

[0696] What the program does

[0697] Image acquisition and preprocessing

[0698] Video acquisition

[0699] The server obtains video data from the surveillance camera in real time, for example, by receiving the video stream using the RTSP protocol.

[0700] Pretreatment

[0701] The server performs preprocessing such as noise removal on the video data divided into frames, and improves the quality of the data using the OpenCV library.

[0702] Motion detection and behavior analysis

[0703] Motion Detection

[0704] The server analyzes the pre-processed video data and calculates the difference between consecutive frames to detect moving objects. It identifies and records moving objects using bounding boxes.

[0705] Behavioral analysis

[0706] The server analyzes the location information, movement speed, and movement direction of the moving object, and tracks the object and analyzes its behavior using a Kaliman filter and tracking algorithm.

[0707] Emotion analysis

[0708] emotion recognition

[0709] In addition to analyzing the behavior of moving objects, the server uses an emotion engine to analyze emotions from the user's facial expressions and movements. For example, it uses facial recognition technology to detect suspicious expressions or tension.

[0710] Emotional Data Evaluation

[0711] The server evaluates the likelihood of criminal behavior based on the recognized emotional data, and performs a comprehensive analysis of emotional changes and behavioral patterns according to a set algorithm.

[0712] Automated detection of criminal behavior

[0713] Rule-based Decision

[0714] The server uses a pre-configured rules-based system based on behavioral and emotional analysis to detect suspicious behavior, such as quickly picking up an object in a specific area or displaying a suspicious facial expression.

[0715] Alarm Generation

[0716] When the server detects suspicious activity, it generates a real-time alarm and sends a notification to the management terminal.

[0717] Generating evidence

[0718] Data transmission

[0719] The server sends suspicious behavior and emotion data to the generation AI, formats the data in JSON format, and sends it to the generation AI's API endpoint.

[0720] Evidence generation

[0721] The generation AI generates detailed evidence based on the received data, including a detailed description of the behavior, the date and time, the location, and the identity of the people and objects involved.

[0722] Automatic transmission to police

[0723] Data reduction and format conversion

[0724] The server receives the generated evidence and converts it into the appropriate format. The evidence is formatted as an email and includes any relevant images or video links.

[0725] send

[0726] The server automatically sends the evidence information to the police system. Specifically, it sends the evidence information to a dedicated email address for the police using the SMTP protocol.

[0727] Specific examples

[0728] Image acquisition and preprocessing

[0729] Example 1

[0730] The server receives video data from the surveillance cameras in the store via RTSP. The video data is split into 30fps frames and noise is removed using an OpenCV filter.

[0731] Motion detection and behavior analysis

[0732] Example 2

[0733] The server uses background subtraction to detect customers moving around the store, and a Kaliman filter is used to analyze their location and speed to detect unusual behavior.

[0734] Emotion analysis

[0735] Example 3

[0736] The server uses an emotion engine to analyze customer facial expressions in specific areas and detect nervous or suspicious expressions, using machine learning algorithms to recognize emotions in real time.

[0737] Automated detection of criminal behavior

[0738] Example 4

[0739] Based on the results of the behavioral and emotional analysis of the moving object, the server detects the behavior of quickly picking up an item in a specific area and showing a suspicious expression as "theft behavior."

[0740] Generating and sending evidence

[0741] Example 5

[0742] The server sends behavioral and emotional data to the AI, which then generates evidential information including date, time, location, and details of the behavior. The evidential information is then converted into an email and automatically sent to the police system.

[0743] This system will improve public safety by streamlining surveillance operations and enabling rapid crime response through analysis of motion and emotions.

[0744] The processing flow will be explained below.

[0745] Step 1:

[0746] The server connects to the surveillance camera and acquires real-time video data. Specifically, the server receives the video stream using the RTSP protocol. The server continues to acquire data from the camera at regular intervals.

[0747] Step 2:

[0748] The server divides the acquired video data into frames. Specifically, it divides the video data into 30 frames per second (fps) and converts each frame into an easy-to-handle format. Next, preprocessing is performed, such as noise removal and resolution unification. The server performs preprocessing using the OpenCV library.

[0749] Step 3:

[0750] The server analyzes the preprocessed video data to detect moving objects. Specifically, it uses background subtraction and optical flow to calculate the difference between consecutive frames and identify moving areas. The server then uses bounding boxes to surround moving objects.

[0751] Step 4:

[0752] The server tracks the detected moving object and analyzes its behavior. Using a Kaliman filter and local tracking algorithm, the server calculates the object's location, speed, and direction of movement. This allows the server to analyze the object's movement pattern and identify any unnatural behavior.

[0753] Step 5:

[0754] Based on the results of the motion analysis, the server uses an emotion engine to analyze the user's emotions from their facial expressions and movements. Specifically, it uses facial expression recognition technology to detect nervous or suspicious expressions, and generates emotion data based on this.

[0755] Step 6:

[0756] The server evaluates the possibility of criminal behavior based on emotion data and combines it with the results of motion analysis. For example, if a combination of a certain behavioral pattern and emotion is considered to be criminal behavior, the server evaluates the possibility.

[0757] Step 7:

[0758] Based on the results of behavioral and emotional analysis, the server automatically uses a rule-based system to determine the likelihood of criminal behavior. For example, if someone quickly picks up an object in a specific area and shows a suspicious expression, it will be determined to be "theft behavior."

[0759] Step 8:

[0760] If the server detects any criminal activity, it generates an alarm in real time on the management terminal, specifically by sending a notification using WebSocket to prompt immediate action.

[0761] Step 9:

[0762] The server sends suspicious behavior and emotion data to the AI ​​generator. The server formats the data in JSON format and sends it to the AI ​​generator's API endpoint. This process generates detailed evidence.

[0763] Step 10:

[0764] Based on the data received, the generation AI generates detailed evidence, including a detailed description of the behavior, the date and time, the location, and the identity of the people and objects involved.

[0765] Step 11:

[0766] The server receives the generated evidence, converts it into an appropriate format, and formats the evidence into an email with relevant images and video links attached.

[0767] Step 12:

[0768] The server automatically sends evidence information to the police system. Specifically, it sends evidence information to a dedicated email address for the police using the SMTP protocol. This allows evidence information to be shared quickly and securely.

[0769] Example 2

[0770] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0771] Conventional surveillance systems focus on detecting moving objects and analyzing behavior, but they do not analyze the user's emotional state, making it difficult to accurately detect criminal behavior. Furthermore, they lack the ability to quickly and accurately generate evidence of detected criminal behavior and automatically send it to the police. This can delay early response to criminal behavior and potentially reduce public safety.

[0772] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0773] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data and detecting a moving object, means for analyzing the behavior of the detected moving object and analyzing its behavioral pattern using its moving speed and direction, means for analyzing the emotional state of the user using an emotion engine based on the results of the behavioral analysis of the moving object and evaluating the possibility of criminal behavior, means for sending data to an artificial intelligence generated when criminal behavior is detected and generating evidential information, and means for organizing the generated evidential information and automatically transmitting it to the police. This enables accurate detection of criminal behavior and rapid response by comprehensively performing behavioral analysis and emotion analysis of the moving object.

[0774] The "image acquisition device" is a device for acquiring image data such as a surveillance camera.

[0775] "Video data" refers to data that expresses acquired video information in a digital format.

[0776] "Preprocessing" refers to performing processes such as noise removal and frame division to make video data easier to analyze.

[0777] A "moving object" refers to an object that changes position between successive video frames.

[0778] "Motion detection" refers to identifying moving objects in video data and recognizing their position and movement.

[0779] "Behavioral analysis" refers to analyzing the location information, movement speed, movement direction, etc. of detected moving objects to clarify their behavioral patterns.

[0780] An "emotion engine" refers to a system that analyzes the emotional state of a moving subject from its facial expressions and movements.

[0781] "User" refers to a person or other object that appears in the video data.

[0782] "Possible criminal behavior" refers to assessing whether a target's behavior constitutes a crime based on behavioral and emotional analysis.

[0783] "Generative AI" refers to AI that generates detailed evidentiary information using data evaluated as criminal behavior.

[0784] "Evidence information" refers to information that details criminal behavior and is generated based on analyzed behaviors and emotions.

[0785] "Police" refers to law enforcement agencies that investigate crimes to maintain public safety and order.

[0786] "Data transmission" refers to sending required data to other systems or devices via a network.

[0787] This invention is a system that acquires video data from a video capture device, performs motion detection and behavior analysis, and analyzes user emotions through an emotion engine, detects criminal behavior, and automatically transmits evidence to the police. This system is composed of multiple components, including a video capture device, a server, a generation AI, an emotion engine, and a police system.

[0788] System configuration and processing content

[0789] Image acquisition and preprocessing

[0790] Image acquisition device

[0791] The server acquires video data in real time from video acquisition devices such as surveillance cameras. Specifically, it uses RTSP (Real-Time Streaming Protocol) to receive video streams in the format "rtsp: / / username:password@cameraIP:554 / stream."

[0792] Pretreatment

[0793] The server divides the acquired video data into frames and performs preprocessing such as noise removal and color correction using the OpenCV library, thereby preparing the data for improved analysis accuracy.

[0794] Motion detection and behavior analysis

[0795] Motion Detection

[0796] The server detects motion by applying background subtraction to the pre-processed video data, which analyzes the differences between consecutive frames to identify moving objects.

[0797] Behavioral analysis

[0798] The server analyzes the behavioral patterns of the detected moving object using its location, speed, and direction. Specifically, it uses a Kaliman filter and tracking algorithm to track the object and perform detailed behavioral analysis.

[0799] Emotion analysis

[0800] emotion recognition

[0801] Based on the behavioral analysis results of the moving object, the server uses an emotion engine to analyze the user's emotions from their facial expressions and movements, and uses facial recognition technology to detect suspicious expressions and tension in real time.

[0802] Emotional Data Evaluation

[0803] The server evaluates the likelihood of criminal behavior based on the recognized emotional data, and performs a comprehensive analysis of emotional changes and behavioral patterns according to a specific algorithm.

[0804] Automated detection of criminal behavior

[0805] Rule-based Decision

[0806] The server uses a pre-configured rule-based system to detect suspicious behavior based on the results of behavioral and emotional analysis, such as quickly taking an object from a specific area or displaying a suspicious facial expression, as a "theft attempt."

[0807] Alarm Generation

[0808] If the server detects any suspicious activity, it generates an alarm in real time and sends a notification to the management terminal.

[0809] Evidence generation and transmission

[0810] Data transmission

[0811] The server sends suspicious behavior and emotion data to the generator AI, formatted in JSON and sent through an API endpoint.

[0812] Evidence generation

[0813] Based on the data it receives, the generation AI generates detailed evidence, including a detailed description of the behavior, the date and time, the location, and the identity of the people and objects involved.

[0814] Data reduction and format conversion

[0815] The server receives the generated evidence and converts it into the appropriate format, arranging it into an email format and including related images and video links.

[0816] send

[0817] The server automatically sends the organized evidence information to the police system, specifically to a dedicated email address for the police using the SMTP protocol.

[0818] Specific examples

[0819] Specific examples of image acquisition and preprocessing

[0820] The server connects to "rtsp: / / username:password@cameraIP:554 / stream", acquires video data at 30 frames per second, and performs noise reduction using an OpenCV filter.

[0821] Examples of motion detection and behavior analysis

[0822] The server uses background subtraction to detect customers moving around the store and analyzes their movement speed and location using a Kaliman filter.

[0823] Specific examples of sentiment analysis

[0824] The server uses an emotion engine to analyze customers' facial expressions in specific areas and detect nervous or suspicious expressions in real time.

[0825] Specific examples of automated detection of criminal behavior

[0826] Based on the results of behavioral and emotional analysis, the server quickly retrieves products from specific areas and detects any behavior showing suspicious facial expressions as "theft behavior."

[0827] Specific examples of evidence generation and transmission

[0828] The server sends behavioral and emotional data to the generating AI, which then organizes the generated evidence information in an appropriate format and automatically sends it to the police.

[0829] Prompt Sentence Examples

[0830] "Please explain in detail the processing steps of a system that uses video footage acquired from surveillance cameras to detect motion, analyze behavior and emotions, automatically detect criminal activity, and transmit evidence to the police."

[0831] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0832] Step 1:

[0833] The server acquires video data in real time from a video acquisition device. The input is streaming video from a surveillance camera, which the server receives using the RTSP protocol. Specifically, it connects to a URL in the format "rtsp: / / username:password@cameraIP:554 / stream" to acquire the video data. The output is raw video data.

[0834] Step 2:

[0835] The server divides the acquired video data into frames and performs preprocessing. The input is the video data acquired in step 1, and the server uses the OpenCV library to perform noise removal and color correction. Specifically, it removes noise using "cv2.GaussianBlur(image, (5, 5), 0)". The output is preprocessed, high-quality video data.

[0836] Step 3:

[0837] The server analyzes the preprocessed video data and detects moving objects. The input is the preprocessed video data obtained in step 2, and the server detects moving objects using background subtraction. Specifically, it calculates the difference between consecutive frames and marks moving objects with bounding boxes. The output is video data containing the position information of moving objects.

[0838] Step 4:

[0839] The server analyzes the behavior of the detected moving object. The input is the video data containing the location information of the moving object obtained in step 3, and the server analyzes the moving speed and direction of the moving object using a Kaliman filter or tracking algorithm. For example, a tracking algorithm is used to compare the moving object's current position with its past positions and analyze its behavioral patterns. The output is the data resulting from the behavioral analysis.

[0840] Step 5:

[0841] The server uses an emotion engine to analyze the user's emotional state based on the behavioral analysis results. The input is the behavioral analysis results obtained in step 4, and the server uses facial recognition technology to recognize emotions from facial expressions. Specifically, it uses a deep learning model to detect suspicious expressions and tension. The output is the analyzed emotional data.

[0842] Step 6:

[0843] The server comprehensively evaluates the emotional data and behavioral analysis data to determine the likelihood of criminal behavior. The input is the data obtained in steps 4 and 5, and the server determines the likelihood of criminal behavior based on set rules. As a specific example, if a person quickly picks up an item in a specific area and shows a suspicious expression, it will be determined to be "theft behavior." The output is an evaluation result indicating the likelihood of criminal behavior.

[0844] Step 7:

[0845] If criminal behavior is detected, the server sends data to the generation AI. The input is the evaluation result obtained in step 6, and the server sends the data in JSON format to the API endpoint of the generation AI. The output is the data sent to the generation AI.

[0846] Step 8:

[0847] The generation AI generates detailed evidential information based on the received data. The input is the data sent in step 7, and the generation AI generates evidential information including a detailed description of the action, date and time, location, and identification information of people and objects involved. The output is the generated evidential information.

[0848] Step 9:

[0849] The server organizes the generated evidence and converts it into the appropriate format. The input is the evidence obtained in step 8, and the server formats the evidence into an email and includes related image and video links. The output is the evidence converted into email format.

[0850] Step 10:

[0851] The server automatically sends the organized evidence information to the police. The input is the evidence information in email format obtained in step 9, which the server sends to the police's dedicated email address using the SMTP protocol. The output is the evidence information sent to the police system.

[0852] (Application example 2)

[0853] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0854] Conventional security systems require manual review of surveillance camera footage, making it difficult to detect criminal behavior in real time and respond quickly. Furthermore, simply reviewing surveillance footage makes it difficult to grasp the details of a criminal's behavior or the victim's emotional state, resulting in low crime detection accuracy. Furthermore, reporting to the police must be done manually, making it difficult to respond quickly.

[0855] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0856] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data to detect moving objects, means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior, means for sending a real-time alert to a smart device when criminal behavior is detected, means for sending data to a generating artificial intelligence when criminal behavior is detected to generate evidence information, and means for organizing the generated evidence information and automatically sending it to the police. This enables the detection of criminal behavior and rapid reporting, making it possible to improve the efficiency of surveillance work and public safety.

[0857] An "image capture device" is a device such as a surveillance camera that captures image data and transfers it to a processing device such as a server.

[0858] "Splitting into frames" refers to the process of splitting continuous video data into still images (frames) at regular intervals.

[0859] "Preprocessing" is the process of performing initial processing such as noise removal and image quality adjustment on acquired video data to make it easier to analyze.

[0860] "Motion detection" refers to the process of identifying moving objects in video data and identifying their position and range.

[0861] "Behavior analysis" is a process of analyzing the position information, movement speed, and movement direction of a moving object and extracting a specific behavior pattern.

[0862] "Evaluating the possibility of criminal behavior" refers to the process of analyzing the behavior of a detected moving object and determining whether or not that behavior constitutes a crime.

[0863] A "smart device" is a portable information terminal with advanced processing capabilities, such as a smartphone, tablet, or smart glasses.

[0864] "Real-time alerts" are notifications that are generated immediately and send a warning when a specific condition is detected.

[0865] "Generative AI" is an AI technology used to generate detailed evidential information based on acquired data.

[0866] "Evidence information" is information that includes a detailed description of the detected criminal activity, as well as the people involved, dates, times, and locations.

[0867] "Automatic transmission to police" is a process in which the generated evidence information is formatted into a specific format and automatically transmitted to the police system.

[0868] This invention provides a system that acquires video data from a video capture device, performs real-time motion detection and behavior analysis, captures signs of criminal behavior, sends alerts to smart devices, generates evidence information using artificial intelligence, and automatically notifies the police.

[0869] System Configuration

[0870] 1. Image acquisition device

[0871] In this system, surveillance cameras installed in stores and public facilities function as video capture devices, and the video data is sent to a server in real time.

[0872] 2. Server

[0873] The server performs the following steps:

[0874] Video Acquisition:

[0875] The server acquires video data in real time from the video acquisition device, and the video data is streamed using the RTSP protocol.

[0876] Pretreatment:

[0877] The server divides the acquired video data into frames and performs preprocessing such as noise removal using OpenCV.

[0878] Motion detection:

[0879] The preprocessed video data is analyzed and moving objects are detected using background subtraction and a Kaliman filter.

[0880] Behavior analysis:

[0881] The location, speed, and direction of the detected moving objects are analyzed to evaluate their behavioral patterns.Face detection is also performed using the dlib library to analyze their emotional states.

[0882] Criminal Behavior Assessment:

[0883] Based on the results of behavioral and sentiment analysis, a pre-defined rules-based algorithm is used to assess the likelihood of criminal behavior.

[0884] 3. Smart Devices

[0885] If criminal activity is detected, the server sends the information in real time to a smart device (smartphone, smart glasses, etc.) and notifies the user with an alert.

[0886] 4. Generative Artificial Intelligence

[0887] The server sends the behavioral data and emotional data of the moving object to the AI ​​generator, which then generates detailed evidence information, including the date, time, location, details of the behavior, and emotional data.

[0888] 5. Police System

[0889] The generated evidence information is automatically sent to the police. The server uses the SMTP protocol to send the evidence information to the police's dedicated email address.

[0890] Specific examples

[0891] Image acquisition and preprocessing

[0892] For example, video data is acquired from a surveillance camera installed in a store using the RTSP protocol, divided into 30 fps frames, and noise is removed through an OpenCV filter.

[0893] Motion detection and behavior analysis

[0894] It uses background subtraction to detect customers moving around the store, and a Kaliman filter to analyze their location and speed. It also uses the dlib library for face detection and emotion analysis.

[0895] Automatic detection and alerting of criminal activity

[0896] The system detects in real time any suspicious behavior such as quickly picking up an item in a specific area as "theft behavior" and sends an alert to a smart device.

[0897] Generate evidence and automatically send it to the police

[0898] The server sends behavioral and emotional data to the generating AI, which then converts the generated evidence information into an email and automatically sends it to the police system.

[0899] Prompt Sentence Examples

[0900] "Write a program that performs speed anomaly detection using bounding box averaging in a cluster setting. This program uses OpenCV to analyze video data acquired from a surveillance camera and detect objects moving at speeds above a certain threshold."

[0901] In this way, it is possible to improve public safety by making surveillance more efficient and enabling a quicker response to crime.

[0902] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0903] Step 1:

[0904] The server acquires video data in real time from a video acquisition device. The input is a video stream sent using the RTSP protocol, and the output is video data divided into frames. Specifically, the server receives the video captured by the surveillance camera as is and divides the video into frames.

[0905] Step 2:

[0906] The server performs preprocessing on the acquired video data. The input is video divided into frames, and the output is preprocessed frame data that has undergone noise removal and image quality adjustment. Specifically, OpenCV is used to remove noise from the video, and brightness and contrast are adjusted as necessary.

[0907] Step 3:

[0908] The server analyzes the preprocessed video data and performs moving object detection. The input is the preprocessed frame data, and the output is the position information and bounding box of the detected moving object. Specifically, moving objects are extracted using background subtraction, and their position information is obtained using a Kaliman filter.

[0909] Step 4:

[0910] The server analyzes the behavior of the detected moving object. The input is the object's location information, and the output is the object's behavior pattern, movement speed, and movement direction data. Specifically, an analysis algorithm is used to extract the object's movement pattern and detect abnormalities in its movement.

[0911] Step 5:

[0912] The server uses the dlib library to recognize the faces of detected moving objects and analyze their emotional states. The input is frame data of the moving objects, and the output is emotion-analyzed information. Specifically, it detects faces and infers emotions from their facial expressions.

[0913] Step 6:

[0914] The server evaluates the likelihood of criminal behavior based on the results of behavioral and emotional analysis. The input is behavioral patterns and emotional data, and the output is the evaluation result of criminal behavior. Specifically, it uses a pre-configured rule-based model to determine whether the detected behavior and emotion match the behavioral criteria.

[0915] Step 7:

[0916] If criminal behavior is detected, the server sends a real-time alert to the smart device. The input is the criminal behavior evaluation result, and the output is an alert notification to the smart device. Specifically, the detected information is sent to a smartphone or smart glasses via a notification application.

[0917] Step 8:

[0918] When criminal behavior is detected, the server sends data to the generation AI to generate evidence. The input is behavioral data and emotional data, and the output is the generated evidence. Specifically, the generation AI model uses prompt sentences to generate detailed evidence.

[0919] Step 9:

[0920] The server organizes the generated evidence information and automatically sends it to the police. The input is the generated evidence information, and the output is a notification of completion of transmission to the police system. Specifically, the evidence information is formatted appropriately and sent to a dedicated email address for the police using the SMTP protocol.

[0921] Prompt Sentence Examples

[0922] "Write a program that performs speed anomaly detection using bounding box averaging in a cluster setting. This program uses OpenCV to analyze video data acquired from a surveillance camera and detect objects moving at speeds above a certain threshold."

[0923] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0924] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0925] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0926] [Third embodiment]

[0927] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0928] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0929] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0930] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0931] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0932] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0933] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0934] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0935] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0936] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0937] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0938] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0939] The overall system configuration and program processing flow for implementing this invention are shown below. Specifically, it explains how the server, terminals, and users work together to operate the system, automatically detect criminal behavior, and generate and transmit evidence information.

[0940] System Configuration

[0941] The system mainly consists of the following components:

[0942] 1. Video capture devices - Various surveillance cameras in stores, public facilities, etc.

[0943] 2. Server - Receives data from the video capture device, analyzes it, and generates and transmits evidence information.

[0944] 3. Generative AI - An artificial intelligence system that generates detailed evidence information based on data on detected criminal behavior.

[0945] 4. Police system - receives and responds to the evidence generated.

[0946] What the program does

[0947] Image acquisition and preprocessing

[0948] Video acquisition

[0949] The server acquires video data from the surveillance camera in real time.

[0950] For example, a server uses the RTSP protocol to periodically connect to a video capture device and receive a video stream.

[0951] Pretreatment

[0952] The video data acquired by the server is divided into frames.

[0953] Preprocessing such as noise removal is performed on each frame to improve data quality.

[0954] For example, apply a denoising filter using the OpenCV library.

[0955] Motion detection and behavior analysis

[0956] Motion Detection

[0957] The server analyzes the pre-processed video data, calculates the differences between frames, and detects moving objects.

[0958] If a moving object is detected, its location is identified and recorded using a bounding box.

[0959] Behavioral analysis

[0960] The server analyzes the location, speed, direction, etc. of the detected moving object.

[0961] The Kaliman filter and tracking algorithms are used to track the target and analyze its behavioral patterns.

[0962] For example, identifying criminal behavior such as quickly removing an object within a certain area.

[0963] Automated detection of criminal behavior

[0964] Rule-based Decision

[0965] The server applies rules to assess the likelihood of criminal behavior based on the analyzed behavioral data.

[0966] Automatically detects suspicious behavior based on pre-defined rules.

[0967] Alarm Generation

[0968] If the server detects suspicious activity, it generates a real-time alarm and notifies the management terminal.

[0969] Generating evidence

[0970] Data transmission

[0971] The server sends details of the suspicious activity to the generating artificial intelligence.

[0972] For example, behavioral data is sent to the generation AI in JSON format.

[0973] Evidence generation

[0974] The generation AI generates detailed evidence information based on the data it receives.

[0975] The evidence information includes a detailed description of the action, date and time, location information, etc.

[0976] Automatic transmission to police

[0977] Data reduction and format conversion

[0978] The server receives the evidence information from the generation AI and converts it into the appropriate format.

[0979] Convert evidence into email format and attach links and relevant images.

[0980] send

[0981] The server automatically sends evidence information to the police system.

[0982] For security reasons, SMTP or other communication protocols are used.

[0983] Specific examples

[0984] Image acquisition and preprocessing

[0985] Example 1

[0986] The server acquires video data from the surveillance cameras in the store via RTSP.

[0987] The video data is divided into 30fps frames and noise is removed using an OpenCV filter.

[0988] Motion detection and behavior analysis

[0989] Example 2

[0990] The server uses background subtraction to detect customers moving around the store.

[0991] A Kaliman filter is used to analyze the customer's location and movement speed to detect unnatural behavior.

[0992] Automated detection of criminal behavior

[0993] Example 3

[0994] The server detects the act of quickly removing an item from a specific area as "theft."

[0995] If any suspicious activity is detected, a real-time notification is sent to the management terminal via WebSocket.

[0996] Generating and sending evidence

[0997] Example 4

[0998] The server sends behavioral data to the generation AI, which then generates evidentiary information including date, time, location, and behavioral details.

[0999] Evidence information is converted into email and automatically sent to the police system.

[1000] Through these processes, this system will improve the efficiency of surveillance work and enable rapid crime response.

[1001] The processing flow will be explained below.

[1002] Step 1:

[1003] The server connects to the surveillance camera and acquires real-time video data. Specifically, the server receives the video stream using the RTSP protocol. The server continues to acquire data from the camera at regular intervals.

[1004] Step 2:

[1005] The server divides the acquired video data into frames. Specifically, it divides the video data into 30 frames per second (fps) and converts each frame into an easy-to-handle format. Next, preprocessing is performed, such as noise removal and resolution unification. The server performs preprocessing using the OpenCV library.

[1006] Step 3:

[1007] The server analyzes the preprocessed video data to detect moving objects. Specifically, it uses background subtraction and optical flow to calculate the difference between consecutive frames and identify moving areas. The server then uses bounding boxes to surround moving objects.

[1008] Step 4:

[1009] The server tracks the detected moving object and analyzes its behavior. Using a Kaliman filter and local tracking algorithm, the server calculates the object's location, speed, and direction of movement. This allows the server to analyze the object's movement pattern and identify any unnatural behavior.

[1010] Step 5:

[1011] The server evaluates the likelihood of criminal activity based on the results of behavioral analysis. Specifically, it uses a pre-configured rule-based system to automatically detect suspicious behavior. For example, quickly retrieving an object in a specific area is detected as criminal activity.

[1012] Step 6:

[1013] If the server detects suspicious activity, it generates a real-time alarm. Specifically, it uses WebSocket to send a notification to the management terminal, prompting immediate action. The server notifies the management terminal with detailed data about the suspicious activity.

[1014] Step 7:

[1015] The server sends suspicious behavior data to the generation AI, which then formats the behavior data in JSON format and sends it to the generation AI's API endpoint, which then starts generating detailed evidence.

[1016] Step 8:

[1017] The generation AI generates detailed evidence information based on the received data. Specifically, it generates a detailed description of the behavior, the date and time, the location, and the identification information of related people and objects. The generated evidence information is returned to the server.

[1018] Step 9:

[1019] The server receives the generated evidence and converts it into the appropriate format. The evidence is formatted as an email and includes any relevant images or video links.

[1020] Step 10:

[1021] The server automatically sends evidence information to the police system. Specifically, it sends evidence information to a dedicated police email address using the SMTP protocol. This allows evidence information to be shared quickly and securely.

[1022] Example 1

[1023] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1024] Conventional surveillance systems require manual analysis of video data obtained from surveillance cameras, making it difficult for limited personnel to review large volumes of data. Furthermore, the accuracy of motion detection and behavior analysis is low, making it difficult to detect suspicious activity. This has led to issues with early detection of criminal activity and rapid response.

[1025] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1026] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data and detecting moving objects, means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior, means for transmitting data to a generation artificial intelligence to generate evidence information when criminal behavior is detected, means for organizing the generated evidence information and automatically transmitting it to the police, means for acquiring video data from a surveillance camera in real time using the RTSP protocol, means for applying a noise reduction filter to the acquired video data using the OpenCV library, means for detecting moving objects using background subtraction, means for analyzing the behavioral patterns of moving objects using a Kaliman filter or tracking algorithm, and means for notifying a management terminal of suspicious behavior in real time using WebSocket. This enables automatic analysis of large amounts of video data, highly accurate detection of suspicious behavior, and rapid detection and response to criminal behavior.

[1027] The "image acquisition device" is a device for acquiring image data such as a surveillance camera.

[1028] "Video data" refers to digital video information obtained from a video capture device such as a surveillance camera.

[1029] "Server" means a central system for analyzing video data received from video capture devices and detecting and responding to criminal activity.

[1030] "Preprocessing" is a process of improving the quality of acquired video data by performing processes such as noise removal and frame division.

[1031] A "frame unit" is a unit into which video data is divided into individual still images.

[1032] A "moving object" is a moving object or person that is detected by the difference between frames.

[1033] "Motion detection" is the process of analyzing pre-processed video data and calculating the differences between frames to find moving objects.

[1034] "Behavioral analysis" refers to analyzing the location information, movement speed, and movement direction of a detected moving object to understand its behavioral patterns.

[1035] "Assessing the possibility of criminal behavior" refers to applying pre-set rules to the analyzed behavioral data to determine whether or not there is a possibility of criminal behavior.

[1036] "Generative AI" is an AI system that automatically generates detailed evidence information based on data on criminal behavior.

[1037] "Evidence information" is data containing detailed information about criminal activity, including date, time, location, and activity details.

[1038] "Automatic transmission to police" refers to the process of converting the generated evidence information into an appropriate format and automatically transmitting it to the police system.

[1039] The "RTSP protocol" is a communication protocol for acquiring video streams in real time.

[1040] The "OpenCV library" is an open-source library for performing computer vision tasks.

[1041] A "noise reduction filter" is a tool used to remove unnecessary noise from video data and improve the quality of the data.

[1042] "Background subtraction" is a technique used for motion detection, which detects moving objects by comparing them with the background.

[1043] The "Kaliman filter" is an algorithm for tracking the location information of moving objects, and estimates and predicts their location.

[1044] A "tracking algorithm" is an algorithm for tracking the successive positions of a moving object.

[1045] "WebSocket" is a protocol for two-way communication between a server and a management terminal.

[1046] A "management terminal" is a terminal device for receiving notifications of detected criminal behavior, and is usually used by an administrator.

[1047] The overall system configuration and specific program processing for implementing this invention are described below. The system consists of a video capture device including a surveillance camera, a server that analyzes and generates and transmits evidence information, a generation AI, and a police system.

[1048] Video data acquisition and preprocessing

[1049] The server receives video data from surveillance cameras in real time using the RTSP protocol. For example, it receives a video stream from a surveillance camera in a store, divides it into frames, and applies a noise reduction filter to each frame using the OpenCV library. This improves the quality of the data and makes the subsequent analysis process smoother.

[1050] Motion detection and behavior analysis

[1051] The server analyzes the preprocessed video data and detects moving objects using background subtraction. If a moving object is detected, its location is identified using a bounding box and its location information is recorded. The server then analyzes the location, speed, and direction of the detected moving object using a Kaliman filter and tracking algorithm to understand its behavioral patterns.

[1052] Automated detection of criminal behavior

[1053] The server then applies pre-defined rules to assess the likelihood of criminal activity based on the analyzed behavioral data. For example, a rule could be applied that defines "taking an item from a shelf within five seconds" as theft, and if suspicious activity is detected, a real-time alarm is generated and a notification is sent to the management terminal via WebSocket.

[1054] Generating and sending evidence

[1055] The server sends the detected behavioral data in JSON format to the generation AI. The generation AI generates detailed evidence information based on the received data, creating content including date, time, location, and behavior details. The server then converts this evidence information into email format and automatically sends it to the police system using the SMTP protocol.

[1056] Specific examples

[1057] For example, the server acquires a 30fps video stream from a store's surveillance cameras using the RTSP protocol and removes noise using OpenCV's GaussianBlur filter. Next, it detects moving people using background subtraction and tracks their location using a Kaliman filter. If the server detects someone quickly removing an item in a specific area, it sends an alarm notification to the management terminal in real time via WebSocket. The server then sends the behavioral data to the generation AI, which creates detailed evidence information. The server then converts this evidence information into email format and sends it to the police system using the SMTP protocol.

[1058] Example prompts for generative AI models

[1059] For example, the following prompt could be entered into the generation AI: "Generate detailed evidence information based on the following data: Data: Date and time: October 10, 2023, Location: Store A, Action: A person quickly removed an item from a specific area."

[1060] Through the above process, this invention automatically analyzes large amounts of video data, detects suspicious activity with high accuracy, and enables rapid detection and response to criminal activity. Furthermore, by automatically transmitting the generated detailed evidence information to the police, it supports rapid legal response.

[1061] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1062] Step 1:

[1063] The server acquires video data from a surveillance camera. As input, it uses the RTSP stream URL of the surveillance camera. Specifically, the server connects to the surveillance camera using the RTSP protocol and receives the video stream in real time. As output, it obtains the stream data.

[1064] Step 2:

[1065] The server divides the acquired video data into frames. The input is real-time video stream data. Specifically, the server divides the video stream into 30 frames per second (30 fps). The output is individual frame data.

[1066] Step 3:

[1067] The server performs noise reduction on each frame. As input, it receives the segmented frame data. Specifically, it applies a Gaussian Blur filter using the OpenCV library to remove noise from the frame. As output, it obtains high-quality frame data.

[1068] Step 4:

[1069] The server analyzes the preprocessed frame data and detects moving objects. The input is noise-removed frame data. Specifically, it uses background subtraction to detect movement between frames and identifies the location of moving objects using bounding boxes. The output is the location information of the moving objects.

[1070] Step 5:

[1071] The server analyzes the behavior of the detected moving object. The input is the location information of the moving object. Specifically, it uses a Kaliman filter and tracking algorithm to analyze the object's location, speed, and direction of movement, and identifies its behavioral pattern. The output is the analyzed behavioral data.

[1072] Step 6:

[1073] The server evaluates the likelihood of criminal behavior based on the analyzed behavioral data. The input is the analyzed behavioral data. Specific actions are detected by applying pre-set rules. For example, "taking an item from a shelf within five seconds" is evaluated as theft. The output is a notification of suspicious behavior.

[1074] Step 7:

[1075] If the server detects suspicious behavior, it generates an alarm and notifies the management terminal. The input is a notification of the detected suspicious behavior. The specific operation is to send a real-time notification to the management terminal using the WebSocket protocol. The output is a notification displayed on the management terminal.

[1076] Step 8:

[1077] The server sends behavioral data to the generation AI to generate evidential information. The input is detailed data on suspicious behavior. Specifically, the behavioral data is sent to the generation AI in JSON format, and the AI ​​generates evidential information including date, time, location, and details of the behavior. The generated evidential information is obtained as output.

[1078] Step 9:

[1079] The server automatically sends the generated evidence information to the police. The input is the evidence information obtained from the generation AI. Specifically, the server converts the evidence information into email format, attaches links and related images, and sends it to the police system using the SMTP protocol. The output is the evidence information sent to the police system.

[1080] (Application example 1)

[1081] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1082] Automatic detection systems for criminal behavior using surveillance cameras are essential for responding quickly when a crime occurs. However, current systems not only need to analyze video data from surveillance cameras in real time, but also need the ability to quickly and accurately generate and transmit detailed evidence information when criminal behavior is detected. Real-time alert generation and notification are also required to enable police to respond immediately. To achieve this, an effective system is required that integrates motion detection, behavior analysis, the generation of evidence information using generative AI, and automatic transmission functions.

[1083] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1084] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data and detecting moving objects, means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior, means for transmitting data to a generating AI and generating evidential information when criminal behavior is detected, means for organizing the generated evidential information and automatically transmitting it to the police, and means for saving screenshots and transmitting images to the police by email when criminal behavior is detected. This makes it possible to realize a system that detects criminal behavior in real time, quickly and accurately generates and transmits evidence, and enables the police to respond immediately.

[1085] An "image capture device" is a device for capturing image data in real time, such as a surveillance camera or video camera.

[1086] "Video data" refers to digital information that is organized for each frame of a captured video.

[1087] "Preprocessing" refers to the process of processing the acquired video data, such as by removing noise and dividing the data into frames, in order to improve the quality.

[1088] "Motion detection" is the process of identifying moving objects in video using differences between frames.

[1089] "Behavioral analysis" is a method of analyzing the movements of moving objects based on their location, speed, direction, etc., to find specific behavioral patterns.

[1090] The "means for assessing the likelihood of criminal activity" is a process that includes a rule-based algorithm for assessing and detecting the likelihood of criminal activity based on the analyzed data.

[1091] "Generative artificial intelligence" is an AI system that generates detailed evidential information based on specified data.

[1092] "Evidential information" is information necessary to prove criminal activity, such as the date, time, location, and details of the action.

[1093] "Automatic transmission" is the process of quickly transmitting generated evidence information to the police using a pre-defined protocol.

[1094] "Real-time alert" is a warning system that immediately notifies you of any suspicious behavior when it is detected, prompting you to take action.

[1095] A "screenshot" is a technique for capturing video data at a specific point in time and saving it as a still image.

[1096] "Email transmission" is a communication method for transmitting information in the form of email via the Internet.

[1097] To implement this invention, the following specific system configuration and processing procedures are required. The entire system operates through the cooperation of three parties: a server, a terminal, and a user. Automatic detection of criminal behavior and generation and transmission of evidence information are carried out according to the following procedures.

[1098] System Configuration

[1099] 1. Video acquisition device: A device such as a surveillance camera installed in stores and public facilities that acquires video data in real time.

[1100] 2. Server: Plays the central role of receiving and analyzing the video data sent from the video capture device, and generating and sending evidence information.

[1101] 3. Generative AI model: An artificial intelligence system that generates detailed evidential information based on data sent from the server.

[1102] 4. Police system: A system for receiving and responding to the evidence information generated.

[1103] What the program does

[1104] Image acquisition and preprocessing

[1105] The server acquires video data in real time from a surveillance camera. A typical surveillance camera is used as the video acquisition device. The acquired video data is divided into frames and preprocessed using OpenCV filters, such as noise removal.

[1106] Motion detection and behavior analysis

[1107] The server analyzes the preprocessed video data and calculates the difference between frames to detect moving objects. Using a Kaliman filter and tracking algorithm, the server analyzes the location, speed, and direction of the detected moving object to identify its behavioral pattern. For example, the action of quickly removing an object within a certain area can be detected as "theft."

[1108] Automated detection of criminal behavior

[1109] The server uses rule-based decision-making to assess the likelihood of criminal activity based on the analyzed behavioral data. If suspicious activity is detected, it generates a real-time alarm and notifies the management terminal. It also has the ability to save screenshots and send images via email to the police if criminal activity is detected.

[1110] Generating and sending evidence

[1111] The server sends details of suspicious behavior to a generative AI model, which then generates evidence including the date, time, location, and details of the behavior. The server then organizes the evidence, converts it into an appropriate format, and automatically transmits it to the police system. A secure communication protocol, such as SMTP, is used for transmission.

[1112] Specific examples

[1113] 1. Example of image acquisition and pre-processing:

[1114] The server acquires video data from the surveillance cameras inside the store using the RTSP protocol, divides it into 30fps frames, and removes noise using an OpenCV filter.

[1115] 2. Examples of motion detection and behavior analysis:

[1116] The server uses background subtraction to detect customers moving around the store, and uses a Kaliman filter to analyze the customer's location and movement speed to detect any unnatural behavior.

[1117] 3. Examples of evidence generation and transmission to police:

[1118] The server sends the behavioral data to the generation AI, which then generates evidential information including the date, time, location, and details of the behavior. The generated evidential information is converted into an email and automatically sent to the police system.

[1119] Prompt Sentence Examples

[1120] "Write a script that grabs frames from an RTSP stream, detects suspicious activity, saves the image, and notifies the police."

[1121] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1122] Step 1:

[1123] The server acquires video data from the video acquisition device. Specifically, the server connects to the surveillance camera via the RTSP protocol and receives the video stream in real time. The input of this step is the video stream provided by the surveillance camera, and the output is the video data received by the server.

[1124] Step 2:

[1125] The server divides the acquired video data into frames and performs preprocessing. Specifically, the video data is divided into 30 fps frames and noise is removed using an OpenCV filter. The input to this step is the video data received in real time, and the output is the preprocessed frame data.

[1126] Step 3:

[1127] The server analyzes the preprocessed video data to detect moving objects. Specifically, it uses background subtraction to calculate the difference between frames and identify moving objects. The input to this step is the preprocessed frame data, and the output is data with identified moving objects.

[1128] Step 4:

[1129] The server analyzes the behavior of the detected moving object. Specifically, it analyzes the location information, movement speed, and movement direction of the moving object using a Kaliman filter or tracking algorithm. The input of this step is the data on the identified moving object, and the output is the analyzed behavior information of the moving object.

[1130] Step 5:

[1131] The server evaluates the likelihood of criminal behavior based on the analyzed behavioral data. Specifically, it uses rule-based judgment to identify suspicious behavior. The input to this step is the analyzed behavioral information of the moving object, and the output is data evaluating the likelihood of criminal behavior.

[1132] Step 6:

[1133] When criminal behavior is detected, the server sends the data to the generation AI to generate evidence. Specifically, the behavioral data is sent in JSON format to the generation AI model, which generates detailed evidence. The input to this step is the data determined to be criminal behavior, and the output is the generated evidence.

[1134] Step 7:

[1135] The server organizes the generated evidence information and automatically sends it to the police. Specifically, it converts the evidence information into an appropriate format and sends it to the police system using the SMTP protocol. The input to this step is the evidence information from the generative AI model, and the output is the evidence data sent to the police.

[1136] Step 8:

[1137] If the server detects a criminal activity, it saves a screenshot and sends the image to the police by email. Specifically, it sends a warning message along with the saved screenshot by email. The input of this step is the video data at the time when the suspicious activity was detected, and the output is the screenshot and warning message sent to the police.

[1138] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1139] This invention is a system that acquires video data from a video capture device, performs moving object detection and behavior analysis, and analyzes user emotions through an emotion engine, detects criminal behavior, and automatically transmits evidence of the behavior to the police. The system configuration, program processing flow, and specific examples are described below.

[1140] System Configuration

[1141] The system mainly consists of the following components:

[1142] 1. Video capture device - Surveillance cameras installed in stores, public facilities, etc.

[1143] 2. Server - Receives data from the video capture device, performs motion detection, behavior analysis, emotion analysis, and generates and transmits evidence information.

[1144] 3. Generative AI - Artificial intelligence that generates detailed evidence from data that detects criminal behavior.

[1145] 4. Emotion Engine - Analyzes the user's facial expressions and movements from video to recognize their emotional state.

[1146] 5. Police system - receives and responds to generated evidence information.

[1147] What the program does

[1148] Image acquisition and preprocessing

[1149] Video acquisition

[1150] The server obtains video data from the surveillance camera in real time, for example, by receiving the video stream using the RTSP protocol.

[1151] Pretreatment

[1152] The server performs preprocessing such as noise removal on the video data divided into frames, and improves the quality of the data using the OpenCV library.

[1153] Motion detection and behavior analysis

[1154] Motion Detection

[1155] The server analyzes the pre-processed video data and calculates the difference between consecutive frames to detect moving objects. It identifies and records moving objects using bounding boxes.

[1156] Behavioral analysis

[1157] The server analyzes the location information, movement speed, and movement direction of the moving object, and tracks the object and analyzes its behavior using a Kaliman filter and tracking algorithm.

[1158] Emotion analysis

[1159] emotion recognition

[1160] In addition to analyzing the behavior of moving objects, the server uses an emotion engine to analyze emotions from the user's facial expressions and movements. For example, it uses facial recognition technology to detect suspicious expressions or tension.

[1161] Emotional Data Evaluation

[1162] The server evaluates the likelihood of criminal behavior based on the recognized emotional data, and performs a comprehensive analysis of emotional changes and behavioral patterns according to a set algorithm.

[1163] Automated detection of criminal behavior

[1164] Rule-based Decision

[1165] The server uses a pre-configured rules-based system based on behavioral and emotional analysis to detect suspicious behavior, such as quickly picking up an object in a specific area or displaying a suspicious facial expression.

[1166] Alarm Generation

[1167] When the server detects suspicious activity, it generates a real-time alarm and sends a notification to the management terminal.

[1168] Generating evidence

[1169] Data transmission

[1170] The server sends suspicious behavior and emotion data to the generation AI, formats the data in JSON format, and sends it to the generation AI's API endpoint.

[1171] Evidence generation

[1172] The generation AI generates detailed evidence based on the received data, including a detailed description of the behavior, the date and time, the location, and the identity of the people and objects involved.

[1173] Automatic transmission to police

[1174] Data reduction and format conversion

[1175] The server receives the generated evidence and converts it into the appropriate format. The evidence is formatted as an email and includes any relevant images or video links.

[1176] send

[1177] The server automatically sends the evidence information to the police system. Specifically, it sends the evidence information to a dedicated email address for the police using the SMTP protocol.

[1178] Specific examples

[1179] Image acquisition and preprocessing

[1180] Example 1

[1181] The server receives video data from the surveillance cameras in the store via RTSP. The video data is split into 30fps frames and noise is removed using an OpenCV filter.

[1182] Motion detection and behavior analysis

[1183] Example 2

[1184] The server uses background subtraction to detect customers moving around the store, and a Kaliman filter is used to analyze their location and speed to detect unusual behavior.

[1185] Emotion analysis

[1186] Example 3

[1187] The server uses an emotion engine to analyze customer facial expressions in specific areas and detect nervous or suspicious expressions, using machine learning algorithms to recognize emotions in real time.

[1188] Automated detection of criminal behavior

[1189] Example 4

[1190] Based on the results of the behavioral and emotional analysis of the moving object, the server detects the behavior of quickly picking up an item in a specific area and showing a suspicious expression as "theft behavior."

[1191] Generating and sending evidence

[1192] Example 5

[1193] The server sends behavioral and emotional data to the AI, which then generates evidential information including date, time, location, and details of the behavior. The evidential information is then converted into an email and automatically sent to the police system.

[1194] This system will improve public safety by streamlining surveillance operations and enabling rapid crime response through analysis of motion and emotions.

[1195] The processing flow will be explained below.

[1196] Step 1:

[1197] The server connects to the surveillance camera and acquires real-time video data. Specifically, the server receives the video stream using the RTSP protocol. The server continues to acquire data from the camera at regular intervals.

[1198] Step 2:

[1199] The server divides the acquired video data into frames. Specifically, it divides the video data into 30 frames per second (fps) and converts each frame into an easy-to-handle format. Next, preprocessing is performed, such as noise removal and resolution unification. The server performs preprocessing using the OpenCV library.

[1200] Step 3:

[1201] The server analyzes the preprocessed video data to detect moving objects. Specifically, it uses background subtraction and optical flow to calculate the difference between consecutive frames and identify moving areas. The server then uses bounding boxes to surround moving objects.

[1202] Step 4:

[1203] The server tracks the detected moving object and analyzes its behavior. Using a Kaliman filter and local tracking algorithm, the server calculates the object's location, speed, and direction of movement. This allows the server to analyze the object's movement pattern and identify any unnatural behavior.

[1204] Step 5:

[1205] Based on the results of the motion analysis, the server uses an emotion engine to analyze the user's emotions from their facial expressions and movements. Specifically, it uses facial expression recognition technology to detect nervous or suspicious expressions, and generates emotion data based on this.

[1206] Step 6:

[1207] The server evaluates the possibility of criminal behavior based on emotion data and combines it with the results of motion analysis. For example, if a combination of a certain behavioral pattern and emotion is considered to be criminal behavior, the server evaluates the possibility.

[1208] Step 7:

[1209] Based on the results of behavioral and emotional analysis, the server automatically uses a rule-based system to determine the likelihood of criminal behavior. For example, if someone quickly picks up an object in a specific area and shows a suspicious expression, it will be determined to be "theft behavior."

[1210] Step 8:

[1211] If the server detects any criminal activity, it generates an alarm in real time on the management terminal, specifically by sending a notification using WebSocket to prompt immediate action.

[1212] Step 9:

[1213] The server sends suspicious behavior and emotion data to the AI ​​generator. The server formats the data in JSON format and sends it to the AI ​​generator's API endpoint. This process generates detailed evidence.

[1214] Step 10:

[1215] Based on the data received, the generation AI generates detailed evidence, including a detailed description of the behavior, the date and time, the location, and the identity of the people and objects involved.

[1216] Step 11:

[1217] The server receives the generated evidence, converts it into an appropriate format, and formats the evidence into an email with relevant images and video links attached.

[1218] Step 12:

[1219] The server automatically sends evidence information to the police system. Specifically, it sends evidence information to a dedicated email address for the police using the SMTP protocol. This allows evidence information to be shared quickly and securely.

[1220] Example 2

[1221] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1222] Conventional surveillance systems focus on detecting moving objects and analyzing behavior, but they do not analyze the user's emotional state, making it difficult to accurately detect criminal behavior. Furthermore, they lack the ability to quickly and accurately generate evidence of detected criminal behavior and automatically send it to the police. This can delay early response to criminal behavior and potentially reduce public safety.

[1223] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1224] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data and detecting a moving object, means for analyzing the behavior of the detected moving object and analyzing its behavioral pattern using its moving speed and direction, means for analyzing the emotional state of the user using an emotion engine based on the results of the behavioral analysis of the moving object and evaluating the possibility of criminal behavior, means for sending data to an artificial intelligence generated when criminal behavior is detected and generating evidential information, and means for organizing the generated evidential information and automatically transmitting it to the police. This enables accurate detection of criminal behavior and rapid response by comprehensively performing behavioral analysis and emotion analysis of the moving object.

[1225] The "image acquisition device" is a device for acquiring image data such as a surveillance camera.

[1226] "Video data" refers to data that expresses acquired video information in a digital format.

[1227] "Preprocessing" refers to performing processes such as noise removal and frame division to make video data easier to analyze.

[1228] A "moving object" refers to an object that changes position between successive video frames.

[1229] "Motion detection" refers to identifying moving objects in video data and recognizing their position and movement.

[1230] "Behavioral analysis" refers to analyzing the location information, movement speed, movement direction, etc. of detected moving objects to clarify their behavioral patterns.

[1231] An "emotion engine" refers to a system that analyzes the emotional state of a moving subject from its facial expressions and movements.

[1232] "User" refers to a person or other object that appears in the video data.

[1233] "Possible criminal behavior" refers to assessing whether a target's behavior constitutes a crime based on behavioral and emotional analysis.

[1234] "Generative AI" refers to AI that generates detailed evidentiary information using data evaluated as criminal behavior.

[1235] "Evidence information" refers to information that details criminal behavior and is generated based on analyzed behaviors and emotions.

[1236] "Police" refers to law enforcement agencies that investigate crimes to maintain public safety and order.

[1237] "Data transmission" refers to sending required data to other systems or devices via a network.

[1238] This invention is a system that acquires video data from a video capture device, performs motion detection and behavior analysis, and analyzes user emotions through an emotion engine, detects criminal behavior, and automatically transmits evidence to the police. This system is composed of multiple components, including a video capture device, a server, a generation AI, an emotion engine, and a police system.

[1239] System configuration and processing content

[1240] Image acquisition and preprocessing

[1241] Image acquisition device

[1242] The server acquires video data in real time from video acquisition devices such as surveillance cameras. Specifically, it uses RTSP (Real-Time Streaming Protocol) to receive video streams in the format "rtsp: / / username:password@cameraIP:554 / stream."

[1243] Pretreatment

[1244] The server divides the acquired video data into frames and performs preprocessing such as noise removal and color correction using the OpenCV library, thereby preparing the data for improved analysis accuracy.

[1245] Motion detection and behavior analysis

[1246] Motion Detection

[1247] The server detects motion by applying background subtraction to the pre-processed video data, which analyzes the differences between consecutive frames to identify moving objects.

[1248] Behavioral analysis

[1249] The server analyzes the behavioral patterns of the detected moving object using its location, speed, and direction. Specifically, it uses a Kaliman filter and tracking algorithm to track the object and perform detailed behavioral analysis.

[1250] Emotion analysis

[1251] emotion recognition

[1252] Based on the behavioral analysis results of the moving object, the server uses an emotion engine to analyze the user's emotions from their facial expressions and movements, and uses facial recognition technology to detect suspicious expressions and tension in real time.

[1253] Emotional Data Evaluation

[1254] The server evaluates the likelihood of criminal behavior based on the recognized emotional data, and performs a comprehensive analysis of emotional changes and behavioral patterns according to a specific algorithm.

[1255] Automated detection of criminal behavior

[1256] Rule-based Decision

[1257] The server uses a pre-configured rule-based system to detect suspicious behavior based on the results of behavioral and emotional analysis, such as quickly taking an object from a specific area or displaying a suspicious facial expression, as a "theft attempt."

[1258] Alarm Generation

[1259] If the server detects any suspicious activity, it generates an alarm in real time and sends a notification to the management terminal.

[1260] Evidence generation and transmission

[1261] Data transmission

[1262] The server sends suspicious behavior and emotion data to the generator AI, formatted in JSON and sent through an API endpoint.

[1263] Evidence generation

[1264] Based on the data it receives, the generation AI generates detailed evidence, including a detailed description of the behavior, the date and time, the location, and the identity of the people and objects involved.

[1265] Data reduction and format conversion

[1266] The server receives the generated evidence and converts it into the appropriate format, arranging it into an email format and including related images and video links.

[1267] send

[1268] The server automatically sends the organized evidence information to the police system, specifically to a dedicated email address for the police using the SMTP protocol.

[1269] Specific examples

[1270] Specific examples of image acquisition and preprocessing

[1271] The server connects to "rtsp: / / username:password@cameraIP:554 / stream", acquires video data at 30 frames per second, and performs noise reduction using an OpenCV filter.

[1272] Examples of motion detection and behavior analysis

[1273] The server uses background subtraction to detect customers moving around the store and analyzes their movement speed and location using a Kaliman filter.

[1274] Specific examples of sentiment analysis

[1275] The server uses an emotion engine to analyze customers' facial expressions in specific areas and detect nervous or suspicious expressions in real time.

[1276] Specific examples of automated detection of criminal behavior

[1277] Based on the results of behavioral and emotional analysis, the server quickly retrieves products from specific areas and detects any behavior showing suspicious facial expressions as "theft behavior."

[1278] Specific examples of evidence generation and transmission

[1279] The server sends behavioral and emotional data to the generating AI, which then organizes the generated evidence information in an appropriate format and automatically sends it to the police.

[1280] Prompt Sentence Examples

[1281] "Please explain in detail the processing steps of a system that uses video footage acquired from surveillance cameras to detect motion, analyze behavior and emotions, automatically detect criminal activity, and transmit evidence to the police."

[1282] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1283] Step 1:

[1284] The server acquires video data in real time from a video acquisition device. The input is streaming video from a surveillance camera, which the server receives using the RTSP protocol. Specifically, it connects to a URL in the format "rtsp: / / username:password@cameraIP:554 / stream" to acquire the video data. The output is raw video data.

[1285] Step 2:

[1286] The server divides the acquired video data into frames and performs preprocessing. The input is the video data acquired in step 1, and the server uses the OpenCV library to perform noise removal and color correction. Specifically, it removes noise using "cv2.GaussianBlur(image, (5, 5), 0)". The output is preprocessed, high-quality video data.

[1287] Step 3:

[1288] The server analyzes the preprocessed video data and detects moving objects. The input is the preprocessed video data obtained in step 2, and the server detects moving objects using background subtraction. Specifically, it calculates the difference between consecutive frames and marks moving objects with bounding boxes. The output is video data containing the position information of moving objects.

[1289] Step 4:

[1290] The server analyzes the behavior of the detected moving object. The input is the video data containing the location information of the moving object obtained in step 3, and the server analyzes the moving speed and direction of the moving object using a Kaliman filter or tracking algorithm. For example, a tracking algorithm is used to compare the moving object's current position with its past positions and analyze its behavioral patterns. The output is the data resulting from the behavioral analysis.

[1291] Step 5:

[1292] The server uses an emotion engine to analyze the user's emotional state based on the behavioral analysis results. The input is the behavioral analysis results obtained in step 4, and the server uses facial recognition technology to recognize emotions from facial expressions. Specifically, it uses a deep learning model to detect suspicious expressions and tension. The output is the analyzed emotional data.

[1293] Step 6:

[1294] The server comprehensively evaluates the emotional data and behavioral analysis data to determine the likelihood of criminal behavior. The input is the data obtained in steps 4 and 5, and the server determines the likelihood of criminal behavior based on set rules. As a specific example, if a person quickly picks up an item in a specific area and shows a suspicious expression, it will be determined to be "theft behavior." The output is an evaluation result indicating the likelihood of criminal behavior.

[1295] Step 7:

[1296] If criminal behavior is detected, the server sends data to the generation AI. The input is the evaluation result obtained in step 6, and the server sends the data in JSON format to the API endpoint of the generation AI. The output is the data sent to the generation AI.

[1297] Step 8:

[1298] The generation AI generates detailed evidential information based on the received data. The input is the data sent in step 7, and the generation AI generates evidential information including a detailed description of the action, date and time, location, and identification information of people and objects involved. The output is the generated evidential information.

[1299] Step 9:

[1300] The server organizes the generated evidence and converts it into the appropriate format. The input is the evidence obtained in step 8, and the server formats the evidence into an email and includes related image and video links. The output is the evidence converted into email format.

[1301] Step 10:

[1302] The server automatically sends the organized evidence information to the police. The input is the evidence information in email format obtained in step 9, which the server sends to the police's dedicated email address using the SMTP protocol. The output is the evidence information sent to the police system.

[1303] (Application example 2)

[1304] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1305] Conventional security systems require manual review of surveillance camera footage, making it difficult to detect criminal behavior in real time and respond quickly. Furthermore, simply reviewing surveillance footage makes it difficult to grasp the details of a criminal's behavior or the victim's emotional state, resulting in low crime detection accuracy. Furthermore, reporting to the police must be done manually, making it difficult to respond quickly.

[1306] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1307] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data to detect moving objects, means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior, means for sending a real-time alert to a smart device when criminal behavior is detected, means for sending data to a generating artificial intelligence when criminal behavior is detected to generate evidence information, and means for organizing the generated evidence information and automatically sending it to the police. This enables the detection of criminal behavior and rapid reporting, making it possible to improve the efficiency of surveillance work and public safety.

[1308] An "image capture device" is a device such as a surveillance camera that captures image data and transfers it to a processing device such as a server.

[1309] "Splitting into frames" refers to the process of splitting continuous video data into still images (frames) at regular intervals.

[1310] "Preprocessing" is the process of performing initial processing such as noise removal and image quality adjustment on acquired video data to make it easier to analyze.

[1311] "Motion detection" refers to the process of identifying moving objects in video data and identifying their position and range.

[1312] "Behavior analysis" is a process of analyzing the position information, movement speed, and movement direction of a moving object and extracting a specific behavior pattern.

[1313] "Evaluating the possibility of criminal behavior" refers to the process of analyzing the behavior of a detected moving object and determining whether or not that behavior constitutes a crime.

[1314] A "smart device" is a portable information terminal with advanced processing capabilities, such as a smartphone, tablet, or smart glasses.

[1315] "Real-time alerts" are notifications that are generated immediately and send a warning when a specific condition is detected.

[1316] "Generative AI" is an AI technology used to generate detailed evidential information based on acquired data.

[1317] "Evidence information" is information that includes a detailed description of the detected criminal activity, as well as the people involved, dates, times, and locations.

[1318] "Automatic transmission to police" is a process in which the generated evidence information is formatted into a specific format and automatically transmitted to the police system.

[1319] This invention provides a system that acquires video data from a video capture device, performs real-time motion detection and behavior analysis, captures signs of criminal behavior, sends alerts to smart devices, generates evidence information using artificial intelligence, and automatically notifies the police.

[1320] System Configuration

[1321] 1. Image acquisition device

[1322] In this system, surveillance cameras installed in stores and public facilities function as video capture devices, and the video data is sent to a server in real time.

[1323] 2. Server

[1324] The server performs the following steps:

[1325] Video Acquisition:

[1326] The server acquires video data in real time from the video acquisition device, and the video data is streamed using the RTSP protocol.

[1327] Pretreatment:

[1328] The server divides the acquired video data into frames and performs preprocessing such as noise removal using OpenCV.

[1329] Motion detection:

[1330] The preprocessed video data is analyzed and moving objects are detected using background subtraction and a Kaliman filter.

[1331] Behavior analysis:

[1332] The location, speed, and direction of the detected moving objects are analyzed to evaluate their behavioral patterns.Face detection is also performed using the dlib library to analyze their emotional states.

[1333] Criminal Behavior Assessment:

[1334] Based on the results of behavioral and sentiment analysis, a pre-defined rules-based algorithm is used to assess the likelihood of criminal behavior.

[1335] 3. Smart Devices

[1336] If criminal activity is detected, the server sends the information in real time to a smart device (smartphone, smart glasses, etc.) and notifies the user with an alert.

[1337] 4. Generative Artificial Intelligence

[1338] The server sends the behavioral data and emotional data of the moving object to the AI ​​generator, which then generates detailed evidence information, including the date, time, location, details of the behavior, and emotional data.

[1339] 5. Police System

[1340] The generated evidence information is automatically sent to the police. The server uses the SMTP protocol to send the evidence information to the police's dedicated email address.

[1341] Specific examples

[1342] Image acquisition and preprocessing

[1343] For example, video data is acquired from a surveillance camera installed in a store using the RTSP protocol, divided into 30 fps frames, and noise is removed through an OpenCV filter.

[1344] Motion detection and behavior analysis

[1345] It uses background subtraction to detect customers moving around the store, and a Kaliman filter to analyze their location and speed. It also uses the dlib library for face detection and emotion analysis.

[1346] Automatic detection and alerting of criminal activity

[1347] The system detects in real time any suspicious behavior such as quickly picking up an item in a specific area as "theft behavior" and sends an alert to a smart device.

[1348] Generate evidence and automatically send it to the police

[1349] The server sends behavioral and emotional data to the generating AI, which then converts the generated evidence information into an email and automatically sends it to the police system.

[1350] Prompt Sentence Examples

[1351] "Write a program that performs speed anomaly detection using bounding box averaging in a cluster setting. This program uses OpenCV to analyze video data acquired from a surveillance camera and detect objects moving at speeds above a certain threshold."

[1352] In this way, it is possible to improve public safety by making surveillance more efficient and enabling a quicker response to crime.

[1353] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1354] Step 1:

[1355] The server acquires video data in real time from a video acquisition device. The input is a video stream sent using the RTSP protocol, and the output is video data divided into frames. Specifically, the server receives the video captured by the surveillance camera as is and divides the video into frames.

[1356] Step 2:

[1357] The server performs preprocessing on the acquired video data. The input is video divided into frames, and the output is preprocessed frame data that has undergone noise removal and image quality adjustment. Specifically, OpenCV is used to remove noise from the video, and brightness and contrast are adjusted as necessary.

[1358] Step 3:

[1359] The server analyzes the preprocessed video data and performs moving object detection. The input is the preprocessed frame data, and the output is the position information and bounding box of the detected moving object. Specifically, moving objects are extracted using background subtraction, and their position information is obtained using a Kaliman filter.

[1360] Step 4:

[1361] The server analyzes the behavior of the detected moving object. The input is the object's location information, and the output is the object's behavior pattern, movement speed, and movement direction data. Specifically, an analysis algorithm is used to extract the object's movement pattern and detect abnormalities in its movement.

[1362] Step 5:

[1363] The server uses the dlib library to recognize the faces of detected moving objects and analyze their emotional states. The input is frame data of the moving objects, and the output is emotion-analyzed information. Specifically, it detects faces and infers emotions from their facial expressions.

[1364] Step 6:

[1365] The server evaluates the likelihood of criminal behavior based on the results of behavioral and emotional analysis. The input is behavioral patterns and emotional data, and the output is the evaluation result of criminal behavior. Specifically, it uses a pre-configured rule-based model to determine whether the detected behavior and emotion match the behavioral criteria.

[1366] Step 7:

[1367] If criminal behavior is detected, the server sends a real-time alert to the smart device. The input is the criminal behavior evaluation result, and the output is an alert notification to the smart device. Specifically, the detected information is sent to a smartphone or smart glasses via a notification application.

[1368] Step 8:

[1369] When criminal behavior is detected, the server sends data to the generation AI to generate evidence. The input is behavioral data and emotional data, and the output is the generated evidence. Specifically, the generation AI model uses prompt sentences to generate detailed evidence.

[1370] Step 9:

[1371] The server organizes the generated evidence information and automatically sends it to the police. The input is the generated evidence information, and the output is a notification of completion of transmission to the police system. Specifically, the evidence information is formatted appropriately and sent to a dedicated email address for the police using the SMTP protocol.

[1372] Prompt Sentence Examples

[1373] "Write a program that performs speed anomaly detection using bounding box averaging in a cluster setting. This program uses OpenCV to analyze video data acquired from a surveillance camera and detect objects moving at speeds above a certain threshold."

[1374] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1375] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1376] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1377] [Fourth embodiment]

[1378] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1379] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1380] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1381] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1382] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1383] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1384] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1385] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1386] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1387] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1388] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1389] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1390] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1391] The overall system configuration and program processing flow for implementing this invention are shown below. Specifically, it explains how the server, terminals, and users work together to operate the system, automatically detect criminal behavior, and generate and transmit evidence information.

[1392] System Configuration

[1393] The system mainly consists of the following components:

[1394] 1. Video capture devices - Various surveillance cameras in stores, public facilities, etc.

[1395] 2. Server - Receives data from the video capture device, analyzes it, and generates and transmits evidence information.

[1396] 3. Generative AI - An artificial intelligence system that generates detailed evidence information based on data on detected criminal behavior.

[1397] 4. Police system - receives and responds to the evidence generated.

[1398] What the program does

[1399] Image acquisition and preprocessing

[1400] Video acquisition

[1401] The server acquires video data from the surveillance camera in real time.

[1402] For example, a server uses the RTSP protocol to periodically connect to a video capture device and receive a video stream.

[1403] Pretreatment

[1404] The video data acquired by the server is divided into frames.

[1405] Preprocessing such as noise removal is performed on each frame to improve data quality.

[1406] For example, apply a denoising filter using the OpenCV library.

[1407] Motion detection and behavior analysis

[1408] Motion Detection

[1409] The server analyzes the pre-processed video data, calculates the differences between frames, and detects moving objects.

[1410] If a moving object is detected, its location is identified and recorded using a bounding box.

[1411] Behavioral analysis

[1412] The server analyzes the location, speed, direction, etc. of the detected moving object.

[1413] The Kaliman filter and tracking algorithms are used to track the target and analyze its behavioral patterns.

[1414] For example, identifying criminal behavior such as quickly removing an object within a certain area.

[1415] Automated detection of criminal behavior

[1416] Rule-based Decision

[1417] The server applies rules to assess the likelihood of criminal behavior based on the analyzed behavioral data.

[1418] Automatically detects suspicious behavior based on pre-defined rules.

[1419] Alarm Generation

[1420] If the server detects suspicious activity, it generates a real-time alarm and notifies the management terminal.

[1421] Generating evidence

[1422] Data transmission

[1423] The server sends details of the suspicious activity to the generating artificial intelligence.

[1424] For example, behavioral data is sent to the generation AI in JSON format.

[1425] Evidence generation

[1426] The generation AI generates detailed evidence information based on the data it receives.

[1427] The evidence information includes a detailed description of the action, date and time, location information, etc.

[1428] Automatic transmission to police

[1429] Data reduction and format conversion

[1430] The server receives the evidence information from the generation AI and converts it into the appropriate format.

[1431] Convert evidence into email format and attach links and relevant images.

[1432] send

[1433] The server automatically sends evidence information to the police system.

[1434] For security reasons, SMTP or other communication protocols are used.

[1435] Specific examples

[1436] Image acquisition and preprocessing

[1437] Example 1

[1438] The server acquires video data from the surveillance cameras in the store via RTSP.

[1439] The video data is divided into 30fps frames and noise is removed using an OpenCV filter.

[1440] Motion detection and behavior analysis

[1441] Example 2

[1442] The server uses background subtraction to detect customers moving around the store.

[1443] A Kaliman filter is used to analyze the customer's location and movement speed to detect unnatural behavior.

[1444] Automated detection of criminal behavior

[1445] Example 3

[1446] The server detects the act of quickly removing an item from a specific area as "theft."

[1447] If any suspicious activity is detected, a real-time notification is sent to the management terminal via WebSocket.

[1448] Generating and sending evidence

[1449] Example 4

[1450] The server sends behavioral data to the generation AI, which then generates evidentiary information including date, time, location, and behavioral details.

[1451] Evidence information is converted into email and automatically sent to the police system.

[1452] Through these processes, this system will improve the efficiency of surveillance work and enable rapid crime response.

[1453] The processing flow will be explained below.

[1454] Step 1:

[1455] The server connects to the surveillance camera and acquires real-time video data. Specifically, the server receives the video stream using the RTSP protocol. The server continues to acquire data from the camera at regular intervals.

[1456] Step 2:

[1457] The server divides the acquired video data into frames. Specifically, it divides the video data into 30 frames per second (fps) and converts each frame into an easy-to-handle format. Next, preprocessing is performed, such as noise removal and resolution unification. The server performs preprocessing using the OpenCV library.

[1458] Step 3:

[1459] The server analyzes the preprocessed video data to detect moving objects. Specifically, it uses background subtraction and optical flow to calculate the difference between consecutive frames and identify moving areas. The server then uses bounding boxes to surround moving objects.

[1460] Step 4:

[1461] The server tracks the detected moving object and analyzes its behavior. Using a Kaliman filter and local tracking algorithm, the server calculates the object's location, speed, and direction of movement. This allows the server to analyze the object's movement pattern and identify any unnatural behavior.

[1462] Step 5:

[1463] The server evaluates the likelihood of criminal activity based on the results of behavioral analysis. Specifically, it uses a pre-configured rule-based system to automatically detect suspicious behavior. For example, quickly retrieving an object in a specific area is detected as criminal activity.

[1464] Step 6:

[1465] If the server detects suspicious activity, it generates a real-time alarm. Specifically, it uses WebSocket to send a notification to the management terminal, prompting immediate action. The server notifies the management terminal with detailed data about the suspicious activity.

[1466] Step 7:

[1467] The server sends suspicious behavior data to the generation AI, which then formats the behavior data in JSON format and sends it to the generation AI's API endpoint, which then starts generating detailed evidence.

[1468] Step 8:

[1469] The generation AI generates detailed evidence information based on the received data. Specifically, it generates a detailed description of the behavior, the date and time, the location, and the identification information of related people and objects. The generated evidence information is returned to the server.

[1470] Step 9:

[1471] The server receives the generated evidence and converts it into the appropriate format. The evidence is formatted as an email and includes any relevant images or video links.

[1472] Step 10:

[1473] The server automatically sends evidence information to the police system. Specifically, it sends evidence information to a dedicated police email address using the SMTP protocol. This allows evidence information to be shared quickly and securely.

[1474] Example 1

[1475] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1476] Conventional surveillance systems require manual analysis of video data obtained from surveillance cameras, making it difficult for limited personnel to review large volumes of data. Furthermore, the accuracy of motion detection and behavior analysis is low, making it difficult to detect suspicious activity. This has led to issues with early detection of criminal activity and rapid response.

[1477] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1478] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data and detecting moving objects, means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior, means for transmitting data to a generation artificial intelligence to generate evidence information when criminal behavior is detected, means for organizing the generated evidence information and automatically transmitting it to the police, means for acquiring video data from a surveillance camera in real time using the RTSP protocol, means for applying a noise reduction filter to the acquired video data using the OpenCV library, means for detecting moving objects using background subtraction, means for analyzing the behavioral patterns of moving objects using a Kaliman filter or tracking algorithm, and means for notifying a management terminal of suspicious behavior in real time using WebSocket. This enables automatic analysis of large amounts of video data, highly accurate detection of suspicious behavior, and rapid detection and response to criminal behavior.

[1479] The "image acquisition device" is a device for acquiring image data such as a surveillance camera.

[1480] "Video data" refers to digital video information obtained from a video capture device such as a surveillance camera.

[1481] "Server" means a central system for analyzing video data received from video capture devices and detecting and responding to criminal activity.

[1482] "Preprocessing" is a process of improving the quality of acquired video data by performing processes such as noise removal and frame division.

[1483] A "frame unit" is a unit into which video data is divided into individual still images.

[1484] A "moving object" is a moving object or person that is detected by the difference between frames.

[1485] "Motion detection" is the process of analyzing pre-processed video data and calculating the differences between frames to find moving objects.

[1486] "Behavioral analysis" refers to analyzing the location information, movement speed, and movement direction of a detected moving object to understand its behavioral patterns.

[1487] "Assessing the possibility of criminal behavior" refers to applying pre-set rules to the analyzed behavioral data to determine whether or not there is a possibility of criminal behavior.

[1488] "Generative AI" is an AI system that automatically generates detailed evidence information based on data on criminal behavior.

[1489] "Evidence information" is data containing detailed information about criminal activity, including date, time, location, and activity details.

[1490] "Automatic transmission to police" refers to the process of converting the generated evidence information into an appropriate format and automatically transmitting it to the police system.

[1491] The "RTSP protocol" is a communication protocol for acquiring video streams in real time.

[1492] The "OpenCV library" is an open-source library for performing computer vision tasks.

[1493] A "noise reduction filter" is a tool used to remove unnecessary noise from video data and improve the quality of the data.

[1494] "Background subtraction" is a technique used for motion detection, which detects moving objects by comparing them with the background.

[1495] The "Kaliman filter" is an algorithm for tracking the location information of moving objects, and estimates and predicts their location.

[1496] A "tracking algorithm" is an algorithm for tracking the successive positions of a moving object.

[1497] "WebSocket" is a protocol for two-way communication between a server and a management terminal.

[1498] A "management terminal" is a terminal device for receiving notifications of detected criminal behavior, and is usually used by an administrator.

[1499] The overall system configuration and specific program processing for implementing this invention are described below. The system consists of a video capture device including a surveillance camera, a server that analyzes and generates and transmits evidence information, a generation AI, and a police system.

[1500] Video data acquisition and preprocessing

[1501] The server receives video data from surveillance cameras in real time using the RTSP protocol. For example, it receives a video stream from a surveillance camera in a store, divides it into frames, and applies a noise reduction filter to each frame using the OpenCV library. This improves the quality of the data and makes the subsequent analysis process smoother.

[1502] Motion detection and behavior analysis

[1503] The server analyzes the preprocessed video data and detects moving objects using background subtraction. If a moving object is detected, its location is identified using a bounding box and its location information is recorded. The server then analyzes the location, speed, and direction of the detected moving object using a Kaliman filter and tracking algorithm to understand its behavioral patterns.

[1504] Automated detection of criminal behavior

[1505] The server then applies pre-defined rules to assess the likelihood of criminal activity based on the analyzed behavioral data. For example, a rule could be applied that defines "taking an item from a shelf within five seconds" as theft, and if suspicious activity is detected, a real-time alarm is generated and a notification is sent to the management terminal via WebSocket.

[1506] Generating and sending evidence

[1507] The server sends the detected behavioral data in JSON format to the generation AI. The generation AI generates detailed evidence information based on the received data, creating content including date, time, location, and behavior details. The server then converts this evidence information into email format and automatically sends it to the police system using the SMTP protocol.

[1508] Specific examples

[1509] For example, the server acquires a 30fps video stream from a store's surveillance cameras using the RTSP protocol and removes noise using OpenCV's GaussianBlur filter. Next, it detects moving people using background subtraction and tracks their location using a Kaliman filter. If the server detects someone quickly removing an item in a specific area, it sends an alarm notification to the management terminal in real time via WebSocket. The server then sends the behavioral data to the generation AI, which creates detailed evidence information. The server then converts this evidence information into email format and sends it to the police system using the SMTP protocol.

[1510] Example prompts for generative AI models

[1511] For example, the following prompt could be entered into the generation AI: "Generate detailed evidence information based on the following data: Data: Date and time: October 10, 2023, Location: Store A, Action: A person quickly removed an item from a specific area."

[1512] Through the above process, this invention automatically analyzes large amounts of video data, detects suspicious activity with high accuracy, and enables rapid detection and response to criminal activity. Furthermore, by automatically transmitting the generated detailed evidence information to the police, it supports rapid legal response.

[1513] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1514] Step 1:

[1515] The server acquires video data from a surveillance camera. As input, it uses the RTSP stream URL of the surveillance camera. Specifically, the server connects to the surveillance camera using the RTSP protocol and receives the video stream in real time. As output, it obtains the stream data.

[1516] Step 2:

[1517] The server divides the acquired video data into frames. The input is real-time video stream data. Specifically, the server divides the video stream into 30 frames per second (30 fps). The output is individual frame data.

[1518] Step 3:

[1519] The server performs noise reduction on each frame. As input, it receives the segmented frame data. Specifically, it applies a Gaussian Blur filter using the OpenCV library to remove noise from the frame. As output, it obtains high-quality frame data.

[1520] Step 4:

[1521] The server analyzes the preprocessed frame data and detects moving objects. The input is noise-removed frame data. Specifically, it uses background subtraction to detect movement between frames and identifies the location of moving objects using bounding boxes. The output is the location information of the moving objects.

[1522] Step 5:

[1523] The server analyzes the behavior of the detected moving object. The input is the location information of the moving object. Specifically, it uses a Kaliman filter and tracking algorithm to analyze the object's location, speed, and direction of movement, and identifies its behavioral pattern. The output is the analyzed behavioral data.

[1524] Step 6:

[1525] The server evaluates the likelihood of criminal behavior based on the analyzed behavioral data. The input is the analyzed behavioral data. Specific actions are detected by applying pre-set rules. For example, "taking an item from a shelf within five seconds" is evaluated as theft. The output is a notification of suspicious behavior.

[1526] Step 7:

[1527] If the server detects suspicious behavior, it generates an alarm and notifies the management terminal. The input is a notification of the detected suspicious behavior. The specific operation is to send a real-time notification to the management terminal using the WebSocket protocol. The output is a notification displayed on the management terminal.

[1528] Step 8:

[1529] The server sends behavioral data to the generation AI to generate evidential information. The input is detailed data on suspicious behavior. Specifically, the behavioral data is sent to the generation AI in JSON format, and the AI ​​generates evidential information including date, time, location, and details of the behavior. The generated evidential information is obtained as output.

[1530] Step 9:

[1531] The server automatically sends the generated evidence information to the police. The input is the evidence information obtained from the generation AI. Specifically, the server converts the evidence information into email format, attaches links and related images, and sends it to the police system using the SMTP protocol. The output is the evidence information sent to the police system.

[1532] (Application example 1)

[1533] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1534] Automatic detection systems for criminal behavior using surveillance cameras are essential for responding quickly when a crime occurs. However, current systems not only need to analyze video data from surveillance cameras in real time, but also need the ability to quickly and accurately generate and transmit detailed evidence information when criminal behavior is detected. Real-time alert generation and notification are also required to enable police to respond immediately. To achieve this, an effective system is required that integrates motion detection, behavior analysis, the generation of evidence information using generative AI, and automatic transmission functions.

[1535] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1536] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data and detecting moving objects, means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior, means for transmitting data to a generating AI and generating evidential information when criminal behavior is detected, means for organizing the generated evidential information and automatically transmitting it to the police, and means for saving screenshots and transmitting images to the police by email when criminal behavior is detected. This makes it possible to realize a system that detects criminal behavior in real time, quickly and accurately generates and transmits evidence, and enables the police to respond immediately.

[1537] An "image capture device" is a device for capturing image data in real time, such as a surveillance camera or video camera.

[1538] "Video data" refers to digital information that is organized for each frame of a captured video.

[1539] "Preprocessing" refers to the process of processing the acquired video data, such as by removing noise and dividing the data into frames, in order to improve the quality.

[1540] "Motion detection" is the process of identifying moving objects in video using differences between frames.

[1541] "Behavioral analysis" is a method of analyzing the movements of moving objects based on their location, speed, direction, etc., to find specific behavioral patterns.

[1542] The "means for assessing the likelihood of criminal activity" is a process that includes a rule-based algorithm for assessing and detecting the likelihood of criminal activity based on the analyzed data.

[1543] "Generative artificial intelligence" is an AI system that generates detailed evidential information based on specified data.

[1544] "Evidential information" is information necessary to prove criminal activity, such as the date, time, location, and details of the action.

[1545] "Automatic transmission" is the process of quickly transmitting generated evidence information to the police using a pre-defined protocol.

[1546] "Real-time alert" is a warning system that immediately notifies you of any suspicious behavior when it is detected, prompting you to take action.

[1547] A "screenshot" is a technique for capturing video data at a specific point in time and saving it as a still image.

[1548] "Email transmission" is a communication method for transmitting information in the form of email via the Internet.

[1549] To implement this invention, the following specific system configuration and processing procedures are required. The entire system operates through the cooperation of three parties: a server, a terminal, and a user. Automatic detection of criminal behavior and generation and transmission of evidence information are carried out according to the following procedures.

[1550] System Configuration

[1551] 1. Video acquisition device: A device such as a surveillance camera installed in stores and public facilities that acquires video data in real time.

[1552] 2. Server: Plays the central role of receiving and analyzing the video data sent from the video capture device, and generating and sending evidence information.

[1553] 3. Generative AI model: An artificial intelligence system that generates detailed evidential information based on data sent from the server.

[1554] 4. Police system: A system for receiving and responding to the evidence information generated.

[1555] What the program does

[1556] Image acquisition and preprocessing

[1557] The server acquires video data in real time from a surveillance camera. A typical surveillance camera is used as the video acquisition device. The acquired video data is divided into frames and preprocessed using OpenCV filters, such as noise removal.

[1558] Motion detection and behavior analysis

[1559] The server analyzes the preprocessed video data and calculates the difference between frames to detect moving objects. Using a Kaliman filter and tracking algorithm, the server analyzes the location, speed, and direction of the detected moving object to identify its behavioral pattern. For example, the action of quickly removing an object within a certain area can be detected as "theft."

[1560] Automated detection of criminal behavior

[1561] The server uses rule-based decision-making to assess the likelihood of criminal activity based on the analyzed behavioral data. If suspicious activity is detected, it generates a real-time alarm and notifies the management terminal. It also has the ability to save screenshots and send images via email to the police if criminal activity is detected.

[1562] Generating and sending evidence

[1563] The server sends details of suspicious behavior to a generative AI model, which then generates evidence including the date, time, location, and details of the behavior. The server then organizes the evidence, converts it into an appropriate format, and automatically transmits it to the police system. A secure communication protocol, such as SMTP, is used for transmission.

[1564] Specific examples

[1565] 1. Example of image acquisition and pre-processing:

[1566] The server acquires video data from the surveillance cameras inside the store using the RTSP protocol, divides it into 30fps frames, and removes noise using an OpenCV filter.

[1567] 2. Examples of motion detection and behavior analysis:

[1568] The server uses background subtraction to detect customers moving around the store, and uses a Kaliman filter to analyze the customer's location and movement speed to detect any unnatural behavior.

[1569] 3. Examples of evidence generation and transmission to police:

[1570] The server sends the behavioral data to the generation AI, which then generates evidential information including the date, time, location, and details of the behavior. The generated evidential information is converted into an email and automatically sent to the police system.

[1571] Prompt Sentence Examples

[1572] "Write a script that grabs frames from an RTSP stream, detects suspicious activity, saves the image, and notifies the police."

[1573] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1574] Step 1:

[1575] The server acquires video data from the video acquisition device. Specifically, the server connects to the surveillance camera via the RTSP protocol and receives the video stream in real time. The input of this step is the video stream provided by the surveillance camera, and the output is the video data received by the server.

[1576] Step 2:

[1577] The server divides the acquired video data into frames and performs preprocessing. Specifically, the video data is divided into 30 fps frames and noise is removed using an OpenCV filter. The input to this step is the video data received in real time, and the output is the preprocessed frame data.

[1578] Step 3:

[1579] The server analyzes the preprocessed video data to detect moving objects. Specifically, it uses background subtraction to calculate the difference between frames and identify moving objects. The input to this step is the preprocessed frame data, and the output is data with identified moving objects.

[1580] Step 4:

[1581] The server analyzes the behavior of the detected moving object. Specifically, it analyzes the location information, movement speed, and movement direction of the moving object using a Kaliman filter or tracking algorithm. The input of this step is the data on the identified moving object, and the output is the analyzed behavior information of the moving object.

[1582] Step 5:

[1583] The server evaluates the likelihood of criminal behavior based on the analyzed behavioral data. Specifically, it uses rule-based judgment to identify suspicious behavior. The input to this step is the analyzed behavioral information of the moving object, and the output is data evaluating the likelihood of criminal behavior.

[1584] Step 6:

[1585] When criminal behavior is detected, the server sends the data to the generation AI to generate evidence. Specifically, the behavioral data is sent in JSON format to the generation AI model, which generates detailed evidence. The input to this step is the data determined to be criminal behavior, and the output is the generated evidence.

[1586] Step 7:

[1587] The server organizes the generated evidence information and automatically sends it to the police. Specifically, it converts the evidence information into an appropriate format and sends it to the police system using the SMTP protocol. The input to this step is the evidence information from the generative AI model, and the output is the evidence data sent to the police.

[1588] Step 8:

[1589] If the server detects a criminal activity, it saves a screenshot and sends the image to the police by email. Specifically, it sends a warning message along with the saved screenshot by email. The input of this step is the video data at the time when the suspicious activity was detected, and the output is the screenshot and warning message sent to the police.

[1590] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1591] This invention is a system that acquires video data from a video capture device, performs moving object detection and behavior analysis, and analyzes user emotions through an emotion engine, detects criminal behavior, and automatically transmits evidence of the behavior to the police. The system configuration, program processing flow, and specific examples are described below.

[1592] System Configuration

[1593] The system mainly consists of the following components:

[1594] 1. Video capture device - Surveillance cameras installed in stores, public facilities, etc.

[1595] 2. Server - Receives data from the video capture device, performs motion detection, behavior analysis, emotion analysis, and generates and transmits evidence information.

[1596] 3. Generative AI - Artificial intelligence that generates detailed evidence from data that detects criminal behavior.

[1597] 4. Emotion Engine - Analyzes the user's facial expressions and movements from video to recognize their emotional state.

[1598] 5. Police system - receives and responds to generated evidence information.

[1599] What the program does

[1600] Image acquisition and preprocessing

[1601] Video acquisition

[1602] The server obtains video data from the surveillance camera in real time, for example, by receiving the video stream using the RTSP protocol.

[1603] Pretreatment

[1604] The server performs preprocessing such as noise removal on the video data divided into frames, and improves the quality of the data using the OpenCV library.

[1605] Motion detection and behavior analysis

[1606] Motion Detection

[1607] The server analyzes the pre-processed video data and calculates the difference between consecutive frames to detect moving objects. It identifies and records moving objects using bounding boxes.

[1608] Behavioral analysis

[1609] The server analyzes the location information, movement speed, and movement direction of the moving object, and tracks the object and analyzes its behavior using a Kaliman filter and tracking algorithm.

[1610] Emotion analysis

[1611] emotion recognition

[1612] In addition to analyzing the behavior of moving objects, the server uses an emotion engine to analyze emotions from the user's facial expressions and movements. For example, it uses facial recognition technology to detect suspicious expressions or tension.

[1613] Emotional Data Evaluation

[1614] The server evaluates the likelihood of criminal behavior based on the recognized emotional data, and performs a comprehensive analysis of emotional changes and behavioral patterns according to a set algorithm.

[1615] Automated detection of criminal behavior

[1616] Rule-based Decision

[1617] The server uses a pre-configured rules-based system based on behavioral and emotional analysis to detect suspicious behavior, such as quickly picking up an object in a specific area or displaying a suspicious facial expression.

[1618] Alarm Generation

[1619] When the server detects suspicious activity, it generates a real-time alarm and sends a notification to the management terminal.

[1620] Generating evidence

[1621] Data transmission

[1622] The server sends suspicious behavior and emotion data to the generation AI, formats the data in JSON format, and sends it to the generation AI's API endpoint.

[1623] Evidence generation

[1624] The generation AI generates detailed evidence based on the received data, including a detailed description of the behavior, the date and time, the location, and the identity of the people and objects involved.

[1625] Automatic transmission to police

[1626] Data reduction and format conversion

[1627] The server receives the generated evidence and converts it into the appropriate format. The evidence is formatted as an email and includes any relevant images or video links.

[1628] send

[1629] The server automatically sends the evidence information to the police system. Specifically, it sends the evidence information to a dedicated email address for the police using the SMTP protocol.

[1630] Specific examples

[1631] Image acquisition and preprocessing

[1632] Example 1

[1633] The server receives video data from the surveillance cameras in the store via RTSP. The video data is split into 30fps frames and noise is removed using an OpenCV filter.

[1634] Motion detection and behavior analysis

[1635] Example 2

[1636] The server uses background subtraction to detect customers moving around the store, and a Kaliman filter is used to analyze their location and speed to detect unusual behavior.

[1637] Emotion analysis

[1638] Example 3

[1639] The server uses an emotion engine to analyze customer facial expressions in specific areas and detect nervous or suspicious expressions, using machine learning algorithms to recognize emotions in real time.

[1640] Automated detection of criminal behavior

[1641] Example 4

[1642] Based on the results of the behavioral and emotional analysis of the moving object, the server detects the behavior of quickly picking up an item in a specific area and showing a suspicious expression as "theft behavior."

[1643] Generating and sending evidence

[1644] Example 5

[1645] The server sends behavioral and emotional data to the AI, which then generates evidential information including date, time, location, and details of the behavior. The evidential information is then converted into an email and automatically sent to the police system.

[1646] This system will improve public safety by streamlining surveillance operations and enabling rapid crime response through analysis of motion and emotions.

[1647] The processing flow will be explained below.

[1648] Step 1:

[1649] The server connects to the surveillance camera and acquires real-time video data. Specifically, the server receives the video stream using the RTSP protocol. The server continues to acquire data from the camera at regular intervals.

[1650] Step 2:

[1651] The server divides the acquired video data into frames. Specifically, it divides the video data into 30 frames per second (fps) and converts each frame into an easy-to-handle format. Next, preprocessing is performed, such as noise removal and resolution unification. The server performs preprocessing using the OpenCV library.

[1652] Step 3:

[1653] The server analyzes the preprocessed video data to detect moving objects. Specifically, it uses background subtraction and optical flow to calculate the difference between consecutive frames and identify moving areas. The server then uses bounding boxes to surround moving objects.

[1654] Step 4:

[1655] The server tracks the detected moving object and analyzes its behavior. Using a Kaliman filter and local tracking algorithm, the server calculates the object's location, speed, and direction of movement. This allows the server to analyze the object's movement pattern and identify any unnatural behavior.

[1656] Step 5:

[1657] Based on the results of the motion analysis, the server uses an emotion engine to analyze the user's emotions from their facial expressions and movements. Specifically, it uses facial expression recognition technology to detect nervous or suspicious expressions, and generates emotion data based on this.

[1658] Step 6:

[1659] The server evaluates the possibility of criminal behavior based on emotion data and combines it with the results of motion analysis. For example, if a combination of a certain behavioral pattern and emotion is considered to be criminal behavior, the server evaluates the possibility.

[1660] Step 7:

[1661] Based on the results of behavioral and emotional analysis, the server automatically uses a rule-based system to determine the likelihood of criminal behavior. For example, if someone quickly picks up an object in a specific area and shows a suspicious expression, it will be determined to be "theft behavior."

[1662] Step 8:

[1663] If the server detects any criminal activity, it generates an alarm in real time on the management terminal, specifically by sending a notification using WebSocket to prompt immediate action.

[1664] Step 9:

[1665] The server sends suspicious behavior and emotion data to the AI ​​generator. The server formats the data in JSON format and sends it to the AI ​​generator's API endpoint. This process generates detailed evidence.

[1666] Step 10:

[1667] Based on the data received, the generation AI generates detailed evidence, including a detailed description of the behavior, the date and time, the location, and the identity of the people and objects involved.

[1668] Step 11:

[1669] The server receives the generated evidence, converts it into an appropriate format, and formats the evidence into an email with relevant images and video links attached.

[1670] Step 12:

[1671] The server automatically sends evidence information to the police system. Specifically, it sends evidence information to a dedicated email address for the police using the SMTP protocol. This allows evidence information to be shared quickly and securely.

[1672] Example 2

[1673] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1674] Conventional surveillance systems focus on detecting moving objects and analyzing behavior, but they do not analyze the user's emotional state, making it difficult to accurately detect criminal behavior. Furthermore, they lack the ability to quickly and accurately generate evidence of detected criminal behavior and automatically send it to the police. This can delay early response to criminal behavior and potentially reduce public safety.

[1675] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1676] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data and detecting a moving object, means for analyzing the behavior of the detected moving object and analyzing its behavioral pattern using its moving speed and direction, means for analyzing the emotional state of the user using an emotion engine based on the results of the behavioral analysis of the moving object and evaluating the possibility of criminal behavior, means for sending data to an artificial intelligence generated when criminal behavior is detected and generating evidential information, and means for organizing the generated evidential information and automatically transmitting it to the police. This enables accurate detection of criminal behavior and rapid response by comprehensively performing behavioral analysis and emotion analysis of the moving object.

[1677] The "image acquisition device" is a device for acquiring image data such as a surveillance camera.

[1678] "Video data" refers to data that expresses acquired video information in a digital format.

[1679] "Preprocessing" refers to performing processes such as noise removal and frame division to make video data easier to analyze.

[1680] A "moving object" refers to an object that changes position between successive video frames.

[1681] "Motion detection" refers to identifying moving objects in video data and recognizing their position and movement.

[1682] "Behavioral analysis" refers to analyzing the location information, movement speed, movement direction, etc. of detected moving objects to clarify their behavioral patterns.

[1683] An "emotion engine" refers to a system that analyzes the emotional state of a moving subject from its facial expressions and movements.

[1684] "User" refers to a person or other object that appears in the video data.

[1685] "Possible criminal behavior" refers to assessing whether a target's behavior constitutes a crime based on behavioral and emotional analysis.

[1686] "Generative AI" refers to AI that generates detailed evidentiary information using data evaluated as criminal behavior.

[1687] "Evidence information" refers to information that details criminal behavior and is generated based on analyzed behaviors and emotions.

[1688] "Police" refers to law enforcement agencies that investigate crimes to maintain public safety and order.

[1689] "Data transmission" refers to sending required data to other systems or devices via a network.

[1690] This invention is a system that acquires video data from a video capture device, performs motion detection and behavior analysis, and analyzes user emotions through an emotion engine, detects criminal behavior, and automatically transmits evidence to the police. This system is composed of multiple components, including a video capture device, a server, a generation AI, an emotion engine, and a police system.

[1691] System configuration and processing content

[1692] Image acquisition and preprocessing

[1693] Image acquisition device

[1694] The server acquires video data in real time from video acquisition devices such as surveillance cameras. Specifically, it uses RTSP (Real-Time Streaming Protocol) to receive video streams in the format "rtsp: / / username:password@cameraIP:554 / stream."

[1695] Pretreatment

[1696] The server divides the acquired video data into frames and performs preprocessing such as noise removal and color correction using the OpenCV library, thereby preparing the data for improved analysis accuracy.

[1697] Motion detection and behavior analysis

[1698] Motion Detection

[1699] The server detects motion by applying background subtraction to the pre-processed video data, which analyzes the differences between consecutive frames to identify moving objects.

[1700] Behavioral analysis

[1701] The server analyzes the behavioral patterns of the detected moving object using its location, speed, and direction. Specifically, it uses a Kaliman filter and tracking algorithm to track the object and perform detailed behavioral analysis.

[1702] Emotion analysis

[1703] emotion recognition

[1704] Based on the behavioral analysis results of the moving object, the server uses an emotion engine to analyze the user's emotions from their facial expressions and movements, and uses facial recognition technology to detect suspicious expressions and tension in real time.

[1705] Emotional Data Evaluation

[1706] The server evaluates the likelihood of criminal behavior based on the recognized emotional data, and performs a comprehensive analysis of emotional changes and behavioral patterns according to a specific algorithm.

[1707] Automated detection of criminal behavior

[1708] Rule-based Decision

[1709] The server uses a pre-configured rule-based system to detect suspicious behavior based on the results of behavioral and emotional analysis, such as quickly taking an object from a specific area or displaying a suspicious facial expression, as a "theft attempt."

[1710] Alarm Generation

[1711] If the server detects any suspicious activity, it generates an alarm in real time and sends a notification to the management terminal.

[1712] Evidence generation and transmission

[1713] Data transmission

[1714] The server sends suspicious behavior and emotion data to the generator AI, formatted in JSON and sent through an API endpoint.

[1715] Evidence generation

[1716] Based on the data it receives, the generation AI generates detailed evidence, including a detailed description of the behavior, the date and time, the location, and the identity of the people and objects involved.

[1717] Data reduction and format conversion

[1718] The server receives the generated evidence and converts it into the appropriate format, arranging it into an email format and including related images and video links.

[1719] send

[1720] The server automatically sends the organized evidence information to the police system, specifically to a dedicated email address for the police using the SMTP protocol.

[1721] Specific examples

[1722] Specific examples of image acquisition and preprocessing

[1723] The server connects to "rtsp: / / username:password@cameraIP:554 / stream", acquires video data at 30 frames per second, and performs noise reduction using an OpenCV filter.

[1724] Examples of motion detection and behavior analysis

[1725] The server uses background subtraction to detect customers moving around the store and analyzes their movement speed and location using a Kaliman filter.

[1726] Specific examples of sentiment analysis

[1727] The server uses an emotion engine to analyze customers' facial expressions in specific areas and detect nervous or suspicious expressions in real time.

[1728] Specific examples of automated detection of criminal behavior

[1729] Based on the results of behavioral and emotional analysis, the server quickly retrieves products from specific areas and detects any behavior showing suspicious facial expressions as "theft behavior."

[1730] Specific examples of evidence generation and transmission

[1731] The server sends behavioral and emotional data to the generating AI, which then organizes the generated evidence information in an appropriate format and automatically sends it to the police.

[1732] Prompt Sentence Examples

[1733] "Please explain in detail the processing steps of a system that uses video footage acquired from surveillance cameras to detect motion, analyze behavior and emotions, automatically detect criminal activity, and transmit evidence to the police."

[1734] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1735] Step 1:

[1736] The server acquires video data in real time from a video acquisition device. The input is streaming video from a surveillance camera, which the server receives using the RTSP protocol. Specifically, it connects to a URL in the format "rtsp: / / username:password@cameraIP:554 / stream" to acquire the video data. The output is raw video data.

[1737] Step 2:

[1738] The server divides the acquired video data into frames and performs preprocessing. The input is the video data acquired in step 1, and the server uses the OpenCV library to perform noise removal and color correction. Specifically, it removes noise using "cv2.GaussianBlur(image, (5, 5), 0)". The output is preprocessed, high-quality video data.

[1739] Step 3:

[1740] The server analyzes the preprocessed video data and detects moving objects. The input is the preprocessed video data obtained in step 2, and the server detects moving objects using background subtraction. Specifically, it calculates the difference between consecutive frames and marks moving objects with bounding boxes. The output is video data containing the position information of moving objects.

[1741] Step 4:

[1742] The server analyzes the behavior of the detected moving object. The input is the video data containing the location information of the moving object obtained in step 3, and the server analyzes the moving speed and direction of the moving object using a Kaliman filter or tracking algorithm. For example, a tracking algorithm is used to compare the moving object's current position with its past positions and analyze its behavioral patterns. The output is the data resulting from the behavioral analysis.

[1743] Step 5:

[1744] The server uses an emotion engine to analyze the user's emotional state based on the behavioral analysis results. The input is the behavioral analysis results obtained in step 4, and the server uses facial recognition technology to recognize emotions from facial expressions. Specifically, it uses a deep learning model to detect suspicious expressions and tension. The output is the analyzed emotional data.

[1745] Step 6:

[1746] The server comprehensively evaluates the emotional data and behavioral analysis data to determine the likelihood of criminal behavior. The input is the data obtained in steps 4 and 5, and the server determines the likelihood of criminal behavior based on set rules. As a specific example, if a person quickly picks up an item in a specific area and shows a suspicious expression, it will be determined to be "theft behavior." The output is an evaluation result indicating the likelihood of criminal behavior.

[1747] Step 7:

[1748] If criminal behavior is detected, the server sends data to the generation AI. The input is the evaluation result obtained in step 6, and the server sends the data in JSON format to the API endpoint of the generation AI. The output is the data sent to the generation AI.

[1749] Step 8:

[1750] The generation AI generates detailed evidential information based on the received data. The input is the data sent in step 7, and the generation AI generates evidential information including a detailed description of the action, date and time, location, and identification information of people and objects involved. The output is the generated evidential information.

[1751] Step 9:

[1752] The server organizes the generated evidence and converts it into the appropriate format. The input is the evidence obtained in step 8, and the server formats the evidence into an email and includes related image and video links. The output is the evidence converted into email format.

[1753] Step 10:

[1754] The server automatically sends the organized evidence information to the police. The input is the evidence information in email format obtained in step 9, which the server sends to the police's dedicated email address using the SMTP protocol. The output is the evidence information sent to the police system.

[1755] (Application example 2)

[1756] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1757] Conventional security systems require manual review of surveillance camera footage, making it difficult to detect criminal behavior in real time and respond quickly. Furthermore, simply reviewing surveillance footage makes it difficult to grasp the details of a criminal's behavior or the victim's emotional state, resulting in low crime detection accuracy. Furthermore, reporting to the police must be done manually, making it difficult to respond quickly.

[1758] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1759] In this invention, the server includes means for acquiring video data from a video acquisition device, means for dividing the acquired video data into frames and performing preprocessing, means for analyzing the preprocessed video data to detect moving objects, means for analyzing the behavior of the detected moving objects and evaluating the possibility of criminal behavior, means for sending a real-time alert to a smart device when criminal behavior is detected, means for sending data to a generating artificial intelligence when criminal behavior is detected to generate evidence information, and means for organizing the generated evidence information and automatically sending it to the police. This enables the detection of criminal behavior and rapid reporting, making it possible to improve the efficiency of surveillance work and public safety.

[1760] An "image capture device" is a device such as a surveillance camera that captures image data and transfers it to a processing device such as a server.

[1761] "Splitting into frames" refers to the process of splitting continuous video data into still images (frames) at regular intervals.

[1762] "Preprocessing" is the process of performing initial processing such as noise removal and image quality adjustment on acquired video data to make it easier to analyze.

[1763] "Motion detection" refers to the process of identifying moving objects in video data and identifying their position and range.

[1764] "Behavior analysis" is a process of analyzing the position information, movement speed, and movement direction of a moving object and extracting a specific behavior pattern.

[1765] "Evaluating the possibility of criminal behavior" refers to the process of analyzing the behavior of a detected moving object and determining whether or not that behavior constitutes a crime.

[1766] A "smart device" is a portable information terminal with advanced processing capabilities, such as a smartphone, tablet, or smart glasses.

[1767] "Real-time alerts" are notifications that are generated immediately and send a warning when a specific condition is detected.

[1768] "Generative AI" is an AI technology used to generate detailed evidential information based on acquired data.

[1769] "Evidence information" is information that includes a detailed description of the detected criminal activity, as well as the people involved, dates, times, and locations.

[1770] "Automatic transmission to police" is a process in which the generated evidence information is formatted into a specific format and automatically transmitted to the police system.

[1771] This invention provides a system that acquires video data from a video capture device, performs real-time motion detection and behavior analysis, captures signs of criminal behavior, sends alerts to smart devices, generates evidence information using artificial intelligence, and automatically notifies the police.

[1772] System Configuration

[1773] 1. Image acquisition device

[1774] In this system, surveillance cameras installed in stores and public facilities function as video capture devices, and the video data is sent to a server in real time.

[1775] 2. Server

[1776] The server performs the following steps:

[1777] Video Acquisition:

[1778] The server acquires video data in real time from the video acquisition device, and the video data is streamed using the RTSP protocol.

[1779] Pretreatment:

[1780] The server divides the acquired video data into frames and performs preprocessing such as noise removal using OpenCV.

[1781] Motion detection:

[1782] The preprocessed video data is analyzed and moving objects are detected using background subtraction and a Kaliman filter.

[1783] Behavior analysis:

[1784] The location, speed, and direction of the detected moving objects are analyzed to evaluate their behavioral patterns.Face detection is also performed using the dlib library to analyze their emotional states.

[1785] Criminal Behavior Assessment:

[1786] Based on the results of behavioral and sentiment analysis, a pre-defined rules-based algorithm is used to assess the likelihood of criminal behavior.

[1787] 3. Smart Devices

[1788] If criminal activity is detected, the server sends the information in real time to a smart device (smartphone, smart glasses, etc.) and notifies the user with an alert.

[1789] 4. Generative Artificial Intelligence

[1790] The server sends the behavioral data and emotional data of the moving object to the AI ​​generator, which then generates detailed evidence information, including the date, time, location, details of the behavior, and emotional data.

[1791] 5. Police System

[1792] The generated evidence information is automatically sent to the police. The server uses the SMTP protocol to send the evidence information to the police's dedicated email address.

[1793] Specific examples

[1794] Image acquisition and preprocessing

[1795] For example, video data is acquired from a surveillance camera installed in a store using the RTSP protocol, divided into 30 fps frames, and noise is removed through an OpenCV filter.

[1796] Motion detection and behavior analysis

[1797] It uses background subtraction to detect customers moving around the store, and a Kaliman filter to analyze their location and speed. It also uses the dlib library for face detection and emotion analysis.

[1798] Automatic detection and alerting of criminal activity

[1799] The system detects in real time any suspicious behavior such as quickly picking up an item in a specific area as "theft behavior" and sends an alert to a smart device.

[1800] Generate evidence and automatically send it to the police

[1801] The server sends behavioral and emotional data to the generating AI, which then converts the generated evidence information into an email and automatically sends it to the police system.

[1802] Prompt Sentence Examples

[1803] "Write a program that performs speed anomaly detection using bounding box averaging in a cluster setting. This program uses OpenCV to analyze video data acquired from a surveillance camera and detect objects moving at speeds above a certain threshold."

[1804] In this way, it is possible to improve public safety by making surveillance more efficient and enabling a quicker response to crime.

[1805] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1806] Step 1:

[1807] The server acquires video data in real time from a video acquisition device. The input is a video stream sent using the RTSP protocol, and the output is video data divided into frames. Specifically, the server receives the video captured by the surveillance camera as is and divides the video into frames.

[1808] Step 2:

[1809] The server performs preprocessing on the acquired video data. The input is video divided into frames, and the output is preprocessed frame data that has undergone noise removal and image quality adjustment. Specifically, OpenCV is used to remove noise from the video, and brightness and contrast are adjusted as necessary.

[1810] Step 3:

[1811] The server analyzes the preprocessed video data and performs moving object detection. The input is the preprocessed frame data, and the output is the position information and bounding box of the detected moving object. Specifically, moving objects are extracted using background subtraction, and their position information is obtained using a Kaliman filter.

[1812] Step 4:

[1813] The server analyzes the behavior of the detected moving object. The input is the object's location information, and the output is the object's behavior pattern, movement speed, and movement direction data. Specifically, an analysis algorithm is used to extract the object's movement pattern and detect abnormalities in its movement.

[1814] Step 5:

[1815] The server uses the dlib library to recognize the faces of detected moving objects and analyze their emotional states. The input is frame data of the moving objects, and the output is emotion-analyzed information. Specifically, it detects faces and infers emotions from their facial expressions.

[1816] Step 6:

[1817] The server evaluates the likelihood of criminal behavior based on the results of behavioral and emotional analysis. The input is behavioral patterns and emotional data, and the output is the evaluation result of criminal behavior. Specifically, it uses a pre-configured rule-based model to determine whether the detected behavior and emotion match the behavioral criteria.

[1818] Step 7:

[1819] If criminal behavior is detected, the server sends a real-time alert to the smart device. The input is the criminal behavior evaluation result, and the output is an alert notification to the smart device. Specifically, the detected information is sent to a smartphone or smart glasses via a notification application.

[1820] Step 8:

[1821] When criminal behavior is detected, the server sends data to the generation AI to generate evidence. The input is behavioral data and emotional data, and the output is the generated evidence. Specifically, the generation AI model uses prompt sentences to generate detailed evidence.

[1822] Step 9:

[1823] The server organizes the generated evidence information and automatically sends it to the police. The input is the generated evidence information, and the output is a notification of completion of transmission to the police system. Specifically, the evidence information is formatted appropriately and sent to a dedicated email address for the police using the SMTP protocol.

[1824] Prompt Sentence Examples

[1825] "Write a program that performs speed anomaly detection using bounding box averaging in a cluster setting. This program uses OpenCV to analyze video data acquired from a surveillance camera and detect objects moving at speeds above a certain threshold."

[1826] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1827] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1828] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1829] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1830] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1831] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1832] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1833] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1834] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1835] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1836] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1837] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1838] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1839] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1840] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1841] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. An...

Claims

1. means for acquiring video data from a video acquisition device; A means for dividing the acquired video data into frames and performing preprocessing; means for analyzing the preprocessed video data and detecting moving objects; A means for analyzing the behavior of the detected moving object and assessing the possibility of criminal behavior; A means for transmitting data to a generating artificial intelligence when a criminal behavior is detected, and causing the generating artificial intelligence to generate evidence information; A means of organizing the generated evidence information and automatically sending it to the police; A system including:

2. 2. The system according to claim 1, wherein the means for analyzing the behavior of the moving object analyzes the behavior pattern using position information, moving speed, and moving direction of the moving object.

3. 2. The system of claim 1, wherein the means for generating evidence information generates information including details of date, time, location, and action.

Images