System

Abstract

A system is provided.SOLUTION: A system includes a means for acquiring video data in a store, a means for analyzing the acquired video data and specifying a face or action, a means for collating the analyzed face or action with a database, a means for generating and notifying an alert on the basis of a collation result, and a means for displaying alert information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, many commercial facilities and stores have installed security cameras to prevent shoplifting and other fraudulent activities. However, constantly monitoring security camera footage is difficult, and there is a problem of a lack of human resources, especially in large stores that cover a wide area. In addition, there is no system that can immediately detect suspicious behavior and notify store staff, making it difficult to respond quickly. There is a need to solve these problems and provide an efficient surveillance system to prevent shoplifting. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for acquiring video data from within a store, a means for analyzing the acquired video data and identifying faces and behaviors, a means for comparing the analyzed faces and behaviors with a database, a means for generating and notifying alerts based on the comparison results, and a means for displaying the alert information. This enables an automated shoplifting monitoring system that covers a wide area of ​​the store, and is capable of instantly detecting suspicious behavior or suspicious individuals and notifying store staff. Furthermore, the alert information can be displayed in a chat format, and advance photos of suspicious individuals can be captured and registered in a database, enabling efficient and rapid response.

[0006] "In-store video data" refers to video information captured by security cameras installed inside the store.

[0007] The "acquiring means" refers to a device or program that has the function of acquiring video data from a camera and transmitting it to a subsequent system for necessary processing.

[0008] "Means for analysis" refers to algorithms or devices that receive video data and perform processing such as facial recognition and behavioral analysis.

[0009] "Means for identification" refers to a device or program that has the function of identifying individual people and their actions from the analyzed video data and extracting the necessary information.

[0010] The "matching means" refers to a device or program that has the function of comparing the analyzed data on people and their behavior with a pre-registered database and determining whether or not there is a match.

[0011] The "means for generating an alert" refers to a device or program that has the function of generating a warning or notification when suspicious behavior or a suspicious person is detected based on the matching results.

[0012] The "notification means" refers to a device or program that has the function of quickly transmitting the generated alert to a store clerk's terminal or the like.

[0013] "Alert information" is a warning related to suspicious behavior or a person requiring attention, and specifically includes information such as a facial image, clothing, location information, and details of the behavior.

[0014] The "display means" refers to a device or program that visually displays the received alert information.

[0015] "Chat format" refers to a format that allows text messages to be sent and received in an interactive format, and is a format that conveys alert information in an easy-to-understand manner.

[0016] "Preliminary photographs of suspicious individuals" refer to facial image data of individuals who are suspected of shoplifting or other suspicious behavior in the past.

[0017] A "database" is a system that stores and manages information such as facial images and behavioral patterns required for analysis and matching. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention relates to a surveillance system aimed at preventing shoplifting and detecting suspicious behavior in a store. The system analyzes video data obtained from security cameras in real time to detect suspicious behavior or suspicious individuals. An alert is then sent to the store clerk's terminal, enabling a prompt response.

[0040] System configuration

[0041] 1. Video data acquisition method

[0042] The terminal acquires video data in real time from multiple security cameras installed within the store.

[0043] The terminal compresses the acquired video data and transmits it to the server.

[0044] 2. Video data analysis methods

[0045] The server analyzes the received video data and applies facial recognition and behavioral analysis algorithms.

[0046] The server performs facial recognition and extracts the detected facial images for each frame.

[0047] The server uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements.

[0048] 3. Database matching method

[0049] The server matches the newly detected face data with the existing database.

[0050] The server compares the analyzed behavioral data with past behavioral data to identify suspicious behavioral patterns.

[0051] 4. Alert Generation and Notification Methods

[0052] If the server identifies a person of interest or suspicious activity, it generates an alert.

[0053] The alert information includes a person's facial image, clothing, location information, and details of their behavior.

[0054] The server transmits the generated alert information to the store clerk's terminal and notifies the store clerk.

[0055] 5. Displaying alert information

[0056] The store clerk's device receives the alert and displays the alert information in chat format.

[0057] The store clerk's terminal screen displays the person's clothing, location, and behavior details in real time.

[0058] Program processing

[0059] 1. Device operation

[0060] The device continuously acquires video data from the store's security cameras and transmits it to the server.

[0061] The device acquires images from multiple cameras simultaneously, efficiently collecting data.

[0062] 2. Server Data Analysis

[0063] The server performs facial recognition on the received video data and applies a highly accurate facial recognition algorithm.

[0064] The server analyzes the behavior of people in the video and runs algorithms to detect abnormal behavior.

[0065] 3. Database Matching

[0066] The server compares the newly detected facial data with the database and identifies facial data with high matching scores.

[0067] Based on the results of behavioral analysis, the server compares suspicious behavioral patterns with past data to identify potentially risky behavior.

[0068] 4. Alert generation and notification

[0069] If the server detects an abnormality, it will immediately generate an alert and create a notification message.

[0070] The alert message includes details of the suspicious person's facial image, clothing, location, and behavior.

[0071] 5. Assistance via store clerk's terminal

[0072] The user (store clerk) receives the alert and checks the information displayed in chat format.

[0073] The user goes to a specific location in the store and takes appropriate action against the detected suspicious person.

[0074] Specific examples

[0075] For example, imagine a store with security cameras monitoring the store 24 / 7. Devices continuously capture data from these cameras and send it to a server. The server uses facial recognition technology to identify people in the footage and check whether they are on a blacklist in a database. At the same time, the server applies behavioral analysis algorithms to detect if a person repeatedly behaves suspiciously in front of a particular product.

[0076] In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an abnormally long time." This alert is sent to the store clerk's device, which displays detailed information in chat format. Based on this information, the store clerk rushes to the scene, confirms the possibility of shoplifting, and takes appropriate action.

[0077] This system makes it possible to efficiently and quickly detect suspicious activity over a wide area within a store and immediately notify store staff, significantly strengthening shoplifting prevention and improving store security.

[0078] The processing flow will be explained below.

[0079] Step 1: Acquire video data

[0080] The terminal acquires video data in real time from security cameras installed inside the store.

[0081] The video data acquired by the terminal is compressed and sent to a server via the Internet.

[0082] Step 2: Receiving video data

[0083] The server receives the video data transmitted from the terminal.

[0084] The server temporarily stores the received video data.

[0085] Step 3: Performing facial recognition

[0086] The server applies a facial recognition algorithm to the received video data.

[0087] The server cuts out the detected face for each frame and generates face image data.

[0088] Step 4: Performing behavioral analysis

[0089] The server applies an algorithm to track the movements of each person in the video data.

[0090] The server detects certain behavioral patterns (e.g., repeatedly returning to the same location).

[0091] Step 5: Check against the database

[0092] The server matches the newly detected face data with the existing database.

[0093] The server checks whether there is a person in the database that matches the facial data.

[0094] Step 6: Evaluate the match results

[0095] Based on the matched results, the server evaluates whether there is a person of interest or suspicious activity.

[0096] If the server identifies suspicious activity, it records the details of that activity.

[0097] Step 7: Generate an alert

[0098] If the server identifies a person of interest or suspicious activity, it generates an alert.

[0099] The server includes details of the face image, clothing, location, and behavior in the alert information.

[0100] Step 8: Sending an alert

[0101] The server transmits the generated alert information to the store clerk's terminal.

[0102] The alert sent from the server is set to be delivered to multiple store clerk terminals simultaneously.

[0103] Step 9: Viewing Alerts

[0104] The alert information received by the store clerk's terminal is displayed in chat format.

[0105] The store clerk's terminal screen displays the person's clothing, location, and details of their behavior in real time.

[0106] Step 10: Staff response

[0107] The user (store clerk) checks the alert and refers to the displayed detailed information.

[0108] The user takes appropriate action and shares the information with other store staff as needed.

[0109] By using the above steps, the monitoring system of the present invention can efficiently and quickly detect suspicious behavior or suspicious individuals, and immediately issue an alert to prompt a response.

[0110] Example 1

[0111] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0112] From a security perspective, it is important to efficiently and quickly detect shoplifting and other suspicious behavior in stores and respond appropriately. However, current systems lack sufficient accuracy in analyzing video data and the speed of notification, resulting in the overlooking of suspicious behavior and delayed response. Other issues include the accuracy of the process of comparing suspect individuals and behavioral patterns with a database, and the efficiency of communicating information to store staff.

[0113] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0114] In this invention, the server includes means for acquiring video data of the monitored area, means for compressing the acquired video data and sending it to the server, means for analyzing the received video data and performing facial recognition and behavioral analysis, means for comparing the analyzed facial and behavioral data with a database, means for generating and notifying an alert based on the comparison results, and means for displaying the alert information. This makes it possible to detect suspicious behavior over a wide area within the store with high accuracy and immediately notify store staff.

[0115] "Surveillance Area" means a physical space that is subject to surveillance for a particular security purpose.

[0116] "Video data" refers to video image information collected by imaging equipment such as cameras installed in a monitored area.

[0117] "Compression" is the process of changing the format of data using a specific algorithm to reduce the volume of the data.

[0118] A "server" is a computer system used to provide a particular service and process data over a network.

[0119] "Facial recognition" is a technology that detects people's faces from video data and identifies individual faces using specific algorithms.

[0120] "Behavioral analysis" is a technology that analyzes the movements of people in video data and identifies specific behaviors and patterns.

[0121] A "database" is a system for storing, managing, and searching data in an organized manner.

[0122] "Matching" is the process of comparing the analyzed data with information from existing databases.

[0123] An "alert" is a warning message that is generated when a particular condition is met.

[0124] "Notification" is the act of sending generated alert information to a designated recipient.

[0125] "Means" refers to a method or device used to achieve a particular purpose.

[0126] "Display" is the act of visually showing information.

[0127] The present invention relates to a surveillance system aimed at preventing shoplifting and detecting suspicious behavior in stores. Specifically, the system analyzes video data obtained from security cameras in real time to detect suspicious behavior and suspicious individuals. Furthermore, based on the results, the system sends alerts to store clerk terminals to prompt prompt action.

[0128] Hardware and Software Configuration

[0129] Acquiring video data of the monitored area

[0130] The device acquires video data in real time from multiple security cameras (e.g., IP cameras) installed in the store. The device temporarily stores the video from the cameras in a buffer for easy access.

[0131] Video data compression and transmission

[0132] The video data acquired by the device is compressed and sent to the server using an efficient video compression format such as H.264. The compressed data is sent via the network to the specified server address.

[0133] Video data analysis

[0134] The server starts analyzing the received video data. It applies a high-precision facial recognition algorithm (e.g., OpenCV or Dlib) to detect faces in the video. The detected faces are extracted from each frame and saved as individual image data. In parallel, the server uses a behavior analysis algorithm (e.g., an RNN- or LSTM-based model) to analyze the behavior of people in the video.

[0135] Database Matching

[0136] The server compares the newly detected face data with an existing database (e.g., an SQL database) by calculating the distance between vectors of facial features. Similarly, behavioral data is compared with past behavioral patterns.

[0137] Alert generation and notification

[0138] If the server identifies a suspicious person or suspicious behavior, it generates an alert, which includes details of the detected face, clothing, location, and behavior. The alert message is generated in real time and pushed to the store clerk's device.

[0139] Viewing Alerts

[0140] When the device receives an alert, the alert information is displayed in chat format, allowing store staff to check the notification in real time on the device screen. The alert information includes details of the person's clothing, location, and behavior.

[0141] Staff response

[0142] The user (store clerk) receives the alert and checks the displayed information. The clerk then promptly takes action based on the alert. For example, they go to the specific location where the suspicious behavior was reported and take appropriate action against the suspicious person.

[0143] Specific examples

[0144] For example, suppose a store is monitored 24 hours a day by security cameras. A device acquires video data from multiple cameras in the store and sends it to a server. The server analyzes the received data and performs highly accurate facial recognition and behavioral analysis. This makes it possible to check whether a specific person is included in the blacklist of individuals in the past database.

[0145] At the same time, the server applies a behavioral analysis algorithm to detect suspicious behavior in front of a specific product. Based on this, the server generates an alert such as "A man wearing a red jacket is lingering in front of a specific product for an unusually long time." This alert is sent to the store clerk's device, and detailed information is displayed in chat format.

[0146] Using this information, store staff can rush to the scene, confirm the possibility of shoplifting, and take appropriate action. This system will enable the efficient and rapid detection of suspicious activity across a wide area of ​​the store, significantly strengthening the prevention of shoplifting.

[0147] Prompt Sentence Examples

[0148] 1. A store clerk can detect someone spending an abnormally long time at a particular shelf.

[0149] 2. The server detects suspicious behavior and sends an alert to the store clerk's terminal.

[0150] 3. The store staff checks the alert and takes appropriate action.

[0151] In this way, the present invention provides a system that efficiently and quickly strengthens security within a store and contributes to preventing fraudulent activities such as shoplifting.

[0152] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0153] Step 1: Acquire video data

[0154] The device acquires video data in real time from multiple security cameras installed in the store. Specifically, the video captured by the cameras is temporarily stored in a buffer, enabling continuous data collection.

[0155] Input: Real-time video data from security cameras.

[0156] Output: Uncompressed video data stored in a buffer.

[0157] Step 2: Compress and send the data

[0158] The video data acquired by the device is compressed and sent to the server using an efficient video compression format such as H.264, and the data is sent to the server via the network.

[0159] Input: Uncompressed video data.

[0160] Output: Compressed video data.

[0161] Step 3: Analyzing the video data

[0162] The server analyzes the received compressed video data. First, a face recognition algorithm (e.g., OpenCV or Dlib) is applied to detect faces in the video. The detected faces are extracted frame by frame and saved as individual image data. At the same time, a behavior analysis algorithm (e.g., an RNN or LSTM-based model) is applied to analyze the person's behavior.

[0163] Input: Compressed video data.

[0164] Output: Detected face images and behavior analysis results.

[0165] Step 4: Check against the database

[0166] The server matches newly detected facial data with an existing database (e.g., an SQL database), calculates the distance between facial feature vectors, and identifies potential matches of suspicious individuals. Behavioral data is similarly compared with past behavioral patterns.

[0167] Input: Detected face images and behavior analysis results.

[0168] Output: Matching person of interest information and suspicious behavior identification.

[0169] Step 5: Generate an alert

[0170] If the server identifies a suspicious person or behavior, it generates an alert, combining detected facial images, clothing, location information, and behavior details to create an alert message.

[0171] Input: Matching person of interest information and suspicious activity identification results.

[0172] Output: The alert message.

[0173] Step 6: Notification and display of alerts

[0174] The server sends the generated alert information to the store clerk's terminal, which displays the alert information in a pop-up or chat window format so that the store clerk can check it immediately.

[0175] Input: Alert message.

[0176] Output: Alert information displayed on the terminal.

[0177] Step 7: Staff response

[0178] The user (store clerk) receives the alert and checks the displayed information. The clerk follows the instructions in the alert to go to the specific location and deal with the suspicious person.

[0179] Input: The alert information displayed on the terminal.

[0180] Output: On-site response actions.

[0181] (Application example 1)

[0182] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0183] While conventional systems have the ability to analyze security camera footage to detect suspicious behavior and suspicious individuals, they have limited means of providing information to store staff so that they can respond quickly.In addition, there is a lack of ways for store staff patrolling the store to identify suspicious behavior in real time, making it difficult to respond quickly.

[0184] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0185] In this invention, the server includes means for acquiring video data of the store, means for analyzing the acquired video data and identifying faces and behaviors, means for comparing the analyzed faces and behaviors with a database on the server, means for generating and notifying an alert based on the comparison results, and means for displaying the alert information on the smart glasses. This enables store staff wearing the smart glasses to identify suspicious behavior in real time while patrolling and respond quickly.

[0186] "In-store video data" refers to images and video information obtained in real time from security cameras installed within a store.

[0187] The "acquisition means" refers to hardware and software components for receiving video data output from a security camera and storing or transmitting it.

[0188] "Means for analyzing and identifying faces and behaviors" refers to facial recognition algorithms and behavior analysis algorithms for identifying people's faces and analyzing their behaviors based on the captured video data.

[0189] The "database on the server" is a data management system used to store analyzed facial and behavioral data and compare it with past data.

[0190] The "matching means" is an algorithm that compares real-time facial and behavioral data with existing data in a database on the server to confirm a match.

[0191] "Means for generating and notifying alerts" refers to a system that creates an alert when suspicious behavior or a suspicious individual is detected and immediately notifies relevant parties of that information.

[0192] The "means for displaying alert information on smart glasses" refers to communication and display technology for displaying the generated alert information on the display of smart glasses worn by the store clerk.

[0193] This invention is a surveillance system aimed at preventing shoplifting in a store and detecting suspicious individuals at an early stage. This system acquires video data from multiple security cameras in real time, analyzes the data, detects suspicious behavior and individuals of interest, and notifies store staff.

[0194] Hardware configuration:

[0195] Security cameras: These are installed at strategic locations within the store and are used to capture footage in real time.

[0196] Server: A computer with powerful computing power for receiving and analyzing video data.

[0197] Smart glasses: worn by store associates and used to receive and display alert information in real time.

[0198] Software configuration:

[0199] OpenCV: A library for acquiring and processing security camera footage in real time.

[0200] Keras: A deep learning framework used to implement face recognition and behavior analysis algorithms.

[0201] Flask: A lightweight web framework for receiving and analyzing video data on the server side.

[0202] Requests: An HTTP library for data communication between smart glasses and a server.

[0203] Overview of program processing:

[0204] The device continuously captures video data from security cameras in the store in real time and sends it to a server. The server then applies facial recognition and behavioral analysis algorithms to the received video data to analyze the behavioral patterns of specific individuals. The analysis results are compared with existing data in a database, and an alert is generated if a match is found or if abnormal behavior is detected. The generated alert is then sent to the smart glasses, which notify the store staff in real time.

[0205] Examples:

[0206] For example, consider a store where security cameras are monitoring the shop 24 hours a day. The server receives this video data in real time and uses a facial recognition algorithm to identify individuals. It then checks whether the identified individuals are on a blacklist in the database. At the same time, the server applies a behavioral analysis algorithm to detect that an individual is repeatedly lingering in front of a particular product for an abnormally long time. In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an abnormally long time." This alert is sent to smart glasses, and the information is displayed in chat format on the smart glasses worn by the store clerk. Based on this information, the store clerk can head to the scene, confirm the possibility of shoplifting, and take appropriate action.

[0207] Example prompt sentence:

[0208] "The system analyzes security camera footage within stores in real time and generates an alert if it determines something is suspicious. The alert includes details of the face, clothing, location, and behavior. This alert is displayed on the smart glasses, enabling a prompt response."

[0209] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0210] Step 1:

[0211] Device operation

[0212] The device acquires video data from security cameras inside the store in real time.

[0213] Input: Video data from a security camera.

[0214] Data processing: The terminal compresses the acquired video data and converts it into a format that can be sent to the server (for example, JPEG format).

[0215] Output: Compressed video data.

[0216] Specific operation: The device continuously acquires data from multiple security cameras and transmits it to the server in real time.

[0217] Step 2:

[0218] Server data reception

[0219] A server receives the compressed video data.

[0220] Input: Compressed video data sent from the device.

[0221] Data processing: The server decompresses the compressed video data and converts it into a format that can be used as the original video data.

[0222] Output: Decompressed video data.

[0223] Specific operation: The server sequentially decompresses the received video data and prepares it for subsequent analysis processing.

[0224] Step 3:

[0225] Facial Recognition and Behavioral Analysis

[0226] The server analyzes the decompressed video data and applies facial recognition and behavioral analysis algorithms.

[0227] Input: Decompressed video data.

[0228] Data computation: The server uses a facial recognition algorithm (e.g., a Keras model) to identify faces in the video and applies a behavior analysis algorithm to analyze people's behavior.

[0229] Output: Analyzed face and behavioral data.

[0230] Specific operation: The server extracts the person's face from each video frame and analyzes their behavioral patterns.

[0231] Step 4:

[0232] Database collation

[0233] The server compares the analyzed facial and behavioral data with a database on the server.

[0234] Input: Parsed face and behavioral data.

[0235] Data calculation: The server compares the data with data in an existing database to identify facial data and suspicious behavior patterns with high matching scores.

[0236] Output: Matching results (face data and behavioral patterns with high matching scores).

[0237] Specific operation: The server uses a matching algorithm to match the blacklist in the database and identify matching data.

[0238] Step 5:

[0239] Generate alerts

[0240] The server generates an alert based on the match result.

[0241] Input: Matching results (face data and behavioral patterns with high matching scores).

[0242] Data processing: The server generates alert information (a person's face image, clothing, location information, and details of their behavior).

[0243] Output: Alert information.

[0244] What it does: The server immediately creates an alert when a suspicious person or activity is detected.

[0245] Step 6:

[0246] Sending and Viewing Alerts

[0247] The server transmits the generated alert information to the smart glasses for display.

[0248] Input: Alert information.

[0249] Data processing: The server converts the alert information into a format suitable for the smart glasses and sends it.

[0250] Output: Alert information displayed on smart glasses.

[0251] Specific operation: The server composes a notification message and sends it to the smart glasses, which receive the notification and the store clerk confirms it.

[0252] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0253] This invention relates to a surveillance system aimed at preventing shoplifting and detecting suspicious behavior in stores. This system analyzes video data obtained from security cameras in real time, and performs facial recognition and behavior analysis. In addition, by combining it with an emotion engine that recognizes user emotions, it enables more accurate detection of suspicious behavior and quicker response.

[0254] System configuration

[0255] 1. Video data acquisition method

[0256] The terminal acquires video data in real time from security cameras installed inside the store.

[0257] The video data acquired by the terminal is compressed and sent to a server via the Internet.

[0258] 2. Video data analysis methods

[0259] The server analyzes the received video data and applies facial recognition and behavioral analysis algorithms.

[0260] The server performs facial recognition and extracts the detected facial images for each frame.

[0261] The server uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements.

[0262] 3. Emotion Engine

[0263] The emotion engine built into the server identifies the user's emotion based on the facial image detected during face recognition.

[0264] The emotion engine generates user emotion data (e.g., impatience, anger, anxiety) and reflects it in the behavioral analysis results.

[0265] 4. Database matching method

[0266] The server matches the newly detected facial and emotion data with the existing database.

[0267] The server checks whether there is a person in the database that matches the facial data.

[0268] 5. Evaluation of matching results

[0269] Based on the matched results, the server evaluates whether there is a person of interest or suspicious activity.

[0270] If the server identifies suspicious behavior, it records details of the behavior and emotions.

[0271] 6. Alert Generation and Notification Methods

[0272] If the server identifies a person of interest or suspicious activity, it generates an alert.

[0273] The alert information includes a person's facial image, clothing, location information, behavior details, and emotional state.

[0274] The server transmits the generated alert information to the store clerk's terminal and notifies the store clerk.

[0275] 7. How to display alert information

[0276] The store clerk's device receives the alert and displays the alert information in chat format.

[0277] The store clerk's terminal screen displays the person's clothing, location, behavioral details, and emotional state in real time.

[0278] Program processing

[0279] 1. Device operation

[0280] The device continuously acquires video data from the store's security cameras and transmits it to the server.

[0281] The device acquires images from multiple cameras simultaneously, efficiently collecting data.

[0282] 2. Server Data Analysis

[0283] The server performs facial recognition on the received video data and applies a highly accurate facial recognition algorithm.

[0284] The server analyzes the behavior of people in the video and runs algorithms to detect abnormal behavior.

[0285] 3. Emotion recognition

[0286] The server's emotion engine analyzes the user's emotions based on the facial data and generates emotion data.

[0287] The server evaluates this emotional data in combination with the results of behavioral analysis.

[0288] 4. Database Matching

[0289] The server matches the newly detected facial and emotion data against a database to identify matches.

[0290] The server performs a risk assessment based on the facial data and emotional state that match closely.

[0291] 5. Alert generation and notification

[0292] If the server detects an abnormality, it immediately generates an alert and creates a notification message.

[0293] The alert message includes a facial image of the suspicious person, their clothing, location, and details of their behavior and emotions.

[0294] 6. Assistance via store clerk's terminal

[0295] The user (store clerk) receives the alert and checks the information displayed in chat format.

[0296] The user goes to a specific location in the store and takes appropriate action against the detected suspicious person.

[0297] Specific examples

[0298] For example, imagine a store with security cameras monitoring the store 24 hours a day. A device continuously captures data from these cameras and sends it to a server. The server uses facial recognition technology to identify people in the footage and check whether they are on a blacklist in a database. At the same time, an emotion engine analyzes the emotional data from the person's face to detect whether they are expressing anxiety or impatience.

[0299] In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an unusually long time and appears anxious." This alert is sent to the store clerk's device, which displays detailed information in a chat format. Based on this information, the store clerk rushes to the scene, confirms the possibility of shoplifting, and takes appropriate action.

[0300] This system makes it possible to efficiently and quickly monitor suspicious behavior and emotional states over a wide area within a store, and immediately notify store staff to prompt them to take action, thereby significantly strengthening shoplifting prevention and improving store security.

[0301] The processing flow will be explained below.

[0302] Step 1: Acquire video data

[0303] The terminal acquires video data in real time from security cameras installed inside the store.

[0304] The video data acquired by the terminal is compressed and sent to a server via the Internet.

[0305] Step 2: Receiving and saving video data

[0306] The server receives the video data transmitted from the terminal.

[0307] The video data received by the server is temporarily stored in a storage system.

[0308] Step 3: Performing facial recognition

[0309] The server analyzes the video data stored in the storage system and applies a facial recognition algorithm.

[0310] The server cuts out the detected face for each frame and generates face image data.

[0311] Step 4: Performing behavioral analysis

[0312] The server runs an algorithm that tracks the movements of each person in the video data.

[0313] The server detects specific behavioral patterns (e.g., staying in the same place for a certain period of time or going back and forth to the same place multiple times).

[0314] Step 5: Performing Emotion Recognition

[0315] The server applies an emotion engine based on the facial image data to analyze the user's emotions.

[0316] The server generates user emotion data and evaluates it in combination with behavioral data.

[0317] Step 6: Check against the database

[0318] The server compares the newly detected facial and emotion data with a pre-registered database.

[0319] The server identifies matches to the facial and emotion data in the database.

[0320] Step 7: Evaluate the match results

[0321] The server evaluates suspicious behavior and people of interest based on the matching results.

[0322] The server performs a risk assessment based on the detected behavior and emotions and determines the severity of the alert.

[0323] Step 8: Generate an alert

[0324] If the server identifies a person of interest or suspicious activity, it generates an alert.

[0325] The alert information includes facial image, clothing, location information, behavioral details, and emotional state.

[0326] Step 9: Sending an alert

[0327] The server transmits the generated alert information to the store clerk's terminal.

[0328] The server is set so that alert information is sent to multiple store clerk terminals simultaneously.

[0329] Step 10: Viewing Alerts

[0330] The alert information received by the store clerk's terminal is displayed in chat format.

[0331] The employee's device screen displays a person's facial image, clothing, location, behavioral details, and emotional state in real time.

[0332] Step 11: Staff response

[0333] The user (store clerk) checks the alert and rushes to the scene based on the detailed information displayed.

[0334] The user takes appropriate action against the detected suspicious person.

[0335] The information confirmed by the user is shared with other store staff, and monitoring is strengthened in collaboration.

[0336] By implementing the above steps, the monitoring system of the present invention can efficiently and quickly detect suspicious behavior and suspicious individuals, and immediately issue alerts to prompt responses. Furthermore, by combining it with an emotion engine, advanced risk assessment that takes into account changes in emotions becomes possible.

[0337] Example 2

[0338] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0339] Conventional security systems have had difficulty detecting suspicious behavior in stores in real time and responding quickly. Furthermore, conventional systems have had problems with low accuracy in facial recognition and behavioral analysis, resulting in frequent false positives and missed detections. Furthermore, the method of notifying store staff was inefficient, limiting effective action in situations where immediate action was required.

[0340] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring data within the store, a means for processing the acquired data to identify personal information and behavioral patterns of people, and a means for comparing the identified personal information and behavioral patterns with recording media. This makes it possible to perform highly accurate face recognition and behavioral analysis, detect suspicious behavior or suspicious people in real time, and quickly notify store staff. In addition, by displaying warning information in an interactive format, store staff can immediately understand the situation and take appropriate action.

[0341] "In-store" refers to the space inside a building such as a commercial facility or retail store.

[0342] "Data" refers to a collection of information such as video, images, audio, and sensor information.

[0343] "Means of acquisition" refers to methods of collecting data using devices such as cameras and sensors.

[0344] "Processing" refers to performing operations such as analysis and conversion on acquired data.

[0345] "Person" refers to the human being who is the subject of surveillance.

[0346] "Personal information" refers to information that can identify a specific individual, such as a name or facial image.

[0347] "Behavioral patterns" refer to the characteristics of a person's movements or actions at a particular time.

[0348] "Means of identification" refers to methods that use facial recognition and behavioral analysis algorithms to distinguish a person's characteristics and behavior.

[0349] "Recording media" refers to storage or databases for saving information.

[0350] "Matching" refers to comparing detected information with existing data to determine whether it matches.

[0351] "Warning" refers to a notification or alert that occurs when an abnormality is detected.

[0352] "Interactive" refers to the way information is presented in the form of chats and messages.

[0353] This invention relates to a surveillance system that strengthens security in stores, prevents shoplifting, and detects suspicious behavior. Specifically, this system analyzes video data acquired from security cameras installed in stores in real time to perform facial recognition and behavior analysis.

[0354] Hardware and Software Configuration

[0355] This system consists of the following hardware and software:

[0356] 1. Device:

[0357] Security cameras: Multiple cameras are installed within the store and continuously capture footage.

[0358] Network connection: A high-speed internet connection to transmit captured video data to the server.

[0359] 2. Server:

[0360] Video analytics software: Software for running high-precision facial recognition algorithms (e.g., OpenCV, Dlib, etc.) and behavioral analysis algorithms (e.g., YOLO, OpenPose, etc.).

[0361] Database: A recording medium for storing existing person of interest data and emotion data.

[0362] Emotion engine: A software module for analyzing user emotions.

[0363] Program processing

[0364] The overall system operates as follows.

[0365] 1. Device operation

[0366] The terminal continuously captures video data from security cameras installed in the store, efficiently compresses it, and sends it to the server. Data compression is particularly important when handling high-resolution video data.

[0367] 2. Server Data Analysis

[0368] The server applies a facial recognition algorithm to the received video data to extract facial images of people for each frame, and then uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements.

[0369] 3. Emotion recognition

[0370] The server's emotion engine analyzes the user's emotions based on the facial data and generates emotion data, which is then combined with the results of behavioral analysis to determine the overall risk level.

[0371] 4. Database Matching

[0372] The server then compares the newly detected facial and emotional data against existing databases to identify matches, specifically against historical blacklists and emotional databases.

[0373] 5. Alert generation and notification

[0374] If the server detects an anomaly, it immediately generates an alert and creates a notification message. The alert includes details of the suspicious person's face, clothing, location, behavior, and emotions. This information is sent to the store clerk's terminal and displayed interactively.

[0375] 6. Staff Service

[0376] The user (store clerk) receives an alert on their device, checks the detailed alert information in chat format, rushes to the scene, and takes appropriate action against the detected suspicious person.

[0377] Specific examples

[0378] For example, consider a store where security cameras monitor the store 24 hours a day. Devices continuously capture data from these cameras and send it to a server. The server uses facial recognition technology to identify individuals in the footage and check whether they are on a database of suspicious individuals. At the same time, an emotion engine analyzes the emotional data from the individual's face and detects whether they are expressing anxiety or impatience. In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an unusually long time and appears anxious." This alert is then sent to a store employee's device, which displays detailed information in a chat format. Based on this information, the employee rushes to the scene, confirms the possibility of shoplifting, and takes appropriate action. This system allows for efficient and rapid monitoring of suspicious behavior and emotional states across a wide area of ​​the store, and immediately notifies employees to prompt action. This significantly strengthens shoplifting prevention and improves store security.

[0379] Prompt Sentence Examples

[0380] Examples of prompts to be input to a generative AI model include:

[0381] "How can we create a system that sends notifications when security camera footage shows signs of shoplifting?"

[0382] "How can we combine facial recognition and behavioral analysis to detect suspicious behavior in stores?"

[0383] "Describe an algorithm that uses an emotion engine to detect emotions like anxiety and anger to identify suspicious behavior in real time."

[0384] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0385] Step 1:

[0386] The terminal acquires video data in real time from security cameras installed inside the store. Specifically, each security camera captures video frame by frame and sends it to the terminal. The input is the video data acquired from the security camera, and the output is compressed video data. The terminal performs this continuously to efficiently collect data.

[0387] Step 2:

[0388] The video data acquired by the device is compressed and sent to a server via the Internet. Specifically, the video data is encoded using a compression algorithm such as H.264 to reduce the data volume. The input is raw video data from the security camera, and the output is compressed video data. The compressed data is sent to the server.

[0389] Step 3:

[0390] The server applies a facial recognition algorithm to the video data it receives. Specifically, it uses libraries such as OpenCV and Dlib to detect human faces in each frame and extract facial images. The input is compressed video data, and the output is a list of detected facial images. The server performs this process continuously, achieving highly accurate facial recognition.

[0391] Step 4:

[0392] The server uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements. Specific algorithms, such as YOLO and OpenPose, are used to identify the person's movements and location. The input is frame-by-frame video data, and the output is analyzed behavioral data. The server specializes in detecting unusual behavioral patterns.

[0393] Step 5:

[0394] The server's emotion engine analyzes the user's emotions based on the facial data and generates emotion data. Specifically, it uses an algorithm that analyzes facial expressions to identify the user's psychological state. The input is the detected facial image, and the output is emotion data (e.g., impatience, anger, anxiety). The server then combines this with the results of behavioral analysis and evaluates it.

[0395] Step 6:

[0396] The server compares newly detected facial and emotional data against an existing database to identify matches. Specifically, it uses an algorithm that directly compares new data with the existing database. The input is the newly analyzed facial and emotional data, and the output is whether or not there are any matching records. The server then performs a risk assessment.

[0397] Step 7:

[0398] If the server detects an anomaly, it immediately generates an alert and creates a notification message. Specifically, it runs a program that automatically generates an alert message after detecting an anomaly. The input is the anomaly detection result data, and the output is an alert message. The alert includes details of the suspicious person's face image, clothing, location, behavior, and emotions.

[0399] Step 8:

[0400] After the alert information generated by the server is sent to the store clerk's terminal, the user (store clerk) receives the alert on the terminal and checks the information displayed in chat format. The input is the alert message from the server, and the output is the alert information displayed on the terminal. The store clerk checks this and rushes to the scene depending on the situation.

[0401] Step 9:

[0402] The user (store clerk) rushes to the scene based on the alert information and takes appropriate action against the detected suspicious person. Specific actions include actually checking the situation and, if necessary, coordinating with security guards. The input is detailed information displayed in chat format, and the output is the response results.

[0403] (Application example 2)

[0404] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0405] Conventional suspicious behavior detection systems in stores rely solely on simple behavioral analysis, making them insufficient for detecting shoplifting and other suspicious behavior. Furthermore, simple video analysis cannot capture detailed behavior or psychological states, making it difficult for store staff to take appropriate action on the spot. Furthermore, notification functions are often inadequate, resulting in delayed real-time responses. For these reasons, there was a demand for a system that could provide more accurate and rapid detection of suspicious behavior and countermeasures.

[0406] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring in-store video data, means for analyzing the acquired video data and identifying faces and behaviors, means for comparing the identified faces and behaviors with a database, means for generating and notifying an alert based on the comparison results, means for displaying alert information, means for recording the notification message, emotion engine means for analyzing emotions based on the face recognition results, and means for sending real-time notifications to the terminal. This combines face recognition and emotion analysis to enable advanced and precise detection of suspicious behavior, allowing store staff to respond quickly in real time.

[0407] "In-store video data" refers to real-time video information captured by video capture devices such as security cameras installed in the store.

[0408] "Means of acquisition" refers to the devices and methods used to collect video data from security cameras and other sources and input it into the system.

[0409] "Means for analyzing and identifying faces and behavior" refers to algorithms and software that analyze video data obtained from security cameras and recognize the faces of people in the footage and identify their behavioral patterns.

[0410] "Database matching" refers to the process of comparing analyzed facial and behavioral data with a pre-registered database to identify matching individuals and behaviors.

[0411] "Means for generating and notifying alerts" refers to a system that creates warning information when suspicious behavior or individuals are detected and notifies store staff and other relevant parties.

[0412] The "means for displaying alert information" refers to a device or interface for displaying the generated alert information in a form that can be easily confirmed by store staff and other relevant parties.

[0413] The "means for recording notification messages" refers to a system for saving the generated alerts and notification contents so that they can be referenced later.

[0414] "Emotion engine means for analyzing emotions" refers to algorithms or software that recognize and analyze a person's emotions based on the results of facial recognition, and identify emotional states such as anxiety or fear.

[0415] "Means for real-time notification" refers to a communication method that immediately notifies the store clerk's terminal of detected suspicious behavior or emotional state, urging immediate action.

[0416] The system of the present invention is intended to prevent shoplifting and detect suspicious behavior in a store, and is configured using the following hardware and software.

[0417] System Configuration

[0418] 1. Security cameras

[0419] These cameras are installed in stores to monitor customer movements in real time, making it possible to continuously acquire video data.

[0420] 2. Video data acquisition method

[0421] This method continuously acquires video data from security cameras, compresses it, and sends it to a server. The specific software used is OpenCV.

[0422] 3. Video data analysis methods

[0423] Using this method, the server analyzes the received video data, performs facial recognition and behavior analysis routines, and uses a face recognition library (e.g., face_recognition) to extract and analyze face images frame by frame.

[0424] 4. Emotion Engine

[0425] The server is equipped with a trained emotion recognition model that uses machine learning frameworks such as Keras to identify customer emotions based on facial recognition results.

[0426] 5. Database matching methods

[0427] The server checks the newly detected facial and emotional data against an existing database containing pre-existing images of suspect people to see if there is a match.

[0428] 6. Alert Generation and Notification Methods

[0429] Based on the match, the server generates an alert, which includes the person's facial image, clothing, location, behavioral details, and emotional state, and sends it to the store clerk's smartphone in real time using Twilio.

[0430] 7. Alert Information Display Method

[0431] The user (store clerk) receives the alert via an application installed on their smartphone and can check it in chat format, enabling a quick response in the store.

[0432] Specific examples

[0433] For example, suppose a store has security cameras monitoring the store 24 hours a day. The server continuously collects data from these cameras and analyzes the video data in real time. Specifically, it uses facial recognition technology to identify people in the video and check whether they are included in a database of suspicious individuals.

[0434] At the same time, the emotion engine analyzes the emotional data from the person's face and detects whether the person is expressing anxiety or impatience. In this case, the server generates an alert stating, "A person wearing a red jacket is lingering in front of a particular product for an abnormally long time and is displaying an anxious expression." This alert is sent to the store clerk's device, which displays detailed information in a chat format. Based on this information, the store clerk can rush to the scene, confirm the possibility of shoplifting, and take appropriate action.

[0435] Prompt Sentence Examples

[0436] "Generate application code that acquires video data from security cameras in real time, detects suspicious behavior using an emotion engine, analyzes customer emotions such as anxiety, anger, and fear, and notifies store staff on their smartphones."

[0437] This system makes it possible to efficiently and quickly monitor suspicious behavior and customer emotional states over a wide area within a store, instantly notifying store staff and encouraging them to take prompt action, thereby strengthening shoplifting prevention measures and improving security throughout the store.

[0438] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0439] Step 1:

[0440] A means for acquiring video data from security cameras acquires video data in real time from security cameras in the store. This data is continuously acquired and used for subsequent analysis steps. The input is raw video data and the output is compressed video data.

[0441] Step 2:

[0442] The device transmits the video data acquired from the security camera to the server. The specific input is the compressed video data, and the output is the status of completion of transmission to the server. This process is performed via the network.

[0443] Step 3:

[0444] The server analyzes the received video data and performs facial recognition and behavior analysis. The input is the received video data, and the output is the recognized face image and behavior data. Specifically, OpenCV and the face_recognition library are used.

[0445] Step 4:

[0446] The server performs emotion analysis based on the facial recognition results. The input is the analyzed facial image, and the output is emotion data (e.g., anxiety, impatience, anger). The emotion engine uses Keras and a pre-trained neural network model.

[0447] Step 5:

[0448] The server compares the newly detected face and emotion data with an existing database. The input is face and emotion data, and the output is the matching result. The database contains previous images of suspicious people.

[0449] Step 6:

[0450] If the server detects an anomaly, it generates an alert. The input is the matching result and emotion data, and the output is alert information. The alert information includes the detected person's face image, clothing, location information, behavior details, and emotional state.

[0451] Step 7:

[0452] The server uses Twilio to send alert information to the store clerk's smartphone. The input is the alert information, and the output is a notification to the store clerk's terminal. This notification includes detailed information about the person.

[0453] Step 8:

[0454] The user (store clerk) checks the alert through an application installed on the device. The input is the received alert information, and the output is instructions for the store clerk to act. The store clerk checks the detailed information in chat format and responds promptly.

[0455] summary

[0456] This series of processes makes it possible to efficiently and quickly detect suspicious behavior and emotional states within a store, and immediately notify store staff to prompt them to take action, thereby strengthening shoplifting prevention measures and improving security throughout the store.

[0457] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0458] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0459] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0460] [Second embodiment]

[0461] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0462] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0463] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0464] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0465] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0466] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0467] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0468] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0469] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0470] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0471] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0472] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0473] The present invention relates to a surveillance system for preventing shoplifting and detecting suspicious behavior in a store. The system analyzes video data obtained from security cameras in real time to detect suspicious behavior or suspicious individuals. An alert is then sent to the store clerk's terminal, enabling a prompt response.

[0474] System configuration

[0475] 1. Video data acquisition method

[0476] The terminal acquires video data in real time from multiple security cameras installed within the store.

[0477] The terminal compresses the acquired video data and transmits it to the server.

[0478] 2. Video data analysis methods

[0479] The server analyzes the received video data and applies facial recognition and behavioral analysis algorithms.

[0480] The server performs facial recognition and extracts the detected facial images for each frame.

[0481] The server uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements.

[0482] 3. Database matching method

[0483] The server matches the newly detected face data with the existing database.

[0484] The server compares the analyzed behavioral data with past behavioral data to identify suspicious behavioral patterns.

[0485] 4. Alert Generation and Notification Methods

[0486] If the server identifies a person of interest or suspicious activity, it generates an alert.

[0487] The alert information includes a person's facial image, clothing, location information, and details of their behavior.

[0488] The server transmits the generated alert information to the store clerk's terminal and notifies the store clerk.

[0489] 5. Displaying alert information

[0490] The store clerk's device receives the alert and displays the alert information in chat format.

[0491] The store clerk's terminal screen displays the person's clothing, location, and behavior details in real time.

[0492] Program processing

[0493] 1. Device operation

[0494] The device continuously acquires video data from the store's security cameras and transmits it to the server.

[0495] The device acquires images from multiple cameras simultaneously, efficiently collecting data.

[0496] 2. Server Data Analysis

[0497] The server performs facial recognition on the received video data and applies a highly accurate facial recognition algorithm.

[0498] The server analyzes the behavior of people in the video and runs algorithms to detect abnormal behavior.

[0499] 3. Database Matching

[0500] The server compares the newly detected facial data with the database to identify facial data with high matching scores.

[0501] Based on the results of behavioral analysis, the server compares suspicious behavioral patterns with past data to identify potentially risky behavior.

[0502] 4. Alert generation and notification

[0503] If the server detects an abnormality, it will immediately generate an alert and create a notification message.

[0504] The alert message includes details of the suspicious person's facial image, clothing, location, and behavior.

[0505] 5. Assistance via store clerk's terminal

[0506] The user (store clerk) receives the alert and checks the information displayed in chat format.

[0507] The user goes to a specific location in the store and takes appropriate action against the detected suspicious person.

[0508] Specific examples

[0509] For example, imagine a store with security cameras monitoring the store 24 / 7. Devices continuously capture data from these cameras and send it to a server. The server uses facial recognition technology to identify people in the footage and check whether they are on a blacklist in a database. At the same time, the server applies behavioral analysis algorithms to detect if a person repeatedly behaves suspiciously in front of a particular product.

[0510] In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an abnormally long time." This alert is sent to the store clerk's device, which displays detailed information in chat format. Based on this information, the store clerk rushes to the scene, confirms the possibility of shoplifting, and takes appropriate action.

[0511] This system makes it possible to efficiently and quickly detect suspicious activity over a wide area within a store and immediately notify store staff, significantly strengthening shoplifting prevention and improving store security.

[0512] The processing flow will be explained below.

[0513] Step 1: Acquire video data

[0514] The terminal acquires video data in real time from security cameras installed inside the store.

[0515] The video data acquired by the terminal is compressed and sent to a server via the Internet.

[0516] Step 2: Receiving video data

[0517] The server receives the video data transmitted from the terminal.

[0518] The server temporarily stores the received video data.

[0519] Step 3: Performing facial recognition

[0520] The server applies a facial recognition algorithm to the received video data.

[0521] The server cuts out the detected face for each frame and generates face image data.

[0522] Step 4: Performing behavioral analysis

[0523] The server applies an algorithm to track the movements of each person in the video data.

[0524] The server detects certain behavioral patterns (e.g., repeatedly returning to the same location).

[0525] Step 5: Check against the database

[0526] The server matches the newly detected face data with the existing database.

[0527] The server checks whether there is a person in the database that matches the facial data.

[0528] Step 6: Evaluate the match results

[0529] Based on the matched results, the server evaluates whether there is a person of interest or suspicious activity.

[0530] If the server identifies suspicious activity, it records the details of that activity.

[0531] Step 7: Generate an alert

[0532] If the server identifies a person of interest or suspicious activity, it generates an alert.

[0533] The server includes details of the face image, clothing, location, and behavior in the alert information.

[0534] Step 8: Sending an alert

[0535] The server transmits the generated alert information to the store clerk's terminal.

[0536] The alert sent from the server is set to be delivered to multiple store clerk terminals simultaneously.

[0537] Step 9: Viewing Alerts

[0538] The alert information received by the store clerk's terminal is displayed in chat format.

[0539] The store clerk's terminal screen displays the person's clothing, location, and details of their behavior in real time.

[0540] Step 10: Staff response

[0541] The user (store clerk) checks the alert and refers to the displayed detailed information.

[0542] The user takes appropriate action and shares the information with other store staff as needed.

[0543] By using the above steps, the monitoring system of the present invention can efficiently and quickly detect suspicious behavior or suspicious individuals, and immediately issue an alert to prompt a response.

[0544] Example 1

[0545] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0546] From a security perspective, it is important to efficiently and quickly detect shoplifting and other suspicious behavior in stores and respond appropriately. However, current systems lack sufficient accuracy in analyzing video data and the speed of notification, resulting in the overlooking of suspicious behavior and delayed response. Other issues include the accuracy of the process of comparing suspect individuals and behavioral patterns with a database, and the efficiency of communicating information to store staff.

[0547] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0548] In this invention, the server includes means for acquiring video data of the monitored area, means for compressing the acquired video data and sending it to the server, means for analyzing the received video data and performing facial recognition and behavioral analysis, means for comparing the analyzed facial and behavioral data with a database, means for generating and notifying an alert based on the comparison results, and means for displaying the alert information. This makes it possible to detect suspicious behavior over a wide area within the store with high accuracy and immediately notify store staff.

[0549] "Surveillance Area" means a physical space that is subject to surveillance for a particular security purpose.

[0550] "Video data" refers to video image information collected by imaging equipment such as cameras installed in a monitored area.

[0551] "Compression" is the process of changing the format of data using a specific algorithm to reduce the volume of the data.

[0552] A "server" is a computer system used to provide a particular service and process data over a network.

[0553] "Facial recognition" is a technology that detects people's faces from video data and identifies individual faces using specific algorithms.

[0554] "Behavioral analysis" is a technology that analyzes the movements of people in video data and identifies specific behaviors and patterns.

[0555] A "database" is a system for storing, managing, and searching data in an organized manner.

[0556] "Matching" is the process of comparing the analyzed data with information from existing databases.

[0557] An "alert" is a warning message that is generated when a particular condition is met.

[0558] "Notification" is the act of sending generated alert information to a designated recipient.

[0559] "Means" refers to a method or device used to achieve a particular purpose.

[0560] "Display" is the act of visually showing information.

[0561] The present invention relates to a surveillance system aimed at preventing shoplifting and detecting suspicious behavior in stores. Specifically, the system analyzes video data obtained from security cameras in real time to detect suspicious behavior and suspicious individuals. Furthermore, based on the results, the system sends alerts to store clerk terminals to prompt prompt action.

[0562] Hardware and Software Configuration

[0563] Acquiring video data of the monitored area

[0564] The device acquires video data in real time from multiple security cameras (e.g., IP cameras) installed in the store. The device temporarily stores the video from the cameras in a buffer for easy access.

[0565] Video data compression and transmission

[0566] The video data acquired by the device is compressed and sent to the server using an efficient video compression format such as H.264. The compressed data is sent via the network to the specified server address.

[0567] Video data analysis

[0568] The server starts analyzing the received video data. It applies a high-precision facial recognition algorithm (e.g., OpenCV or Dlib) to detect faces in the video. The detected faces are extracted from each frame and saved as individual image data. In parallel, the server uses a behavior analysis algorithm (e.g., an RNN- or LSTM-based model) to analyze the behavior of people in the video.

[0569] Database Matching

[0570] The server compares the newly detected face data with an existing database (e.g., an SQL database) by calculating the distance between the vectors of facial features. Similarly, behavioral data is compared with past behavioral patterns.

[0571] Alert generation and notification

[0572] If the server identifies a suspicious person or suspicious behavior, it generates an alert, which includes details of the detected face, clothing, location, and behavior. The alert message is generated in real time and pushed to the store clerk's device.

[0573] Viewing Alerts

[0574] When the device receives an alert, the alert information is displayed in chat format, allowing store staff to check the notification in real time on the device screen. The alert information includes details of the person's clothing, location, and behavior.

[0575] Staff response

[0576] The user (store clerk) receives the alert and checks the displayed information. The clerk then promptly takes action based on the alert. For example, they go to the specific location where the suspicious behavior was reported and take appropriate action against the suspicious person.

[0577] Specific examples

[0578] For example, suppose a store is monitored 24 hours a day by security cameras. A device acquires video data from multiple cameras in the store and sends it to a server. The server analyzes the received data and performs highly accurate facial recognition and behavioral analysis. This makes it possible to check whether a specific person is included in the blacklist of individuals in the past database.

[0579] At the same time, the server applies a behavioral analysis algorithm to detect suspicious behavior in front of a specific product. Based on this, the server generates an alert such as "A man wearing a red jacket is lingering in front of a specific product for an unusually long time." This alert is sent to the store clerk's device, and detailed information is displayed in chat format.

[0580] Using this information, store staff can rush to the scene, confirm the possibility of shoplifting, and take appropriate action. This system will enable the efficient and rapid detection of suspicious activity across a wide area of ​​the store, significantly strengthening the prevention of shoplifting.

[0581] Prompt Sentence Examples

[0582] 1. A store clerk can detect someone spending an abnormally long time at a particular shelf.

[0583] 2. The server detects suspicious behavior and sends an alert to the store clerk's terminal.

[0584] 3. The store staff checks the alert and takes appropriate action.

[0585] In this way, the present invention provides a system that efficiently and quickly strengthens security within a store and contributes to preventing fraudulent activities such as shoplifting.

[0586] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0587] Step 1: Acquire video data

[0588] The device acquires video data in real time from multiple security cameras installed in the store. Specifically, the video captured by the cameras is temporarily stored in a buffer, enabling continuous data collection.

[0589] Input: Real-time video data from security cameras.

[0590] Output: Uncompressed video data stored in a buffer.

[0591] Step 2: Compress and send the data

[0592] The video data acquired by the device is compressed and sent to the server using an efficient video compression format such as H.264, and the data is sent to the server via the network.

[0593] Input: Uncompressed video data.

[0594] Output: Compressed video data.

[0595] Step 3: Analyzing the video data

[0596] The server analyzes the received compressed video data. First, a face recognition algorithm (e.g., OpenCV or Dlib) is applied to detect faces in the video. The detected faces are extracted frame by frame and saved as individual image data. At the same time, a behavior analysis algorithm (e.g., an RNN or LSTM-based model) is applied to analyze the person's behavior.

[0597] Input: Compressed video data.

[0598] Output: Detected face images and behavior analysis results.

[0599] Step 4: Check against the database

[0600] The server matches newly detected facial data with an existing database (e.g., an SQL database), calculates the distance between facial feature vectors, and identifies potential matches of suspicious individuals. Behavioral data is similarly compared with past behavioral patterns.

[0601] Input: Detected face images and behavior analysis results.

[0602] Output: Matching person of interest information and suspicious behavior identification.

[0603] Step 5: Generate an alert

[0604] If the server identifies a suspicious person or behavior, it generates an alert, combining detected facial images, clothing, location information, and behavior details to create an alert message.

[0605] Input: Matching person of interest information and suspicious activity identification results.

[0606] Output: The alert message.

[0607] Step 6: Notification and display of alerts

[0608] The server sends the generated alert information to the store clerk's terminal, which displays the alert information in a pop-up or chat window format so that the store clerk can check it immediately.

[0609] Input: Alert message.

[0610] Output: Alert information displayed on the terminal.

[0611] Step 7: Staff response

[0612] The user (store clerk) receives the alert and checks the displayed information. The clerk follows the instructions in the alert to go to the specific location and deal with the suspicious person.

[0613] Input: The alert information displayed on the terminal.

[0614] Output: On-site response actions.

[0615] (Application example 1)

[0616] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0617] While conventional systems have the ability to analyze security camera footage to detect suspicious behavior and suspicious individuals, they have limited means of providing information to store staff so that they can respond quickly.In addition, there is a lack of ways for store staff patrolling the store to identify suspicious behavior in real time, making it difficult to respond quickly.

[0618] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0619] In this invention, the server includes means for acquiring video data of the store, means for analyzing the acquired video data and identifying faces and behaviors, means for comparing the analyzed faces and behaviors with a database on the server, means for generating and notifying an alert based on the comparison results, and means for displaying the alert information on the smart glasses. This enables store staff wearing the smart glasses to identify suspicious behavior in real time while patrolling and respond quickly.

[0620] "In-store video data" refers to images and video information obtained in real time from security cameras installed within a store.

[0621] The "acquisition means" refers to hardware and software components for receiving video data output from a security camera and storing or transmitting it.

[0622] "Means for analyzing and identifying faces and behaviors" refers to facial recognition algorithms and behavior analysis algorithms for identifying people's faces and analyzing their behaviors based on the captured video data.

[0623] The "database on the server" is a data management system used to store analyzed facial and behavioral data and compare it with past data.

[0624] The "means of matching" is an algorithm that compares facial and behavioral data acquired in real time with existing data in a database on the server to confirm a match.

[0625] "Means for generating and notifying alerts" refers to a system that creates an alert when suspicious behavior or suspicious individuals are detected and immediately notifies relevant parties of that information.

[0626] The "means for displaying alert information on smart glasses" refers to communication and display technology for displaying the generated alert information on the display of smart glasses worn by the store clerk.

[0627] This invention is a surveillance system aimed at preventing shoplifting in a store and detecting suspicious individuals at an early stage. This system acquires video data from multiple security cameras in real time, analyzes the data, detects suspicious behavior and individuals of interest, and notifies store staff.

[0628] Hardware configuration:

[0629] Security cameras: These are installed at strategic locations within the store and are used to capture footage in real time.

[0630] Server: A computer with powerful computing power for receiving and analyzing video data.

[0631] Smart glasses: worn by store associates and used to receive and display alert information in real time.

[0632] Software configuration:

[0633] OpenCV: A library for acquiring and processing security camera footage in real time.

[0634] Keras: A deep learning framework used to implement face recognition and behavior analysis algorithms.

[0635] Flask: A lightweight web framework for receiving and analyzing video data on the server side.

[0636] Requests: An HTTP library for data communication between smart glasses and a server.

[0637] Overview of program processing:

[0638] The device continuously captures video data from security cameras in the store in real time and sends it to a server. The server then applies facial recognition and behavioral analysis algorithms to the received video data to analyze the behavioral patterns of specific individuals. The analysis results are compared with existing data in a database, and an alert is generated if a match is found or if abnormal behavior is detected. The generated alert is then sent to the smart glasses, which notify the store staff in real time.

[0639] Examples:

[0640] For example, consider a store where security cameras are monitoring the shop 24 hours a day. The server receives this video data in real time and uses a facial recognition algorithm to identify individuals. It then checks whether the identified individuals are on a blacklist in the database. At the same time, the server applies a behavioral analysis algorithm to detect that an individual is repeatedly lingering in front of a particular product for an abnormally long time. In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an abnormally long time." This alert is sent to smart glasses, and the information is displayed in chat format on the smart glasses worn by the store clerk. Based on this information, the store clerk can head to the scene, confirm the possibility of shoplifting, and take appropriate action.

[0641] Example prompt sentence:

[0642] "The system analyzes security camera footage within stores in real time and generates an alert if it determines something is suspicious. The alert includes details of the face, clothing, location, and behavior. This alert is displayed on the smart glasses, enabling a prompt response."

[0643] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0644] Step 1:

[0645] Device operation

[0646] The device acquires video data from security cameras inside the store in real time.

[0647] Input: Video data from a security camera.

[0648] Data processing: The terminal compresses the acquired video data and converts it into a format that can be sent to the server (for example, JPEG format).

[0649] Output: Compressed video data.

[0650] Specific operation: The device continuously acquires data from multiple security cameras and transmits it to the server in real time.

[0651] Step 2:

[0652] Server data reception

[0653] A server receives the compressed video data.

[0654] Input: Compressed video data sent from the device.

[0655] Data processing: The server decompresses the compressed video data and converts it into a format that can be used as the original video data.

[0656] Output: Decompressed video data.

[0657] Specific operation: The server sequentially decompresses the received video data and prepares it for subsequent analysis processing.

[0658] Step 3:

[0659] Facial Recognition and Behavioral Analysis

[0660] The server analyzes the decompressed video data and applies facial recognition and behavioral analysis algorithms.

[0661] Input: Decompressed video data.

[0662] Data computation: The server uses a facial recognition algorithm (e.g., a Keras model) to identify faces in the video and applies a behavior analysis algorithm to analyze people's behavior.

[0663] Output: Analyzed face and behavioral data.

[0664] Specific operation: The server extracts people's faces from each video frame and analyzes their behavioral patterns.

[0665] Step 4:

[0666] Database collation

[0667] The server compares the analyzed facial and behavioral data with a database on the server.

[0668] Input: Parsed face and behavioral data.

[0669] Data calculation: The server compares the data with data in an existing database to identify facial data and suspicious behavior patterns with high matching scores.

[0670] Output: Matching results (face data and behavioral patterns with high matching scores).

[0671] Specific operation: The server uses a matching algorithm to match the blacklist in the database and identify matching data.

[0672] Step 5:

[0673] Generate alerts

[0674] The server generates an alert based on the match result.

[0675] Input: Matching results (face data and behavioral patterns with high matching scores).

[0676] Data processing: The server generates alert information (a person's face image, clothing, location information, and details of their behavior).

[0677] Output: Alert information.

[0678] What it does: The server immediately creates an alert when a suspicious person or activity is detected.

[0679] Step 6:

[0680] Sending and Viewing Alerts

[0681] The server transmits the generated alert information to the smart glasses for display.

[0682] Input: Alert information.

[0683] Data processing: The server converts the alert information into a format suitable for the smart glasses and sends it.

[0684] Output: Alert information displayed on smart glasses.

[0685] Specific operation: The server composes a notification message and sends it to the smart glasses, which receive the notification and the store clerk confirms it.

[0686] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0687] This invention relates to a surveillance system aimed at preventing shoplifting and detecting suspicious behavior in stores. This system analyzes video data obtained from security cameras in real time, and performs facial recognition and behavior analysis. In addition, by combining it with an emotion engine that recognizes user emotions, it enables more accurate detection of suspicious behavior and quicker response.

[0688] System configuration

[0689] 1. Video data acquisition method

[0690] The terminal acquires video data in real time from security cameras installed inside the store.

[0691] The video data acquired by the terminal is compressed and sent to a server via the Internet.

[0692] 2. Video data analysis methods

[0693] The server analyzes the received video data and applies facial recognition and behavioral analysis algorithms.

[0694] The server performs facial recognition and extracts the detected facial images for each frame.

[0695] The server uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements.

[0696] 3. Emotion Engine

[0697] The emotion engine built into the server identifies the user's emotion based on the facial image detected during face recognition.

[0698] The emotion engine generates user emotion data (e.g., impatience, anger, anxiety) and reflects it in the behavioral analysis results.

[0699] 4. Database matching method

[0700] The server matches the newly detected facial and emotion data with the existing database.

[0701] The server checks whether there is a person in the database that matches the facial data.

[0702] 5. Evaluation of matching results

[0703] Based on the matched results, the server evaluates whether there is a person of interest or suspicious activity.

[0704] If the server identifies suspicious behavior, it records details of the behavior and emotions.

[0705] 6. Alert Generation and Notification Methods

[0706] If the server identifies a person of interest or suspicious activity, it generates an alert.

[0707] The alert information includes a person's facial image, clothing, location information, behavior details, and emotional state.

[0708] The server transmits the generated alert information to the store clerk's terminal and notifies the store clerk.

[0709] 7. How to display alert information

[0710] The store clerk's device receives the alert and displays the alert information in chat format.

[0711] The store clerk's terminal screen displays the person's clothing, location, behavioral details, and emotional state in real time.

[0712] Program processing

[0713] 1. Device operation

[0714] The device continuously acquires video data from the store's security cameras and transmits it to the server.

[0715] The device acquires images from multiple cameras simultaneously, efficiently collecting data.

[0716] 2. Server Data Analysis

[0717] The server performs facial recognition on the received video data and applies a highly accurate facial recognition algorithm.

[0718] The server analyzes the behavior of people in the video and runs algorithms to detect abnormal behavior.

[0719] 3. Emotion recognition

[0720] The server's emotion engine analyzes the user's emotions based on the facial data and generates emotion data.

[0721] The server evaluates this emotional data in combination with the results of behavioral analysis.

[0722] 4. Database Matching

[0723] The server matches the newly detected facial and emotion data against a database to identify matches.

[0724] The server performs a risk assessment based on the facial data and emotional state that match closely.

[0725] 5. Alert generation and notification

[0726] If the server detects an abnormality, it immediately generates an alert and creates a notification message.

[0727] The alert message includes a facial image of the suspicious person, their clothing, location, and details of their behavior and emotions.

[0728] 6. Assistance via store clerk's terminal

[0729] The user (store clerk) receives the alert and checks the information displayed in chat format.

[0730] The user goes to a specific location in the store and takes appropriate action against the detected suspicious person.

[0731] Specific examples

[0732] For example, imagine a store with security cameras monitoring the store 24 hours a day. A device continuously captures data from these cameras and sends it to a server. The server uses facial recognition technology to identify people in the footage and check whether they are on a blacklist in a database. At the same time, an emotion engine analyzes the emotional data from the person's face to detect whether they are expressing anxiety or impatience.

[0733] In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an unusually long time and appears anxious." This alert is sent to the store clerk's device, which displays detailed information in a chat format. Based on this information, the store clerk rushes to the scene, confirms the possibility of shoplifting, and takes appropriate action.

[0734] This system makes it possible to efficiently and quickly monitor suspicious behavior and emotional states over a wide area within a store, and immediately notify store staff to prompt them to take action, thereby significantly strengthening shoplifting prevention and improving store security.

[0735] The processing flow will be explained below.

[0736] Step 1: Acquire video data

[0737] The terminal acquires video data in real time from security cameras installed inside the store.

[0738] The video data acquired by the terminal is compressed and sent to a server via the Internet.

[0739] Step 2: Receiving and saving video data

[0740] The server receives the video data transmitted from the terminal.

[0741] The video data received by the server is temporarily stored in a storage system.

[0742] Step 3: Performing facial recognition

[0743] The server analyzes the video data stored in the storage system and applies a facial recognition algorithm.

[0744] The server cuts out the detected face for each frame and generates face image data.

[0745] Step 4: Performing behavioral analysis

[0746] The server runs an algorithm that tracks the movements of each person in the video data.

[0747] The server detects specific behavioral patterns (e.g., staying in the same place for a certain period of time or going back and forth to the same place multiple times).

[0748] Step 5: Performing Emotion Recognition

[0749] The server applies an emotion engine based on the facial image data to analyze the user's emotions.

[0750] The server generates user emotion data and evaluates it in combination with behavioral data.

[0751] Step 6: Check against the database

[0752] The server compares the newly detected facial and emotion data with a pre-registered database.

[0753] The server identifies matches to the facial and emotion data in the database.

[0754] Step 7: Evaluate the match results

[0755] The server evaluates suspicious behavior and people of interest based on the matching results.

[0756] The server performs a risk assessment based on the detected behavior and emotions and determines the severity of the alert.

[0757] Step 8: Generate an alert

[0758] If the server identifies a person of interest or suspicious activity, it generates an alert.

[0759] The alert information includes facial image, clothing, location information, behavioral details, and emotional state.

[0760] Step 9: Sending alerts

[0761] The server transmits the generated alert information to the store clerk's terminal.

[0762] The server is set so that alert information is sent to multiple store clerk terminals simultaneously.

[0763] Step 10: Viewing Alerts

[0764] The alert information received by the store clerk's terminal is displayed in chat format.

[0765] The employee's device screen displays a person's facial image, clothing, location, behavioral details, and emotional state in real time.

[0766] Step 11: Staff response

[0767] The user (store clerk) checks the alert and rushes to the scene based on the detailed information displayed.

[0768] The user takes appropriate action against the detected suspicious person.

[0769] The information confirmed by the user is shared with other store staff, and monitoring is strengthened in collaboration.

[0770] By implementing the above steps, the monitoring system of the present invention can efficiently and quickly detect suspicious behavior and suspicious individuals, and immediately issue alerts to prompt responses. Furthermore, by combining it with an emotion engine, advanced risk assessment that takes into account changes in emotions becomes possible.

[0771] Example 2

[0772] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0773] Conventional security systems have had difficulty detecting suspicious behavior in stores in real time and responding quickly. Furthermore, conventional systems have had problems with low accuracy in facial recognition and behavioral analysis, resulting in frequent false positives and missed detections. Furthermore, the method of notifying store staff was inefficient, limiting effective action in situations where immediate action was required.

[0774] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring data within the store, a means for processing the acquired data to identify personal information and behavioral patterns of people, and a means for comparing the identified personal information and behavioral patterns with recording media. This makes it possible to perform highly accurate face recognition and behavioral analysis, detect suspicious behavior or suspicious people in real time, and quickly notify store staff. In addition, by displaying warning information in an interactive format, store staff can immediately understand the situation and take appropriate action.

[0775] "In-store" refers to the space inside a building such as a commercial facility or retail store.

[0776] "Data" refers to a collection of information such as video, images, audio, and sensor information.

[0777] "Means of acquisition" refers to methods of collecting data using devices such as cameras and sensors.

[0778] "Processing" refers to performing operations such as analysis and conversion on acquired data.

[0779] "Person" refers to the human being who is the subject of surveillance.

[0780] "Personal information" refers to information that can identify a specific individual, such as a name or facial image.

[0781] "Behavioral patterns" refer to the characteristics of a person's movements or actions at a particular time.

[0782] "Means of identification" refers to methods that use facial recognition and behavioral analysis algorithms to distinguish a person's characteristics and behavior.

[0783] "Recording media" refers to storage or databases for saving information.

[0784] "Matching" refers to comparing detected information with existing data to determine whether it matches.

[0785] "Warning" refers to a notification or alert that occurs when an abnormality is detected.

[0786] "Interactive" refers to the way information is presented in the form of chats and messages.

[0787] This invention relates to a surveillance system that strengthens security in stores, prevents shoplifting, and detects suspicious behavior. Specifically, this system analyzes video data acquired from security cameras installed in stores in real time to perform facial recognition and behavior analysis.

[0788] Hardware and Software Configuration

[0789] This system consists of the following hardware and software:

[0790] 1. Device:

[0791] Security cameras: Multiple cameras are installed within the store and continuously capture footage.

[0792] Network connection: A high-speed internet connection to transmit captured video data to the server.

[0793] 2. Server:

[0794] Video analytics software: Software for running high-precision facial recognition algorithms (e.g., OpenCV, Dlib, etc.) and behavioral analysis algorithms (e.g., YOLO, OpenPose, etc.).

[0795] Database: A recording medium for storing existing person of interest data and emotion data.

[0796] Emotion engine: A software module for analyzing user emotions.

[0797] Program processing

[0798] The overall system operates as follows.

[0799] 1. Device operation

[0800] The terminal continuously captures video data from security cameras installed in the store, efficiently compresses it, and sends it to the server. Data compression is particularly important when handling high-resolution video data.

[0801] 2. Server Data Analysis

[0802] The server applies a facial recognition algorithm to the received video data to extract facial images of people for each frame, and then uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements.

[0803] 3. Emotion recognition

[0804] The server's emotion engine analyzes the user's emotions based on the facial data and generates emotion data, which is then combined with the results of behavioral analysis to determine the overall risk level.

[0805] 4. Database Matching

[0806] The server then compares the newly detected facial and emotional data against existing databases to identify matches, specifically against historical blacklists and emotional databases.

[0807] 5. Alert generation and notification

[0808] If the server detects an anomaly, it immediately generates an alert and creates a notification message. The alert includes details of the suspicious person's face, clothing, location, behavior, and emotions. This information is sent to the store clerk's terminal and displayed interactively.

[0809] 6. Staff Service

[0810] The user (store clerk) receives an alert on their device, checks the detailed alert information in chat format, rushes to the scene, and takes appropriate action against the detected suspicious person.

[0811] Specific examples

[0812] For example, consider a store where security cameras monitor the store 24 hours a day. Devices continuously capture data from these cameras and send it to a server. The server uses facial recognition technology to identify individuals in the footage and check whether they are on a database of suspicious individuals. At the same time, an emotion engine analyzes the emotional data from the individual's face and detects whether they are expressing anxiety or impatience. In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an unusually long time and appears anxious." This alert is then sent to a store employee's device, which displays detailed information in a chat format. Based on this information, the employee rushes to the scene, confirms the possibility of shoplifting, and takes appropriate action. This system allows for efficient and rapid monitoring of suspicious behavior and emotional states across a wide area of ​​the store, and immediately notifies employees to prompt action. This significantly strengthens shoplifting prevention and improves store security.

[0813] Prompt Sentence Examples

[0814] Examples of prompts to be input to a generative AI model include:

[0815] "How can we create a system that sends notifications when security camera footage shows signs of shoplifting?"

[0816] "How can we combine facial recognition and behavioral analysis to detect suspicious behavior in stores?"

[0817] "Describe an algorithm that uses an emotion engine to detect emotions like anxiety and anger to identify suspicious behavior in real time."

[0818] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0819] Step 1:

[0820] The terminal acquires video data in real time from security cameras installed inside the store. Specifically, each security camera captures video frame by frame and sends it to the terminal. The input is the video data acquired from the security camera, and the output is compressed video data. The terminal performs this continuously to efficiently collect data.

[0821] Step 2:

[0822] The video data acquired by the device is compressed and sent to a server via the Internet. Specifically, the video data is encoded using a compression algorithm such as H.264 to reduce the data volume. The input is raw video data from the security camera, and the output is compressed video data. The compressed data is sent to the server.

[0823] Step 3:

[0824] The server applies a facial recognition algorithm to the video data it receives. Specifically, it uses libraries such as OpenCV and Dlib to detect human faces in each frame and extract facial images. The input is compressed video data, and the output is a list of detected facial images. The server performs this process continuously, achieving highly accurate facial recognition.

[0825] Step 4:

[0826] The server uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements. Specific algorithms, such as YOLO and OpenPose, are used to identify the person's movements and location. The input is frame-by-frame video data, and the output is analyzed behavioral data. The server specializes in detecting unusual behavioral patterns.

[0827] Step 5:

[0828] The server's emotion engine analyzes the user's emotions based on the facial data and generates emotion data. Specifically, it uses an algorithm that analyzes facial expressions to identify the user's psychological state. The input is the detected facial image, and the output is emotion data (e.g., impatience, anger, anxiety). The server then combines this with the results of behavioral analysis and evaluates it.

[0829] Step 6:

[0830] The server compares newly detected facial and emotional data against an existing database to identify matches. Specifically, it uses an algorithm that directly compares new data with the existing database. The input is the newly analyzed facial and emotional data, and the output is whether or not there are any matching records. The server then performs a risk assessment.

[0831] Step 7:

[0832] If the server detects an anomaly, it immediately generates an alert and creates a notification message. Specifically, it runs a program that automatically generates an alert message after detecting an anomaly. The input is the anomaly detection result data, and the output is an alert message. The alert includes details of the suspicious person's face image, clothing, location, behavior, and emotions.

[0833] Step 8:

[0834] After the alert information generated by the server is sent to the store clerk's terminal, the user (store clerk) receives the alert on the terminal and checks the information displayed in chat format. The input is the alert message from the server, and the output is the alert information displayed on the terminal. The store clerk checks this and rushes to the scene depending on the situation.

[0835] Step 9:

[0836] The user (store clerk) rushes to the scene based on the alert information and takes appropriate action against the detected suspicious person. Specific actions include actually checking the situation and, if necessary, coordinating with security guards. The input is detailed information displayed in chat format, and the output is the response results.

[0837] (Application example 2)

[0838] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0839] Conventional suspicious behavior detection systems in stores rely solely on simple behavioral analysis, making them insufficient for detecting shoplifting and other suspicious behavior. Furthermore, simple video analysis cannot capture detailed behavior or psychological states, making it difficult for store staff to take appropriate action on the spot. Furthermore, notification functions are often inadequate, resulting in delayed real-time responses. For these reasons, there was a demand for a system that could provide more accurate and rapid detection of suspicious behavior and countermeasures.

[0840] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring in-store video data, means for analyzing the acquired video data and identifying faces and behaviors, means for comparing the identified faces and behaviors with a database, means for generating and notifying an alert based on the comparison results, means for displaying alert information, means for recording the notification message, emotion engine means for analyzing emotions based on the face recognition results, and means for sending real-time notifications to the terminal. This combines face recognition and emotion analysis to enable advanced and precise detection of suspicious behavior, allowing store staff to respond quickly in real time.

[0841] "In-store video data" refers to real-time video information captured by video capture devices such as security cameras installed in the store.

[0842] "Means of acquisition" refers to the devices and methods used to collect video data from security cameras and other sources and input it into the system.

[0843] "Means for analyzing and identifying faces and behavior" refers to algorithms and software that analyze video data obtained from security cameras and recognize the faces of people in the footage and identify their behavioral patterns.

[0844] "Database matching" refers to the process of comparing analyzed facial and behavioral data with a pre-registered database to identify matching people and behaviors.

[0845] "Means for generating and notifying alerts" refers to a system that creates warning information when suspicious behavior or individuals are detected and notifies store staff and other relevant parties.

[0846] The "means for displaying alert information" refers to a device or interface for displaying the generated alert information in a form that can be easily confirmed by store staff and other relevant parties.

[0847] The "means for recording notification messages" refers to a system for saving the generated alerts and notification contents so that they can be referenced later.

[0848] "Emotion engine means for analyzing emotions" refers to algorithms or software that recognize and analyze a person's emotions based on the results of facial recognition, and identify emotional states such as anxiety or fear.

[0849] "Means for real-time notification" refers to a communication method that immediately notifies the store clerk's terminal of detected suspicious behavior or emotional state, urging immediate action.

[0850] The system of the present invention is intended to prevent shoplifting and detect suspicious behavior in a store, and is configured using the following hardware and software.

[0851] System Configuration

[0852] 1. Security cameras

[0853] These cameras are installed in stores to monitor customer movements in real time, making it possible to continuously acquire video data.

[0854] 2. Video data acquisition method

[0855] This method continuously acquires video data from security cameras, compresses it, and sends it to a server. The specific software used is OpenCV.

[0856] 3. Video data analysis methods

[0857] Using this method, the server analyzes the received video data, performs face recognition and behavior analysis routines, and uses a face recognition library (e.g., face_recognition) to extract face images and analyze them frame by frame.

[0858] 4. Emotion Engine

[0859] The server is equipped with a trained emotion recognition model that uses machine learning frameworks such as Keras to identify customer emotions based on facial recognition results.

[0860] 5. Database matching methods

[0861] The server checks the newly detected facial and emotional data against an existing database containing pre-existing images of suspect people to see if there is a match.

[0862] 6. Alert Generation and Notification Methods

[0863] Based on the match, the server generates an alert, which includes the person's facial image, clothing, location, behavioral details, and emotional state, and sends it to the store clerk's smartphone in real time using Twilio.

[0864] 7. Alert Information Display Method

[0865] The user (store clerk) receives the alert via an application installed on their smartphone and can check it in chat format, enabling a quick response in the store.

[0866] Specific examples

[0867] For example, suppose a store has security cameras monitoring the store 24 hours a day. The server continuously collects data from these cameras and analyzes the video data in real time. Specifically, it uses facial recognition technology to identify people in the video and check whether they are included in a database of suspicious individuals.

[0868] At the same time, the emotion engine analyzes the emotional data from the person's face and detects whether the person is expressing anxiety or impatience. In this case, the server generates an alert stating, "A person wearing a red jacket is lingering in front of a particular product for an abnormally long time and is displaying an anxious expression." This alert is sent to the store clerk's device, which displays detailed information in a chat format. Based on this information, the store clerk can rush to the scene, confirm the possibility of shoplifting, and take appropriate action.

[0869] Prompt Sentence Examples

[0870] "Your task is to generate code for an application that acquires video data from security cameras in real time, detects suspicious behavior using an emotion engine, analyzes customer emotions such as anxiety, anger, and fear, and notifies store staff on their smartphones."

[0871] This system makes it possible to efficiently and quickly monitor suspicious behavior and customer emotional states over a wide area within a store, and instantly notify store staff to promptly respond, thereby strengthening shoplifting prevention measures and improving security throughout the store.

[0872] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0873] Step 1:

[0874] A means for acquiring video data from security cameras acquires video data from security cameras in the store in real time. This data is continuously acquired and used for subsequent analysis steps. The input is raw video data and the output is compressed video data.

[0875] Step 2:

[0876] The device sends the video data acquired from the security camera to the server. The specific input is the compressed video data, and the output is the status of completion of transmission to the server. This process is performed via the network.

[0877] Step 3:

[0878] The server analyzes the received video data and performs facial recognition and behavior analysis. The input is the received video data, and the output is the recognized face image and behavior data. Specifically, OpenCV and the face_recognition library are used.

[0879] Step 4:

[0880] The server performs emotion analysis based on the facial recognition results. The input is the analyzed facial image, and the output is emotion data (e.g., anxiety, impatience, anger). The emotion engine uses Keras and a pre-trained neural network model.

[0881] Step 5:

[0882] The server compares the newly detected face and emotion data with an existing database. The input is face and emotion data, and the output is the matching result. The database contains pre-recorded images of suspicious people.

[0883] Step 6:

[0884] If the server detects an anomaly, it generates an alert. The input is the matching result and emotion data, and the output is alert information. The alert information includes the detected person's face image, clothing, location information, behavior details, and emotional state.

[0885] Step 7:

[0886] The server uses Twilio to send alert information to the store clerk's smartphone. The input is the alert information, and the output is a notification to the store clerk's terminal. This notification includes detailed information about the person.

[0887] Step 8:

[0888] The user (store clerk) checks the alert through an application installed on the device. The input is the received alert information, and the output is instructions for the store clerk to act. The store clerk checks the detailed information in chat format and responds promptly.

[0889] summary

[0890] This series of processes makes it possible to efficiently and quickly detect suspicious behavior and emotional states within a store, immediately notifying store staff and prompting them to take action, thereby strengthening shoplifting prevention measures and improving security throughout the store.

[0891] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0892] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0893] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0894] [Third embodiment]

[0895] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0896] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0897] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0898] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0899] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0900] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0901] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0902] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0903] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0904] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0905] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0906] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0907] The present invention relates to a surveillance system for preventing shoplifting and detecting suspicious behavior in a store. The system analyzes video data obtained from security cameras in real time to detect suspicious behavior or suspicious individuals. An alert is then sent to the store clerk's terminal, enabling a prompt response.

[0908] System configuration

[0909] 1. Video data acquisition method

[0910] The terminal acquires video data in real time from multiple security cameras installed within the store.

[0911] The terminal compresses the acquired video data and transmits it to the server.

[0912] 2. Video data analysis methods

[0913] The server analyzes the received video data and applies facial recognition and behavioral analysis algorithms.

[0914] The server performs facial recognition and extracts the detected facial images for each frame.

[0915] The server uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements.

[0916] 3. Database matching method

[0917] The server matches the newly detected face data with the existing database.

[0918] The server compares the analyzed behavioral data with past behavioral data to identify suspicious behavioral patterns.

[0919] 4. Alert Generation and Notification Methods

[0920] If the server identifies a person of interest or suspicious activity, it generates an alert.

[0921] The alert information includes a person's facial image, clothing, location information, and details of their behavior.

[0922] The server transmits the generated alert information to the store clerk's terminal and notifies the store clerk.

[0923] 5. Displaying alert information

[0924] The store clerk's device receives the alert and displays the alert information in chat format.

[0925] The store clerk's terminal screen displays the person's clothing, location, and behavior details in real time.

[0926] Program processing

[0927] 1. Device operation

[0928] The device continuously acquires video data from the store's security cameras and transmits it to the server.

[0929] The device acquires images from multiple cameras simultaneously, efficiently collecting data.

[0930] 2. Server Data Analysis

[0931] The server performs facial recognition on the received video data and applies a highly accurate facial recognition algorithm.

[0932] The server analyzes the behavior of people in the video and runs algorithms to detect abnormal behavior.

[0933] 3. Database Matching

[0934] The server compares the newly detected facial data with the database to identify facial data with high matching scores.

[0935] Based on the results of behavioral analysis, the server compares suspicious behavioral patterns with past data to identify potentially risky behavior.

[0936] 4. Alert generation and notification

[0937] If the server detects an abnormality, it will immediately generate an alert and create a notification message.

[0938] The alert message includes details of the suspicious person's facial image, clothing, location, and behavior.

[0939] 5. Assistance via store clerk's terminal

[0940] The user (store clerk) receives the alert and checks the information displayed in chat format.

[0941] The user goes to a specific location in the store and takes appropriate action against the detected suspicious person.

[0942] Specific examples

[0943] For example, imagine a store with security cameras monitoring the store 24 / 7. Devices continuously capture data from these cameras and send it to a server. The server uses facial recognition technology to identify people in the footage and check whether they are on a blacklist in a database. At the same time, the server applies behavioral analysis algorithms to detect if a person repeatedly behaves suspiciously in front of a particular product.

[0944] In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an abnormally long time." This alert is sent to the store clerk's device, which displays detailed information in chat format. Based on this information, the store clerk rushes to the scene, confirms the possibility of shoplifting, and takes appropriate action.

[0945] This system makes it possible to efficiently and quickly detect suspicious activity over a wide area within a store and immediately notify store staff, significantly strengthening shoplifting prevention and improving store security.

[0946] The processing flow will be explained below.

[0947] Step 1: Acquire video data

[0948] The terminal acquires video data in real time from security cameras installed inside the store.

[0949] The video data acquired by the terminal is compressed and sent to a server via the Internet.

[0950] Step 2: Receiving video data

[0951] The server receives the video data transmitted from the terminal.

[0952] The server temporarily stores the received video data.

[0953] Step 3: Performing facial recognition

[0954] The server applies a facial recognition algorithm to the received video data.

[0955] The server cuts out the detected face for each frame and generates face image data.

[0956] Step 4: Performing behavioral analysis

[0957] The server applies an algorithm to track the movements of each person in the video data.

[0958] The server detects certain behavioral patterns (e.g., repeatedly returning to the same location).

[0959] Step 5: Check against the database

[0960] The server matches the newly detected face data with the existing database.

[0961] The server checks whether there is a person in the database that matches the facial data.

[0962] Step 6: Evaluate the match results

[0963] Based on the matched results, the server evaluates whether there is a person of interest or suspicious activity.

[0964] If the server identifies suspicious activity, it records the details of that activity.

[0965] Step 7: Generate an alert

[0966] If the server identifies a person of interest or suspicious activity, it generates an alert.

[0967] The server includes details of the face image, clothing, location, and behavior in the alert information.

[0968] Step 8: Sending an alert

[0969] The server transmits the generated alert information to the store clerk's terminal.

[0970] The alert sent from the server is set to be delivered to multiple store clerk terminals simultaneously.

[0971] Step 9: Viewing Alerts

[0972] The alert information received by the store clerk's terminal is displayed in chat format.

[0973] The store clerk's terminal screen displays the person's clothing, location, and details of their behavior in real time.

[0974] Step 10: Staff response

[0975] The user (store clerk) checks the alert and refers to the displayed detailed information.

[0976] The user takes appropriate action and shares the information with other store staff as needed.

[0977] By using the above steps, the monitoring system of the present invention can efficiently and quickly detect suspicious behavior or suspicious individuals, and immediately issue an alert to prompt a response.

[0978] Example 1

[0979] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0980] From a security perspective, it is important to efficiently and quickly detect shoplifting and other suspicious behavior in stores and respond appropriately. However, current systems lack sufficient accuracy in analyzing video data and the speed of notification, resulting in the overlooking of suspicious behavior and delayed response. Other issues include the accuracy of the process of comparing suspect individuals and behavioral patterns with a database, and the efficiency of communicating information to store staff.

[0981] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0982] In this invention, the server includes means for acquiring video data of the monitored area, means for compressing the acquired video data and sending it to the server, means for analyzing the received video data and performing facial recognition and behavioral analysis, means for comparing the analyzed facial and behavioral data with a database, means for generating and notifying an alert based on the comparison results, and means for displaying the alert information. This makes it possible to detect suspicious behavior over a wide area within the store with high accuracy and immediately notify store staff.

[0983] "Surveillance Area" means a physical space that is subject to surveillance for a particular security purpose.

[0984] "Video data" refers to video image information collected by imaging equipment such as cameras installed in a monitored area.

[0985] "Compression" is the process of changing the format of data using a specific algorithm to reduce the volume of the data.

[0986] A "server" is a computer system used to provide a particular service and process data over a network.

[0987] "Facial recognition" is a technology that detects people's faces from video data and identifies individual faces using specific algorithms.

[0988] "Behavioral analysis" is a technology that analyzes the movements of people in video data and identifies specific behaviors and patterns.

[0989] A "database" is a system for storing, managing, and searching data in an organized manner.

[0990] "Matching" is the process of comparing the analyzed data with information from existing databases.

[0991] An "alert" is a warning message that is generated when a particular condition is met.

[0992] "Notification" is the act of sending generated alert information to a designated recipient.

[0993] "Means" refers to a method or device used to achieve a particular purpose.

[0994] "Display" is the act of visually showing information.

[0995] The present invention relates to a surveillance system aimed at preventing shoplifting and detecting suspicious behavior in stores. Specifically, the system analyzes video data obtained from security cameras in real time to detect suspicious behavior and suspicious individuals. Furthermore, based on the results, the system sends alerts to store clerk terminals to prompt prompt action.

[0996] Hardware and Software Configuration

[0997] Acquiring video data of the monitored area

[0998] The device acquires video data in real time from multiple security cameras (e.g., IP cameras) installed in the store. The device temporarily stores the video from the cameras in a buffer for easy access.

[0999] Video data compression and transmission

[1000] The video data acquired by the device is compressed and sent to the server using an efficient video compression format such as H.264. The compressed data is sent via the network to the specified server address.

[1001] Video data analysis

[1002] The server starts analyzing the received video data. It applies a high-precision facial recognition algorithm (e.g., OpenCV or Dlib) to detect faces in the video. The detected faces are extracted from each frame and saved as individual image data. In parallel, the server uses a behavior analysis algorithm (e.g., an RNN- or LSTM-based model) to analyze the behavior of people in the video.

[1003] Database Matching

[1004] The server compares the newly detected face data with an existing database (e.g., an SQL database) by calculating the distance between the vectors of facial features. Similarly, behavioral data is compared with past behavioral patterns.

[1005] Alert generation and notification

[1006] If the server identifies a suspicious person or suspicious behavior, it generates an alert, which includes details of the detected face, clothing, location, and behavior. The alert message is generated in real time and pushed to the store clerk's device.

[1007] Viewing Alerts

[1008] When the device receives an alert, the alert information is displayed in chat format, allowing store staff to check the notification in real time on the device screen. The alert information includes details of the person's clothing, location, and behavior.

[1009] Staff response

[1010] The user (store clerk) receives the alert and checks the displayed information. The clerk then promptly takes action based on the alert. For example, they go to the specific location where the suspicious behavior was reported and take appropriate action against the suspicious person.

[1011] Specific examples

[1012] For example, suppose a store is monitored 24 hours a day by security cameras. A device acquires video data from multiple cameras in the store and sends it to a server. The server analyzes the received data and performs highly accurate facial recognition and behavioral analysis. This makes it possible to check whether a specific person is included in the blacklist of individuals in the past database.

[1013] At the same time, the server applies a behavioral analysis algorithm to detect suspicious behavior in front of a specific product. Based on this, the server generates an alert such as "A man wearing a red jacket is lingering in front of a specific product for an unusually long time." This alert is sent to the store clerk's device, and detailed information is displayed in chat format.

[1014] Using this information, store staff can rush to the scene, confirm the possibility of shoplifting, and take appropriate action. This system will enable the efficient and rapid detection of suspicious activity across a wide area of ​​the store, significantly strengthening the prevention of shoplifting.

[1015] Prompt Sentence Examples

[1016] 1. A store clerk can detect someone spending an abnormally long time at a particular shelf.

[1017] 2. The server detects suspicious behavior and sends an alert to the store clerk's terminal.

[1018] 3. The store staff checks the alert and takes appropriate action.

[1019] In this way, the present invention provides a system that efficiently and quickly strengthens security within a store and contributes to preventing fraudulent activities such as shoplifting.

[1020] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1021] Step 1: Acquire video data

[1022] The device acquires video data in real time from multiple security cameras installed in the store. Specifically, the video captured by the cameras is temporarily stored in a buffer, enabling continuous data collection.

[1023] Input: Real-time video data from security cameras.

[1024] Output: Uncompressed video data stored in a buffer.

[1025] Step 2: Compress and send the data

[1026] The video data acquired by the device is compressed and sent to the server using an efficient video compression format such as H.264, and the data is sent to the server via the network.

[1027] Input: Uncompressed video data.

[1028] Output: Compressed video data.

[1029] Step 3: Analyzing the video data

[1030] The server analyzes the received compressed video data. First, a face recognition algorithm (e.g., OpenCV or Dlib) is applied to detect faces in the video. The detected faces are extracted frame by frame and saved as individual image data. At the same time, a behavior analysis algorithm (e.g., an RNN or LSTM-based model) is applied to analyze the person's behavior.

[1031] Input: Compressed video data.

[1032] Output: Detected face images and behavior analysis results.

[1033] Step 4: Check against the database

[1034] The server matches newly detected facial data with an existing database (e.g., an SQL database), calculates the distance between facial feature vectors, and identifies potential matches of suspicious individuals. Behavioral data is similarly compared with past behavioral patterns.

[1035] Input: Detected face images and behavior analysis results.

[1036] Output: Matching person of interest information and suspicious behavior identification.

[1037] Step 5: Generate an alert

[1038] If the server identifies a suspicious person or behavior, it generates an alert, combining detected facial images, clothing, location information, and behavior details to create an alert message.

[1039] Input: Matching person of interest information and suspicious activity identification results.

[1040] Output: The alert message.

[1041] Step 6: Notification and display of alerts

[1042] The server sends the generated alert information to the store clerk's terminal, which displays the alert information in a pop-up or chat window format so that the store clerk can check it immediately.

[1043] Input: Alert message.

[1044] Output: Alert information displayed on the terminal.

[1045] Step 7: Staff response

[1046] The user (store clerk) receives the alert and checks the displayed information. The clerk follows the instructions in the alert to go to the specific location and deal with the suspicious person.

[1047] Input: The alert information displayed on the terminal.

[1048] Output: On-site response actions.

[1049] (Application example 1)

[1050] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1051] While conventional systems have the ability to analyze security camera footage to detect suspicious behavior and suspicious individuals, they have limited means of providing information to store staff so that they can respond quickly.In addition, there is a lack of ways for store staff patrolling the store to identify suspicious behavior in real time, making it difficult to respond quickly.

[1052] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1053] In this invention, the server includes means for acquiring video data of the store, means for analyzing the acquired video data and identifying faces and behaviors, means for comparing the analyzed faces and behaviors with a database on the server, means for generating and notifying an alert based on the comparison results, and means for displaying the alert information on the smart glasses. This enables store staff wearing the smart glasses to identify suspicious behavior in real time while patrolling and respond quickly.

[1054] "In-store video data" refers to images and video information obtained in real time from security cameras installed within a store.

[1055] The "acquisition means" refers to hardware and software components for receiving video data output from a security camera and storing or transmitting it.

[1056] "Means for analyzing and identifying faces and behaviors" refers to facial recognition algorithms and behavior analysis algorithms for identifying people's faces and analyzing their behaviors based on the captured video data.

[1057] The "database on the server" is a data management system used to store analyzed facial and behavioral data and compare it with past data.

[1058] The "means of matching" is an algorithm that compares facial and behavioral data acquired in real time with existing data in a database on the server to confirm a match.

[1059] "Means for generating and notifying alerts" refers to a system that creates an alert when suspicious behavior or suspicious individuals are detected and immediately notifies relevant parties of that information.

[1060] The "means for displaying alert information on smart glasses" refers to communication and display technology for displaying the generated alert information on the display of smart glasses worn by the store clerk.

[1061] This invention is a surveillance system aimed at preventing shoplifting in a store and detecting suspicious individuals at an early stage. This system acquires video data from multiple security cameras in real time, analyzes the data, detects suspicious behavior and individuals of interest, and notifies store staff.

[1062] Hardware configuration:

[1063] Security cameras: These are installed at strategic locations within the store and are used to capture footage in real time.

[1064] Server: A computer with powerful computing power for receiving and analyzing video data.

[1065] Smart glasses: worn by store associates and used to receive and display alert information in real time.

[1066] Software configuration:

[1067] OpenCV: A library for acquiring and processing security camera footage in real time.

[1068] Keras: A deep learning framework used to implement face recognition and behavior analysis algorithms.

[1069] Flask: A lightweight web framework for receiving and analyzing video data on the server side.

[1070] Requests: An HTTP library for data communication between smart glasses and a server.

[1071] Overview of program processing:

[1072] The device continuously captures video data from security cameras in the store in real time and sends it to a server. The server then applies facial recognition and behavioral analysis algorithms to the received video data to analyze the behavioral patterns of specific individuals. The analysis results are compared with existing data in a database, and an alert is generated if a match is found or if abnormal behavior is detected. The generated alert is then sent to the smart glasses, which notify the store staff in real time.

[1073] Examples:

[1074] For example, consider a store where security cameras are monitoring the shop 24 hours a day. The server receives this video data in real time and uses a facial recognition algorithm to identify individuals. It then checks whether the identified individuals are on a blacklist in the database. At the same time, the server applies a behavioral analysis algorithm to detect that an individual is repeatedly lingering in front of a particular product for an abnormally long time. In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an abnormally long time." This alert is sent to smart glasses, and the information is displayed in chat format on the smart glasses worn by the store clerk. Based on this information, the store clerk can head to the scene, confirm the possibility of shoplifting, and take appropriate action.

[1075] Example prompt sentence:

[1076] "The system analyzes security camera footage within stores in real time and generates an alert if it determines something is suspicious. The alert includes details of the face, clothing, location, and behavior. This alert is displayed on the smart glasses, enabling a prompt response."

[1077] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1078] Step 1:

[1079] Device operation

[1080] The device acquires video data from security cameras inside the store in real time.

[1081] Input: Video data from a security camera.

[1082] Data processing: The terminal compresses the acquired video data and converts it into a format that can be sent to the server (for example, JPEG format).

[1083] Output: Compressed video data.

[1084] Specific operation: The device continuously acquires data from multiple security cameras and transmits it to the server in real time.

[1085] Step 2:

[1086] Server data reception

[1087] A server receives the compressed video data.

[1088] Input: Compressed video data sent from the device.

[1089] Data processing: The server decompresses the compressed video data and converts it into a format that can be used as the original video data.

[1090] Output: Decompressed video data.

[1091] Specific operation: The server sequentially decompresses the received video data and prepares it for subsequent analysis processing.

[1092] Step 3:

[1093] Facial Recognition and Behavioral Analysis

[1094] The server analyzes the decompressed video data and applies facial recognition and behavioral analysis algorithms.

[1095] Input: Decompressed video data.

[1096] Data computation: The server uses a facial recognition algorithm (e.g., a Keras model) to identify faces in the video and applies a behavior analysis algorithm to analyze people's behavior.

[1097] Output: Analyzed face and behavioral data.

[1098] Specific operation: The server extracts people's faces from each video frame and analyzes their behavioral patterns.

[1099] Step 4:

[1100] Database collation

[1101] The server compares the analyzed facial and behavioral data with a database on the server.

[1102] Input: Parsed face and behavioral data.

[1103] Data calculation: The server compares the data with data in an existing database to identify facial data and suspicious behavior patterns with high matching scores.

[1104] Output: Matching results (face data and behavioral patterns with high matching scores).

[1105] Specific operation: The server uses a matching algorithm to match the blacklist in the database and identify matching data.

[1106] Step 5:

[1107] Generate alerts

[1108] The server generates an alert based on the match result.

[1109] Input: Matching results (face data and behavioral patterns with high matching scores).

[1110] Data processing: The server generates alert information (a person's face image, clothing, location information, and details of their behavior).

[1111] Output: Alert information.

[1112] What it does: The server immediately creates an alert when a suspicious person or activity is detected.

[1113] Step 6:

[1114] Sending and Viewing Alerts

[1115] The server transmits the generated alert information to the smart glasses for display.

[1116] Input: Alert information.

[1117] Data processing: The server converts the alert information into a format suitable for the smart glasses and sends it.

[1118] Output: Alert information displayed on smart glasses.

[1119] Specific operation: The server composes a notification message and sends it to the smart glasses, which receive the notification and the store clerk confirms it.

[1120] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1121] This invention relates to a surveillance system aimed at preventing shoplifting and detecting suspicious behavior in stores. This system analyzes video data obtained from security cameras in real time, and performs facial recognition and behavior analysis. In addition, by combining it with an emotion engine that recognizes user emotions, it enables more accurate detection of suspicious behavior and quicker response.

[1122] System configuration

[1123] 1. Video data acquisition method

[1124] The terminal acquires video data in real time from security cameras installed inside the store.

[1125] The video data acquired by the terminal is compressed and sent to a server via the Internet.

[1126] 2. Video data analysis methods

[1127] The server analyzes the received video data and applies facial recognition and behavioral analysis algorithms.

[1128] The server performs facial recognition and extracts the detected facial images for each frame.

[1129] The server uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements.

[1130] 3. Emotion Engine

[1131] The emotion engine built into the server identifies the user's emotion based on the facial image detected during face recognition.

[1132] The emotion engine generates user emotion data (e.g., impatience, anger, anxiety) and reflects it in the behavioral analysis results.

[1133] 4. Database matching method

[1134] The server matches the newly detected facial and emotion data with the existing database.

[1135] The server checks whether there is a person in the database that matches the facial data.

[1136] 5. Evaluation of matching results

[1137] Based on the matched results, the server evaluates whether there is a person of interest or suspicious activity.

[1138] If the server identifies suspicious behavior, it records details of the behavior and emotions.

[1139] 6. Alert Generation and Notification Methods

[1140] If the server identifies a person of interest or suspicious activity, it generates an alert.

[1141] The alert information includes a person's facial image, clothing, location information, behavior details, and emotional state.

[1142] The server transmits the generated alert information to the store clerk's terminal and notifies the store clerk.

[1143] 7. How to display alert information

[1144] The store clerk's device receives the alert and displays the alert information in chat format.

[1145] The store clerk's terminal screen displays the person's clothing, location, behavioral details, and emotional state in real time.

[1146] Program processing

[1147] 1. Device operation

[1148] The device continuously acquires video data from the store's security cameras and transmits it to the server.

[1149] The device acquires images from multiple cameras simultaneously, efficiently collecting data.

[1150] 2. Server Data Analysis

[1151] The server performs facial recognition on the received video data and applies a highly accurate facial recognition algorithm.

[1152] The server analyzes the behavior of people in the video and runs algorithms to detect abnormal behavior.

[1153] 3. Emotion recognition

[1154] The server's emotion engine analyzes the user's emotions based on the facial data and generates emotion data.

[1155] The server evaluates this emotional data in combination with the results of behavioral analysis.

[1156] 4. Database Matching

[1157] The server matches the newly detected facial and emotion data against a database to identify matches.

[1158] The server performs a risk assessment based on the facial data and emotional state that match closely.

[1159] 5. Alert generation and notification

[1160] If the server detects an abnormality, it immediately generates an alert and creates a notification message.

[1161] The alert message includes a facial image of the suspicious person, their clothing, location, and details of their behavior and emotions.

[1162] 6. Assistance via store clerk's terminal

[1163] The user (store clerk) receives the alert and checks the information displayed in chat format.

[1164] The user goes to a specific location in the store and takes appropriate action against the detected suspicious person.

[1165] Specific examples

[1166] For example, imagine a store with security cameras monitoring the store 24 hours a day. A device continuously captures data from these cameras and sends it to a server. The server uses facial recognition technology to identify people in the footage and check whether they are on a blacklist in a database. At the same time, an emotion engine analyzes the emotional data from the person's face to detect whether they are expressing anxiety or impatience.

[1167] In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an unusually long time and appears anxious." This alert is sent to the store clerk's device, which displays detailed information in a chat format. Based on this information, the store clerk rushes to the scene, confirms the possibility of shoplifting, and takes appropriate action.

[1168] This system makes it possible to efficiently and quickly monitor suspicious behavior and emotional states over a wide area within a store, and immediately notify store staff to prompt them to take action, thereby significantly strengthening shoplifting prevention and improving store security.

[1169] The processing flow will be explained below.

[1170] Step 1: Acquire video data

[1171] The terminal acquires video data in real time from security cameras installed inside the store.

[1172] The video data acquired by the terminal is compressed and sent to a server via the Internet.

[1173] Step 2: Receiving and saving video data

[1174] The server receives the video data transmitted from the terminal.

[1175] The video data received by the server is temporarily stored in a storage system.

[1176] Step 3: Performing facial recognition

[1177] The server analyzes the video data stored in the storage system and applies a facial recognition algorithm.

[1178] The server cuts out the detected face for each frame and generates face image data.

[1179] Step 4: Performing behavioral analysis

[1180] The server runs an algorithm that tracks the movements of each person in the video data.

[1181] The server detects specific behavioral patterns (e.g., staying in the same place for a certain period of time or going back and forth to the same place multiple times).

[1182] Step 5: Performing Emotion Recognition

[1183] The server applies an emotion engine based on the facial image data to analyze the user's emotions.

[1184] The server generates user emotion data and evaluates it in combination with behavioral data.

[1185] Step 6: Check against the database

[1186] The server compares the newly detected facial and emotion data with a pre-registered database.

[1187] The server identifies matches to the facial and emotion data in the database.

[1188] Step 7: Evaluate the match results

[1189] The server evaluates suspicious behavior and people of interest based on the matching results.

[1190] The server performs a risk assessment based on the detected behavior and emotions and determines the severity of the alert.

[1191] Step 8: Generate an alert

[1192] If the server identifies a person of interest or suspicious activity, it generates an alert.

[1193] The alert information includes facial image, clothing, location information, behavioral details, and emotional state.

[1194] Step 9: Sending alerts

[1195] The server transmits the generated alert information to the store clerk's terminal.

[1196] The server is set so that alert information is sent to multiple store clerk terminals simultaneously.

[1197] Step 10: Viewing Alerts

[1198] The alert information received by the store clerk's terminal is displayed in chat format.

[1199] The employee's device screen displays a person's facial image, clothing, location, behavioral details, and emotional state in real time.

[1200] Step 11: Staff response

[1201] The user (store clerk) checks the alert and rushes to the scene based on the detailed information displayed.

[1202] The user takes appropriate action against the detected suspicious person.

[1203] The information confirmed by the user is shared with other store staff, and monitoring is strengthened in collaboration.

[1204] By implementing the above steps, the monitoring system of the present invention can efficiently and quickly detect suspicious behavior and suspicious individuals, and immediately issue alerts to prompt responses. Furthermore, by combining it with an emotion engine, advanced risk assessment that takes into account changes in emotions becomes possible.

[1205] Example 2

[1206] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1207] Conventional security systems have had difficulty detecting suspicious behavior in stores in real time and responding quickly. Furthermore, conventional systems have had problems with low accuracy in facial recognition and behavioral analysis, resulting in frequent false positives and missed detections. Furthermore, the method of notifying store staff was inefficient, limiting effective action in situations where immediate action was required.

[1208] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring data within the store, a means for processing the acquired data to identify personal information and behavioral patterns of people, and a means for comparing the identified personal information and behavioral patterns with recording media. This makes it possible to perform highly accurate face recognition and behavioral analysis, detect suspicious behavior or suspicious people in real time, and quickly notify store staff. In addition, by displaying warning information in an interactive format, store staff can immediately understand the situation and take appropriate action.

[1209] "In-store" refers to the space inside a building such as a commercial facility or retail store.

[1210] "Data" refers to a collection of information such as video, images, audio, and sensor information.

[1211] "Means of acquisition" refers to methods of collecting data using devices such as cameras and sensors.

[1212] "Processing" refers to performing operations such as analysis and conversion on acquired data.

[1213] "Person" refers to the human being who is the subject of surveillance.

[1214] "Personal information" refers to information that can identify a specific individual, such as a name or facial image.

[1215] "Behavioral patterns" refer to the characteristics of a person's movements or actions at a particular time.

[1216] "Means of identification" refers to methods that use facial recognition and behavioral analysis algorithms to distinguish a person's characteristics and behavior.

[1217] "Recording media" refers to storage or databases for saving information.

[1218] "Matching" refers to comparing detected information with existing data to determine whether it matches.

[1219] "Warning" refers to a notification or alert that occurs when an abnormality is detected.

[1220] "Interactive" refers to the way information is presented in the form of chats and messages.

[1221] This invention relates to a surveillance system that strengthens security in stores, prevents shoplifting, and detects suspicious behavior. Specifically, this system analyzes video data acquired from security cameras installed in stores in real time to perform facial recognition and behavior analysis.

[1222] Hardware and Software Configuration

[1223] This system consists of the following hardware and software:

[1224] 1. Device:

[1225] Security cameras: Multiple cameras are installed within the store and continuously capture footage.

[1226] Network connection: A high-speed internet connection to transmit captured video data to the server.

[1227] 2. Server:

[1228] Video analytics software: Software for running high-precision facial recognition algorithms (e.g., OpenCV, Dlib, etc.) and behavioral analysis algorithms (e.g., YOLO, OpenPose, etc.).

[1229] Database: A recording medium for storing existing person of interest data and emotion data.

[1230] Emotion engine: A software module for analyzing user emotions.

[1231] Program processing

[1232] The overall system operates as follows.

[1233] 1. Device operation

[1234] The terminal continuously captures video data from security cameras installed in the store, efficiently compresses it, and sends it to the server. Data compression is particularly important when handling high-resolution video data.

[1235] 2. Server Data Analysis

[1236] The server applies a facial recognition algorithm to the received video data to extract facial images of people for each frame, and then uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements.

[1237] 3. Emotion recognition

[1238] The server's emotion engine analyzes the user's emotions based on the facial data and generates emotion data, which is then combined with the results of behavioral analysis to determine the overall risk level.

[1239] 4. Database Matching

[1240] The server then compares the newly detected facial and emotional data against existing databases to identify matches, specifically against historical blacklists and emotional databases.

[1241] 5. Alert generation and notification

[1242] If the server detects an anomaly, it immediately generates an alert and creates a notification message. The alert includes details of the suspicious person's face, clothing, location, behavior, and emotions. This information is sent to the store clerk's terminal and displayed interactively.

[1243] 6. Staff Service

[1244] The user (store clerk) receives an alert on their device, checks the detailed alert information in chat format, rushes to the scene, and takes appropriate action against the detected suspicious person.

[1245] Specific examples

[1246] For example, consider a store where security cameras monitor the store 24 hours a day. Devices continuously capture data from these cameras and send it to a server. The server uses facial recognition technology to identify individuals in the footage and check whether they are on a database of suspicious individuals. At the same time, an emotion engine analyzes the emotional data from the individual's face and detects whether they are expressing anxiety or impatience. In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an unusually long time and appears anxious." This alert is then sent to a store employee's device, which displays detailed information in a chat format. Based on this information, the employee rushes to the scene, confirms the possibility of shoplifting, and takes appropriate action. This system allows for efficient and rapid monitoring of suspicious behavior and emotional states across a wide area of ​​the store, and immediately notifies employees to prompt action. This significantly strengthens shoplifting prevention and improves store security.

[1247] Prompt Sentence Examples

[1248] Examples of prompts to be input to a generative AI model include:

[1249] "How can we create a system that sends notifications when security camera footage shows signs of shoplifting?"

[1250] "How can we combine facial recognition and behavioral analysis to detect suspicious behavior in stores?"

[1251] "Describe an algorithm that uses an emotion engine to detect emotions like anxiety and anger to identify suspicious behavior in real time."

[1252] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1253] Step 1:

[1254] The terminal acquires video data in real time from security cameras installed inside the store. Specifically, each security camera captures video frame by frame and sends it to the terminal. The input is the video data acquired from the security camera, and the output is compressed video data. The terminal performs this continuously to efficiently collect data.

[1255] Step 2:

[1256] The video data acquired by the device is compressed and sent to a server via the Internet. Specifically, the video data is encoded using a compression algorithm such as H.264 to reduce the data volume. The input is raw video data from the security camera, and the output is compressed video data. The compressed data is sent to the server.

[1257] Step 3:

[1258] The server applies a facial recognition algorithm to the video data it receives. Specifically, it uses libraries such as OpenCV and Dlib to detect human faces in each frame and extract facial images. The input is compressed video data, and the output is a list of detected facial images. The server performs this process continuously, achieving highly accurate facial recognition.

[1259] Step 4:

[1260] The server uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements. Specific algorithms, such as YOLO and OpenPose, are used to identify the person's movements and location. The input is frame-by-frame video data, and the output is analyzed behavioral data. The server specializes in detecting unusual behavioral patterns.

[1261] Step 5:

[1262] The server's emotion engine analyzes the user's emotions based on the facial data and generates emotion data. Specifically, it uses an algorithm that analyzes facial expressions to identify the user's psychological state. The input is the detected facial image, and the output is emotion data (e.g., impatience, anger, anxiety). The server then combines this with the results of behavioral analysis and evaluates it.

[1263] Step 6:

[1264] The server compares newly detected facial and emotional data against an existing database to identify matches. Specifically, it uses an algorithm that directly compares new data with the existing database. The input is the newly analyzed facial and emotional data, and the output is whether or not there are any matching records. The server then performs a risk assessment.

[1265] Step 7:

[1266] If the server detects an anomaly, it immediately generates an alert and creates a notification message. Specifically, it runs a program that automatically generates an alert message after detecting an anomaly. The input is the anomaly detection result data, and the output is an alert message. The alert includes details of the suspicious person's face image, clothing, location, behavior, and emotions.

[1267] Step 8:

[1268] After the alert information generated by the server is sent to the store clerk's terminal, the user (store clerk) receives the alert on the terminal and checks the information displayed in chat format. The input is the alert message from the server, and the output is the alert information displayed on the terminal. The store clerk checks this and rushes to the scene depending on the situation.

[1269] Step 9:

[1270] The user (store clerk) rushes to the scene based on the alert information and takes appropriate action against the detected suspicious person. Specific actions include actually checking the situation and, if necessary, coordinating with security guards. The input is detailed information displayed in chat format, and the output is the response results.

[1271] (Application example 2)

[1272] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1273] Conventional suspicious behavior detection systems in stores rely solely on simple behavioral analysis, making them insufficient for detecting shoplifting and other suspicious behavior. Furthermore, simple video analysis cannot capture detailed behavior or psychological states, making it difficult for store staff to take appropriate action on the spot. Furthermore, notification functions are often inadequate, resulting in delayed real-time responses. For these reasons, there was a demand for a system that could provide more accurate and rapid detection of suspicious behavior and countermeasures.

[1274] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring in-store video data, means for analyzing the acquired video data and identifying faces and behaviors, means for comparing the identified faces and behaviors with a database, means for generating and notifying an alert based on the comparison results, means for displaying alert information, means for recording the notification message, emotion engine means for analyzing emotions based on the face recognition results, and means for sending real-time notifications to the terminal. This combines face recognition and emotion analysis to enable advanced and precise detection of suspicious behavior, allowing store staff to respond quickly in real time.

[1275] "In-store video data" refers to real-time video information captured by video capture devices such as security cameras installed in the store.

[1276] "Means of acquisition" refers to the devices and methods used to collect video data from security cameras and other sources and input it into the system.

[1277] "Means for analyzing and identifying faces and behavior" refers to algorithms and software that analyze video data obtained from security cameras and recognize the faces of people in the footage and identify their behavioral patterns.

[1278] "Database matching" refers to the process of comparing analyzed facial and behavioral data with a pre-registered database to identify matching people and behaviors.

[1279] "Means for generating and notifying alerts" refers to a system that creates warning information when suspicious behavior or individuals are detected and notifies store staff and other relevant parties.

[1280] The "means for displaying alert information" refers to a device or interface for displaying the generated alert information in a form that can be easily confirmed by store staff and other relevant parties.

[1281] The "means for recording notification messages" refers to a system for saving the generated alerts and notification contents so that they can be referenced later.

[1282] "Emotion engine means for analyzing emotions" refers to algorithms or software that recognize and analyze a person's emotions based on the results of facial recognition, and identify emotional states such as anxiety or fear.

[1283] "Means for real-time notification" refers to a communication method that immediately notifies the store clerk's terminal of detected suspicious behavior or emotional state, urging immediate action.

[1284] The system of the present invention is intended to prevent shoplifting and detect suspicious behavior in a store, and is configured using the following hardware and software.

[1285] System Configuration

[1286] 1. Security cameras

[1287] These cameras are installed in stores to monitor customer movements in real time, making it possible to continuously acquire video data.

[1288] 2. Video data acquisition method

[1289] This method continuously acquires video data from security cameras, compresses it, and sends it to a server. The specific software used is OpenCV.

[1290] 3. Video data analysis methods

[1291] Using this method, the server analyzes the received video data, performs face recognition and behavior analysis routines, and uses a face recognition library (e.g., face_recognition) to extract face images and analyze them frame by frame.

[1292] 4. Emotion Engine

[1293] The server is equipped with a trained emotion recognition model that uses machine learning frameworks such as Keras to identify customer emotions based on facial recognition results.

[1294] 5. Database matching methods

[1295] The server checks the newly detected facial and emotional data against an existing database containing pre-existing images of suspect people to see if there is a match.

[1296] 6. Alert Generation and Notification Methods

[1297] Based on the match, the server generates an alert, which includes the person's facial image, clothing, location, behavioral details, and emotional state, and sends it to the store clerk's smartphone in real time using Twilio.

[1298] 7. Alert Information Display Method

[1299] The user (store clerk) receives the alert via an application installed on their smartphone and can check it in chat format, enabling a quick response in the store.

[1300] Specific examples

[1301] For example, suppose a store has security cameras monitoring the store 24 hours a day. The server continuously collects data from these cameras and analyzes the video data in real time. Specifically, it uses facial recognition technology to identify people in the video and check whether they are included in a database of suspicious individuals.

[1302] At the same time, the emotion engine analyzes the emotional data from the person's face and detects whether the person is expressing anxiety or impatience. In this case, the server generates an alert stating, "A person wearing a red jacket is lingering in front of a particular product for an abnormally long time and is displaying an anxious expression." This alert is sent to the store clerk's device, which displays detailed information in a chat format. Based on this information, the store clerk can rush to the scene, confirm the possibility of shoplifting, and take appropriate action.

[1303] Prompt Sentence Examples

[1304] "Your task is to generate code for an application that acquires video data from security cameras in real time, detects suspicious behavior using an emotion engine, analyzes customer emotions such as anxiety, anger, and fear, and notifies store staff on their smartphones."

[1305] This system makes it possible to efficiently and quickly monitor suspicious behavior and customer emotional states over a wide area within a store, and instantly notify store staff to promptly respond, thereby strengthening shoplifting prevention measures and improving security throughout the store.

[1306] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1307] Step 1:

[1308] A means for acquiring video data from security cameras acquires video data from security cameras in the store in real time. This data is continuously acquired and used for subsequent analysis steps. The input is raw video data and the output is compressed video data.

[1309] Step 2:

[1310] The device sends the video data acquired from the security camera to the server. The specific input is the compressed video data, and the output is the status of completion of transmission to the server. This process is performed via the network.

[1311] Step 3:

[1312] The server analyzes the received video data and performs facial recognition and behavior analysis. The input is the received video data, and the output is the recognized face image and behavior data. Specifically, OpenCV and the face_recognition library are used.

[1313] Step 4:

[1314] The server performs emotion analysis based on the facial recognition results. The input is the analyzed facial image, and the output is emotion data (e.g., anxiety, impatience, anger). The emotion engine uses Keras and a pre-trained neural network model.

[1315] Step 5:

[1316] The server compares the newly detected face and emotion data with an existing database. The input is face and emotion data, and the output is the matching result. The database contains pre-recorded images of suspicious people.

[1317] Step 6:

[1318] If the server detects an anomaly, it generates an alert. The input is the matching result and emotion data, and the output is alert information. The alert information includes the detected person's face image, clothing, location information, behavior details, and emotional state.

[1319] Step 7:

[1320] The server uses Twilio to send alert information to the store clerk's smartphone. The input is the alert information, and the output is a notification to the store clerk's terminal. This notification includes detailed information about the person.

[1321] Step 8:

[1322] The user (store clerk) checks the alert through an application installed on the device. The input is the received alert information, and the output is instructions for the store clerk to act. The store clerk checks the detailed information in chat format and responds promptly.

[1323] summary

[1324] This series of processes makes it possible to efficiently and quickly detect suspicious behavior and emotional states within a store, immediately notifying store staff and prompting them to take action, thereby strengthening shoplifting prevention measures and improving security throughout the store.

[1325] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1326] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1327] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1328] [Fourth embodiment]

[1329] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1330] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1331] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1332] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1333] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1334] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1335] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1336] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1337] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1338] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1339] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1340] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1341] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1342] The present invention relates to a surveillance system for preventing shoplifting and detecting suspicious behavior in a store. The system analyzes video data obtained from security cameras in real time to detect suspicious behavior or suspicious individuals. An alert is then sent to the store clerk's terminal, enabling a prompt response.

[1343] System configuration

[1344] 1. Video data acquisition method

[1345] The terminal acquires video data in real time from multiple security cameras installed within the store.

[1346] The terminal compresses the acquired video data and transmits it to the server.

[1347] 2. Video data analysis methods

[1348] The server analyzes the received video data and applies facial recognition and behavioral analysis algorithms.

[1349] The server performs facial recognition and extracts the detected facial images for each frame.

[1350] The server uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements.

[1351] 3. Database matching method

[1352] The server matches the newly detected face data with the existing database.

[1353] The server compares the analyzed behavioral data with past behavioral data to identify suspicious behavioral patterns.

[1354] 4. Alert Generation and Notification Methods

[1355] If the server identifies a person of interest or suspicious activity, it generates an alert.

[1356] The alert information includes a person's facial image, clothing, location information, and details of their behavior.

[1357] The server transmits the generated alert information to the store clerk's terminal and notifies the store clerk.

[1358] 5. Displaying alert information

[1359] The store clerk's device receives the alert and displays the alert information in chat format.

[1360] The store clerk's terminal screen displays the person's clothing, location, and behavior details in real time.

[1361] Program processing

[1362] 1. Device operation

[1363] The device continuously acquires video data from the store's security cameras and transmits it to the server.

[1364] The device acquires images from multiple cameras simultaneously, efficiently collecting data.

[1365] 2. Server Data Analysis

[1366] The server performs facial recognition on the received video data and applies a highly accurate facial recognition algorithm.

[1367] The server analyzes the behavior of people in the video and runs algorithms to detect abnormal behavior.

[1368] 3. Database Matching

[1369] The server compares the newly detected facial data with the database to identify facial data with high matching scores.

[1370] Based on the results of behavioral analysis, the server compares suspicious behavioral patterns with past data to identify potentially risky behavior.

[1371] 4. Alert generation and notification

[1372] If the server detects an abnormality, it will immediately generate an alert and create a notification message.

[1373] The alert message includes details of the suspicious person's facial image, clothing, location, and behavior.

[1374] 5. Assistance via store clerk's terminal

[1375] The user (store clerk) receives the alert and checks the information displayed in chat format.

[1376] The user goes to a specific location in the store and takes appropriate action against the detected suspicious person.

[1377] Specific examples

[1378] For example, imagine a store with security cameras monitoring the store 24 / 7. Devices continuously capture data from these cameras and send it to a server. The server uses facial recognition technology to identify people in the footage and check whether they are on a blacklist in a database. At the same time, the server applies behavioral analysis algorithms to detect if a person repeatedly behaves suspiciously in front of a particular product.

[1379] In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an abnormally long time." This alert is sent to the store clerk's device, which displays detailed information in chat format. Based on this information, the store clerk rushes to the scene, confirms the possibility of shoplifting, and takes appropriate action.

[1380] This system makes it possible to efficiently and quickly detect suspicious activity over a wide area within a store and immediately notify store staff, significantly strengthening shoplifting prevention and improving store security.

[1381] The processing flow will be explained below.

[1382] Step 1: Acquire video data

[1383] The terminal acquires video data in real time from security cameras installed inside the store.

[1384] The video data acquired by the terminal is compressed and sent to a server via the Internet.

[1385] Step 2: Receiving video data

[1386] The server receives the video data transmitted from the terminal.

[1387] The server temporarily stores the received video data.

[1388] Step 3: Performing facial recognition

[1389] The server applies a facial recognition algorithm to the received video data.

[1390] The server cuts out the detected face for each frame and generates face image data.

[1391] Step 4: Performing behavioral analysis

[1392] The server applies an algorithm to track the movements of each person in the video data.

[1393] The server detects certain behavioral patterns (e.g., repeatedly returning to the same location).

[1394] Step 5: Check against the database

[1395] The server matches the newly detected face data with the existing database.

[1396] The server checks whether there is a person in the database that matches the facial data.

[1397] Step 6: Evaluate the match results

[1398] Based on the matched results, the server evaluates whether there is a person of interest or suspicious activity.

[1399] If the server identifies suspicious activity, it records the details of that activity.

[1400] Step 7: Generate an alert

[1401] If the server identifies a person of interest or suspicious activity, it generates an alert.

[1402] The server includes details of the face image, clothing, location, and behavior in the alert information.

[1403] Step 8: Sending an alert

[1404] The server transmits the generated alert information to the store clerk's terminal.

[1405] The alert sent from the server is set to be delivered to multiple store clerk terminals simultaneously.

[1406] Step 9: Viewing Alerts

[1407] The alert information received by the store clerk's terminal is displayed in chat format.

[1408] The store clerk's terminal screen displays the person's clothing, location, and details of their behavior in real time.

[1409] Step 10: Staff response

[1410] The user (store clerk) checks the alert and refers to the displayed detailed information.

[1411] The user takes appropriate action and shares the information with other store staff as needed.

[1412] By using the above steps, the monitoring system of the present invention can efficiently and quickly detect suspicious behavior or suspicious individuals, and immediately issue an alert to prompt a response.

[1413] Example 1

[1414] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1415] From a security perspective, it is important to efficiently and quickly detect shoplifting and other suspicious behavior in stores and respond appropriately. However, current systems lack sufficient accuracy in analyzing video data and the speed of notification, resulting in the overlooking of suspicious behavior and delayed response. Other issues include the accuracy of the process of comparing suspect individuals and behavioral patterns with a database, and the efficiency of communicating information to store staff.

[1416] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1417] In this invention, the server includes means for acquiring video data of the monitored area, means for compressing the acquired video data and sending it to the server, means for analyzing the received video data and performing facial recognition and behavioral analysis, means for comparing the analyzed facial and behavioral data with a database, means for generating and notifying an alert based on the comparison results, and means for displaying the alert information. This makes it possible to detect suspicious behavior over a wide area within the store with high accuracy and immediately notify store staff.

[1418] "Surveillance Area" means a physical space that is subject to surveillance for a particular security purpose.

[1419] "Video data" refers to video image information collected by imaging equipment such as cameras installed in a monitored area.

[1420] "Compression" is the process of changing the format of data using a specific algorithm to reduce the volume of the data.

[1421] A "server" is a computer system used to provide a particular service and process data over a network.

[1422] "Facial recognition" is a technology that detects people's faces from video data and identifies individual faces using specific algorithms.

[1423] "Behavioral analysis" is a technology that analyzes the movements of people in video data and identifies specific behaviors and patterns.

[1424] A "database" is a system for storing, managing, and searching data in an organized manner.

[1425] "Matching" is the process of comparing the analyzed data with information from existing databases.

[1426] An "alert" is a warning message that is generated when a particular condition is met.

[1427] "Notification" is the act of sending generated alert information to a designated recipient.

[1428] "Means" refers to a method or device used to achieve a particular purpose.

[1429] "Display" is the act of visually showing information.

[1430] The present invention relates to a surveillance system aimed at preventing shoplifting and detecting suspicious behavior in stores. Specifically, the system analyzes video data obtained from security cameras in real time to detect suspicious behavior and suspicious individuals. Furthermore, based on the results, the system sends alerts to store clerk terminals to prompt prompt action.

[1431] Hardware and Software Configuration

[1432] Acquiring video data of the monitored area

[1433] The device acquires video data in real time from multiple security cameras (e.g., IP cameras) installed in the store. The device temporarily stores the video from the cameras in a buffer for easy access.

[1434] Video data compression and transmission

[1435] The video data acquired by the device is compressed and sent to the server using an efficient video compression format such as H.264. The compressed data is sent via the network to the specified server address.

[1436] Video data analysis

[1437] The server starts analyzing the received video data. It applies a high-precision facial recognition algorithm (e.g., OpenCV or Dlib) to detect faces in the video. The detected faces are extracted from each frame and saved as individual image data. In parallel, the server uses a behavior analysis algorithm (e.g., an RNN- or LSTM-based model) to analyze the behavior of people in the video.

[1438] Database Matching

[1439] The server compares the newly detected face data with an existing database (e.g., an SQL database) by calculating the distance between the vectors of facial features. Similarly, behavioral data is compared with past behavioral patterns.

[1440] Alert generation and notification

[1441] If the server identifies a suspicious person or suspicious behavior, it generates an alert, which includes details of the detected face, clothing, location, and behavior. The alert message is generated in real time and pushed to the store clerk's device.

[1442] Viewing Alerts

[1443] When the device receives an alert, the alert information is displayed in chat format, allowing store staff to check the notification in real time on the device screen. The alert information includes details of the person's clothing, location, and behavior.

[1444] Staff response

[1445] The user (store clerk) receives the alert and checks the displayed information. The clerk then promptly takes action based on the alert. For example, they go to the specific location where the suspicious behavior was reported and take appropriate action against the suspicious person.

[1446] Specific examples

[1447] For example, suppose a store is monitored 24 hours a day by security cameras. A device acquires video data from multiple cameras in the store and sends it to a server. The server analyzes the received data and performs highly accurate facial recognition and behavioral analysis. This makes it possible to check whether a specific person is included in the blacklist of individuals in the past database.

[1448] At the same time, the server applies a behavioral analysis algorithm to detect suspicious behavior in front of a specific product. Based on this, the server generates an alert such as "A man wearing a red jacket is lingering in front of a specific product for an unusually long time." This alert is sent to the store clerk's device, and detailed information is displayed in chat format.

[1449] Using this information, store staff can rush to the scene, confirm the possibility of shoplifting, and take appropriate action. This system will enable the efficient and rapid detection of suspicious activity across a wide area of ​​the store, significantly strengthening the prevention of shoplifting.

[1450] Prompt Sentence Examples

[1451] 1. A store clerk can detect someone spending an abnormally long time at a particular shelf.

[1452] 2. The server detects suspicious behavior and sends an alert to the store clerk's terminal.

[1453] 3. The store staff checks the alert and takes appropriate action.

[1454] In this way, the present invention provides a system that efficiently and quickly strengthens security within a store and contributes to preventing fraudulent activities such as shoplifting.

[1455] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1456] Step 1: Acquire video data

[1457] The device acquires video data in real time from multiple security cameras installed in the store. Specifically, the video captured by the cameras is temporarily stored in a buffer, enabling continuous data collection.

[1458] Input: Real-time video data from security cameras.

[1459] Output: Uncompressed video data stored in a buffer.

[1460] Step 2: Compress and send the data

[1461] The video data acquired by the device is compressed and sent to the server using an efficient video compression format such as H.264, and the data is sent to the server via the network.

[1462] Input: Uncompressed video data.

[1463] Output: Compressed video data.

[1464] Step 3: Analyzing the video data

[1465] The server analyzes the received compressed video data. First, a face recognition algorithm (e.g., OpenCV or Dlib) is applied to detect faces in the video. The detected faces are extracted frame by frame and saved as individual image data. At the same time, a behavior analysis algorithm (e.g., an RNN or LSTM-based model) is applied to analyze the person's behavior.

[1466] Input: Compressed video data.

[1467] Output: Detected face images and behavior analysis results.

[1468] Step 4: Check against the database

[1469] The server matches newly detected facial data with an existing database (e.g., an SQL database), calculates the distance between facial feature vectors, and identifies potential matches of suspicious individuals. Behavioral data is similarly compared with past behavioral patterns.

[1470] Input: Detected face images and behavior analysis results.

[1471] Output: Matching person of interest information and suspicious behavior identification.

[1472] Step 5: Generate an alert

[1473] If the server identifies a suspicious person or behavior, it generates an alert, combining detected facial images, clothing, location information, and behavior details to create an alert message.

[1474] Input: Matching person of interest information and suspicious activity identification results.

[1475] Output: The alert message.

[1476] Step 6: Notification and display of alerts

[1477] The server sends the generated alert information to the store clerk's terminal, which displays the alert information in a pop-up or chat window format so that the store clerk can check it immediately.

[1478] Input: Alert message.

[1479] Output: Alert information displayed on the terminal.

[1480] Step 7: Staff response

[1481] The user (store clerk) receives the alert and checks the displayed information. The clerk follows the instructions in the alert to go to the specific location and deal with the suspicious person.

[1482] Input: The alert information displayed on the terminal.

[1483] Output: On-site response actions.

[1484] (Application example 1)

[1485] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1486] While conventional systems have the ability to analyze security camera footage to detect suspicious behavior and suspicious individuals, they have limited means of providing information to store staff so that they can respond quickly.In addition, there is a lack of ways for store staff patrolling the store to identify suspicious behavior in real time, making it difficult to respond quickly.

[1487] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1488] In this invention, the server includes means for acquiring video data of the store, means for analyzing the acquired video data and identifying faces and behaviors, means for comparing the analyzed faces and behaviors with a database on the server, means for generating and notifying an alert based on the comparison results, and means for displaying the alert information on the smart glasses. This enables store staff wearing the smart glasses to identify suspicious behavior in real time while patrolling and respond quickly.

[1489] "In-store video data" refers to images and video information obtained in real time from security cameras installed within a store.

[1490] The "acquisition means" refers to hardware and software components for receiving video data output from a security camera and storing or transmitting it.

[1491] "Means for analyzing and identifying faces and behaviors" refers to facial recognition algorithms and behavior analysis algorithms for identifying people's faces and analyzing their behaviors based on the captured video data.

[1492] The "database on the server" is a data management system used to store analyzed facial and behavioral data and compare it with past data.

[1493] The "means of matching" is an algorithm that compares facial and behavioral data acquired in real time with existing data in a database on the server to confirm a match.

[1494] "Means for generating and notifying alerts" refers to a system that creates an alert when suspicious behavior or suspicious individuals are detected and immediately notifies relevant parties of that information.

[1495] The "means for displaying alert information on smart glasses" refers to communication and display technology for displaying the generated alert information on the display of smart glasses worn by the store clerk.

[1496] This invention is a surveillance system aimed at preventing shoplifting in a store and detecting suspicious individuals at an early stage. This system acquires video data from multiple security cameras in real time, analyzes the data, detects suspicious behavior and individuals of interest, and notifies store staff.

[1497] Hardware configuration:

[1498] Security cameras: These are installed at strategic locations within the store and are used to capture footage in real time.

[1499] Server: A computer with powerful computing power for receiving and analyzing video data.

[1500] Smart glasses: worn by store associates and used to receive and display alert information in real time.

[1501] Software configuration:

[1502] OpenCV: A library for acquiring and processing security camera footage in real time.

[1503] Keras: A deep learning framework used to implement face recognition and behavior analysis algorithms.

[1504] Flask: A lightweight web framework for receiving and analyzing video data on the server side.

[1505] Requests: An HTTP library for data communication between smart glasses and a server.

[1506] Overview of program processing:

[1507] The device continuously captures video data from security cameras in the store in real time and sends it to a server. The server then applies facial recognition and behavioral analysis algorithms to the received video data to analyze the behavioral patterns of specific individuals. The analysis results are compared with existing data in a database, and an alert is generated if a match is found or if abnormal behavior is detected. The generated alert is then sent to the smart glasses, which notify the store staff in real time.

[1508] Examples:

[1509] For example, consider a store where security cameras are monitoring the shop 24 hours a day. The server receives this video data in real time and uses a facial recognition algorithm to identify individuals. It then checks whether the identified individuals are on a blacklist in the database. At the same time, the server applies a behavioral analysis algorithm to detect that an individual is repeatedly lingering in front of a particular product for an abnormally long time. In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an abnormally long time." This alert is sent to smart glasses, and the information is displayed in chat format on the smart glasses worn by the store clerk. Based on this information, the store clerk can head to the scene, confirm the possibility of shoplifting, and take appropriate action.

[1510] Example prompt sentence:

[1511] "The system analyzes security camera footage within stores in real time and generates an alert if it determines something is suspicious. The alert includes details of the face, clothing, location, and behavior. This alert is displayed on the smart glasses, enabling a prompt response."

[1512] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1513] Step 1:

[1514] Device operation

[1515] The device acquires video data from security cameras inside the store in real time.

[1516] Input: Video data from a security camera.

[1517] Data processing: The terminal compresses the acquired video data and converts it into a format that can be sent to the server (for example, JPEG format).

[1518] Output: Compressed video data.

[1519] Specific operation: The device continuously acquires data from multiple security cameras and transmits it to the server in real time.

[1520] Step 2:

[1521] Server data reception

[1522] A server receives the compressed video data.

[1523] Input: Compressed video data sent from the device.

[1524] Data processing: The server decompresses the compressed video data and converts it into a format that can be used as the original video data.

[1525] Output: Decompressed video data.

[1526] Specific operation: The server sequentially decompresses the received video data and prepares it for subsequent analysis processing.

[1527] Step 3:

[1528] Facial Recognition and Behavioral Analysis

[1529] The server analyzes the decompressed video data and applies facial recognition and behavioral analysis algorithms.

[1530] Input: Decompressed video data.

[1531] Data computation: The server uses a facial recognition algorithm (e.g., a Keras model) to identify faces in the video and applies a behavior analysis algorithm to analyze people's behavior.

[1532] Output: Analyzed face and behavioral data.

[1533] Specific operation: The server extracts people's faces from each video frame and analyzes their behavioral patterns.

[1534] Step 4:

[1535] Database collation

[1536] The server compares the analyzed facial and behavioral data with a database on the server.

[1537] Input: Parsed face and behavioral data.

[1538] Data calculation: The server compares the data with data in an existing database to identify facial data and suspicious behavior patterns with high matching scores.

[1539] Output: Matching results (face data and behavioral patterns with high matching scores).

[1540] Specific operation: The server uses a matching algorithm to match the blacklist in the database and identify matching data.

[1541] Step 5:

[1542] Generate alerts

[1543] The server generates an alert based on the match result.

[1544] Input: Matching results (face data and behavioral patterns with high matching scores).

[1545] Data processing: The server generates alert information (a person's face image, clothing, location information, and details of their behavior).

[1546] Output: Alert information.

[1547] What it does: The server immediately creates an alert when a suspicious person or activity is detected.

[1548] Step 6:

[1549] Sending and Viewing Alerts

[1550] The server transmits the generated alert information to the smart glasses for display.

[1551] Input: Alert information.

[1552] Data processing: The server converts the alert information into a format suitable for the smart glasses and sends it.

[1553] Output: Alert information displayed on smart glasses.

[1554] Specific operation: The server composes a notification message and sends it to the smart glasses, which receive the notification and the store clerk confirms it.

[1555] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1556] This invention relates to a surveillance system aimed at preventing shoplifting and detecting suspicious behavior in stores. This system analyzes video data obtained from security cameras in real time, and performs facial recognition and behavior analysis. In addition, by combining it with an emotion engine that recognizes user emotions, it enables more accurate detection of suspicious behavior and quicker response.

[1557] System configuration

[1558] 1. Video data acquisition method

[1559] The terminal acquires video data in real time from security cameras installed inside the store.

[1560] The video data acquired by the terminal is compressed and sent to a server via the Internet.

[1561] 2. Video data analysis methods

[1562] The server analyzes the received video data and applies facial recognition and behavioral analysis algorithms.

[1563] The server performs facial recognition and extracts the detected facial images for each frame.

[1564] The server uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements.

[1565] 3. Emotion Engine

[1566] The emotion engine built into the server identifies the user's emotion based on the facial image detected during face recognition.

[1567] The emotion engine generates user emotion data (e.g., impatience, anger, anxiety) and reflects it in the behavioral analysis results.

[1568] 4. Database matching method

[1569] The server matches the newly detected facial and emotion data with the existing database.

[1570] The server checks whether there is a person in the database that matches the facial data.

[1571] 5. Evaluation of matching results

[1572] Based on the matched results, the server evaluates whether there is a person of interest or suspicious activity.

[1573] If the server identifies suspicious behavior, it records details of the behavior and emotions.

[1574] 6. Alert Generation and Notification Methods

[1575] If the server identifies a person of interest or suspicious activity, it generates an alert.

[1576] The alert information includes a person's facial image, clothing, location information, behavior details, and emotional state.

[1577] The server transmits the generated alert information to the store clerk's terminal and notifies the store clerk.

[1578] 7. How to display alert information

[1579] The store clerk's device receives the alert and displays the alert information in chat format.

[1580] The store clerk's terminal screen displays the person's clothing, location, behavioral details, and emotional state in real time.

[1581] Program processing

[1582] 1. Device operation

[1583] The device continuously acquires video data from the store's security cameras and transmits it to the server.

[1584] The device acquires images from multiple cameras simultaneously, efficiently collecting data.

[1585] 2. Server Data Analysis

[1586] The server performs facial recognition on the received video data and applies a highly accurate facial recognition algorithm.

[1587] The server analyzes the behavior of people in the video and runs algorithms to detect abnormal behavior.

[1588] 3. Emotion recognition

[1589] The server's emotion engine analyzes the user's emotions based on the facial data and generates emotion data.

[1590] The server evaluates this emotional data in combination with the results of behavioral analysis.

[1591] 4. Database Matching

[1592] The server matches the newly detected facial and emotion data against a database to identify matches.

[1593] The server performs a risk assessment based on the facial data and emotional state that match closely.

[1594] 5. Alert generation and notification

[1595] If the server detects an abnormality, it immediately generates an alert and creates a notification message.

[1596] The alert message includes a facial image of the suspicious person, their clothing, location, and details of their behavior and emotions.

[1597] 6. Assistance via store clerk's terminal

[1598] The user (store clerk) receives the alert and checks the information displayed in chat format.

[1599] The user goes to a specific location in the store and takes appropriate action against the detected suspicious person.

[1600] Specific examples

[1601] For example, imagine a store with security cameras monitoring the store 24 hours a day. A device continuously captures data from these cameras and sends it to a server. The server uses facial recognition technology to identify people in the footage and check whether they are on a blacklist in a database. At the same time, an emotion engine analyzes the emotional data from the person's face to detect whether they are expressing anxiety or impatience.

[1602] In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an unusually long time and appears anxious." This alert is sent to the store clerk's device, which displays detailed information in a chat format. Based on this information, the store clerk rushes to the scene, confirms the possibility of shoplifting, and takes appropriate action.

[1603] This system makes it possible to efficiently and quickly monitor suspicious behavior and emotional states over a wide area within a store, and immediately notify store staff to prompt them to take action, thereby significantly strengthening shoplifting prevention and improving store security.

[1604] The processing flow will be explained below.

[1605] Step 1: Acquire video data

[1606] The terminal acquires video data in real time from security cameras installed inside the store.

[1607] The video data acquired by the terminal is compressed and sent to a server via the Internet.

[1608] Step 2: Receiving and saving video data

[1609] The server receives the video data transmitted from the terminal.

[1610] The video data received by the server is temporarily stored in a storage system.

[1611] Step 3: Performing facial recognition

[1612] The server analyzes the video data stored in the storage system and applies a facial recognition algorithm.

[1613] The server cuts out the detected face for each frame and generates face image data.

[1614] Step 4: Performing behavioral analysis

[1615] The server runs an algorithm that tracks the movements of each person in the video data.

[1616] The server detects specific behavioral patterns (e.g., staying in the same place for a certain period of time or going back and forth to the same place multiple times).

[1617] Step 5: Performing Emotion Recognition

[1618] The server applies an emotion engine based on the facial image data to analyze the user's emotions.

[1619] The server generates user emotion data and evaluates it in combination with behavioral data.

[1620] Step 6: Check against the database

[1621] The server compares the newly detected facial and emotion data with a pre-registered database.

[1622] The server identifies matches to the facial and emotion data in the database.

[1623] Step 7: Evaluate the match results

[1624] The server evaluates suspicious behavior and people of interest based on the matching results.

[1625] The server performs a risk assessment based on the detected behavior and emotions and determines the severity of the alert.

[1626] Step 8: Generate an alert

[1627] If the server identifies a person of interest or suspicious activity, it generates an alert.

[1628] The alert information includes facial image, clothing, location information, behavioral details, and emotional state.

[1629] Step 9: Sending alerts

[1630] The server transmits the generated alert information to the store clerk's terminal.

[1631] The server is set so that alert information is sent to multiple store clerk terminals simultaneously.

[1632] Step 10: Viewing Alerts

[1633] The alert information received by the store clerk's terminal is displayed in chat format.

[1634] The employee's device screen displays a person's facial image, clothing, location, behavioral details, and emotional state in real time.

[1635] Step 11: Staff response

[1636] The user (store clerk) checks the alert and rushes to the scene based on the detailed information displayed.

[1637] The user takes appropriate action against the detected suspicious person.

[1638] The information confirmed by the user is shared with other store staff, and monitoring is strengthened in collaboration.

[1639] By implementing the above steps, the monitoring system of the present invention can efficiently and quickly detect suspicious behavior and suspicious individuals, and immediately issue alerts to prompt responses. Furthermore, by combining it with an emotion engine, advanced risk assessment that takes into account changes in emotions becomes possible.

[1640] Example 2

[1641] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1642] Conventional security systems have had difficulty detecting suspicious behavior in stores in real time and responding quickly. Furthermore, conventional systems have had problems with low accuracy in facial recognition and behavioral analysis, resulting in frequent false positives and missed detections. Furthermore, the method of notifying store staff was inefficient, limiting effective action in situations where immediate action was required.

[1643] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring data within the store, a means for processing the acquired data to identify personal information and behavioral patterns of people, and a means for comparing the identified personal information and behavioral patterns with recording media. This makes it possible to perform highly accurate face recognition and behavioral analysis, detect suspicious behavior or suspicious people in real time, and quickly notify store staff. In addition, by displaying warning information in an interactive format, store staff can immediately understand the situation and take appropriate action.

[1644] "In-store" refers to the space inside a building such as a commercial facility or retail store.

[1645] "Data" refers to a collection of information such as video, images, audio, and sensor information.

[1646] "Means of acquisition" refers to methods of collecting data using devices such as cameras and sensors.

[1647] "Processing" refers to performing operations such as analysis and conversion on acquired data.

[1648] "Person" refers to the human being who is the subject of surveillance.

[1649] "Personal information" refers to information that can identify a specific individual, such as a name or facial image.

[1650] "Behavioral patterns" refer to the characteristics of a person's movements or actions at a particular time.

[1651] "Means of identification" refers to methods that use facial recognition and behavioral analysis algorithms to distinguish a person's characteristics and behavior.

[1652] "Recording media" refers to storage or databases for saving information.

[1653] "Matching" refers to comparing detected information with existing data to determine whether it matches.

[1654] "Warning" refers to a notification or alert that occurs when an abnormality is detected.

[1655] "Interactive" refers to the way information is presented in the form of chats and messages.

[1656] This invention relates to a surveillance system that strengthens security in stores, prevents shoplifting, and detects suspicious behavior. Specifically, this system analyzes video data acquired from security cameras installed in stores in real time to perform facial recognition and behavior analysis.

[1657] Hardware and Software Configuration

[1658] This system consists of the following hardware and software:

[1659] 1. Device:

[1660] Security cameras: Multiple cameras are installed within the store and continuously capture footage.

[1661] Network connection: A high-speed internet connection to transmit captured video data to the server.

[1662] 2. Server:

[1663] Video analytics software: Software for running high-precision facial recognition algorithms (e.g., OpenCV, Dlib, etc.) and behavioral analysis algorithms (e.g., YOLO, OpenPose, etc.).

[1664] Database: A recording medium for storing existing person of interest data and emotion data.

[1665] Emotion engine: A software module for analyzing user emotions.

[1666] Program processing

[1667] The overall system operates as follows.

[1668] 1. Device operation

[1669] The terminal continuously captures video data from security cameras installed in the store, efficiently compresses it, and sends it to the server. Data compression is particularly important when handling high-resolution video data.

[1670] 2. Server Data Analysis

[1671] The server applies a facial recognition algorithm to the received video data to extract facial images of people for each frame, and then uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements.

[1672] 3. Emotion recognition

[1673] The server's emotion engine analyzes the user's emotions based on the facial data and generates emotion data, which is then combined with the results of behavioral analysis to determine the overall risk level.

[1674] 4. Database Matching

[1675] The server then compares the newly detected facial and emotional data against existing databases to identify matches, specifically against historical blacklists and emotional databases.

[1676] 5. Alert generation and notification

[1677] If the server detects an anomaly, it immediately generates an alert and creates a notification message. The alert includes details of the suspicious person's face, clothing, location, behavior, and emotions. This information is sent to the store clerk's terminal and displayed interactively.

[1678] 6. Staff Service

[1679] The user (store clerk) receives an alert on their device, checks the detailed alert information in chat format, rushes to the scene, and takes appropriate action against the detected suspicious person.

[1680] Specific examples

[1681] For example, consider a store where security cameras monitor the store 24 hours a day. Devices continuously capture data from these cameras and send it to a server. The server uses facial recognition technology to identify individuals in the footage and check whether they are on a database of suspicious individuals. At the same time, an emotion engine analyzes the emotional data from the individual's face and detects whether they are expressing anxiety or impatience. In this case, the server generates an alert stating, "A man wearing a red jacket is lingering in front of a particular product for an unusually long time and appears anxious." This alert is then sent to a store employee's device, which displays detailed information in a chat format. Based on this information, the employee rushes to the scene, confirms the possibility of shoplifting, and takes appropriate action. This system allows for efficient and rapid monitoring of suspicious behavior and emotional states across a wide area of ​​the store, and immediately notifies employees to prompt action. This significantly strengthens shoplifting prevention and improves store security.

[1682] Prompt Sentence Examples

[1683] Examples of prompts to be input to a generative AI model include:

[1684] "How can we create a system that sends notifications when security camera footage shows signs of shoplifting?"

[1685] "How can we combine facial recognition and behavioral analysis to detect suspicious behavior in stores?"

[1686] "Describe an algorithm that uses an emotion engine to detect emotions like anxiety and anger to identify suspicious behavior in real time."

[1687] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1688] Step 1:

[1689] The terminal acquires video data in real time from security cameras installed inside the store. Specifically, each security camera captures video frame by frame and sends it to the terminal. The input is the video data acquired from the security camera, and the output is compressed video data. The terminal performs this continuously to efficiently collect data.

[1690] Step 2:

[1691] The video data acquired by the device is compressed and sent to a server via the Internet. Specifically, the video data is encoded using a compression algorithm such as H.264 to reduce the data volume. The input is raw video data from the security camera, and the output is compressed video data. The compressed data is sent to the server.

[1692] Step 3:

[1693] The server applies a facial recognition algorithm to the video data it receives. Specifically, it uses libraries such as OpenCV and Dlib to detect human faces in each frame and extract facial images. The input is compressed video data, and the output is a list of detected facial images. The server performs this process continuously, achieving highly accurate facial recognition.

[1694] Step 4:

[1695] The server uses a behavioral analysis algorithm to analyze the behavior of people in the video and detect suspicious movements. Specific algorithms, such as YOLO and OpenPose, are used to identify the person's movements and location. The input is frame-by-frame video data, and the output is analyzed behavioral data. The server specializes in detecting unusual behavioral patterns.

[1696] Step 5:

[1697] The server's emotion engine analyzes the user's emotions based on the facial data and generates emotion data. Specifically, it uses an algorithm that analyzes facial expressions to identify the user's psychological state. The input is the detected facial image, and the output is emotion data (e.g., impatience, anger, anxiety). The server then combines this with the results of behavioral analysis and evaluates it.

[1698] Step 6:

[1699] The server compares newly detected facial and emotional data against an existing database to identify matches. Specifically, it uses an algorithm that directly compares new data with the existing database. The input is the newly analyzed facial and emotional data, and the output is whether or not there are any matching records. The server then performs a risk assessment.

[1700] Step 7:

[1701] If the server detects an anomaly, it immediately generates an alert and creates a notification message. Specifically, it runs a program that automatically generates an alert message after detecting an anomaly. The input is the anomaly detection result data, and the output is an alert message. The alert includes details of the suspicious person's face image, clothing, location, behavior, and emotions.

[1702] Step 8:

[1703] After the alert information generated by the server is sent to the store clerk's terminal, the user (store clerk) receives the alert on the terminal and checks the information displayed in chat format. The input is the alert message from the server, and the output is the alert information displayed on the terminal. The store clerk checks this and rushes to the scene depending on the situation.

[1704] Step 9:

[1705] The user (store clerk) rushes to the scene based on the alert information and takes appropriate action against the detected suspicious person. Specific actions include actually checking the situation and, if necessary, coordinating with security guards. The input is detailed information displayed in chat format, and the output is the response results.

[1706] (Application example 2)

[1707] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1708] Conventional suspicious behavior detection systems in stores rely solely on simple behavioral analysis, making them insufficient for detecting shoplifting and other suspicious behavior. Furthermore, simple video analysis cannot capture detailed behavior or psychological states, making it difficult for store staff to take appropriate action on the spot. Furthermore, notification functions are often inadequate, resulting in delayed real-time responses. For these reasons, there was a demand for a system that could provide more accurate and rapid detection of suspicious behavior and countermeasures.

[1709] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring in-store video data, means for analyzing the acquired video data and identifying faces and behaviors, means for comparing the identified faces and behaviors with a database, means for generating and notifying an alert based on the comparison results, means for displaying alert information, means for recording the notification message, emotion engine means for analyzing emotions based on the face recognition results, and means for sending real-time notifications to the terminal. This combines face recognition and emotion analysis to enable advanced and precise detection of suspicious behavior, allowing store staff to respond quickly in real time.

[1710] "In-store video data" refers to real-time video information captured by video capture devices such as security cameras installed in the store.

[1711] "Means of acquisition" refers to the devices and methods used to collect video data from security cameras and other sources and input it into the system.

[1712] "Means for analyzing and identifying faces and behavior" refers to algorithms and software that analyze video data obtained from security cameras and recognize the faces of people in the footage and identify their behavioral patterns.

[1713] "Database matching" refers to the process of comparing analyzed facial and behavioral data with a pre-registered database to identify matching people and behaviors.

[1714] "Means for generating and notifying alerts" refers to a system that creates warning information when suspicious behavior or individuals are detected and notifies store staff and other relevant parties.

[1715] The "means for displaying alert information" refers to a device or interface for displaying the generated alert information in a form that can be easily confirmed by store staff and other relevant parties.

[1716] The "means for recording notification messages" refers to a system for saving the generated alerts and notification contents so that they can be referenced later.

[1717] "Emotion engine means for analyzing emotions" refers to algorithms or software that recognize and analyze a person's emotions based on the results of facial recognition, and identify emotional states such as anxiety or fear.

[1718] "Means for real-time notification" refers to a communication method that immediately notifies the store clerk's terminal of detected suspicious behavior or emotional state, urging immediate action.

[1719] The system of the present invention is intended to prevent shoplifting and detect suspicious behavior in a store, and is configured using the following hardware and software.

[1720] System Configuration

[1721] 1. Security cameras

[1722] These cameras are installed in stores to monitor customer movements in real time, making it possible to continuously acquire video data.

[1723] 2. Video data acquisition method

[1724] This method continuously acquires video data from security cameras, compresses it, and sends it to a server. The specific software used is OpenCV.

[1725] 3. Video data analysis methods

[1726] Using this method, the server analyzes the received video data, performs face recognition and behavior analysis routines, and uses a face recognition library (e.g., face_recognition) to extract face images and analyze them frame by frame.

[1727] 4. Emotion Engine

[1728] The server is equipped with a trained emotion recognition model that uses machine learning frameworks such as Keras to identify customer emotions based on facial recognition results.

[1729] 5. Database matching methods

[1730] The server checks the newly detected facial and emotional data against an existing database containing pre-existing images of suspect people to see if there is a match.

[1731] 6. Alert Generation and Notification Methods

[1732] Based on the match, the server generates an alert, which includes the person's facial image, clothing, location, behavioral details, and emotional state, and sends it to the store clerk's smartphone in real time using Twilio.

[1733] 7. Alert Information Display Method

[1734] The user (store clerk) receives the alert via an application installed on their smartphone and can check it in chat format, enabling a quick response in the store.

[1735] Specific examples

[1736] For example, suppose a store has security cameras monitoring the store 24 hours a day. The server continuously collects data from these cameras and analyzes the video data in real time. Specifically, it uses facial recognition technology to identify people in the video and check whether they are included in a database of suspicious individuals.

[1737] At the same time, the emotion engine analyzes the emotional data from the person's face and detects whether the person is expressing anxiety or impatience. In this case, the server generates an alert stating, "A person wearing a red jacket is lingering in front of a particular product for an abnormally long time and is displaying an anxious expression." This alert is sent to the store clerk's device, which displays detailed information in a chat format. Based on this information, the store clerk can rush to the scene, confirm the possibility of shoplifting, and take appropriate action.

[1738] Prompt Sentence Examples

[1739] "Your task is to generate code for an application that acquires video data from security cameras in real time, detects suspicious behavior using an emotion engine, analyzes customer emotions such as anxiety, anger, and fear, and notifies store staff on their smartphones."

[1740] This system makes it possible to efficiently and quickly monitor suspicious behavior and customer emotional states over a wide area within a store, and instantly notify store staff to promptly respond, thereby strengthening shoplifting prevention measures and improving security throughout the store.

[1741] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1742] Step 1:

[1743] A means for acquiring video data from security cameras acquires video data from security cameras in the store in real time. This data is continuously acquired and used for subsequent analysis steps. The input is raw video data and the output is compressed video data.

[1744] Step 2:

[1745] The device sends the video data acquired from the security camera to the server. The specific input is the compressed video data, and the output is the status of completion of transmission to the server. This process is performed via the network.

[1746] Step 3:

[1747] The server analyzes the received video data and performs facial recognition and behavior analysis. The input is the received video data, and the output is the recognized face image and behavior data. Specifically, OpenCV and the face_recognition library are used.

[1748] Step 4:

[1749] The server performs emotion analysis based on the facial recognition results. The input is the analyzed facial image, and the output is emotion data (e.g., anxiety, impatience, anger). The emotion engine uses Keras and a pre-trained neural network model.

[1750] Step 5:

[1751] The server compares the newly detected face and emotion data with an existing database. The input is face and emotion data, and the output is the matching result. The database contains pre-recorded images of suspicious people.

[1752] Step 6:

[1753] If the server detects an anomaly, it generates an alert. The input is the matching result and emotion data, and the output is alert information. The alert information includes the detected person's face image, clothing, location information, behavior details, and emotional state.

[1754] Step 7:

[1755] The server uses Twilio to send alert information to the store clerk's smartphone. The input is the alert information, and the output is a notification to the store clerk's terminal. This notification includes detailed information about the person.

[1756] Step 8:

[1757] The user (store clerk) checks the alert through an application installed on the device. The input is the received alert information, and the output is instructions for the store clerk to act. The store clerk checks the detailed information in chat format and responds promptly.

[1758] summary

[1759] This series of processes makes it possible to efficiently and quickly detect suspicious behavior and emotional states within a store, immediately notifying store staff and prompting them to take action, thereby strengthening shoplifting prevention measures and improving security throughout the store.

[1760] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1761] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1762] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1763] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1764] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1765] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1766] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1767] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1768] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1769] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1770] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1771] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1772] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1773] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1774] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1775] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1776] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1777] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1778] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1779] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1780] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1781] The following is further disclosed regarding the above embodiment.

[1782] (Claim 1)

[1783] A means of acquiring video data inside the store;

[1784] A means of analyzing the acquired video data and identifying faces and actions,

[1785] A means of matching the analyzed faces and behaviors with a database;

[1786] A means for generating and notifying alerts based on the matching results;

[1787] A system including a means for displaying alert information.

[1788] (Claim 2)

[1789] 10. The system according to claim 1, further comprising means for displaying the alert information in a chat format.

[1790] (Claim 3)

[1791] 10. The system of claim 1, further comprising means for capturing a preliminary photograph of a person of interest and registering it in a database.

[1792] "Example 1"

[1793] (Claim 1)

[1794] means for acquiring video data of a monitored area;

[1795] means for compressing the acquired video data and transmitting it to a server;

[1796] means for analyzing the received video data and performing facial recognition and behavior analysis;

[1797] A means of matching the analyzed face and behavioral data with a database;

[1798] A means for generating and notifying alerts based on the matching results;

[1799] A system including a means for displaying alert information.

[1800] (Claim 2)

[1801] 10. The system according to claim 1, further comprising means for displaying the alert information in a chat format.

[1802] (Claim 3)

[1803] 10. The system of claim 1, further comprising means for capturing a preliminary image of a person of interest and registering it in a database.

[1804] "Application Example 1"

[1805] (Claim 1)

[1806] A means of acquiring video data inside the store;

[1807] A means of analyzing the acquired video data and identifying faces and actions,

[1808] A means of comparing the analyzed faces and behaviors with a database on a server,

[1809] A means for generating and notifying alerts based on the matching results;

[1810] a means for displaying the alert information on the smart glasses;

[1811] ...

[1812] A system including:

[1813] (Claim 2)

[1814] 10. The system according to claim 1, further comprising means for displaying the alert information in a chat format.

[1815] (Claim 3)

[1816] 10. The system of claim 1, further comprising means for capturing a preliminary photograph of a person of interest and registering it in a database.

[1817] "Example 2: Combining Emotion Engines"

[1818] (Claim 1)

[1819] A means of acquiring in-store data;

[1820] A means of processing the acquired data to identify personal information and behavioral patterns of a person;

[1821] A means for matching identified personal information and behavioral patterns with recording media;

[1822] A means for generating and notifying a warning based on the result of the comparison;

[1823] The system includes a means for displaying warning information.

[1824] (Claim 2)

[1825] 10. The system of claim 1, further comprising means for interactively displaying the warning information.

[1826] (Claim 3)

[1827] 2. The system according to claim 1, further comprising means for capturing a preliminary image of the person of interest and registering it in a recording medium.

[1828] "Application example 2 when combining emotion engines"

[1829] (Claim 1)

[1830] A means of acquiring video data inside the store;

[1831] A means of analyzing the acquired video data and identifying faces and actions,

[1832] a means for matching identified faces and behaviors with a database;

[1833] A means for generating and notifying alerts based on the matching results;

[1834] a means for displaying the alert information;

[1835] means for recording a notification message;

[1836] an emotion engine means for analyzing emotions based on the face recognition result;

[1837] a means for providing real-time notifications to the terminal;

[1838] A system including:

[1839] (Claim 2)

[1840] 10. The system according to claim 1, further comprising means for displaying the alert information in a chat format.

[1841] (Claim 3)

[1842] 10. The system of claim 1, further comprising means for capturing a preliminary image of a person of interest and registering it in a database. [Explanation of symbols]

[1843] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring video data inside the store; A means of analyzing the acquired video data and identifying faces and actions, A means of matching the analyzed faces and behaviors with a database; A means for generating and notifying alerts based on the matching results; A system including a means for displaying alert information.

2. The system according to claim 1 , further comprising means for displaying the alert information in a chat format.

3. 2. The system of claim 1, further comprising means for capturing a preliminary photograph of a person of interest and registering it in a database.

Images