system

Abstract

We provide the system. [Solution] A means for receiving and processing video data from a monitoring device, A means for extracting vehicle identification information based on the video data, A means for detecting an anomaly by comparing the identification information with a pre-prepared database of abnormal vehicles, A means for notifying a display device of an abnormality when it is detected, and for generating notification data, Means for transmitting the notification data to an external organization, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, vehicle thefts and the use of illegally modified vehicles in parking lots are increasing, and there are concerns about the accompanying deterioration of public security. Conventional surveillance systems mainly rely on manual surveillance, making it difficult to detect abnormalities immediately and respond quickly. Also, even when an abnormality is discovered, notifications to the police and related agencies are often delayed, and there is a problem that the crime deterrence force is not sufficient. The present invention aims to solve such problems.

Means for Solving the Problems

[0005] This invention provides a means for receiving and analyzing video data from a monitoring device in real time and extracting vehicle identification information based on that data. Furthermore, it allows for the detection of anomalies by comparing this identification information with an abnormal vehicle database. When an anomaly is detected, the system provides a function to quickly notify a display device of the information, generate notification data, and immediately transmit it to an external organization. This enables immediate detection of anomalies and prompt response.

[0006] A "surveillance device" is a device installed to acquire video footage of a monitored area.

[0007] "Video data" refers to video-format information acquired from surveillance equipment.

[0008] "Processing" refers to a series of operations that analyze video data, including data transformation and filtering.

[0009] "Vehicle identification information" refers to information used to identify a vehicle, and may include license plates and vehicle shape.

[0010] The "Abnormal Vehicle Database" is a database that records information on stolen vehicles, illegally modified vehicles, and other such vehicles.

[0011] "Verification" refers to the process of comparing identification information with database information to confirm a match.

[0012] "Anomaly" refers to an unexpected event or a state that deviates from the standard, and in this context, it primarily refers to the detection of illegal vehicles.

[0013] A "display device" is an output device used to visually present information or notifications.

[0014] "Notification data" refers to a dataset containing detailed information about detected anomalies.

[0015] "External institution" refers to an organization or institution existing outside the system, and mainly refers to the police in the present invention.

Brief Description of Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention provides a system for immediately detecting and quickly responding to illegal vehicle activity in monitored areas such as parking lots. This system works in conjunction with multiple monitoring devices and provides a platform for analyzing acquired video data in real time.

[0038] System Configuration

[0039] 1. Server Role

[0040] The server continuously receives video data from the monitoring device and applies a generative AI model to extract vehicle identification information. This identification information includes the license plate and vehicle shape information of the identified vehicle. Based on this, the server compares it with a database of abnormal vehicles and processes the information further if an abnormality is detected.

[0041] 2. The role of the terminal

[0042] The terminal receives anomaly information transmitted from the server and notifies the monitoring personnel visually and audibly. This allows the monitoring personnel to understand the situation in real time and take necessary actions quickly.

[0043] 3. User Roles

[0044] Based on the information provided by the device, users can check for anomalies and, if necessary, manually provide additional information to external organizations, such as the police. This allows for smoother on-site response and investigation.

[0045] Specific example

[0046] For example, in the underground parking lot of a shopping mall, a surveillance system provides real-time video. A server receives this video and recognizes the license plates of vehicles. If the recognized license plate matches a list of stolen vehicles, the server immediately detects this anomaly and sends the relevant information to a terminal. The terminal displays this information visually and alerts the person in charge. The person in charge reviews the video, contacts the police if necessary, and organizes the response at the scene. Because this entire process is automated and in real time, it enables a faster and more reliable response than traditional methods.

[0047] This implementation allows for the immediate detection and prompt response to any misuse of vehicles within the surveillance area. This system contributes to crime prevention and rapid response after an incident occurs, thereby ensuring safe public spaces.

[0048] The following describes the processing flow.

[0049] Step 1:

[0050] The server receives streaming video data transmitted from the monitoring device in real time. This includes extracting frame data that contains the vehicle to be processed.

[0051] Step 2:

[0052] The server applies a generative AI model to the received video data to extract vehicle license plate information. During this process, it performs noise reduction and format conversion to prepare the data for easier analysis.

[0053] Step 3:

[0054] The server compares the extracted license plate information with a database of abnormal vehicles. Here, it makes a determination to identify vehicles suspected of being stolen or illegally modified.

[0055] Step 4:

[0056] If an anomaly is detected as a result of the matching process, the server organizes relevant data such as snapshots and location information of the vehicle and generates notification data.

[0057] Step 5:

[0058] The server sends the generated notification data to the terminal, informing the monitoring personnel of the details of the anomaly. This notification includes an alert to prompt immediate action.

[0059] Step 6:

[0060] The terminal provides the monitoring personnel with received notification data visually and audibly. This allows the personnel to quickly assess the situation and take necessary countermeasures.

[0061] Step 7:

[0062] The user will check the abnormal situation based on the information from the device. If necessary, the user will contact the police or other relevant organizations directly to expedite the response at the scene.

[0063] (Example 1)

[0064] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0065] There is a need to quickly and accurately detect anomalies in moving objects within the monitoring area and prompt immediate responses based on those detections. In particular, it is crucial to efficiently identify anomalies without relying on human intervention and to facilitate smooth cooperation with external organizations.

[0066] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0067] In this invention, the server includes means for receiving and analyzing visual information from a video acquisition means, means for extracting identification information of a moving object based on the visual information, and means for comparing the identification information with a pre-prepared set of abnormal data to detect abnormalities. This enables automatic and real-time detection of abnormalities within a monitoring area.

[0068] A "video acquisition means" is a mechanism for acquiring visual information from a surveillance area and supplying it to a processing system.

[0069] "Visual information" refers to images and video data provided by surveillance devices.

[0070] "Means of analysis" refers to the processes and techniques used to process acquired visual information and extract necessary identification information.

[0071] "Identification information for moving objects" refers to data used to identify moving objects such as vehicles, and includes information such as license plates and shapes.

[0072] An "anomalous data set" refers to a pre-prepared collection of information about an abnormal state or event.

[0073] "Notification information" refers to information that is sent to external organizations or monitors when an anomaly is detected.

[0074] A "generative AI model" refers to a machine learning model used to extract useful information from visual information.

[0075] "Analysis of identification information" refers to the process of interpreting detected identification information and identifying anomalies.

[0076] A "warning signal" refers to a visual or auditory notification used to alert a monitor when an anomaly is detected.

[0077] A "terminal device" refers to a device that provides notification information received from a server to a monitor and prompts them to take necessary actions.

[0078] This invention is a system that enables rapid detection of anomalies within a monitored area and immediate response. Specifically, it operates through the cooperation of three entities: a server, a terminal, and a user.

[0079] Server role:

[0080] The server receives visual information acquired from monitoring devices. This process uses network communication technology, such as a streaming protocol. The received data is analyzed using a generative AI model to extract identification information for moving objects. The model is developed using software frameworks such as TENSORFLOW® and PyTorch, and identifies the license plates and shapes of moving objects. The identification information is compared with a set of anomaly data, and if an anomaly is detected, information processing is performed. A specific example is identifying a license plate that matches a list of stolen vehicles. In this case, the server immediately generates notification information.

[0081] Terminal role:

[0082] The terminal receives notification information from the server and provides visual and auditory alerts to the monitor. This includes displaying information via a GUI and using alarm sounds. The terminal communicates with the server using the TCP / IP protocol to ensure stable data reception.

[0083] User roles:

[0084] Users make on-site decisions based on information provided by their devices. For example, they can identify vehicles that have been flagged as abnormal on the monitoring screen and provide information to external organizations such as the police if necessary. This process enables rapid response and coordination.

[0085] An example of a prompt message for the generated AI model is, "Extract vehicle license plates from the surveillance footage and compare them with the existing list of stolen vehicles." This allows users to effectively utilize the AI ​​model under specific instructions.

[0086] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0087] Step 1:

[0088] The server receives visual information in real time from the monitoring device. Visual data is supplied as input, and a network connection is used to receive it stably. The received data is then directly passed on to the generation AI model for analysis.

[0089] Step 2:

[0090] The server analyzes the received visual information using a generating AI model. The input here is the visual data acquired in step 1. The AI ​​model, developed using, for example, TensorFlow, performs data calculations to extract the license plate and shape of moving objects from the visual data. The output is the identification information of the moving objects.

[0091] Step 3:

[0092] The server compares the generated identification information with the anomaly data set. The input is the identification information, which is the output of step 2. The server executes a database query to perform the comparison. For example, it uses SQL to check if the license plate exists in the stolen vehicle list. The output is the result of whether or not an anomaly exists.

[0093] Step 4:

[0094] The server generates notification information and sends it to the terminal when an anomaly is detected. The input is the anomaly detection result, which is the output of step 3. The generated notification information includes the license plate number and location information of the vehicle in question. This information is then passed to the terminal.

[0095] Step 5:

[0096] The terminal alerts the monitor based on notification information received from the server. The input is the notification information generated in step 4. The terminal displays the information visually using a GUI and also provides auditory attention by sounding an alarm. The output is an immediate status notification to the monitor.

[0097] Step 6:

[0098] The user checks for anomalies based on the information provided by the terminal. The input is the output information from step 5. The user checks the data on the monitor and assesses the situation on site. Based on this assessment, they notify an external organization if necessary. The output is the information to be provided to the external organization and preparations for a rapid response.

[0099] (Application Example 1)

[0100] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0101] Rapid detection and response to misconduct in monitored areas is a critical issue in modern society. However, conventional monitoring systems often require manual response when anomalies are detected, making rapid response difficult. Furthermore, coordination with external organizations when anomalies are detected can be slow, resulting in delayed responses.

[0102] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0103] In this invention, the server includes means for receiving and processing signal data from monitoring means, means for extracting identification information of a moving object based on the signal data, and means for comparing the identification information with a pre-prepared record of abnormal moving objects to detect anomalies. This makes it possible to detect fraudulent activity within the monitoring area in real time and to respond quickly. Furthermore, by notifying information visually and audibly on smart devices, it supports immediate response by personnel and facilitates smooth communication with external organizations.

[0104] "Monitoring means" refers to a device and system that have the function of continuously observing the conditions of a monitoring area and acquiring data.

[0105] "Signal data" refers to digital information such as video and audio that is acquired and processed by monitoring devices.

[0106] "Processing means" refers to hardware or software technology for analyzing signal data received from monitoring means and extracting useful information.

[0107] "Moving objects" refer to objects moving within the monitoring area, specifically including vehicles and people.

[0108] "Identification information" refers to data or characteristics used to identify or recognize a moving object.

[0109] An "anomalous movement record" is a database that collects and records information about suspicious or illegally moving objects in advance.

[0110] "Means of comparison" refers to a device or method that determines whether a moving object is abnormal by comparing extracted identification information with records of abnormal moving objects.

[0111] "Means for detecting anomalies" refers to technology that has the function of determining that a moving object is abnormal as a result of a comparison and taking the necessary next action.

[0112] "Means of visual and auditory notification" refers to devices or methods that convey information to the person in charge through visual displays or audio when an abnormality is detected.

[0113] "External organizations" refer to third-party facilities or institutions that need to be notified or coordinated when an anomaly occurs.

[0114] This invention is a system for rapidly detecting and responding to fraudulent activity in parking lots and public surveillance areas. The system primarily consists of three elements: a server, terminals, and users.

[0115] The server receives signal data from the monitoring device and analyzes the data using a generative AI model. Specifically, the server uses software libraries such as TensorFlow and OpenCV to extract identification information for moving objects from the video data. The extracted identification information is compared with the record of abnormal moving objects, and if an anomaly is detected, the process proceeds to the next step.

[0116] The terminal receives anomaly information from the server and notifies the responsible person in real time. This notification utilizes a smart device to convey the anomaly visually and audibly. The smart device can be a smartphone or another portable device. The responsible person who receives the notification will check the anomaly information and contact external organizations if necessary.

[0117] Users take immediate action based on anomaly information. This includes reviewing camera footage and promptly reporting to external agencies (such as the police). In this way, the system provides a framework for detecting fraudulent activity on the spot and taking immediate countermeasures.

[0118] For example, consider a scenario where a shopping mall parking lot is monitored. In this case, the surveillance camera transmits signal data of moving objects to a server. The generated AI model analyzes the data using a prompt message such as, "Please determine whether there is a suspicious vehicle in this parking lot footage," and matches it to an anomaly based on an identification mark. When the server detects an anomaly, it transmits the information to smart devices in real time, allowing personnel to immediately review the footage and report to security or the police as needed. This collaboration enables a faster and more reliable response than before.

[0119] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0120] Step 1:

[0121] The server receives signal data from the monitoring device. The input is a video stream from the surveillance camera, and this data is ready for analysis as output. First, the server converts the data into a playable format and separates it into frames.

[0122] Step 2:

[0123] The server analyzes the video data using a generative AI model. Here, the separated frames become input, and feature data of moving objects is output as identification information. The server prompts the AI ​​model to perform calculations using the prompt message, "Determine whether there is a suspicious vehicle in this parking lot video."

[0124] Step 3:

[0125] The server compares the extracted identification information with the abnormal movement record. Based on the identification mark and vehicle shape information, it performs a database search and checks for matching information. The input is the identification information from the previous step, and the output is a flag indicating whether or not an abnormality occurred.

[0126] Step 4:

[0127] The terminal receives anomaly detection results from the server and notifies the responsible person visually and audibly. The input consists of anomaly detection flags and related information, and the output is data displayed as an alert. The terminal provides this as an audio alert or screen display using a mobile notification API.

[0128] Step 5:

[0129] Users receive notifications from their devices and quickly check the situation. Inputs are alert information from the device, and outputs are the user's actions, such as reporting to the police. Users can operate the device to view video and related information and, if necessary, share information with external organizations.

[0130] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0131] This invention combines a system for detecting illegal vehicle activity in a monitored area with an emotion engine that recognizes the user's emotional state. This allows the system to check the user's emotions when an abnormal event occurs and the system notifies them, enabling more effective responses based on that information.

[0132] System Configuration

[0133] 1. Server Role

[0134] The server receives video data from the monitoring device and extracts vehicle identification information using a generated AI model. Based on this information, it compares it with a database of abnormal vehicles and has the function of detecting anomalies. In addition, it receives analysis results from the emotion engine and generates more detailed notification data.

[0135] 2. The role of the terminal

[0136] The terminal receives anomaly notification information sent from the server and notifies the monitoring personnel. Furthermore, the terminal captures the user's voice and facial expressions through the camera and other input devices and transmits this information to the emotion engine to analyze the user's emotional state.

[0137] 3. User Roles

[0138] Users immediately check the situation upon receiving a notification from their device. By analyzing the user's emotions, the system evaluates the user's stress level and optimizes how information is presented by changing the notification interface as needed.

[0139] Specific example

[0140] In a shopping mall parking lot, surveillance equipment provides real-time video. A server reads vehicle license plates and compares them to a list of suspicious vehicles to detect stolen vehicles. This information is sent to a terminal, which simultaneously sends the user's voice tone and facial expressions to an emotion engine to analyze the user's emotional state. For example, if the user is showing high levels of stress, the terminal can change how notifications are displayed, temporarily softening the alert sound or simplifying the interface. This reduces the burden on the user while enabling them to take necessary actions smoothly.

[0141] In this way, the present invention combines automated monitoring with user feedback to not only detect anomalies but also optimize the entire process of responding to them. This system makes a significant contribution to creating a safer environment and effective risk management.

[0142] The following describes the processing flow.

[0143] Step 1:

[0144] The server receives video data in real time from the monitoring device. The video data includes information about vehicles entering and exiting the parking lot.

[0145] Step 2:

[0146] The server analyzes the received video data using a generating AI model and extracts the vehicle's license plate number. This extracted license plate information is then compared against a database of abnormal vehicles.

[0147] Step 3:

[0148] If the server detects a stolen or illegally modified vehicle as a result of the verification process, it generates notification data indicating that an anomaly has been detected. This data includes detailed information about the vehicle in question.

[0149] Step 4:

[0150] The terminal receives notification data sent from the server and issues an alert to the monitoring personnel. The alert is immediately communicated visually and audibly.

[0151] Step 5:

[0152] The device also transmits the user's voice and facial expressions to the emotion engine via the camera and microphone. Based on this data, the user's emotional state is analyzed.

[0153] Step 6:

[0154] The emotion engine analyzes the user's emotional state and evaluates their stress level and type of emotion. This result is then fed back to the device.

[0155] Step 7:

[0156] The device adjusts how alerts are displayed based on feedback from the emotion engine. For example, if the user is stressed, the alert sound may be softened and the displayed content simplified.

[0157] Step 8:

[0158] Users can quickly check the situation based on coordinated notifications from their devices and contact external organizations if necessary. Emotional state is taken into consideration, allowing users to respond in the most optimal mental state.

[0159] (Example 2)

[0160] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0161] Modern surveillance systems require rapid detection and response to abnormal behavior. However, conventional methods fail to adequately address the need for responses that consider the emotional state of users, in addition to simply detecting anomalies. As a result, users experience increased stress when receiving information, making it difficult to make quick and appropriate decisions.

[0162] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0163] In this invention, the server includes means for receiving and analyzing digital information from a monitoring device, means for extracting identification information of transportation equipment based on the digital information, and means for collecting audio and video data to acquire and evaluate the user's emotional state. This makes it possible to detect abnormal behavior and present information according to the user's emotional state.

[0164] A "monitoring device" is a device used to acquire digital information in real time within a specific area.

[0165] "Digital information" refers to visual or audio data acquired from a monitoring device in a processable format.

[0166] "Transportation equipment" refers to equipment used for moving goods or people, such as vehicles.

[0167] "Identification information" refers to unique data used to distinguish transportation equipment from others, and includes registration identification codes, etc.

[0168] The "Abnormal Transportation Equipment Database" is a database that records information about abnormal behavior or specific conditions related to transportation equipment.

[0169] A "human interface" refers to a device that includes display devices and audio outputs for information exchange between a system and a user.

[0170] An "analysis device" is a computer system that receives, processes, and analyzes digital information and related data.

[0171] "Emotional state" refers to a state that is evaluated based on an analysis of the psychological or physiological characteristics exhibited by the user.

[0172] This invention is a system that detects fraudulent activity involving transportation equipment in a monitored area and evaluates the emotional state of users. To achieve this, the server, terminals, and users each play important roles.

[0173] Server Embodiment

[0174] The server receives digital information from monitoring devices and analyzes it using a generated AI model. The server extracts identification information for transportation equipment and detects anomalies by comparing it with a database of abnormal transportation equipment. The collected data is also used to evaluate the user's emotional state via an emotion engine.

[0175] Terminal embodiment

[0176] The device receives notifications of anomalies and communicates the information to the user through a human interface. Furthermore, the device uses its camera and microphone to collect the user's voice and video data, and an emotion engine analyzes their emotional state. Based on this information, the notification interface is adjusted according to the user's stress level.

[0177] User roles

[0178] Users receive anomaly notifications through their devices and check the situation. Optimizing the information provided based on their emotional state enables quick decision-making and response. For example, if a user is stressed, the notification method is adjusted to reduce their burden.

[0179] Specific example

[0180] A concrete example is a scenario in a shopping mall parking lot where a monitoring device acquires vehicle license plates in real time, and a server compares them with a database of abnormal transport equipment to detect stolen vehicles. The terminal receives this notification and simultaneously sends the user's voice and facial expressions to an emotion engine to analyze their emotional state. Based on these results, the terminal adjusts the notification interface to reduce stress.

[0181] Example of a prompt

[0182] "Please tell me how to extract a vehicle's license plate number from surveillance camera footage and determine if it is an illegal vehicle."

[0183] "Please explain in detail the process of analyzing the user's emotional state from their voice and facial expressions."

[0184] The system configured in this way enables a swift and effective response to fraudulent activities involving transportation equipment, as well as the provision of information while considering the burden on users.

[0185] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0186] Step 1:

[0187] The server receives digital information as input from the monitoring device. This digital information includes video data of transportation equipment within the monitoring area. The server uses a generation AI model to detect transportation equipment from the video and extracts features such as license plates as identification information. This process outputs data that can be compared with the abnormal transportation equipment database.

[0188] Step 2:

[0189] The server compares the identification information extracted in Step 1 with the database of abnormal transport equipment. By comparing the input identification information with the data in the database, it detects whether or not abnormal activity has occurred. This result is generated as abnormality notification data and is output when an abnormality occurs.

[0190] Step 3:

[0191] If an anomaly is detected, the server outputs an anomaly notification data to the terminal. The notification data contains detailed information about the anomaly that occurred. This allows the terminal to quickly understand the anomaly and is provided with the information needed to proceed to the next step.

[0192] Step 4:

[0193] The terminal receives abnormal notification data from the server as input and notifies the user through a human interface. These notifications include visual warnings and audio alerts. This allows the user to immediately recognize the situation and begin on-site response.

[0194] Step 5:

[0195] The device uses a camera and microphone to collect the user's voice and facial expressions as input. The input emotional data is analyzed by an emotion engine to evaluate the user's emotional state. Through this data processing and calculation, the user's stress level and anxiety level are quantified, and the analysis results are output.

[0196] Step 6:

[0197] The server receives analysis results from the emotion engine and adjusts the notification interface based on the user's emotional state. Specifically, it presents information by lowering the volume of alert sounds, simplifying the layout of the notification screen, and so on. This process reduces the burden on the user and streamlines the response process.

[0198] (Application Example 2)

[0199] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0200] While detecting vehicle misconduct in monitored areas is crucial, conventional systems focused solely on detecting anomalies, neglecting consideration for the user's psychological state and stress levels. This presented a challenge: notifications of abnormal events could potentially place an excessive burden and stress on users.

[0201] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0202] In this invention, the server includes means for receiving and processing video data from a monitoring device, means for extracting vehicle identification information based on the video data, means for comparing the identification information with a pre-prepared abnormal vehicle database to detect abnormalities, means for notifying a display device of the detected abnormality and generating notification data, means for transmitting the notification data to an external organization, means for capturing and analyzing the user's voice and facial expressions, and means for dynamically adjusting the notification method based on the user's emotional state. This enables flexible information presentation that takes into account the user's psychological state as well as abnormality detection.

[0203] A "monitoring device" is a device that collects video data in a specific area, and this video data is used for anomaly detection.

[0204] "Video data" refers to image information acquired by cameras and other optical devices, and serves as the basis for extracting vehicle identification information.

[0205] "Vehicle identification information" refers to information used to identify a vehicle, and usually refers to the vehicle's license plate information.

[0206] The "abnormal vehicle database" is a database that stores identification information of vehicles suspected of engaging in illegal activities, and is used to cross-reference this information with the information of detected vehicles.

[0207] "User emotional state" refers to information that represents the user's current psychological response and stress level, and is analyzed from information such as voice and facial expressions.

[0208] "Dynamically adjusting notification methods" means automatically changing how information is presented according to the user's emotional state, thereby enabling users to receive information more appropriately.

[0209] The system for realizing this invention consists of four main elements: a monitoring device, a server, a terminal, and a user. The server receives video data from the monitoring device and extracts vehicle identification information using a generated AI model. This identification information is compared with a pre-prepared database of abnormal vehicles, and when an abnormality is detected, the server notifies the display device of this information and generates detailed notification data. Furthermore, the server uses an emotion engine to analyze the user's emotional state and dynamically adjusts the notification method based on the results.

[0210] The device receives anomaly notification information from the server and captures the user's voice and facial expressions via its camera and microphone, transmitting this information to the emotion engine. The device then analyzes the user's emotional state and adjusts the notification interface as needed.

[0211] Users receive notifications from their devices and quickly check the situation. By analyzing the user's emotions, the system assesses the user's stress level and reduces the user's burden by providing appropriate information.

[0212] As a concrete example, consider the use of smart glasses by security guards at a large-scale event venue. The smart glasses collect video footage of the venue in real time, and a server identifies suspicious vehicles and individuals. The system also captures the user's voice and facial expressions, and can adjust the warning sound and simplify the notification interface based on the user's stress level when they receive information about a suspicious vehicle. An example of a prompt to the generated AI model would be, "Analyze the emotions this user exhibits when they detect inappropriate activity and select the most appropriate notification method."

[0213] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0214] Step 1:

[0215] The server receives video data in real time from the monitoring device. The input is raw video data from the monitoring device, which is digitized using an image processing algorithm. The output is identifiable video data.

[0216] Step 2:

[0217] The server extracts vehicle identification information from the received video data using a generative AI model. The input is the identifiable video data obtained in step 1, and the AI ​​model analyzes the data to identify vehicle identification information such as license plates. The output is the vehicle identification information.

[0218] Step 3:

[0219] The server compares the extracted vehicle identification information with the abnormal vehicle database. The input is the vehicle identification information, which is the output data from step 2. By comparing this information with the database, the server determines whether or not an abnormality has been detected. The output is the result of whether or not an abnormality was detected.

[0220] Step 4:

[0221] If an anomaly is detected, the server sends the generated notification data to the terminal. The input is the anomaly detection result obtained in step 3, and the server creates the notification data based on this information. The output is the notification data to be sent to the terminal.

[0222] Step 5:

[0223] The terminal receives anomaly notification data sent from the server and notifies the user. The input is the notification data sent in step 4, and the terminal performs the action of displaying the information on the display device. The output is the warning that the user receives visually and audibly.

[0224] Step 6:

[0225] The device uses a camera and microphone to capture the user's voice and facial expressions and sends them to the emotion engine. The input is the user's real-time voice and video information, and the user's emotional state is evaluated through emotion analysis. The output is data on the user's emotional state.

[0226] Step 7:

[0227] The server dynamically adjusts the notification method based on the results of the sentiment analysis. The input is the user's sentiment data obtained in step 6, and the server optimizes actions such as adjusting the alert sound volume and the display method of the interface. The output is the adjusted notification interface.

[0228] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0229] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0230] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0231] [Second Embodiment]

[0232] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0233] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0234] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0235] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0236] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0237] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0238] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0239] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0240] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0241] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0242] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0243] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0244] This invention provides a system for immediately detecting and quickly responding to illegal vehicle activity in monitored areas such as parking lots. This system works in conjunction with multiple monitoring devices and provides a platform for analyzing acquired video data in real time.

[0245] System Configuration

[0246] 1. Server Role

[0247] The server continuously receives video data from the monitoring device and applies a generative AI model to extract vehicle identification information. This identification information includes the license plate and vehicle shape information of the identified vehicle. Based on this, the server compares it with a database of abnormal vehicles and processes the information further if an abnormality is detected.

[0248] 2. The role of the terminal

[0249] The terminal receives anomaly information transmitted from the server and notifies the monitoring personnel visually and audibly. This allows the monitoring personnel to understand the situation in real time and take necessary actions quickly.

[0250] 3. User Roles

[0251] Based on the information provided by the device, users can check for anomalies and, if necessary, manually provide additional information to external organizations, such as the police. This allows for smoother on-site response and investigation.

[0252] Specific example

[0253] For example, in the underground parking lot of a shopping mall, a surveillance system provides real-time video. A server receives this video and recognizes the license plates of vehicles. If the recognized license plate matches a list of stolen vehicles, the server immediately detects this anomaly and sends the relevant information to a terminal. The terminal displays this information visually and alerts the person in charge. The person in charge reviews the video, contacts the police if necessary, and organizes the response at the scene. Because this entire process is automated and in real time, it enables a faster and more reliable response than traditional methods.

[0254] This implementation allows for the immediate detection and prompt response to any misuse of vehicles within the surveillance area. This system contributes to crime prevention and rapid response after an incident occurs, thereby ensuring safe public spaces.

[0255] The following describes the processing flow.

[0256] Step 1:

[0257] The server receives streaming video data transmitted from the monitoring device in real time. This includes extracting frame data that contains the vehicle to be processed.

[0258] Step 2:

[0259] The server applies a generative AI model to the received video data to extract vehicle license plate information. During this process, it performs noise reduction and format conversion to prepare the data for easier analysis.

[0260] Step 3:

[0261] The server compares the extracted license plate information with a database of abnormal vehicles. Here, it makes a determination to identify vehicles suspected of being stolen or illegally modified.

[0262] Step 4:

[0263] If an anomaly is detected as a result of the matching process, the server organizes relevant data such as snapshots and location information of the vehicle and generates notification data.

[0264] Step 5:

[0265] The server sends the generated notification data to the terminal, informing the monitoring personnel of the details of the anomaly. This notification includes an alert to prompt immediate action.

[0266] Step 6:

[0267] The terminal provides the monitoring personnel with received notification data visually and audibly. This allows the personnel to quickly assess the situation and take necessary countermeasures.

[0268] Step 7:

[0269] The user will check the abnormal situation based on the information from the device. If necessary, the user will contact the police or other relevant organizations directly to expedite the response at the scene.

[0270] (Example 1)

[0271] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0272] There is a need to quickly and accurately detect anomalies in moving objects within the monitoring area and prompt immediate responses based on those detections. In particular, it is crucial to efficiently identify anomalies without relying on human intervention and to facilitate smooth cooperation with external organizations.

[0273] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0274] In this invention, the server includes means for receiving and analyzing visual information from a video acquisition means, means for extracting identification information of a moving object based on the visual information, and means for comparing the identification information with a pre-prepared set of abnormal data to detect abnormalities. This enables automatic and real-time detection of abnormalities within a monitoring area.

[0275] A "video acquisition means" is a mechanism for acquiring visual information from a surveillance area and supplying it to a processing system.

[0276] "Visual information" refers to images and video data provided by surveillance devices.

[0277] "Means of analysis" refers to the processes and techniques used to process acquired visual information and extract necessary identification information.

[0278] "Identification information of a moving object" refers to data for identifying a moving object such as a vehicle, and includes, for example, license plate numbers and shape information.

[0279] "Abnormal data set" refers to a collection of pre-prepared information regarding abnormal states or events.

[0280] "Notification information" refers to information transmitted to external agencies or monitors when an abnormality is detected.

[0281] "Generated AI model" means a machine learning model used to extract useful information from visual information.

[0282] "Analysis of identification information" refers to the process of interpreting the detected identification information to identify abnormalities.

[0283] "Warning signal" refers to a visual or auditory notification to alert the monitor when an abnormality is discovered.

[0284] "Terminal device" refers to a device that provides the notification information received from the server to the monitor and prompts necessary actions.

[0285] This invention is a system that can quickly detect abnormalities within a monitored area and enable immediate response. Specifically, three entities: the server, the terminal, and the user cooperate to operate.

[0286] Role of the server:

[0287] The server receives visual information acquired from monitoring devices. This process uses network communication technology, such as a streaming protocol. The received data is analyzed using a generative AI model to extract identification information for moving objects. The model is developed using software frameworks such as TensorFlow and PyTorch, and identifies the license plates and shapes of moving objects. The identification information is compared with a set of anomaly data, and if an anomaly is detected, information processing is performed. A specific example is identifying a license plate that matches a list of stolen vehicles. In this case, the server immediately generates notification information.

[0288] Terminal role:

[0289] The terminal receives notification information from the server and provides visual and auditory alerts to the monitor. This includes displaying information via a GUI and using alarm sounds. The terminal communicates with the server using the TCP / IP protocol to ensure stable data reception.

[0290] User roles:

[0291] Users make on-site decisions based on information provided by their devices. For example, they can identify vehicles that have been flagged as abnormal on the monitoring screen and provide information to external organizations such as the police if necessary. This process enables rapid response and coordination.

[0292] An example of a prompt message for the generated AI model is, "Extract vehicle license plates from the surveillance footage and compare them with the existing list of stolen vehicles." This allows users to effectively utilize the AI ​​model under specific instructions.

[0293] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0294] Step 1:

[0295] The server receives visual information in real time from the monitoring device. Visual data is supplied as input, and a network connection is used to receive it stably. The received data is then directly passed on to the generation AI model for analysis.

[0296] Step 2:

[0297] The server analyzes the received visual information using a generating AI model. The input here is the visual data acquired in step 1. The AI ​​model, developed using, for example, TensorFlow, performs data calculations to extract the license plate and shape of moving objects from the visual data. The output is the identification information of the moving objects.

[0298] Step 3:

[0299] The server compares the generated identification information with the anomaly data set. The input is the identification information, which is the output of step 2. The server executes a database query to perform the comparison. For example, it uses SQL to check if the license plate exists in the stolen vehicle list. The output is the result of whether or not an anomaly exists.

[0300] Step 4:

[0301] The server generates notification information and sends it to the terminal when an anomaly is detected. The input is the anomaly detection result, which is the output of step 3. The generated notification information includes the license plate number and location information of the vehicle in question. This information is then passed to the terminal.

[0302] Step 5:

[0303] The terminal alerts the monitor based on notification information received from the server. The input is the notification information generated in step 4. The terminal displays the information visually using a GUI and also provides auditory attention by sounding an alarm. The output is an immediate status notification to the monitor.

[0304] Step 6:

[0305] The user checks for abnormalities based on the information provided by the terminal. The input is the output information of step 5. The user checks the data on the monitor and determines the situation at the site. Based on this determination, if necessary, the user reports to an external agency. The output is the information provided to the external agency and the preparation for a prompt response.

[0306] (Application Example 1)

[0307] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0308] The prompt detection and response to illegal acts in the monitoring area are important issues in modern society. However, in conventional monitoring systems, it is necessary to manually respond when detecting abnormalities, and prompt response is often difficult. In addition, when an abnormality is detected, cooperation with an external organization may not be smoothly carried out, and as a result, the response may be delayed, which has been a problem.

[0309] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0310] In this invention, the server includes means for receiving and processing signal data from the monitoring means, means for extracting identification information of the moving object based on the signal data, and means for collating the identification information with a pre-prepared record of abnormal moving objects to detect abnormalities. Thereby, it becomes possible to detect illegal acts in the monitoring area in real time and respond promptly. In addition, by notifying information visually and auditorily on the smart device, it is possible to support the immediate response by the person in charge and smoothly contact an external organization.

[0311] The "monitoring means" is a device and its system that continuously observes the situation in the monitoring area and acquires data.

[0312] "Signal data" refers to digital information such as video and audio that is acquired and processed by monitoring devices.

[0313] "Processing means" refers to hardware or software technology for analyzing signal data received from monitoring means and extracting useful information.

[0314] "Moving objects" refer to objects moving within the monitoring area, specifically including vehicles and people.

[0315] "Identification information" refers to data or characteristics used to identify or recognize a moving object.

[0316] An "anomalous movement record" is a database that collects and records information about suspicious or illegally moving objects in advance.

[0317] "Means of comparison" refers to a device or method that determines whether a moving object is abnormal by comparing extracted identification information with records of abnormal moving objects.

[0318] "Means for detecting anomalies" refers to technology that has the function of determining that a moving object is abnormal as a result of a comparison and taking the necessary next action.

[0319] "Means of visual and auditory notification" refers to devices or methods that convey information to the person in charge through visual displays or audio when an abnormality is detected.

[0320] "External organizations" refer to third-party facilities or institutions that need to be notified or coordinated when an anomaly occurs.

[0321] This invention is a system for rapidly detecting and responding to fraudulent activity in parking lots and public surveillance areas. The system primarily consists of three elements: a server, terminals, and users.

[0322] The server receives signal data from the monitoring device and analyzes the data using a generative AI model. Specifically, the server uses software libraries such as TensorFlow and OpenCV to extract identification information for moving objects from the video data. The extracted identification information is compared with the record of abnormal moving objects, and if an anomaly is detected, the process proceeds to the next step.

[0323] The terminal receives anomaly information from the server and notifies the responsible person in real time. This notification utilizes a smart device to convey the anomaly visually and audibly. The smart device can be a smartphone or another portable device. The responsible person who receives the notification will check the anomaly information and contact external organizations if necessary.

[0324] Users take immediate action based on anomaly information. This includes reviewing camera footage and promptly reporting to external agencies (such as the police). In this way, the system provides a framework for detecting fraudulent activity on the spot and taking immediate countermeasures.

[0325] For example, consider a scenario where a shopping mall parking lot is monitored. In this case, the surveillance camera transmits signal data of moving objects to a server. The generated AI model analyzes the data using a prompt message such as, "Please determine whether there is a suspicious vehicle in this parking lot footage," and matches it to an anomaly based on an identification mark. When the server detects an anomaly, it transmits the information to smart devices in real time, allowing personnel to immediately review the footage and report to security or the police as needed. This collaboration enables a faster and more reliable response than before.

[0326] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0327] Step 1:

[0328] The server receives signal data from the monitoring device. The input is a video stream from the surveillance camera, and this data is ready for analysis as output. First, the server converts the data into a playable format and separates it into frames.

[0329] Step 2:

[0330] The server analyzes the video data using a generative AI model. Here, the separated frames become input, and feature data of moving objects is output as identification information. The server prompts the AI ​​model to perform calculations using the prompt message, "Determine whether there is a suspicious vehicle in this parking lot video."

[0331] Step 3:

[0332] The server compares the extracted identification information with the abnormal movement record. Based on the identification mark and vehicle shape information, it performs a database search and checks for matching information. The input is the identification information from the previous step, and the output is a flag indicating whether or not an abnormality occurred.

[0333] Step 4:

[0334] The terminal receives anomaly detection results from the server and notifies the responsible person visually and audibly. The input consists of anomaly detection flags and related information, and the output is data displayed as an alert. The terminal provides this as an audio alert or screen display using a mobile notification API.

[0335] Step 5:

[0336] Users receive notifications from their devices and quickly check the situation. Inputs are alert information from the device, and outputs are the user's actions, such as reporting to the police. Users can operate the device to view video and related information and, if necessary, share information with external organizations.

[0337] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0338] This invention combines a system for detecting illegal vehicle activity in a monitored area with an emotion engine that recognizes the user's emotional state. This allows the system to check the user's emotions when an abnormal event occurs and the system notifies them, enabling more effective responses based on that information.

[0339] System Configuration

[0340] 1. Server Role

[0341] The server receives video data from the monitoring device and extracts vehicle identification information using a generated AI model. Based on this information, it compares it with a database of abnormal vehicles and has the function of detecting anomalies. In addition, it receives analysis results from the emotion engine and generates more detailed notification data.

[0342] 2. The role of the terminal

[0343] The terminal receives anomaly notification information sent from the server and notifies the monitoring personnel. Furthermore, the terminal captures the user's voice and facial expressions through the camera and other input devices and transmits this information to the emotion engine to analyze the user's emotional state.

[0344] 3. User Roles

[0345] Users immediately check the situation upon receiving a notification from their device. By analyzing the user's emotions, the system evaluates the user's stress level and optimizes how information is presented by changing the notification interface as needed.

[0346] Specific example

[0347] In a shopping mall parking lot, surveillance equipment provides real-time video. A server reads vehicle license plates and compares them to a list of suspicious vehicles to detect stolen vehicles. This information is sent to a terminal, which simultaneously sends the user's voice tone and facial expressions to an emotion engine to analyze the user's emotional state. For example, if the user is showing high levels of stress, the terminal can change how notifications are displayed, temporarily softening the alert sound or simplifying the interface. This reduces the burden on the user while enabling them to take necessary actions smoothly.

[0348] In this way, the present invention combines automated monitoring with user feedback to not only detect anomalies but also optimize the entire process of responding to them. This system makes a significant contribution to creating a safer environment and effective risk management.

[0349] The following describes the processing flow.

[0350] Step 1:

[0351] The server receives video data in real time from the monitoring device. The video data includes information about vehicles entering and exiting the parking lot.

[0352] Step 2:

[0353] The server analyzes the received video data using a generating AI model and extracts the vehicle's license plate number. This extracted license plate information is then compared against a database of abnormal vehicles.

[0354] Step 3:

[0355] If the server detects a stolen or illegally modified vehicle as a result of the verification process, it generates notification data indicating that an anomaly has been detected. This data includes detailed information about the vehicle in question.

[0356] Step 4:

[0357] The terminal receives notification data sent from the server and issues an alert to the monitoring personnel. The alert is immediately communicated visually and audibly.

[0358] Step 5:

[0359] The device also transmits the user's voice and facial expressions to the emotion engine via the camera and microphone. Based on this data, the user's emotional state is analyzed.

[0360] Step 6:

[0361] The emotion engine analyzes the user's emotional state and evaluates their stress level and type of emotion. This result is then fed back to the device.

[0362] Step 7:

[0363] The device adjusts how alerts are displayed based on feedback from the emotion engine. For example, if the user is stressed, the alert sound may be softened and the displayed content simplified.

[0364] Step 8:

[0365] Users can quickly check the situation based on coordinated notifications from their devices and contact external organizations if necessary. Emotional state is taken into consideration, allowing users to respond in the most optimal mental state.

[0366] (Example 2)

[0367] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0368] Modern surveillance systems require rapid detection and response to abnormal behavior. However, conventional methods fail to adequately address the need for responses that consider the emotional state of users, in addition to simply detecting anomalies. As a result, users experience increased stress when receiving information, making it difficult to make quick and appropriate decisions.

[0369] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0370] In this invention, the server includes means for receiving and analyzing digital information from a monitoring device, means for extracting identification information of transportation equipment based on the digital information, and means for collecting audio and video data to acquire and evaluate the user's emotional state. This makes it possible to detect abnormal behavior and present information according to the user's emotional state.

[0371] A "monitoring device" is a device used to acquire digital information in real time within a specific area.

[0372] "Digital information" refers to visual or audio data acquired from a monitoring device in a processable format.

[0373] "Transportation equipment" refers to equipment used for moving goods or people, such as vehicles.

[0374] "Identification information" refers to unique data used to distinguish transportation equipment from others, and includes registration identification codes, etc.

[0375] The "Abnormal Transportation Equipment Database" is a database that records information about abnormal behavior or specific conditions related to transportation equipment.

[0376] A "human interface" refers to a device that includes display devices and audio outputs for information exchange between a system and a user.

[0377] An "analysis device" is a computer system that receives, processes, and analyzes digital information and related data.

[0378] "Emotional state" refers to a state that is evaluated based on an analysis of the psychological or physiological characteristics exhibited by the user.

[0379] This invention is a system that detects fraudulent activity involving transportation equipment in a monitored area and evaluates the emotional state of users. To achieve this, the server, terminals, and users each play important roles.

[0380] Server Embodiment

[0381] The server receives digital information from monitoring devices and analyzes it using a generated AI model. The server extracts identification information for transportation equipment and detects anomalies by comparing it with a database of abnormal transportation equipment. The collected data is also used to evaluate the user's emotional state via an emotion engine.

[0382] Terminal embodiment

[0383] The device receives notifications of anomalies and communicates the information to the user through a human interface. Furthermore, the device uses its camera and microphone to collect the user's voice and video data, and an emotion engine analyzes their emotional state. Based on this information, the notification interface is adjusted according to the user's stress level.

[0384] User roles

[0385] Users receive anomaly notifications through their devices and check the situation. Optimizing the information provided based on their emotional state enables quick decision-making and response. For example, if a user is stressed, the notification method is adjusted to reduce their burden.

[0386] Specific example

[0387] A concrete example is a scenario in a shopping mall parking lot where a monitoring device acquires vehicle license plates in real time, and a server compares them with a database of abnormal transport equipment to detect stolen vehicles. The terminal receives this notification and simultaneously sends the user's voice and facial expressions to an emotion engine to analyze their emotional state. Based on these results, the terminal adjusts the notification interface to reduce stress.

[0388] Example of a prompt

[0389] "Please tell me how to extract a vehicle's license plate number from surveillance camera footage and determine if it is an illegal vehicle."

[0390] "Please explain in detail the process of analyzing the user's emotional state from their voice and facial expressions."

[0391] The system configured in this way enables a swift and effective response to fraudulent activities involving transportation equipment, as well as the provision of information while considering the burden on users.

[0392] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0393] Step 1:

[0394] The server receives digital information as input from the monitoring device. This digital information includes video data of transportation equipment within the monitoring area. The server uses a generation AI model to detect transportation equipment from the video and extracts features such as license plates as identification information. This process outputs data that can be compared with the abnormal transportation equipment database.

[0395] Step 2:

[0396] The server compares the identification information extracted in Step 1 with the database of abnormal transport equipment. By comparing the input identification information with the data in the database, it detects whether or not abnormal activity has occurred. This result is generated as abnormality notification data and is output when an abnormality occurs.

[0397] Step 3:

[0398] If an anomaly is detected, the server outputs an anomaly notification data to the terminal. The notification data contains detailed information about the anomaly that occurred. This allows the terminal to quickly understand the anomaly and is provided with the information needed to proceed to the next step.

[0399] Step 4:

[0400] The terminal receives abnormal notification data from the server as input and notifies the user through a human interface. These notifications include visual warnings and audio alerts. This allows the user to immediately recognize the situation and begin on-site response.

[0401] Step 5:

[0402] The device uses a camera and microphone to collect the user's voice and facial expressions as input. The input emotional data is analyzed by an emotion engine to evaluate the user's emotional state. Through this data processing and calculation, the user's stress level and anxiety level are quantified, and the analysis results are output.

[0403] Step 6:

[0404] The server receives analysis results from the emotion engine and adjusts the notification interface based on the user's emotional state. Specifically, it presents information by lowering the volume of alert sounds, simplifying the layout of the notification screen, and so on. This process reduces the burden on the user and streamlines the response process.

[0405] (Application Example 2)

[0406] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0407] While detecting vehicle misconduct in monitored areas is crucial, conventional systems focused solely on detecting anomalies, neglecting consideration for the user's psychological state and stress levels. This presented a challenge: notifications of abnormal events could potentially place an excessive burden and stress on users.

[0408] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0409] In this invention, the server includes means for receiving and processing video data from a monitoring device, means for extracting vehicle identification information based on the video data, means for comparing the identification information with a pre-prepared abnormal vehicle database to detect abnormalities, means for notifying a display device of the detected abnormality and generating notification data, means for transmitting the notification data to an external organization, means for capturing and analyzing the user's voice and facial expressions, and means for dynamically adjusting the notification method based on the user's emotional state. This enables flexible information presentation that takes into account the user's psychological state as well as abnormality detection.

[0410] A "monitoring device" is a device that collects video data in a specific area, and this video data is used for anomaly detection.

[0411] "Video data" refers to image information acquired by cameras and other optical devices, and serves as the basis for extracting vehicle identification information.

[0412] "Vehicle identification information" refers to information used to identify a vehicle, and usually refers to the vehicle's license plate information.

[0413] The "abnormal vehicle database" is a database that stores identification information of vehicles suspected of engaging in illegal activities, and is used to cross-reference this information with the information of detected vehicles.

[0414] "User emotional state" refers to information that represents the user's current psychological response and stress level, and is analyzed from information such as voice and facial expressions.

[0415] "Dynamically adjusting notification methods" means automatically changing how information is presented according to the user's emotional state, thereby enabling users to receive information more appropriately.

[0416] The system for realizing this invention consists of four main elements: a monitoring device, a server, a terminal, and a user. The server receives video data from the monitoring device and extracts vehicle identification information using a generated AI model. This identification information is compared with a pre-prepared database of abnormal vehicles, and when an abnormality is detected, the server notifies the display device of this information and generates detailed notification data. Furthermore, the server uses an emotion engine to analyze the user's emotional state and dynamically adjusts the notification method based on the results.

[0417] The device receives anomaly notification information from the server and captures the user's voice and facial expressions via its camera and microphone, transmitting this information to the emotion engine. The device then analyzes the user's emotional state and adjusts the notification interface as needed.

[0418] Users receive notifications from their devices and quickly check the situation. By analyzing the user's emotions, the system assesses the user's stress level and reduces the user's burden by providing appropriate information.

[0419] As a concrete example, consider the use of smart glasses by security guards at a large-scale event venue. The smart glasses collect video footage of the venue in real time, and a server identifies suspicious vehicles and individuals. The system also captures the user's voice and facial expressions, and can adjust the warning sound and simplify the notification interface based on the user's stress level when they receive information about a suspicious vehicle. An example of a prompt to the generated AI model would be, "Analyze the emotions this user exhibits when they detect inappropriate activity and select the most appropriate notification method."

[0420] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0421] Step 1:

[0422] The server receives video data in real time from the monitoring device. The input is raw video data from the monitoring device, which is digitized using an image processing algorithm. The output is identifiable video data.

[0423] Step 2:

[0424] The server extracts vehicle identification information from the received video data using a generative AI model. The input is the identifiable video data obtained in step 1, and the AI ​​model analyzes the data to identify vehicle identification information such as license plates. The output is the vehicle identification information.

[0425] Step 3:

[0426] The server compares the extracted vehicle identification information with the abnormal vehicle database. The input is the vehicle identification information, which is the output data from step 2. By comparing this information with the database, the server determines whether or not an abnormality has been detected. The output is the result of whether or not an abnormality was detected.

[0427] Step 4:

[0428] If an anomaly is detected, the server sends the generated notification data to the terminal. The input is the anomaly detection result obtained in step 3, and the server creates the notification data based on this information. The output is the notification data to be sent to the terminal.

[0429] Step 5:

[0430] The terminal receives anomaly notification data sent from the server and notifies the user. The input is the notification data sent in step 4, and the terminal performs the action of displaying the information on the display device. The output is the warning that the user receives visually and audibly.

[0431] Step 6:

[0432] The device uses a camera and microphone to capture the user's voice and facial expressions and sends them to the emotion engine. The input is the user's real-time voice and video information, and the user's emotional state is evaluated through emotion analysis. The output is data on the user's emotional state.

[0433] Step 7:

[0434] The server dynamically adjusts the notification method based on the results of the sentiment analysis. The input is the user's sentiment data obtained in step 6, and the server optimizes actions such as adjusting the alert sound volume and the display method of the interface. The output is the adjusted notification interface.

[0435] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0436] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0437] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0438] [Third Embodiment]

[0439] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0440] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0441] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0442] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0443] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0444] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0445] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0446] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0447] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0448] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0449] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0450] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0451] This invention provides a system for immediately detecting and quickly responding to illegal vehicle activity in monitored areas such as parking lots. This system works in conjunction with multiple monitoring devices and provides a platform for analyzing acquired video data in real time.

[0452] System Configuration

[0453] 1. Server Role

[0454] The server continuously receives video data from the monitoring device and applies a generative AI model to extract vehicle identification information. This identification information includes the license plate and vehicle shape information of the identified vehicle. Based on this, the server compares it with a database of abnormal vehicles and processes the information further if an abnormality is detected.

[0455] 2. The role of the terminal

[0456] The terminal receives anomaly information transmitted from the server and notifies the monitoring personnel visually and audibly. This allows the monitoring personnel to understand the situation in real time and take necessary actions quickly.

[0457] 3. User Roles

[0458] Based on the information provided by the device, users can check for anomalies and, if necessary, manually provide additional information to external organizations, such as the police. This allows for smoother on-site response and investigation.

[0459] Specific example

[0460] For example, in the underground parking lot of a shopping mall, a surveillance system provides real-time video. A server receives this video and recognizes the license plates of vehicles. If the recognized license plate matches a list of stolen vehicles, the server immediately detects this anomaly and sends the relevant information to a terminal. The terminal displays this information visually and alerts the person in charge. The person in charge reviews the video, contacts the police if necessary, and organizes the response at the scene. Because this entire process is automated and in real time, it enables a faster and more reliable response than traditional methods.

[0461] This implementation allows for the immediate detection and prompt response to any misuse of vehicles within the surveillance area. This system contributes to crime prevention and rapid response after an incident occurs, thereby ensuring safe public spaces.

[0462] The following describes the processing flow.

[0463] Step 1:

[0464] The server receives streaming video data transmitted from the monitoring device in real time. This includes extracting frame data that contains the vehicle to be processed.

[0465] Step 2:

[0466] The server applies a generative AI model to the received video data to extract vehicle license plate information. During this process, it performs noise reduction and format conversion to prepare the data for easier analysis.

[0467] Step 3:

[0468] The server compares the extracted license plate information with a database of abnormal vehicles. Here, it makes a determination to identify vehicles suspected of being stolen or illegally modified.

[0469] Step 4:

[0470] If an anomaly is detected as a result of the matching process, the server organizes relevant data such as snapshots and location information of the vehicle and generates notification data.

[0471] Step 5:

[0472] The server sends the generated notification data to the terminal, informing the monitoring personnel of the details of the anomaly. This notification includes an alert to prompt immediate action.

[0473] Step 6:

[0474] The terminal provides the monitoring personnel with received notification data visually and audibly. This allows the personnel to quickly assess the situation and take necessary countermeasures.

[0475] Step 7:

[0476] The user will check the abnormal situation based on the information from the device. If necessary, the user will contact the police or other relevant organizations directly to expedite the response at the scene.

[0477] (Example 1)

[0478] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0479] There is a need to quickly and accurately detect anomalies in moving objects within the monitoring area and prompt immediate responses based on those detections. In particular, it is crucial to efficiently identify anomalies without relying on human intervention and to facilitate smooth cooperation with external organizations.

[0480] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0481] In this invention, the server includes means for receiving and analyzing visual information from a video acquisition means, means for extracting identification information of a moving object based on the visual information, and means for comparing the identification information with a pre-prepared set of abnormal data to detect abnormalities. This enables automatic and real-time detection of abnormalities within a monitoring area.

[0482] A "video acquisition means" is a mechanism for acquiring visual information from a surveillance area and supplying it to a processing system.

[0483] "Visual information" refers to images and video data provided by surveillance devices.

[0484] "Means of analysis" refers to the processes and techniques used to process acquired visual information and extract necessary identification information.

[0485] "Identification information for moving objects" refers to data used to identify moving objects such as vehicles, and includes information such as license plates and shapes.

[0486] An "anomalous data set" refers to a pre-prepared collection of information about an abnormal state or event.

[0487] "Notification information" refers to information that is sent to external organizations or monitors when an anomaly is detected.

[0488] A "generative AI model" refers to a machine learning model used to extract useful information from visual information.

[0489] "Analysis of identification information" refers to the process of interpreting detected identification information and identifying anomalies.

[0490] A "warning signal" refers to a visual or auditory notification used to alert a monitor when an anomaly is detected.

[0491] A "terminal device" refers to a device that provides notification information received from a server to a monitor and prompts them to take necessary actions.

[0492] This invention is a system that enables rapid detection of anomalies within a monitored area and immediate response. Specifically, it operates through the cooperation of three entities: a server, a terminal, and a user.

[0493] Server role:

[0494] The server receives visual information acquired from monitoring devices. This process uses network communication technology, such as a streaming protocol. The received data is analyzed using a generative AI model to extract identification information for moving objects. The model is developed using software frameworks such as TensorFlow and PyTorch, and identifies the license plates and shapes of moving objects. The identification information is compared with a set of anomaly data, and if an anomaly is detected, information processing is performed. A specific example is identifying a license plate that matches a list of stolen vehicles. In this case, the server immediately generates notification information.

[0495] Terminal role:

[0496] The terminal receives notification information from the server and provides visual and auditory alerts to the monitor. This includes displaying information via a GUI and using alarm sounds. The terminal communicates with the server using the TCP / IP protocol to ensure stable data reception.

[0497] User roles:

[0498] Users make on-site decisions based on information provided by their devices. For example, they can identify vehicles that have been flagged as abnormal on the monitoring screen and provide information to external organizations such as the police if necessary. This process enables rapid response and coordination.

[0499] An example of a prompt message for the generated AI model is, "Extract vehicle license plates from the surveillance footage and compare them with the existing list of stolen vehicles." This allows users to effectively utilize the AI ​​model under specific instructions.

[0500] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0501] Step 1:

[0502] The server receives visual information in real time from the monitoring device. Visual data is supplied as input, and a network connection is used to receive it stably. The received data is then directly passed on to the generation AI model for analysis.

[0503] Step 2:

[0504] The server analyzes the received visual information using a generating AI model. The input here is the visual data acquired in step 1. The AI ​​model, developed using, for example, TensorFlow, performs data calculations to extract the license plate and shape of moving objects from the visual data. The output is the identification information of the moving objects.

[0505] Step 3:

[0506] The server compares the generated identification information with the anomaly data set. The input is the identification information, which is the output of step 2. The server executes a database query to perform the comparison. For example, it uses SQL to check if the license plate exists in the stolen vehicle list. The output is the result of whether or not an anomaly exists.

[0507] Step 4:

[0508] The server generates notification information and sends it to the terminal when an anomaly is detected. The input is the anomaly detection result, which is the output of step 3. The generated notification information includes the license plate number and location information of the vehicle in question. This information is then passed to the terminal.

[0509] Step 5:

[0510] The terminal alerts the monitor based on notification information received from the server. The input is the notification information generated in step 4. The terminal displays the information visually using a GUI and also provides auditory attention by sounding an alarm. The output is an immediate status notification to the monitor.

[0511] Step 6:

[0512] The user checks for anomalies based on the information provided by the terminal. The input is the output information from step 5. The user checks the data on the monitor and assesses the situation on site. Based on this assessment, they notify an external organization if necessary. The output is the information to be provided to the external organization and preparations for a rapid response.

[0513] (Application Example 1)

[0514] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0515] Rapid detection and response to misconduct in monitored areas is a critical issue in modern society. However, conventional monitoring systems often require manual response when anomalies are detected, making rapid response difficult. Furthermore, coordination with external organizations when anomalies are detected can be slow, resulting in delayed responses.

[0516] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0517] In this invention, the server includes means for receiving and processing signal data from monitoring means, means for extracting identification information of a moving object based on the signal data, and means for comparing the identification information with a pre-prepared record of abnormal moving objects to detect anomalies. This makes it possible to detect fraudulent activity within the monitoring area in real time and to respond quickly. Furthermore, by notifying information visually and audibly on smart devices, it supports immediate response by personnel and facilitates smooth communication with external organizations.

[0518] "Monitoring means" refers to a device and system that have the function of continuously observing the conditions of a monitoring area and acquiring data.

[0519] "Signal data" refers to digital information such as video and audio that is acquired and processed by monitoring devices.

[0520] "Processing means" refers to hardware or software technology for analyzing signal data received from monitoring means and extracting useful information.

[0521] "Moving objects" refer to objects moving within the monitoring area, specifically including vehicles and people.

[0522] "Identification information" refers to data or characteristics used to identify or recognize a moving object.

[0523] An "anomalous movement record" is a database that collects and records information about suspicious or illegally moving objects in advance.

[0524] "Means of comparison" refers to a device or method that determines whether a moving object is abnormal by comparing extracted identification information with records of abnormal moving objects.

[0525] "Means for detecting anomalies" refers to technology that has the function of determining that a moving object is abnormal as a result of a comparison and taking the necessary next action.

[0526] "Means of visual and auditory notification" refers to devices or methods that convey information to the person in charge through visual displays or audio when an abnormality is detected.

[0527] "External organizations" refer to third-party facilities or institutions that need to be notified or coordinated when an anomaly occurs.

[0528] This invention is a system for rapidly detecting and responding to fraudulent activity in parking lots and public surveillance areas. The system primarily consists of three elements: a server, terminals, and users.

[0529] The server receives signal data from the monitoring device and analyzes the data using a generative AI model. Specifically, the server uses software libraries such as TensorFlow and OpenCV to extract identification information for moving objects from the video data. The extracted identification information is compared with the record of abnormal moving objects, and if an anomaly is detected, the process proceeds to the next step.

[0530] The terminal receives anomaly information from the server and notifies the responsible person in real time. This notification utilizes a smart device to convey the anomaly visually and audibly. The smart device can be a smartphone or another portable device. The responsible person who receives the notification will check the anomaly information and contact external organizations if necessary.

[0531] Users take immediate action based on anomaly information. This includes reviewing camera footage and promptly reporting to external agencies (such as the police). In this way, the system provides a framework for detecting fraudulent activity on the spot and taking immediate countermeasures.

[0532] For example, consider a scenario where a shopping mall parking lot is monitored. In this case, the surveillance camera transmits signal data of moving objects to a server. The generated AI model analyzes the data using a prompt message such as, "Please determine whether there is a suspicious vehicle in this parking lot footage," and matches it to an anomaly based on an identification mark. When the server detects an anomaly, it transmits the information to smart devices in real time, allowing personnel to immediately review the footage and report to security or the police as needed. This collaboration enables a faster and more reliable response than before.

[0533] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0534] Step 1:

[0535] The server receives signal data from the monitoring device. The input is a video stream from the surveillance camera, and this data is ready for analysis as output. First, the server converts the data into a playable format and separates it into frames.

[0536] Step 2:

[0537] The server analyzes the video data using a generative AI model. Here, the separated frames become input, and feature data of moving objects is output as identification information. The server prompts the AI ​​model to perform calculations using the prompt message, "Determine whether there is a suspicious vehicle in this parking lot video."

[0538] Step 3:

[0539] The server compares the extracted identification information with the abnormal movement record. Based on the identification mark and vehicle shape information, it performs a database search and checks for matching information. The input is the identification information from the previous step, and the output is a flag indicating whether or not an abnormality occurred.

[0540] Step 4:

[0541] The terminal receives anomaly detection results from the server and notifies the responsible person visually and audibly. The input consists of anomaly detection flags and related information, and the output is data displayed as an alert. The terminal provides this as an audio alert or screen display using a mobile notification API.

[0542] Step 5:

[0543] Users receive notifications from their devices and quickly check the situation. Inputs are alert information from the device, and outputs are the user's actions, such as reporting to the police. Users can operate the device to view video and related information and, if necessary, share information with external organizations.

[0544] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0545] This invention combines a system for detecting illegal vehicle activity in a monitored area with an emotion engine that recognizes the user's emotional state. This allows the system to check the user's emotions when an abnormal event occurs and the system notifies them, enabling more effective responses based on that information.

[0546] System Configuration

[0547] 1. Server Role

[0548] The server receives video data from the monitoring device and extracts vehicle identification information using a generated AI model. Based on this information, it compares it with a database of abnormal vehicles and has the function of detecting anomalies. In addition, it receives analysis results from the emotion engine and generates more detailed notification data.

[0549] 2. The role of the terminal

[0550] The terminal receives anomaly notification information sent from the server and notifies the monitoring personnel. Furthermore, the terminal captures the user's voice and facial expressions through the camera and other input devices and transmits this information to the emotion engine to analyze the user's emotional state.

[0551] 3. User Roles

[0552] Users immediately check the situation upon receiving a notification from their device. By analyzing the user's emotions, the system evaluates the user's stress level and optimizes how information is presented by changing the notification interface as needed.

[0553] Specific example

[0554] In a shopping mall parking lot, surveillance equipment provides real-time video. A server reads vehicle license plates and compares them to a list of suspicious vehicles to detect stolen vehicles. This information is sent to a terminal, which simultaneously sends the user's voice tone and facial expressions to an emotion engine to analyze the user's emotional state. For example, if the user is showing high levels of stress, the terminal can change how notifications are displayed, temporarily softening the alert sound or simplifying the interface. This reduces the burden on the user while enabling them to take necessary actions smoothly.

[0555] In this way, the present invention combines automated monitoring with user feedback to not only detect anomalies but also optimize the entire process of responding to them. This system makes a significant contribution to creating a safer environment and effective risk management.

[0556] The following describes the processing flow.

[0557] Step 1:

[0558] The server receives video data in real time from the monitoring device. The video data includes information about vehicles entering and exiting the parking lot.

[0559] Step 2:

[0560] The server analyzes the received video data using a generating AI model and extracts the vehicle's license plate number. This extracted license plate information is then compared against a database of abnormal vehicles.

[0561] Step 3:

[0562] If the server detects a stolen or illegally modified vehicle as a result of the verification process, it generates notification data indicating that an anomaly has been detected. This data includes detailed information about the vehicle in question.

[0563] Step 4:

[0564] The terminal receives notification data sent from the server and issues an alert to the monitoring personnel. The alert is immediately communicated visually and audibly.

[0565] Step 5:

[0566] The device also transmits the user's voice and facial expressions to the emotion engine via the camera and microphone. Based on this data, the user's emotional state is analyzed.

[0567] Step 6:

[0568] The emotion engine analyzes the user's emotional state and evaluates their stress level and type of emotion. This result is then fed back to the device.

[0569] Step 7:

[0570] The device adjusts how alerts are displayed based on feedback from the emotion engine. For example, if the user is stressed, the alert sound may be softened and the displayed content simplified.

[0571] Step 8:

[0572] Users can quickly check the situation based on coordinated notifications from their devices and contact external organizations if necessary. Emotional state is taken into consideration, allowing users to respond in the most optimal mental state.

[0573] (Example 2)

[0574] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0575] Modern surveillance systems require rapid detection and response to abnormal behavior. However, conventional methods fail to adequately address the need for responses that consider the emotional state of users, in addition to simply detecting anomalies. As a result, users experience increased stress when receiving information, making it difficult to make quick and appropriate decisions.

[0576] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0577] In this invention, the server includes means for receiving and analyzing digital information from a monitoring device, means for extracting identification information of transportation equipment based on the digital information, and means for collecting audio and video data to acquire and evaluate the user's emotional state. This makes it possible to detect abnormal behavior and present information according to the user's emotional state.

[0578] A "monitoring device" is a device used to acquire digital information in real time within a specific area.

[0579] "Digital information" refers to visual or audio data acquired from a monitoring device in a processable format.

[0580] "Transportation equipment" refers to equipment used for moving goods or people, such as vehicles.

[0581] "Identification information" refers to unique data used to distinguish transportation equipment from others, and includes registration identification codes, etc.

[0582] The "Abnormal Transportation Equipment Database" is a database that records information about abnormal behavior or specific conditions related to transportation equipment.

[0583] A "human interface" refers to a device that includes display devices and audio outputs for information exchange between a system and a user.

[0584] An "analysis device" is a computer system that receives, processes, and analyzes digital information and related data.

[0585] "Emotional state" refers to a state that is evaluated based on an analysis of the psychological or physiological characteristics exhibited by the user.

[0586] This invention is a system that detects fraudulent activity involving transportation equipment in a monitored area and evaluates the emotional state of users. To achieve this, the server, terminals, and users each play important roles.

[0587] Server Embodiment

[0588] The server receives digital information from monitoring devices and analyzes it using a generated AI model. The server extracts identification information for transportation equipment and detects anomalies by comparing it with a database of abnormal transportation equipment. The collected data is also used to evaluate the user's emotional state via an emotion engine.

[0589] Terminal embodiment

[0590] The device receives notifications of anomalies and communicates the information to the user through a human interface. Furthermore, the device uses its camera and microphone to collect the user's voice and video data, and an emotion engine analyzes their emotional state. Based on this information, the notification interface is adjusted according to the user's stress level.

[0591] User roles

[0592] Users receive anomaly notifications through their devices and check the situation. Optimizing the information provided based on their emotional state enables quick decision-making and response. For example, if a user is stressed, the notification method is adjusted to reduce their burden.

[0593] Specific example

[0594] A concrete example is a scenario in a shopping mall parking lot where a monitoring device acquires vehicle license plates in real time, and a server compares them with a database of abnormal transport equipment to detect stolen vehicles. The terminal receives this notification and simultaneously sends the user's voice and facial expressions to an emotion engine to analyze their emotional state. Based on these results, the terminal adjusts the notification interface to reduce stress.

[0595] Example of a prompt

[0596] "Please tell me how to extract a vehicle's license plate number from surveillance camera footage and determine if it is an illegal vehicle."

[0597] "Please explain in detail the process of analyzing the user's emotional state from their voice and facial expressions."

[0598] The system configured in this way enables a swift and effective response to fraudulent activities involving transportation equipment, as well as the provision of information while considering the burden on users.

[0599] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0600] Step 1:

[0601] The server receives digital information as input from the monitoring device. This digital information includes video data of transportation equipment within the monitoring area. The server uses a generation AI model to detect transportation equipment from the video and extracts features such as license plates as identification information. This process outputs data that can be compared with the abnormal transportation equipment database.

[0602] Step 2:

[0603] The server compares the identification information extracted in Step 1 with the database of abnormal transport equipment. By comparing the input identification information with the data in the database, it detects whether or not abnormal activity has occurred. This result is generated as abnormality notification data and is output when an abnormality occurs.

[0604] Step 3:

[0605] If an anomaly is detected, the server outputs an anomaly notification data to the terminal. The notification data contains detailed information about the anomaly that occurred. This allows the terminal to quickly understand the anomaly and is provided with the information needed to proceed to the next step.

[0606] Step 4:

[0607] The terminal receives abnormal notification data from the server as input and notifies the user through a human interface. These notifications include visual warnings and audio alerts. This allows the user to immediately recognize the situation and begin on-site response.

[0608] Step 5:

[0609] The device uses a camera and microphone to collect the user's voice and facial expressions as input. The input emotional data is analyzed by an emotion engine to evaluate the user's emotional state. Through this data processing and calculation, the user's stress level and anxiety level are quantified, and the analysis results are output.

[0610] Step 6:

[0611] The server receives analysis results from the emotion engine and adjusts the notification interface based on the user's emotional state. Specifically, it presents information by lowering the volume of alert sounds, simplifying the layout of the notification screen, and so on. This process reduces the burden on the user and streamlines the response process.

[0612] (Application Example 2)

[0613] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0614] While detecting vehicle misconduct in monitored areas is crucial, conventional systems focused solely on detecting anomalies, neglecting consideration for the user's psychological state and stress levels. This presented a challenge: notifications of abnormal events could potentially place an excessive burden and stress on users.

[0615] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0616] In this invention, the server includes means for receiving and processing video data from a monitoring device, means for extracting vehicle identification information based on the video data, means for comparing the identification information with a pre-prepared abnormal vehicle database to detect abnormalities, means for notifying a display device of the detected abnormality and generating notification data, means for transmitting the notification data to an external organization, means for capturing and analyzing the user's voice and facial expressions, and means for dynamically adjusting the notification method based on the user's emotional state. This enables flexible information presentation that takes into account the user's psychological state as well as abnormality detection.

[0617] A "monitoring device" is a device that collects video data in a specific area, and this video data is used for anomaly detection.

[0618] "Video data" refers to image information acquired by cameras and other optical devices, and serves as the basis for extracting vehicle identification information.

[0619] "Vehicle identification information" refers to information used to identify a vehicle, and usually refers to the vehicle's license plate information.

[0620] The "abnormal vehicle database" is a database that stores identification information of vehicles suspected of engaging in illegal activities, and is used to cross-reference this information with the information of detected vehicles.

[0621] "User emotional state" refers to information that represents the user's current psychological response and stress level, and is analyzed from information such as voice and facial expressions.

[0622] "Dynamically adjusting notification methods" means automatically changing how information is presented according to the user's emotional state, thereby enabling users to receive information more appropriately.

[0623] The system for realizing this invention consists of four main elements: a monitoring device, a server, a terminal, and a user. The server receives video data from the monitoring device and extracts vehicle identification information using a generated AI model. This identification information is compared with a pre-prepared database of abnormal vehicles, and when an abnormality is detected, the server notifies the display device of this information and generates detailed notification data. Furthermore, the server uses an emotion engine to analyze the user's emotional state and dynamically adjusts the notification method based on the results.

[0624] The device receives anomaly notification information from the server and captures the user's voice and facial expressions via its camera and microphone, transmitting this information to the emotion engine. The device then analyzes the user's emotional state and adjusts the notification interface as needed.

[0625] Users receive notifications from their devices and quickly check the situation. By analyzing the user's emotions, the system assesses the user's stress level and reduces the user's burden by providing appropriate information.

[0626] As a concrete example, consider the use of smart glasses by security guards at a large-scale event venue. The smart glasses collect video footage of the venue in real time, and a server identifies suspicious vehicles and individuals. The system also captures the user's voice and facial expressions, and can adjust the warning sound and simplify the notification interface based on the user's stress level when they receive information about a suspicious vehicle. An example of a prompt to the generated AI model would be, "Analyze the emotions this user exhibits when they detect inappropriate activity and select the most appropriate notification method."

[0627] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0628] Step 1:

[0629] The server receives video data in real time from the monitoring device. The input is raw video data from the monitoring device, which is digitized using an image processing algorithm. The output is identifiable video data.

[0630] Step 2:

[0631] The server extracts vehicle identification information from the received video data using a generative AI model. The input is the identifiable video data obtained in step 1, and the AI ​​model analyzes the data to identify vehicle identification information such as license plates. The output is the vehicle identification information.

[0632] Step 3:

[0633] The server compares the extracted vehicle identification information with the abnormal vehicle database. The input is the vehicle identification information, which is the output data from step 2. By comparing this information with the database, the server determines whether or not an abnormality has been detected. The output is the result of whether or not an abnormality was detected.

[0634] Step 4:

[0635] If an anomaly is detected, the server sends the generated notification data to the terminal. The input is the anomaly detection result obtained in step 3, and the server creates the notification data based on this information. The output is the notification data to be sent to the terminal.

[0636] Step 5:

[0637] The terminal receives anomaly notification data sent from the server and notifies the user. The input is the notification data sent in step 4, and the terminal performs the action of displaying the information on the display device. The output is the warning that the user receives visually and audibly.

[0638] Step 6:

[0639] The device uses a camera and microphone to capture the user's voice and facial expressions and sends them to the emotion engine. The input is the user's real-time voice and video information, and the user's emotional state is evaluated through emotion analysis. The output is data on the user's emotional state.

[0640] Step 7:

[0641] The server dynamically adjusts the notification method based on the results of the sentiment analysis. The input is the user's sentiment data obtained in step 6, and the server optimizes actions such as adjusting the alert sound volume and the display method of the interface. The output is the adjusted notification interface.

[0642] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0643] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0644] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0645] [Fourth Embodiment]

[0646] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0647] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0648] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0649] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0650] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0651] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0652] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0653] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0654] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0655] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0656] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0657] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0658] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0659] This invention provides a system for immediately detecting and quickly responding to illegal vehicle activity in monitored areas such as parking lots. This system works in conjunction with multiple monitoring devices and provides a platform for analyzing acquired video data in real time.

[0660] System Configuration

[0661] 1. Server Role

[0662] The server continuously receives video data from the monitoring device and applies a generative AI model to extract vehicle identification information. This identification information includes the license plate and vehicle shape information of the identified vehicle. Based on this, the server compares it with a database of abnormal vehicles and processes the information further if an abnormality is detected.

[0663] 2. The role of the terminal

[0664] The terminal receives anomaly information transmitted from the server and notifies the monitoring personnel visually and audibly. This allows the monitoring personnel to understand the situation in real time and take necessary actions quickly.

[0665] 3. User Roles

[0666] Based on the information provided by the device, users can check for anomalies and, if necessary, manually provide additional information to external organizations, such as the police. This allows for smoother on-site response and investigation.

[0667] Specific example

[0668] For example, in the underground parking lot of a shopping mall, a surveillance system provides real-time video. A server receives this video and recognizes the license plates of vehicles. If the recognized license plate matches a list of stolen vehicles, the server immediately detects this anomaly and sends the relevant information to a terminal. The terminal displays this information visually and alerts the person in charge. The person in charge reviews the video, contacts the police if necessary, and organizes the response at the scene. Because this entire process is automated and in real time, it enables a faster and more reliable response than traditional methods.

[0669] This implementation allows for the immediate detection and prompt response to any misuse of vehicles within the surveillance area. This system contributes to crime prevention and rapid response after an incident occurs, thereby ensuring safe public spaces.

[0670] The following describes the processing flow.

[0671] Step 1:

[0672] The server receives streaming video data transmitted from the monitoring device in real time. This includes extracting frame data that contains the vehicle to be processed.

[0673] Step 2:

[0674] The server applies a generative AI model to the received video data to extract vehicle license plate information. During this process, it performs noise reduction and format conversion to prepare the data for easier analysis.

[0675] Step 3:

[0676] The server compares the extracted license plate information with a database of abnormal vehicles. Here, it makes a determination to identify vehicles suspected of being stolen or illegally modified.

[0677] Step 4:

[0678] If an anomaly is detected as a result of the matching process, the server organizes relevant data such as snapshots and location information of the vehicle and generates notification data.

[0679] Step 5:

[0680] The server sends the generated notification data to the terminal, informing the monitoring personnel of the details of the anomaly. This notification includes an alert to prompt immediate action.

[0681] Step 6:

[0682] The terminal provides the monitoring personnel with received notification data visually and audibly. This allows the personnel to quickly assess the situation and take necessary countermeasures.

[0683] Step 7:

[0684] The user will check the abnormal situation based on the information from the device. If necessary, the user will contact the police or other relevant organizations directly to expedite the response at the scene.

[0685] (Example 1)

[0686] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0687] There is a need to quickly and accurately detect anomalies in moving objects within the monitoring area and prompt immediate responses based on those detections. In particular, it is crucial to efficiently identify anomalies without relying on human intervention and to facilitate smooth cooperation with external organizations.

[0688] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0689] In this invention, the server includes means for receiving and analyzing visual information from a video acquisition means, means for extracting identification information of a moving object based on the visual information, and means for comparing the identification information with a pre-prepared set of abnormal data to detect abnormalities. This enables automatic and real-time detection of abnormalities within a monitoring area.

[0690] A "video acquisition means" is a mechanism for acquiring visual information from a surveillance area and supplying it to a processing system.

[0691] "Visual information" refers to images and video data provided by surveillance devices.

[0692] "Means of analysis" refers to the processes and techniques used to process acquired visual information and extract necessary identification information.

[0693] "Identification information for moving objects" refers to data used to identify moving objects such as vehicles, and includes information such as license plates and shapes.

[0694] An "anomalous data set" refers to a pre-prepared collection of information about an abnormal state or event.

[0695] "Notification information" refers to information that is sent to external organizations or monitors when an anomaly is detected.

[0696] A "generative AI model" refers to a machine learning model used to extract useful information from visual information.

[0697] "Analysis of identification information" refers to the process of interpreting detected identification information and identifying anomalies.

[0698] A "warning signal" refers to a visual or auditory notification used to alert a monitor when an anomaly is detected.

[0699] A "terminal device" refers to a device that provides notification information received from a server to a monitor and prompts them to take necessary actions.

[0700] This invention is a system that enables rapid detection of anomalies within a monitored area and immediate response. Specifically, it operates through the cooperation of three entities: a server, a terminal, and a user.

[0701] Server role:

[0702] The server receives visual information acquired from monitoring devices. This process uses network communication technology, such as a streaming protocol. The received data is analyzed using a generative AI model to extract identification information for moving objects. The model is developed using software frameworks such as TensorFlow and PyTorch, and identifies the license plates and shapes of moving objects. The identification information is compared with a set of anomaly data, and if an anomaly is detected, information processing is performed. A specific example is identifying a license plate that matches a list of stolen vehicles. In this case, the server immediately generates notification information.

[0703] Terminal role:

[0704] The terminal receives notification information from the server and provides visual and auditory alerts to the monitor. This includes displaying information via a GUI and using alarm sounds. The terminal communicates with the server using the TCP / IP protocol to ensure stable data reception.

[0705] User roles:

[0706] Users make on-site decisions based on information provided by their devices. For example, they can identify vehicles that have been flagged as abnormal on the monitoring screen and provide information to external organizations such as the police if necessary. This process enables rapid response and coordination.

[0707] An example of a prompt message for the generated AI model is, "Extract vehicle license plates from the surveillance footage and compare them with the existing list of stolen vehicles." This allows users to effectively utilize the AI ​​model under specific instructions.

[0708] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0709] Step 1:

[0710] The server receives visual information in real time from the monitoring device. Visual data is supplied as input, and a network connection is used to receive it stably. The received data is then directly passed on to the generation AI model for analysis.

[0711] Step 2:

[0712] The server analyzes the received visual information using a generating AI model. The input here is the visual data acquired in step 1. The AI ​​model, developed using, for example, TensorFlow, performs data calculations to extract the license plate and shape of moving objects from the visual data. The output is the identification information of the moving objects.

[0713] Step 3:

[0714] The server compares the generated identification information with the anomaly data set. The input is the identification information, which is the output of step 2. The server executes a database query to perform the comparison. For example, it uses SQL to check if the license plate exists in the stolen vehicle list. The output is the result of whether or not an anomaly exists.

[0715] Step 4:

[0716] The server generates notification information and sends it to the terminal when an anomaly is detected. The input is the anomaly detection result, which is the output of step 3. The generated notification information includes the license plate number and location information of the vehicle in question. This information is then passed to the terminal.

[0717] Step 5:

[0718] The terminal alerts the monitor based on notification information received from the server. The input is the notification information generated in step 4. The terminal displays the information visually using a GUI and also provides auditory attention by sounding an alarm. The output is an immediate status notification to the monitor.

[0719] Step 6:

[0720] The user checks for anomalies based on the information provided by the terminal. The input is the output information from step 5. The user checks the data on the monitor and assesses the situation on site. Based on this assessment, they notify an external organization if necessary. The output is the information to be provided to the external organization and preparations for a rapid response.

[0721] (Application Example 1)

[0722] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0723] Rapid detection and response to misconduct in monitored areas is a critical issue in modern society. However, conventional monitoring systems often require manual response when anomalies are detected, making rapid response difficult. Furthermore, coordination with external organizations when anomalies are detected can be slow, resulting in delayed responses.

[0724] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0725] In this invention, the server includes means for receiving and processing signal data from monitoring means, means for extracting identification information of a moving object based on the signal data, and means for comparing the identification information with a pre-prepared record of abnormal moving objects to detect anomalies. This makes it possible to detect fraudulent activity within the monitoring area in real time and to respond quickly. Furthermore, by notifying information visually and audibly on smart devices, it supports immediate response by personnel and facilitates smooth communication with external organizations.

[0726] "Monitoring means" refers to a device and system that have the function of continuously observing the conditions of a monitoring area and acquiring data.

[0727] "Signal data" refers to digital information such as video and audio that is acquired and processed by monitoring devices.

[0728] "Processing means" refers to hardware or software technology for analyzing signal data received from monitoring means and extracting useful information.

[0729] "Moving objects" refer to objects moving within the monitoring area, specifically including vehicles and people.

[0730] "Identification information" refers to data or characteristics used to identify or recognize a moving object.

[0731] An "anomalous movement record" is a database that collects and records information about suspicious or illegally moving objects in advance.

[0732] "Means of comparison" refers to a device or method that determines whether a moving object is abnormal by comparing extracted identification information with records of abnormal moving objects.

[0733] "Means for detecting anomalies" refers to technology that has the function of determining that a moving object is abnormal as a result of a comparison and taking the necessary next action.

[0734] "Means of visual and auditory notification" refers to devices or methods that convey information to the person in charge through visual displays or audio when an abnormality is detected.

[0735] "External organizations" refer to third-party facilities or institutions that need to be notified or coordinated when an anomaly occurs.

[0736] This invention is a system for rapidly detecting and responding to fraudulent activity in parking lots and public surveillance areas. The system primarily consists of three elements: a server, terminals, and users.

[0737] The server receives signal data from the monitoring device and analyzes the data using a generative AI model. Specifically, the server uses software libraries such as TensorFlow and OpenCV to extract identification information for moving objects from the video data. The extracted identification information is compared with the record of abnormal moving objects, and if an anomaly is detected, the process proceeds to the next step.

[0738] The terminal receives anomaly information from the server and notifies the responsible person in real time. This notification utilizes a smart device to convey the anomaly visually and audibly. The smart device can be a smartphone or another portable device. The responsible person who receives the notification will check the anomaly information and contact external organizations if necessary.

[0739] Users take immediate action based on anomaly information. This includes reviewing camera footage and promptly reporting to external agencies (such as the police). In this way, the system provides a framework for detecting fraudulent activity on the spot and taking immediate countermeasures.

[0740] For example, consider a scenario where a shopping mall parking lot is monitored. In this case, the surveillance camera transmits signal data of moving objects to a server. The generated AI model analyzes the data using a prompt message such as, "Please determine whether there is a suspicious vehicle in this parking lot footage," and matches it to an anomaly based on an identification mark. When the server detects an anomaly, it transmits the information to smart devices in real time, allowing personnel to immediately review the footage and report to security or the police as needed. This collaboration enables a faster and more reliable response than before.

[0741] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0742] Step 1:

[0743] The server receives signal data from the monitoring device. The input is a video stream from the surveillance camera, and this data is ready for analysis as output. First, the server converts the data into a playable format and separates it into frames.

[0744] Step 2:

[0745] The server analyzes the video data using a generative AI model. Here, the separated frames become input, and feature data of moving objects is output as identification information. The server prompts the AI ​​model to perform calculations using the prompt message, "Determine whether there is a suspicious vehicle in this parking lot video."

[0746] Step 3:

[0747] The server compares the extracted identification information with the abnormal movement record. Based on the identification mark and vehicle shape information, it performs a database search and checks for matching information. The input is the identification information from the previous step, and the output is a flag indicating whether or not an abnormality occurred.

[0748] Step 4:

[0749] The terminal receives anomaly detection results from the server and notifies the responsible person visually and audibly. The input consists of anomaly detection flags and related information, and the output is data displayed as an alert. The terminal provides this as an audio alert or screen display using a mobile notification API.

[0750] Step 5:

[0751] Users receive notifications from their devices and quickly check the situation. Inputs are alert information from the device, and outputs are the user's actions, such as reporting to the police. Users can operate the device to view video and related information and, if necessary, share information with external organizations.

[0752] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0753] This invention combines a system for detecting illegal vehicle activity in a monitored area with an emotion engine that recognizes the user's emotional state. This allows the system to check the user's emotions when an abnormal event occurs and the system notifies them, enabling more effective responses based on that information.

[0754] System Configuration

[0755] 1. Server Role

[0756] The server receives video data from the monitoring device and extracts vehicle identification information using a generated AI model. Based on this information, it compares it with a database of abnormal vehicles and has the function of detecting anomalies. In addition, it receives analysis results from the emotion engine and generates more detailed notification data.

[0757] 2. The role of the terminal

[0758] The terminal receives anomaly notification information sent from the server and notifies the monitoring personnel. Furthermore, the terminal captures the user's voice and facial expressions through the camera and other input devices and transmits this information to the emotion engine to analyze the user's emotional state.

[0759] 3. User Roles

[0760] Users immediately check the situation upon receiving a notification from their device. By analyzing the user's emotions, the system evaluates the user's stress level and optimizes how information is presented by changing the notification interface as needed.

[0761] Specific example

[0762] In a shopping mall parking lot, surveillance equipment provides real-time video. A server reads vehicle license plates and compares them to a list of suspicious vehicles to detect stolen vehicles. This information is sent to a terminal, which simultaneously sends the user's voice tone and facial expressions to an emotion engine to analyze the user's emotional state. For example, if the user is showing high levels of stress, the terminal can change how notifications are displayed, temporarily softening the alert sound or simplifying the interface. This reduces the burden on the user while enabling them to take necessary actions smoothly.

[0763] In this way, the present invention combines automated monitoring with user feedback to not only detect anomalies but also optimize the entire process of responding to them. This system makes a significant contribution to creating a safer environment and effective risk management.

[0764] The following describes the processing flow.

[0765] Step 1:

[0766] The server receives video data in real time from the monitoring device. The video data includes information about vehicles entering and exiting the parking lot.

[0767] Step 2:

[0768] The server analyzes the received video data using a generating AI model and extracts the vehicle's license plate number. This extracted license plate information is then compared against a database of abnormal vehicles.

[0769] Step 3:

[0770] If the server detects a stolen or illegally modified vehicle as a result of the verification process, it generates notification data indicating that an anomaly has been detected. This data includes detailed information about the vehicle in question.

[0771] Step 4:

[0772] The terminal receives notification data sent from the server and issues an alert to the monitoring personnel. The alert is immediately communicated visually and audibly.

[0773] Step 5:

[0774] The device also transmits the user's voice and facial expressions to the emotion engine via the camera and microphone. Based on this data, the user's emotional state is analyzed.

[0775] Step 6:

[0776] The emotion engine analyzes the user's emotional state and evaluates their stress level and type of emotion. This result is then fed back to the device.

[0777] Step 7:

[0778] The device adjusts how alerts are displayed based on feedback from the emotion engine. For example, if the user is stressed, the alert sound may be softened and the displayed content simplified.

[0779] Step 8:

[0780] Users can quickly check the situation based on coordinated notifications from their devices and contact external organizations if necessary. Emotional state is taken into consideration, allowing users to respond in the most optimal mental state.

[0781] (Example 2)

[0782] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0783] Modern surveillance systems require rapid detection and response to abnormal behavior. However, conventional methods fail to adequately address the need for responses that consider the emotional state of users, in addition to simply detecting anomalies. As a result, users experience increased stress when receiving information, making it difficult to make quick and appropriate decisions.

[0784] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0785] In this invention, the server includes means for receiving and analyzing digital information from a monitoring device, means for extracting identification information of transportation equipment based on the digital information, and means for collecting audio and video data to acquire and evaluate the user's emotional state. This makes it possible to detect abnormal behavior and present information according to the user's emotional state.

[0786] A "monitoring device" is a device used to acquire digital information in real time within a specific area.

[0787] "Digital information" refers to visual or audio data acquired from a monitoring device in a processable format.

[0788] "Transportation equipment" refers to equipment used for moving goods or people, such as vehicles.

[0789] "Identification information" refers to unique data used to distinguish transportation equipment from others, and includes registration identification codes, etc.

[0790] The "Abnormal Transportation Equipment Database" is a database that records information about abnormal behavior or specific conditions related to transportation equipment.

[0791] A "human interface" refers to a device that includes display devices and audio outputs for information exchange between a system and a user.

[0792] An "analysis device" is a computer system that receives, processes, and analyzes digital information and related data.

[0793] "Emotional state" refers to a state that is evaluated based on an analysis of the psychological or physiological characteristics exhibited by the user.

[0794] This invention is a system that detects fraudulent activity involving transportation equipment in a monitored area and evaluates the emotional state of users. To achieve this, the server, terminals, and users each play important roles.

[0795] Server Embodiment

[0796] The server receives digital information from monitoring devices and analyzes it using a generated AI model. The server extracts identification information for transportation equipment and detects anomalies by comparing it with a database of abnormal transportation equipment. The collected data is also used to evaluate the user's emotional state via an emotion engine.

[0797] Terminal embodiment

[0798] The device receives notifications of anomalies and communicates the information to the user through a human interface. Furthermore, the device uses its camera and microphone to collect the user's voice and video data, and an emotion engine analyzes their emotional state. Based on this information, the notification interface is adjusted according to the user's stress level.

[0799] User roles

[0800] Users receive anomaly notifications through their devices and check the situation. Optimizing the information provided based on their emotional state enables quick decision-making and response. For example, if a user is stressed, the notification method is adjusted to reduce their burden.

[0801] Specific example

[0802] A concrete example is a scenario in a shopping mall parking lot where a monitoring device acquires vehicle license plates in real time, and a server compares them with a database of abnormal transport equipment to detect stolen vehicles. The terminal receives this notification and simultaneously sends the user's voice and facial expressions to an emotion engine to analyze their emotional state. Based on these results, the terminal adjusts the notification interface to reduce stress.

[0803] Example of a prompt

[0804] "Please tell me how to extract a vehicle's license plate number from surveillance camera footage and determine if it is an illegal vehicle."

[0805] "Please explain in detail the process of analyzing the user's emotional state from their voice and facial expressions."

[0806] The system configured in this way enables a swift and effective response to fraudulent activities involving transportation equipment, as well as the provision of information while considering the burden on users.

[0807] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0808] Step 1:

[0809] The server receives digital information as input from the monitoring device. This digital information includes video data of transportation equipment within the monitoring area. The server uses a generation AI model to detect transportation equipment from the video and extracts features such as license plates as identification information. This process outputs data that can be compared with the abnormal transportation equipment database.

[0810] Step 2:

[0811] The server compares the identification information extracted in Step 1 with the database of abnormal transport equipment. By comparing the input identification information with the data in the database, it detects whether or not abnormal activity has occurred. This result is generated as abnormality notification data and is output when an abnormality occurs.

[0812] Step 3:

[0813] If an anomaly is detected, the server outputs an anomaly notification data to the terminal. The notification data contains detailed information about the anomaly that occurred. This allows the terminal to quickly understand the anomaly and is provided with the information needed to proceed to the next step.

[0814] Step 4:

[0815] The terminal receives abnormal notification data from the server as input and notifies the user through a human interface. These notifications include visual warnings and audio alerts. This allows the user to immediately recognize the situation and begin on-site response.

[0816] Step 5:

[0817] The device uses a camera and microphone to collect the user's voice and facial expressions as input. The input emotional data is analyzed by an emotion engine to evaluate the user's emotional state. Through this data processing and calculation, the user's stress level and anxiety level are quantified, and the analysis results are output.

[0818] Step 6:

[0819] The server receives analysis results from the emotion engine and adjusts the notification interface based on the user's emotional state. Specifically, it presents information by lowering the volume of alert sounds, simplifying the layout of the notification screen, and so on. This process reduces the burden on the user and streamlines the response process.

[0820] (Application Example 2)

[0821] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0822] While detecting vehicle misconduct in monitored areas is crucial, conventional systems focused solely on detecting anomalies, neglecting consideration for the user's psychological state and stress levels. This presented a challenge: notifications of abnormal events could potentially place an excessive burden and stress on users.

[0823] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0824] In this invention, the server includes means for receiving and processing video data from a monitoring device, means for extracting vehicle identification information based on the video data, means for comparing the identification information with a pre-prepared abnormal vehicle database to detect abnormalities, means for notifying a display device of the detected abnormality and generating notification data, means for transmitting the notification data to an external organization, means for capturing and analyzing the user's voice and facial expressions, and means for dynamically adjusting the notification method based on the user's emotional state. This enables flexible information presentation that takes into account the user's psychological state as well as abnormality detection.

[0825] A "monitoring device" is a device that collects video data in a specific area, and this video data is used for anomaly detection.

[0826] "Video data" refers to image information acquired by cameras and other optical devices, and serves as the basis for extracting vehicle identification information.

[0827] "Vehicle identification information" refers to information used to identify a vehicle, and usually refers to the vehicle's license plate information.

[0828] The "abnormal vehicle database" is a database that stores identification information of vehicles suspected of engaging in illegal activities, and is used to cross-reference this information with the information of detected vehicles.

[0829] "User emotional state" refers to information that represents the user's current psychological response and stress level, and is analyzed from information such as voice and facial expressions.

[0830] "Dynamically adjusting notification methods" means automatically changing how information is presented according to the user's emotional state, thereby enabling users to receive information more appropriately.

[0831] The system for realizing this invention consists of four main elements: a monitoring device, a server, a terminal, and a user. The server receives video data from the monitoring device and extracts vehicle identification information using a generated AI model. This identification information is compared with a pre-prepared database of abnormal vehicles, and when an abnormality is detected, the server notifies the display device of this information and generates detailed notification data. Furthermore, the server uses an emotion engine to analyze the user's emotional state and dynamically adjusts the notification method based on the results.

[0832] The device receives anomaly notification information from the server and captures the user's voice and facial expressions via its camera and microphone, transmitting this information to the emotion engine. The device then analyzes the user's emotional state and adjusts the notification interface as needed.

[0833] Users receive notifications from their devices and quickly check the situation. By analyzing the user's emotions, the system assesses the user's stress level and reduces the user's burden by providing appropriate information.

[0834] As a concrete example, consider the use of smart glasses by security guards at a large-scale event venue. The smart glasses collect video footage of the venue in real time, and a server identifies suspicious vehicles and individuals. The system also captures the user's voice and facial expressions, and can adjust the warning sound and simplify the notification interface based on the user's stress level when they receive information about a suspicious vehicle. An example of a prompt to the generated AI model would be, "Analyze the emotions this user exhibits when they detect inappropriate activity and select the most appropriate notification method."

[0835] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0836] Step 1:

[0837] The server receives video data in real time from the monitoring device. The input is raw video data from the monitoring device, which is digitized using an image processing algorithm. The output is identifiable video data.

[0838] Step 2:

[0839] The server extracts vehicle identification information from the received video data using a generative AI model. The input is the identifiable video data obtained in step 1, and the AI ​​model analyzes the data to identify vehicle identification information such as license plates. The output is the vehicle identification information.

[0840] Step 3:

[0841] The server compares the extracted vehicle identification information with the abnormal vehicle database. The input is the vehicle identification information, which is the output data from step 2. By comparing this information with the database, the server determines whether or not an abnormality has been detected. The output is the result of whether or not an abnormality was detected.

[0842] Step 4:

[0843] If an anomaly is detected, the server sends the generated notification data to the terminal. The input is the anomaly detection result obtained in step 3, and the server creates the notification data based on this information. The output is the notification data to be sent to the terminal.

[0844] Step 5:

[0845] The terminal receives anomaly notification data sent from the server and notifies the user. The input is the notification data sent in step 4, and the terminal performs the action of displaying the information on the display device. The output is the warning that the user receives visually and audibly.

[0846] Step 6:

[0847] The device uses a camera and microphone to capture the user's voice and facial expressions and sends them to the emotion engine. The input is the user's real-time voice and video information, and the user's emotional state is evaluated through emotion analysis. The output is data on the user's emotional state.

[0848] Step 7:

[0849] The server dynamically adjusts the notification method based on the results of the sentiment analysis. The input is the user's sentiment data obtained in step 6, and the server optimizes actions such as adjusting the alert sound volume and the display method of the interface. The output is the adjusted notification interface.

[0850] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0851] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0852] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0853] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0854] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0855] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0856] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0857] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0858] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0859] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0860] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0861] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0862] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0863] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0864] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0865] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0866] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0867] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0868] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0869] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0870] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0871] The following is further disclosed regarding the embodiments described above.

[0872] (Claim 1)

[0873] A means for receiving and processing video data from a monitoring device,

[0874] A means for extracting vehicle identification information based on the video data,

[0875] A means for detecting an anomaly by comparing the identification information with a pre-prepared database of abnormal vehicles,

[0876] A means for notifying a display device of an abnormality when it is detected, and for generating notification data,

[0877] Means for transmitting the notification data to an external organization,

[0878] A system that includes this.

[0879] (Claim 2)

[0880] The system according to claim 1, characterized in that it recognizes the license plate as vehicle identification information when detecting an abnormality.

[0881] (Claim 3)

[0882] The system according to claim 1, characterized in that the notification data transmitted to an external organization includes a snapshot of video data and location information of the vehicle.

[0883] "Example 1"

[0884] (Claim 1)

[0885] A means for receiving and analyzing visual information from a video acquisition means,

[0886] Means for extracting identification information of a moving object based on the visual information,

[0887] A means for comparing the identification information with a pre-prepared set of abnormal data to detect an anomaly,

[0888] A means for notifying a display device of an abnormality when one is detected, and for generating notification information,

[0889] Means for transmitting the notification information to an external organization,

[0890] A means of analyzing identification information by applying a generative AI model,

[0891] In the analysis of identification information, means for recognizing the license plate and shape as vehicle identification information,

[0892] A means of providing a warning signal visually and audibly when an abnormality is detected,

[0893] A system that includes this.

[0894] (Claim 2)

[0895] The system according to claim 1, characterized in that the notification information transmitted to an external organization upon detection of an anomaly includes a portion of the visual information and the location information of the moving object.

[0896] (Claim 3)

[0897] The system according to claim 1, characterized in that the terminal device provides abnormal information to the monitor to encourage a prompt response.

[0898] "Application Example 1"

[0899] (Claim 1)

[0900] A means for receiving and processing signal data from a monitoring means,

[0901] A means for extracting identification information of a moving object based on the signal data,

[0902] A means for detecting an anomaly by comparing the identification information with a pre-prepared record of abnormal moving objects,

[0903] A means for notifying a display means of an abnormality when it is detected, and for generating notification information,

[0904] Means for transmitting the notification information to an external organization,

[0905] Means for notifying the aforementioned information visually and audibly on a smart device,

[0906] A system that includes this.

[0907] (Claim 2)

[0908] The system according to claim 1, characterized in that it recognizes an identification mark as identification information for a moving object when detecting an anomaly.

[0909] (Claim 3)

[0910] The system according to claim 1, characterized in that the notification information transmitted to an external organization includes a printed diagram of the signal data and location information of the mobile object.

[0911] "Example 2 of combining an emotion engine"

[0912] (Claim 1)

[0913] A means for receiving and analyzing digital information from a monitoring device,

[0914] A means for extracting identification information of transportation equipment based on the digital information,

[0915] A means for detecting an anomaly by comparing the identification information with a pre-prepared database of abnormal transport equipment,

[0916] A means for notifying the human interface of any detected anomalies and for generating notification data,

[0917] Means for transmitting the notification data to the analysis device,

[0918] A means for collecting audio and video data to acquire and evaluate the emotional state of users,

[0919] Means for adjusting the information presentation means based on the emotional state,

[0920] A system that includes this.

[0921] (Claim 2)

[0922] The system according to claim 1, characterized in that it recognizes a registered identification code as identification information for transport equipment when detecting an anomaly.

[0923] (Claim 3)

[0924] The system according to claim 1, characterized in that the notification data transmitted to the analysis device includes fragments of digital information and geographical data of the transport equipment.

[0925] "Application example 2 when combining with an emotional engine"

[0926] (Claim 1)

[0927] A means for receiving and processing video data from a monitoring device,

[0928] A means for extracting vehicle identification information based on the video data,

[0929] A means for detecting an anomaly by comparing the identification information with a pre-prepared database of abnormal vehicles,

[0930] A means for notifying a display device of an abnormality when it is detected, and for generating notification data,

[0931] Means for transmitting the notification data to an external organization,

[0932] A means of capturing and analyzing the user's voice and facial expressions,

[0933] A means of dynamically adjusting the notification method based on the user's emotional state,

[0934] A system that includes this.

[0935] (Claim 2)

[0936] The system according to claim 1, characterized in that it recognizes the license plate as vehicle identification information when detecting an abnormality.

[0937] (Claim 3)

[0938] The system according to claim 1, characterized in that the notification data transmitted to an external organization includes a snapshot of video data and location information of the vehicle. [Explanation of symbols]

[0939] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving and processing video data from a monitoring device, A means for extracting vehicle identification information based on the video data, A means for detecting an anomaly by comparing the identification information with a pre-prepared database of abnormal vehicles, A means for notifying a display device of an abnormality when one is detected, and for generating notification data, Means for transmitting the notification data to an external organization, A system that includes this.

2. The system according to claim 1, characterized in that it recognizes the license plate as vehicle identification information when detecting an abnormality.

3. The system according to claim 1, characterized in that the notification data transmitted to an external organization includes a snapshot of video data and location information of the vehicle.

Images