System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring a video; means for transmitting the acquired video to a server; means for analyzing the video received by the server; means for comparing an analysis result with a legal database and determining illegality; and means for generating and transmitting feedback to a terminal when illegality is recognized.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional monitoring and reporting systems, it was difficult to immediately identify and report nuisance or illegal activity occurring right in front of one's eyes. Furthermore, ordinary citizens often lack in-depth knowledge of the law, making it difficult to accurately identify and report illegal activity. This results in illegal activity being left unattended, leading to a deterioration in local public safety. The present invention aims to solve these problems and contribute to improving public safety in local areas. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for acquiring video, a means for transmitting the acquired video to a server, a means for analyzing the video received by the server, a means for comparing the analysis results with a legal database to determine illegality, and a means for generating feedback and transmitting it to a terminal if illegality is determined. This system allows users to instantly record illegal activity occurring in front of them and find out which law the activity violates. Furthermore, the analysis results are notified in real time, facilitating reporting and legal consultation, thereby contributing to improving local public safety.

[0006] "Video" refers to real-time video or still images captured by a user using a camera.

[0007] "Means of acquisition" refers to the functions and methods by which a user operates a photographing device to obtain images.

[0008] "Transmitting means" refers to a function or method for sending the acquired video as data to a server via the Internet or other communication network.

[0009] A "server" refers to a computer system that is connected to the Internet and receives, analyzes, and stores data sent from terminals.

[0010] "Means of analysis" refers to the functions and methods for analyzing received video data using technologies such as AI models.

[0011] A "legal database" refers to a collection of data that stores various laws and ordinances in digital format and is used to check which laws a particular act violates.

[0012] "Means of matching" refers to the functions and methods for comparing the analysis results with information in a legal database and determining illegality.

[0013] "Means for generating feedback" refers to the function or method for creating messages or notifications to inform users of the results of any illegal activity that has been identified.

[0014] "Means for sending (feedback)" refers to the method or function for sending the generated feedback to the user's device.

[0015] "User" refers to any individual or organization that uses this system to record and report illegal activity.

[0016] "Device" refers to a smartphone, tablet, or other electronic device operated by a user and used to capture footage and receive feedback.

[0017] "Real-time" refers to a timeframe in which processing or response is near-instant, meaning minimal delay.

[0018] "Reporting" refers to the action taken to inform the relevant authorities of illegal activity. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This system uses AI technology to detect illegal activity based on video footage taken by users, compares the results with the law, and provides feedback. The system includes a video capture device, a server, a legal database, and a communication network connecting them.

[0041] Video acquisition and transmission

[0042] Users use their smartphones or tablets to capture specific scenes, and the captured data is recorded as video or still images. The device then compresses the captured video and sends it to a server via the Internet.

[0043] Video analysis

[0044] The server stores the video data received from the device in a database. The stored data is then analyzed using AI technology. Specifically, machine learning models and deep learning are used to identify actions within the video (e.g., illegal dumping, illegal parking, smoking, etc.).

[0045] Legal Compliance

[0046] The analyzed behavior is compared with a legal database on the server. The legal database stores various laws and ordinances, and determines whether they are illegal. For example, if smoking is detected in a no-smoking area, the system determines whether the behavior violates the local government's smoking ban ordinance.

[0047] Generate and send feedback

[0048] If the action is found to be illegal, the server generates a feedback message containing the relevant legal provisions and ordinances and sends it to the device. For example, a specific message such as "This action violates Article 5 of the smoking ban ordinance of X City" may be displayed.

[0049] Reporting and legal consultation support

[0050] Users can report or seek legal advice based on the feedback messages displayed on their device. For example, by pressing the report button within the app, they can report directly to the relevant authorities (police station or local government). Furthermore, if necessary, the app will provide contact information for lawyers and free legal advice.

[0051] Specific examples

[0052] 1. A user takes a photo of someone littering in a park.

[0053] 2. The device compresses the video and sends it to the server.

[0054] 3. The server analyzes the video and detects illegal dumping of garbage.

[0055] 4. The server determines that the act constitutes illegal dumping based on local government ordinances.

[0056] 5. The server creates feedback stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act," and sends it to the device.

[0057] 6. The user reports the message.

[0058] This will enable users to immediately identify nuisance or illegal behavior and take appropriate action, contributing to improving public safety in the local area.

[0059] This system can solve the problems that traditional reporting systems have by providing real-time analysis and notification, as well as support for reporting and legal consultation.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] The user launches the app and switches to camera mode by tapping the app icon to open it and then pressing the camera button on the screen to switch to camera mode.

[0063] Step 2:

[0064] When a user captures a specific scene, for example, recording someone throwing trash in a park, the camera will record footage within its coverage area.

[0065] Step 3:

[0066] The device temporarily stores the video it has taken. The video data is stored in the device's memory.

[0067] Step 4:

[0068] The device compresses the captured video and prepares it for transmission to the server, converting the video data into an optimized format and establishing an Internet connection.

[0069] Step 5:

[0070] The device sends the video data to the server. Specifically, the data is encrypted using the HTTPS protocol and securely uploaded to the server.

[0071] Step 6:

[0072] The server receives the video data sent from the device and saves it in a specified directory on the server.

[0073] Step 7:

[0074] The server analyzes the received video, specifically using machine learning models and deep learning algorithms to detect specific behaviors within the video, in this case illegal dumping of trash.

[0075] Step 8:

[0076] The server compares the analysis results with a legal database, which stores various laws and regulations, and determines whether the information is illegal based on the analysis results.

[0077] Step 9:

[0078] If the server determines that the action is illegal, it generates a feedback message indicating the result, such as "This action violates Article 16 of the Waste Disposal and Public Cleansing Act."

[0079] Step 10:

[0080] The server sends a feedback message to the terminal, specifically, a generated message sent back to the user's terminal for display.

[0081] Step 11:

[0082] The device receives the feedback message and displays it to the user as a notification, which can be a pop-up notification or an in-app message.

[0083] Step 12:

[0084] Based on the feedback messages received, users can report or seek legal advice. Specifically, by tapping the "Report" or "Legal Advice" button within the app, an interface will appear for contacting the relevant authorities.

[0085] By following these steps, users can quickly identify any illegal activity occurring in front of them and take appropriate action.

[0086] Example 1

[0087] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0088] In modern society, there is a need for real-time identification and response to illegal and nuisance behavior. Conventional reporting systems require users to manually report illegal behavior, making it difficult to respond quickly and accurately. In addition, determining illegality takes time, which makes it difficult to respond quickly even when illegal behavior occurs.

[0089] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0090] In this invention, the server includes a means for a user to acquire video, a means for a terminal to compress the acquired video and transmit it to the server, and a means for the server to store the video received. This allows video captured by a user to be quickly and efficiently transmitted to the server and stored. The server also includes a means for analyzing the stored video using a machine learning model, a means for identifying the analyzed behavior and comparing it with a legal database to determine illegality, and a means for generating a feedback message and transmitting it to the terminal if illegality is determined. This makes it possible to quickly determine illegality based on the analysis results and provide feedback to the user, thereby realizing a rapid response to illegal or nuisance behavior.

[0091] "User" means any person or entity that uses the system to obtain footage and detect or report illegal activity.

[0092] A "terminal" is a hardware device used by a user to capture video, and includes devices such as smartphones and tablets.

[0093] A "server" is a computer system that receives, stores, and analyzes video data sent from a terminal.

[0094] "Video data" refers to digital information such as videos and still images taken by a user using a device.

[0095] "Compression" is a technical process for reducing the file size of video data and is a means of making data transfer more efficient.

[0096] A "machine learning model" is a collection of algorithms used for data analysis and pattern recognition, using artificial intelligence techniques to extract features in video.

[0097] A "legal database" is a database that stores various laws and ordinances in digital format and contains standard information for determining whether an analyzed act is illegal.

[0098] A "feedback message" is a notification generated based on the results of the analysis, and is a message that provides users with a determination of illegality and information on related legal provisions.

[0099] "Real-time" refers to near-instant processing, meaning results are returned within a short time of a user's action.

[0100] "Reporting" refers to the act of a user notifying the relevant authorities of illegal activity that they have discovered, and is done through the functions within the application.

[0101] The present invention is a system that uses AI technology to detect illegal activity based on video footage taken by users, compares the results with the law, and provides feedback. This system includes a video capture device, a server, a legal database, and a communication network connecting them. A specific embodiment of this system is described below.

[0102] Users use a camera device such as a smartphone or tablet to capture a specific scene. For example, they can capture a video of someone throwing away trash in a park. The device records the video data in high quality. Still images can also be captured as needed.

[0103] The device then compresses the captured video data, for example using software such as FFmpeg to reduce the size of the video. This compression process makes data transfer more efficient, and the compressed video data is sent to a server over the internet using the HTTPS protocol.

[0104] When the server receives video data from a device, it first stores it in a database (for example, cloud storage such as Amazon S3). The stored data is also accompanied by metadata such as the user ID and shooting date and time.

[0105] The server then analyzes the stored video data. For this analysis, it uses a machine learning model (e.g., TENSORFLOW (registered trademark) or PyTorch). Specifically, it uses an object detection algorithm such as YOLO (You Only Look Once) to detect different objects and actions in the video. For example, it identifies "littering" in the video.

[0106] The analyzed behavior is compared with a legal database (e.g., MySQL (registered trademark) or PostgreSQL) on the server. The legal database stores various laws and regulations, and it is determined which law the analyzed behavior violates. For example, it is checked whether the behavior violates a specific local smoking ban.

[0107] If the action is found to be illegal, the server generates a feedback message based on the results, including information about the relevant legal provisions and regulations. For example, the generated text might read, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act."

[0108] The generated feedback message is sent by the server to the device, again using the HTTPS protocol to ensure data security, and the device notifies the user of the received message.

[0109] Based on the feedback message, users can report the incident to the relevant authorities by pressing the report button in the app. For example, there is a function to report directly to local governments or police. If necessary, the app also provides legal advice by displaying contact information for lawyers and free consultation services.

[0110] Examples and prompts

[0111] Example 1:

[0112] A user takes a photo of someone throwing trash in a park.

[0113] The device compresses the video and sends it to the server.

[0114] The server analyzes the video and detects illegal dumping of garbage.

[0115] The server determines that the act constitutes illegal dumping based on the local government ordinance.

[0116] The server creates a feedback message saying "This action violates Article 16 of the Waste Disposal and Public Cleansing Act" and sends it to the terminal.

[0117] Based on the message, the user reports it to the appropriate authorities.

[0118] Example prompt sentence:

[0119] A user filmed illegal dumping in a park. The video was sent from the device to the server, which analyzed the footage and identified the illegal dumping. Based on the results, the server checked the relevant legal database to confirm its illegality. A feedback message was then generated and sent to the device. The user received the message and reported it to the relevant authorities.

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Step 1:

[0122] The user acquires the video

[0123] Input: The user captures a specific scene as a video or still image using a device (smartphone or tablet).

[0124] Processing: The user uses a camera application, presses the camera button, and records the subject's actions. The data is then saved in a high-quality format.

[0125] Output: Video files (video or still images) saved on the device.

[0126] Step 2:

[0127] The device compresses the video data and sends it to the server.

[0128] Input: User-shot video files.

[0129] Processing: The device compresses the video data using video compression software such as FFmpeg. This reduces the data size and makes transmission more efficient. After compression, the data is sent to the server using the HTTPS protocol.

[0130] Output: Compressed video data sent to the server.

[0131] Step 3:

[0132] The server receives and stores the video data.

[0133] Input: Compressed video data sent from the terminal.

[0134] Processing: The server stores the received video data in a cloud storage service such as Amazon S3. When storing the data, metadata such as the user ID, shooting date and time, and location information is also stored.

[0135] Output: Compressed video data and its metadata stored in cloud storage.

[0136] Step 4:

[0137] The server analyzes the video data using a machine learning model

[0138] Input: Video data stored in cloud storage.

[0139] Processing: The server uses machine learning libraries such as TensorFlow and PyTorch to apply the YOLO algorithm to recognize objects and actions in the video. For example, it identifies "littering" and "smoking" in the video.

[0140] Output: JSON formatted analysis result data containing the analyzed actions.

[0141] Step 5:

[0142] The server compares the analysis results with a legal database to determine whether they are illegal.

[0143] Input: Analysis result data generated by the machine learning model and legal database.

[0144] Processing: The server retrieves legal information from a MySQL or PostgreSQL database and checks whether the resulting behavior violates specific laws or regulations. For example, does it violate local smoking bans?

[0145] Output: Data containing the result of the illegality determination.

[0146] Step 6:

[0147] The server generates a feedback message

[0148] Input: Illegality determination result data.

[0149] Processing: If the server determines that the action is illegal, it generates a feedback message based on the relevant legal provisions and ordinances. For example, it creates a message stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act."

[0150] Output: The generated feedback message.

[0151] Step 7:

[0152] The server sends a feedback message to the device.

[0153] Input: The generated feedback message.

[0154] Processing: The server sends a feedback message to the terminal using the HTTPS protocol. The terminal notifies the user of the received message.

[0155] Output: The feedback message displayed on the user's terminal.

[0156] Step 8:

[0157] Users can report and seek legal advice based on feedback messages

[0158] Input: The feedback message displayed on the terminal.

[0159] Action: The user can report the incident to the appropriate authorities by pressing the report button within the app, and if necessary, contact a lawyer or legal advice center using the links and contact information in the feedback message.

[0160] Output: The relevant authorities have been notified or legal advice has been provided.

[0161] (Application example 1)

[0162] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0163] Current security systems have the challenge of detecting illegal or suspicious behavior in real time and notifying users immediately. They also lack the feedback needed to respond appropriately to the situation they encounter. As a result, local safety management may not be adequately managed, making real-time responses necessary.

[0164] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0165] In this invention, the server includes means for acquiring video, means for transmitting the acquired video to the server, means for analyzing the video received by the server, means for comparing the analysis result with a legal database to determine illegality, means for generating feedback and transmitting it to the terminal if illegality is determined, means for capturing video in real time using smart glasses, means for transmitting compressed data via the Internet, and means for displaying feedback to the user on a display, thereby making it possible to detect illegal or suspicious behavior in real time and provide immediate feedback to the user.

[0166] "Means for acquiring video" refers to the means by which a user uses a photographic device to record a specific scene or situation as video data.

[0167] The "means for transmitting the acquired video to the server" refers to a means for transmitting the captured video data via a network to a server for temporary storage or analysis.

[0168] "Means for analyzing video received by the server" refers to a means for analyzing the content of received video data using AI or machine learning technology.

[0169] "Means for comparing the analysis results with a legal database and determining illegality" refers to means for comparing the analyzed actions with a legal database and evaluating whether the actions are legally problematic.

[0170] "Means for generating feedback and sending it to the terminal when illegality is confirmed" refers to means for generating and sending a feedback message to notify the user of the result when illegal activity is confirmed.

[0171] "Means for capturing video in real time using smart glasses" refers to a means for recording video of the outside world in real time using the camera of smart glasses worn by the user.

[0172] "Means for transmitting compressed data via the Internet" refers to a means for compressing video data using highly efficient data compression technology and transferring the data to a server via an Internet line.

[0173] The "means for displaying feedback to the user on a display" refers to a means for visually displaying the analysis results and legal feedback on the display of the smart glasses or terminal used by the user.

[0174] The present invention is a system that uses AI technology to detect illegal activities based on video footage taken by users, compares the results with the law, and provides feedback. The specific configuration and procedures for implementing this system are described in detail below.

[0175] Video acquisition and transmission

[0176] The user wears the smart glasses and captures specific scenes or situations in real time. The smart glasses' built-in camera captures images of the outside world and temporarily stores them in the glasses' internal memory. The stored video data is then compressed using highly efficient data compression technology such as H.264 and sent to a server via the Internet.

[0177] Video analysis

[0178] The server stores the video data received from the smart glasses in a database. The stored video data is then analyzed using deep learning frameworks such as TensorFlow and PyTorch. A pre-trained machine learning model identifies activities in the video (e.g., acts of violence, theft, graffiti, etc.).

[0179] Legal Compliance

[0180] The analyzed behavior is compared with a legal database on the server (specifically, MongoDB or ElasticSearch (registered trademark)). The legal database stores various laws and ordinances, and the illegality of the analysis results is determined based on this. For example, if smoking in a public place is detected, it is determined whether the behavior violates the local government's smoking ban ordinance.

[0181] Generate and send feedback

[0182] If the action is deemed illegal, the server generates a feedback message and displays it on the smart glasses' display. The feedback message includes the relevant legal provisions and ordinances, providing the user with the necessary information immediately. For example, a specific message such as "This action violates Article 5 of the smoking ban ordinance of X City" may be displayed.

[0183] Reporting and legal consultation support

[0184] Users can operate the smart glasses interface to report or seek legal advice based on the feedback message. By pressing the report button within the application, users can directly report to the relevant authorities (police station or local government). Furthermore, contact details for experts and a free consultation service are provided for legal advice, if necessary.

[0185] Specific examples

[0186] 1. A user is wearing smart glasses while walking around town.

[0187] 2. Smart glasses capture scenes of street violence in real time.

[0188] 3. The server analyzes the video and detects violent acts.

[0189] 4. The server checks a legal database and determines that the activity violates criminal law.

[0190] 5. The server generates a feedback message saying "This act violates Article XX of the Criminal Code" and displays it on the smart glasses display.

[0191] 6. The user presses the report button on the smart glasses to call the police.

[0192] Prompt Sentence Examples

[0193] "Detect illegal or suspicious activity in the current footage and match it with applicable legal provisions."

[0194] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0195] Step 1: Capturing video with smart glasses

[0196] What happens: The camera in the smart glasses worn by the user captures images of the outside world in real time.

[0197] Input: Video of the user's field of view (environment video).

[0198] Output: Video data (real-time video frames).

[0199] How it works: The smart glasses' camera captures video frames at 30fps and temporarily stores them in its internal memory.

[0200] Step 2: Compress and transmit video data

[0201] What it does: Captured video data is compressed into a highly efficient format (such as H.264) and sent to a server over the Internet.

[0202] Input: Video data.

[0203] Output: Compressed video data.

[0204] How it works: The encoder inside the smart glasses compresses the video data and sends it to the server via Wi-Fi.

[0205] Step 3: Receiving and storing video data on the server

[0206] Processing details: The server decompresses the compressed video data received from the smart glasses and stores it in a database.

[0207] Input: Compressed video data.

[0208] Output: Decompressed video data.

[0209] What happens: The server's decoder decompresses the data and stores it in a database (e.g., MongoDB).

[0210] Step 4: Analyzing the video data

[0211] What it does: The server analyzes the stored video data using machine learning models (e.g., TensorFlow or PyTorch) to identify actions in the video.

[0212] Input: Decompressed video data.

[0213] Output: Analysis results (behavior classification).

[0214] How it works: An AI model running on a server analyzes video frames to detect suspicious or illegal behavior.

[0215] Step 5: Legal database matching

[0216] Processing details: The analyzed behavior is compared with a legal database to determine whether it is illegal.

[0217] Input: Analysis results.

[0218] Output: Legal decision result.

[0219] Specific operation: The server queries a legal database (e.g., Elasticsearch) with the analysis results and compares them with applicable laws and regulations.

[0220] Step 6: Generate and send a feedback message

[0221] What it does: Generates a feedback message and displays it on the smart glasses display.

[0222] Input: Legal decision result.

[0223] Output: The feedback message.

[0224] Specific operation: The server creates a feedback message citing the appropriate legal provisions for illegal activities and sends it to the smart glasses via the Internet.

[0225] Step 7: User Reports and Legal Advice

[0226] What it does: Users use the smart glasses interface to report or seek legal advice based on their feedback.

[0227] Input: The feedback message.

[0228] Output: Report or request for legal advice.

[0229] How it works: When the user presses the report button on the smart glasses, the device will directly report the incident to the police or local government. If necessary, contact details for legal advice will also be displayed on the screen.

[0230] Through these steps, users can obtain video footage in real time, instantly identify illegal or suspicious activity, and receive appropriate feedback to report or seek legal advice.

[0231] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0232] This system uses AI technology and an emotion engine to detect illegal activity based on video footage taken by users, compares the results with the law, and provides feedback. The system includes a video capture device, a server, a legal database, an emotion engine, and a communication network connecting them.

[0233] Video acquisition and transmission

[0234] Users use their smartphones or tablets to capture specific scenes, and the captured data is recorded as video or still images. The device then compresses the captured video and sends it to a server via the Internet.

[0235] Video analysis

[0236] The server stores the video data received from the device in a database. The stored data is then analyzed using AI technology. Specifically, machine learning models and deep learning are used to identify actions within the video (e.g., illegal dumping, illegal parking, smoking, etc.).

[0237] Legal Compliance

[0238] The analyzed behavior is compared with a legal database on the server. The legal database stores various laws and ordinances, and determines whether they are illegal. For example, if smoking is detected in a no-smoking area, the system determines whether the behavior violates the local government's smoking ban ordinance.

[0239] Combining Emotion Engines

[0240] This system incorporates an emotion engine and has been added with the ability to recognize the user's emotional state. The device collects emotional data from the user's voice, facial expressions, text input, etc., and the server analyzes this data using the emotion engine. Specifically, if the user is in a high stress state, feedback reflecting that state is provided.

[0241] Generate and send feedback

[0242] If illegality is found, the server generates a feedback message to reflect the result. The tone of the message and recommended actions change depending on the user's emotional state. For example, if the user is in a state of high stress, a more reassuring message (such as "Don't push yourself, take appropriate action") will be generated and sent to the device.

[0243] Reporting and legal consultation support

[0244] Users can report or seek legal advice based on the feedback messages displayed on their device. For example, by pressing the report button within the app, they can directly report to the relevant authorities (police station or local government). In addition, based on the analysis results of the emotion engine, the app will provide recommended actions to reduce psychological stress and the contact information for lawyers for legal advice.

[0245] Specific examples

[0246] 1. The user takes a picture of someone throwing away trash in a park and says, "What should I do about this?"

[0247] 2. The device compresses the video and audio data and sends it to the server.

[0248] 3. The server analyzes the video to detect illegal dumping, and simultaneously analyzes the audio data with an emotion engine to recognize high stress levels.

[0249] 4. The server determines that the act constitutes illegal dumping based on local government ordinances.

[0250] 5. The server generates feedback stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not force yourself and take appropriate action," and sends it to the device.

[0251] 6. The user reports the message.

[0252] This allows users to instantly identify nuisance or illegal behavior and take appropriate action, and by utilizing the emotion engine, they can deal with the situation while reducing the mental burden.

[0253] The processing flow will be explained below.

[0254] Step 1:

[0255] The user launches the app and switches to camera mode by tapping the app icon on their smartphone or tablet and then pressing the camera button on the screen.

[0256] Step 2:

[0257] The user captures a specific scene, for example, someone throwing out trash, and the camera records the footage within its coverage area.

[0258] Step 3:

[0259] The device also records the user's voice and facial expressions. For example, if the user says, "What should I do about this?", that voice data will also be recorded.

[0260] Step 4:

[0261] The device temporarily stores the video and audio data it captures. The video and audio data are stored in the device's memory.

[0262] Step 5:

[0263] The device compresses the captured video and audio data and prepares it for transmission to the server, converting the captured data into an optimized format and establishing an Internet connection.

[0264] Step 6:

[0265] The device sends the video and audio data to the server, where it is encrypted using the HTTPS protocol and securely uploaded to the server.

[0266] Step 7:

[0267] The server receives the video and audio data sent from the terminal, and saves the received data in a specified directory on the server.

[0268] Step 8:

[0269] The server analyzes the received video, specifically using machine learning models and deep learning algorithms to detect specific behaviors within the video, in this case illegal dumping of trash.

[0270] Step 9:

[0271] The server analyzes the voice and facial expression data it receives using an emotion engine, specifically recognizing emotional states such as high stress or anger.

[0272] Step 10:

[0273] The server compares the video analysis results with a legal database, which stores various laws and regulations, and determines whether the video is illegal based on the analysis results.

[0274] Step 11:

[0275] If the server determines that something is illegal, it generates a feedback message that reflects the results of the emotion engine. For example, if a high stress state is recognized, a message such as "This behavior violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not push yourself and take appropriate action" will be created.

[0276] Step 12:

[0277] The server sends the generated feedback message to the terminal, specifically, sends the generated message back to the user's terminal for display.

[0278] Step 13:

[0279] The device receives the feedback message and displays it to the user as a notification, which can be a pop-up notification or an in-app message.

[0280] Step 14:

[0281] Based on the feedback messages received, users can report or seek legal advice. Specifically, by tapping the "Report" or "Legal Advice" button within the app, an interface will appear for contacting the relevant authorities.

[0282] This allows users to quickly identify illegal activity occurring in front of them and take appropriate action, while also reducing the mental burden by utilizing the emotion engine.

[0283] Example 2

[0284] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0285] Conventional illegal activity detection systems have the problem of low accuracy in analyzing video data, which makes it difficult to quickly determine illegal activity. In addition, users often feel emotional stress when reporting illegal activity, and appropriate feedback may not be provided. This makes it difficult for users to take appropriate action.

[0286] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0287] In this invention, the server includes a means for storing and analyzing data in a database, a means for classifying the analyzed results as specific acts using a machine learning model, and a means for comparing the results with a legal database to determine illegality. This enables highly accurate analysis of video data and rapid determination of illegality. Furthermore, the server includes a means for generating feedback based on the user's emotional data, which can provide appropriate feedback suited to the user's emotional state, reducing stress for the user and encouraging appropriate responses.

[0288] "Means for acquiring images" refers to the ability of a user to capture a specific scene using a device such as a smartphone or tablet.

[0289] "Means for the terminal to compress the captured video and send it to the server" is a function that compresses the captured video data and transfers it to the server via the Internet.

[0290] The "means for storing in a database and analyzing" is a mechanism for storing the video data received by the server and later analyzing the data.

[0291] "Means of using a machine learning model to classify the analyzed results as specific actions" refers to a system that uses machine learning technology to recognize and classify specific actions (e.g., illegal dumping, illegal parking, etc.) from analyzed video data.

[0292] "Means for comparing with a legal database and determining illegality" is a function that compares the analyzed act with the laws and ordinances in the legal database and determines whether the act is illegal.

[0293] "Means for collecting user emotional data" refers to devices and software for collecting data such as the user's voice, facial expressions, and text input.

[0294] The "means for analyzing emotional data" is a system that analyzes collected emotional data using an emotional engine and recognizes the user's emotional state (e.g., high stress level).

[0295] "Means for generating feedback and sending it to the device" refers to a function that generates a feedback message based on the user's emotional state when illegality is recognized, and sends and displays that message on the device.

[0296] "Means for reporting or seeking legal advice" means a system that allows users to report to relevant authorities based on feedback messages or provides contact information for assistance in seeking legal advice.

[0297] This invention is a system that uses AI technology and an emotion engine to detect illegal activity based on video footage taken by users, compares the results with a legal database, and provides feedback.The system is primarily composed of devices such as smartphones and tablets, a server that processes data, a legal database, an emotion engine, and a communication network that connects them.

[0298] Video acquisition and transmission

[0299] Users use their smartphones or tablets to capture specific scenes. The captured data is recorded as video or still images. The device then compresses the video data using algorithms such as the H.264 compression algorithm and sends it to a server via the Internet. For example, if a user captures someone throwing away trash in a park, the video is compressed on the device and sent to the server.

[0300] Video analysis

[0301] The server stores the video data received from the device in a dedicated database (e.g., MongoDB). The server then analyzes the stored video data using a deep learning library such as TensorFlow. This analysis allows it to recognize specific behaviors (e.g., illegal dumping, illegal parking, etc.). For example, the received video data can be analyzed using a deep learning model to identify illegal dumping of garbage in a park.

[0302] Legal Compliance

[0303] The analyzed actions are compared with a legal database (e.g., PostgreSQL) on the server. The legal database stores various laws and ordinances, and determines whether a particular action is illegal. For example, it determines whether illegal dumping of garbage violates local government ordinances.

[0304] Combining Emotion Engines

[0305] This system is equipped with an emotion engine, and the device collects emotional data from the user's voice, facial expressions, text input, etc. The server analyzes this emotional data using an emotion engine (e.g., Microsoft (registered trademark) Emotion API) to recognize the user's emotional state (e.g., high stress level). For example, analyzing voice data in which the user mutters, "What should I do about this?", the system recognizes that the user is in a high stress state.

[0306] Generate and send feedback

[0307] If illegality is found, the server generates a feedback message to that effect. The tone of the message and recommended actions change based on the user's emotional state. For example, if the user is in a state of high stress, a reassuring message will be generated and sent to the device. For example, a message such as, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please take appropriate action without forcing yourself" will be generated.

[0308] Reporting and legal consultation support

[0309] Based on the feedback messages displayed on the device, users can report the incident or seek legal advice. For example, by pressing the report button within the app, users can directly report the incident to the relevant authority (e.g., the local government's environmental protection department). In addition, based on the analysis results of the emotion engine, recommended actions to reduce psychological stress and contact information for legal advice are provided. For example, users can press the report button to report the incident to the police station.

[0310] Prompt Sentence Examples

[0311] An example of a prompt that can be input to a generative AI model is, "Detect illegal dumping of garbage in a park and generate a feedback message based on the user's emotional state."

[0312] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0313] Step 1:

[0314] A user uses a smartphone or tablet to capture a specific scene. The input is the captured video data (video or still images). A specific operation is for the user to capture a scene of someone throwing away trash in a park. The output is the video data stored on the device.

[0315] Step 2:

[0316] The device compresses the captured video data and sends it to the server via the Internet. The input is the video data generated in step 1. Specifically, the captured data is compressed using the H.264 compression algorithm (for example, reducing a 100MB video to 5MB). The output is the compressed video data being sent to the server.

[0317] Step 3:

[0318] The server stores compressed video data received from the device in a database. The input is compressed video data. Specifically, the video data is stored in a database such as MongoDB. The output is video data stored in the database.

[0319] Step 4:

[0320] The server uses a deep learning model (e.g., TensorFlow) to analyze the video data. The input is the video data stored in a database. Specifically, the video data is input into the deep learning model to detect specific behaviors, such as illegal dumping of garbage. The output is specific behavior data as the analysis result.

[0321] Step 5:

[0322] The server compares the analysis results with a legal database to determine whether the act is illegal. The input is specific act data. Specifically, it accesses a legal database such as PostgreSQL and compares the analysis results with various laws and ordinances. The output is a determination of whether the act is illegal.

[0323] Step 6:

[0324] The device collects emotion data from the user's voice, facial expressions, text input, etc. The input is the user's emotion-related data (voice, facial expressions, text). Specifically, it acquires voice data in which the user mutters, "What should I do about this?" The output is emotion data.

[0325] Step 7:

[0326] The server uses an emotion engine (e.g., Microsoft's Emotion API) to analyze emotion data. The input is the collected emotion data. Specific operations include inputting data into the emotion engine and recognizing the user's emotional state (e.g., high stress). The output is the emotional state data resulting from the analysis.

[0327] Step 8:

[0328] The server generates a feedback message based on the illegality determination result and the user's emotional state data. The input is the illegality determination result and the emotional state data. Specific operations include appropriately adjusting the tone and content of the feedback message. A message is generated stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not force yourself and take appropriate action." The output is the generated feedback message.

[0329] Step 9:

[0330] The server sends the generated feedback message to the terminal. The input is the feedback message generated in step 8. As a specific operation, the message is sent to the terminal via the Internet. The output is the feedback message displayed on the terminal.

[0331] Step 10:

[0332] The user can report or seek legal advice based on the feedback message. The input is the feedback message displayed on the terminal. Specific actions include the user pressing the report button to report to the police station or the local government's environmental protection department. The output is the completion of the report to the relevant agency or the contact for legal advice.

[0333] (Application example 2)

[0334] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0335] Previously, systems existed to detect illegal and nuisance behavior, but they simply detected and reported illegal behavior without taking into account the user's emotional state. This made it difficult for them to take appropriate action when users were experiencing high levels of stress or anxiety. Furthermore, they lacked the accuracy of recognizing illegal behavior and the ability to provide appropriate feedback in real time. As a result, users were unsure of how to act, leading to a problem of low enforcement rates.

[0336] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video, means for comparing the analysis results with a legal database to determine illegality, and means for recognizing the user's emotional state and generating an appropriate feedback message. This makes it possible to quickly detect illegal acts from video captured by the user and provide appropriate feedback based on the emotional analysis without burdening the user. The user can take appropriate action based on the feedback message, and can report the act or seek legal advice in real time. This realizes a quick and effective response to illegal acts while reducing the mental burden on the user.

[0337] "Means for acquiring video footage" refers to a function that allows users to record videos and images using devices such as smartphones and cameras.

[0338] The "means for transmitting to a server" is a function for transmitting the acquired video data to a server via the Internet or other communication network.

[0339] "Means for analyzing video" refers to a function that utilizes machine learning models and deep learning technology to analyze received video data and identify specific actions and objects within it.

[0340] The "means of comparing with a legal database to determine illegality" is a function that compares the content of the analyzed video data with pre-registered legal and ordinance data to determine whether it violates laws and regulations.

[0341] "Means for recognizing the user's emotional state" refers to a function that analyzes the user's voice, facial expression, or text input to measure their current emotional state and stress level.

[0342] The "means for generating appropriate feedback messages" is a function for creating messages that reassure users or encourage them to take appropriate action based on the recognized emotional state of the user.

[0343] The "means for sending a feedback message to a user terminal" is a function for sending the generated feedback message to a user terminal such as a smartphone or tablet.

[0344] An "emotion engine" is a technology that includes algorithms and models for analyzing a user's emotions from voice and text data and identifying their emotional state.

[0345] A "machine learning model" is a collection of algorithms that learn from data and recognize patterns to make predictions or classifications for a specific task.

[0346] "Real-time" refers to a process in which data is processed and analyzed within a very short time after it is acquired, and the results are fed back immediately.

[0347] The present invention is a system that analyzes videos taken by users, detects illegal activities using AI technology and an emotion engine, compares the results with the law, and provides feedback. Specific embodiments of the system are described in detail below.

[0348] System Configuration

[0349] This system mainly consists of a user terminal, a server, a legal database, an emotion engine, and a communication network that connects them.

[0350] 1. User Device

[0351] Users use devices such as smartphones or cameras to capture images of the target. The captured image data is generated using the device's camera function. The emotion engine also collects the user's voice and text inputs as needed.

[0352] 2. Server

[0353] The video data captured by the device is sent to a server via a communication network. The server has the following functions:

[0354] 1. Video analysis: Video data is received and analyzed using machine learning models and deep learning techniques to identify actions and objects in the video. This process uses libraries such as TensorFlow and OpenCV.

[0355] 2. Legal matching: The analyzed behavior is compared with a legal database containing various laws and regulations to determine whether it constitutes an illegal act.

[0356] 3. Emotion analysis: Analyze the user's voice and text data using an emotion engine (e.g., TextBlob) to recognize the user's emotional state.

[0357] 4. Feedback Generation: If illegality is detected, an appropriate feedback message is generated taking into account the user's emotional state. This message is sent to the device.

[0358] 3. Communication Network

[0359] Communication between the user device and the server is carried out via a communication network such as the Internet, and video data, analysis results, and feedback messages are sent and received in a timely manner.

[0360] Specific examples

[0361] 1. A user spots someone illegally littering in a park and takes a photo of the incident with their smartphone.

[0362] 2. When taking a photo, the user mutters out loud, "What should I do with this?"

[0363] 3. The smartphone compresses the video and audio data and sends it to the server.

[0364] 4. The server analyzes the video to detect illegal dumping, and simultaneously analyzes the audio data with an emotion engine to recognize high stress levels.

[0365] 5. The server checks the legal database and determines that the illegal dumping violates Article 16 of the Waste Disposal and Public Cleansing Act.

[0366] 6. The server generates a feedback message stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not force yourself and take appropriate action," and sends it to the user's device.

[0367] 7. The user checks the message and reports it to local authorities using the reporting function within the app.

[0368] Example prompts for generative AI models

[0369] "A user's smartphone was used to record the act of littering in a park. Based on the video, AI technology was used to detect illegal dumping, and an emotion engine was used to identify the user's high stress level. The system then generated a feedback message saying, 'Don't push yourself, take appropriate action,' and reported the incident to local authorities."

[0370] In this way, the present invention makes it possible to deal with illegal activities quickly and appropriately while reducing the burden on users.

[0371] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0372] Step 1:

[0373] The user shoots video and inputs audio using a smartphone.

[0374] Input: Video and audio data obtained from the smartphone camera and microphone.

[0375] Output: Recorded video files (video or still images) and audio files.

[0376] Specific operation: The smartphone camera records the target's actions, and the microphone simultaneously records the user's voice.

[0377] Step 2:

[0378] The video and audio data captured by the device is compressed and sent to a server via the Internet.

[0379] Input: Recorded video and audio files.

[0380] Output: Compressed video and audio data.

[0381] Specific operation: Video and audio files are reduced in size using a compression algorithm and sent to the server's API endpoint.

[0382] Step 3:

[0383] The server analyzes the received video data and identifies the actions in the video.

[0384] Input: Compressed video data.

[0385] Output: Analysis results (identification of actions).

[0386] How it works: The server decodes the video data and uses machine learning models (e.g., TensorFlow) to analyze the actions in the video, such as identifying illegal dumping and illegal parking.

[0387] Step 4:

[0388] The server compares the analysis results with a legal database to determine whether they are illegal.

[0389] Input: Analysis results (identification of actions).

[0390] Output: Illegality determination result.

[0391] Specific operation: The analysis results are compared with ordinances and laws registered in a legal database to determine whether the behavior is illegal. For example, if smoking is detected in a local government no-smoking zone, the behavior will be determined to be a violation of the ordinance.

[0392] Step 5:

[0393] The server analyzes the received voice data using an emotion engine to recognize the user's emotional state.

[0394] Input: Audio data.

[0395] Output: Emotional state judgment result.

[0396] What it does: Analyzes voice data and recognizes the user's emotional state. It uses a sentiment analysis model (e.g., TextBlob) to determine emotions (e.g., high stress, anger, relief, etc.) from the user's voice.

[0397] Step 6:

[0398] The server generates an appropriate feedback message based on the results of the illegal act determination and the emotional state determination.

[0399] Input: Illegality judgment results and emotional state judgment results.

[0400] Output: Feedback message.

[0401] Specific operation: The results of the illegality judgment and the emotional state judgment are combined to generate a feedback message for the user. For example, if illegal dumping is detected and the user is in a high stress state, a message such as "This behavior is illegal. Please do not push yourself and take appropriate action" is generated.

[0402] Step 7:

[0403] The server sends the generated feedback message to the terminal.

[0404] Input: Feedback message.

[0405] Output: Messages that are displayed on the user's terminal.

[0406] Specific operation: The generated feedback message is sent to the user's device and displayed on the screen, allowing the user to check the message.

[0407] Step 8:

[0408] The user reports or seeks legal advice based on the feedback message.

[0409] Input: Feedback message.

[0410] Output: Report or provide legal advice.

[0411] Specific actions: The user checks the feedback message displayed on the device and, if necessary, reports the issue to the appropriate authorities using the in-app reporting function, or contacts a lawyer or other professional using the legal consultation function.

[0412] By taking these steps, the system of the present invention can detect illegal activity from video footage taken by users, provide appropriate feedback, and take prompt action.

[0413] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0414] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0415] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0416] [Second embodiment]

[0417] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0418] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0419] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0420] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0421] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0422] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0423] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0424] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0425] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0426] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0427] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0428] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0429] This system uses AI technology to detect illegal activity based on video footage taken by users, compares the results with the law, and provides feedback. The system includes a video capture device, a server, a legal database, and a communication network connecting them.

[0430] Video acquisition and transmission

[0431] Users use their smartphones or tablets to capture specific scenes, and the captured data is recorded as video or still images. The device then compresses the captured video and sends it to a server via the Internet.

[0432] Video analysis

[0433] The server stores the video data received from the device in a database. The stored data is then analyzed using AI technology. Specifically, machine learning models and deep learning are used to identify actions within the video (e.g., illegal dumping, illegal parking, smoking, etc.).

[0434] Legal Compliance

[0435] The analyzed behavior is compared with a legal database on the server. The legal database stores various laws and ordinances, and determines whether they are illegal. For example, if smoking is detected in a no-smoking area, the system determines whether the behavior violates the local government's smoking ban ordinance.

[0436] Generate and send feedback

[0437] If the action is found to be illegal, the server generates a feedback message containing the relevant legal provisions and ordinances and sends it to the device. For example, a specific message such as "This action violates Article 5 of the smoking ban ordinance of X City" may be displayed.

[0438] Reporting and legal consultation support

[0439] Users can report or seek legal advice based on the feedback messages displayed on their device. For example, by pressing the report button within the app, they can report directly to the relevant authorities (police station or local government). Furthermore, if necessary, the app will provide contact information for lawyers and free legal advice.

[0440] Specific examples

[0441] 1. A user takes a photo of someone littering in a park.

[0442] 2. The device compresses the video and sends it to the server.

[0443] 3. The server analyzes the video and detects illegal dumping of garbage.

[0444] 4. The server determines that the act constitutes illegal dumping based on local government ordinances.

[0445] 5. The server creates feedback stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act," and sends it to the device.

[0446] 6. The user reports the message.

[0447] This will enable users to immediately identify nuisance or illegal behavior and take appropriate action, contributing to improving public safety in the local area.

[0448] This system can solve the problems that traditional reporting systems have by providing real-time analysis and notification, as well as support for reporting and legal consultation.

[0449] The processing flow will be explained below.

[0450] Step 1:

[0451] The user launches the app and switches to camera mode by tapping the app icon to open it and then pressing the camera button on the screen to switch to camera mode.

[0452] Step 2:

[0453] When a user captures a specific scene, for example, recording someone throwing trash in a park, the camera will record footage within its coverage area.

[0454] Step 3:

[0455] The device temporarily stores the video it has taken. The video data is stored in the device's memory.

[0456] Step 4:

[0457] The device compresses the captured video and prepares it for transmission to the server, converting the video data into an optimized format and establishing an Internet connection.

[0458] Step 5:

[0459] The device sends the video data to the server. Specifically, the data is encrypted using the HTTPS protocol and securely uploaded to the server.

[0460] Step 6:

[0461] The server receives the video data sent from the device and saves it in a specified directory on the server.

[0462] Step 7:

[0463] The server analyzes the received video, specifically using machine learning models and deep learning algorithms to detect specific behaviors within the video, in this case illegal dumping of trash.

[0464] Step 8:

[0465] The server compares the analysis results with a legal database, which stores various laws and regulations, and determines whether the information is illegal based on the analysis results.

[0466] Step 9:

[0467] If the server determines that the action is illegal, it generates a feedback message indicating the result, such as "This action violates Article 16 of the Waste Disposal and Public Cleansing Act."

[0468] Step 10:

[0469] The server sends a feedback message to the terminal, specifically, a generated message sent back to the user's terminal for display.

[0470] Step 11:

[0471] The device receives the feedback message and displays it to the user as a notification, which can be a pop-up notification or an in-app message.

[0472] Step 12:

[0473] Based on the feedback messages received, users can report or seek legal advice. Specifically, by tapping the "Report" or "Legal Advice" button within the app, an interface will appear for contacting the relevant authorities.

[0474] By following these steps, users can quickly identify any illegal activity occurring in front of them and take appropriate action.

[0475] Example 1

[0476] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0477] In modern society, there is a need for real-time identification and response to illegal and nuisance behavior. Conventional reporting systems require users to manually report illegal behavior, making it difficult to respond quickly and accurately. In addition, determining illegality takes time, which makes it difficult to respond quickly even when illegal behavior occurs.

[0478] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0479] In this invention, the server includes a means for a user to acquire video, a means for a terminal to compress the acquired video and transmit it to the server, and a means for the server to store the video received. This allows video captured by a user to be quickly and efficiently transmitted to the server and stored. The server also includes a means for analyzing the stored video using a machine learning model, a means for identifying the analyzed behavior and comparing it with a legal database to determine illegality, and a means for generating a feedback message and transmitting it to the terminal if illegality is determined. This makes it possible to quickly determine illegality based on the analysis results and provide feedback to the user, thereby realizing a rapid response to illegal or nuisance behavior.

[0480] "User" means any person or entity that uses the system to obtain footage and detect or report illegal activity.

[0481] A "terminal" is a hardware device used by a user to capture video, and includes devices such as smartphones and tablets.

[0482] A "server" is a computer system that receives, stores, and analyzes video data sent from a terminal.

[0483] "Video data" refers to digital information such as videos and still images taken by a user using a device.

[0484] "Compression" is a technical process for reducing the file size of video data and is a means of making data transfer more efficient.

[0485] A "machine learning model" is a collection of algorithms used for data analysis and pattern recognition, using artificial intelligence techniques to extract features in video.

[0486] A "legal database" is a database that stores various laws and ordinances in digital format and contains standard information for determining whether an analyzed act is illegal.

[0487] A "feedback message" is a notification generated based on the results of the analysis, and is a message that provides users with a determination of illegality and information on related legal provisions.

[0488] "Real-time" refers to near-instant processing, meaning results are returned within a short time of a user's action.

[0489] "Reporting" refers to the act of a user notifying the relevant authorities of illegal activity that they have discovered, and is done through the functions within the application.

[0490] The present invention is a system that uses AI technology to detect illegal activity based on video footage taken by users, compares the results with the law, and provides feedback. This system includes a video capture device, a server, a legal database, and a communication network connecting them. A specific embodiment of this system is described below.

[0491] Users use a camera device such as a smartphone or tablet to capture a specific scene. For example, they can capture a video of someone throwing away trash in a park. The device records the video data in high quality. Still images can also be captured as needed.

[0492] The device then compresses the captured video data, for example using software such as FFmpeg to reduce the size of the video. This compression process makes data transfer more efficient, and the compressed video data is sent to a server over the internet using the HTTPS protocol.

[0493] When the server receives video data from a device, it first stores it in a database (for example, cloud storage such as Amazon S3). The stored data is also accompanied by metadata such as the user ID and shooting date and time.

[0494] The server then analyzes the stored video data. For this analysis, it uses machine learning models (e.g., TensorFlow or PyTorch). Specifically, it uses object detection algorithms such as YOLO (You Only Look Once) to detect different objects and actions within the video. For example, it can identify "littering" within the video.

[0495] The analyzed behavior is compared with a legal database (e.g., MySQL or PostgreSQL) on the server. The legal database stores various laws and regulations, and determines which law the analyzed behavior violates. For example, it checks whether the behavior violates a specific local smoking ban.

[0496] If the action is found to be illegal, the server generates a feedback message based on the results, including information about the relevant legal provisions and regulations. For example, the generated text might read, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act."

[0497] The generated feedback message is sent by the server to the device, again using the HTTPS protocol to ensure data security, and the device notifies the user of the received message.

[0498] Based on the feedback message, users can report the incident to the relevant authorities by pressing the report button in the app. For example, there is a function to report directly to local governments or police. If necessary, the app also provides legal advice by displaying contact information for lawyers and free consultation services.

[0499] Examples and prompts

[0500] Example 1:

[0501] A user takes a photo of someone throwing trash in a park.

[0502] The device compresses the video and sends it to the server.

[0503] The server analyzes the video and detects illegal dumping of garbage.

[0504] The server determines that the act constitutes illegal dumping based on the local government ordinance.

[0505] The server creates a feedback message saying "This action violates Article 16 of the Waste Disposal and Public Cleansing Act" and sends it to the terminal.

[0506] Based on the message, the user reports it to the appropriate authorities.

[0507] Example prompt sentence:

[0508] A user filmed illegal dumping in a park. The video was sent from the device to the server, which analyzed the footage and identified the illegal dumping. Based on the results, the server checked the relevant legal database to confirm its illegality. A feedback message was then generated and sent to the device. The user received the message and reported it to the relevant authorities.

[0509] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0510] Step 1:

[0511] The user acquires the video

[0512] Input: The user captures a specific scene as a video or still image using a device (smartphone or tablet).

[0513] Processing: The user uses a camera application, presses the camera button, and records the subject's actions. The data is then saved in a high-quality format.

[0514] Output: Video files (video or still images) saved on the device.

[0515] Step 2:

[0516] The device compresses the video data and sends it to the server.

[0517] Input: User-shot video files.

[0518] Processing: The device compresses the video data using video compression software such as FFmpeg. This reduces the data size and makes transmission more efficient. After compression, the data is sent to the server using the HTTPS protocol.

[0519] Output: Compressed video data sent to the server.

[0520] Step 3:

[0521] The server receives and stores the video data.

[0522] Input: Compressed video data sent from the terminal.

[0523] Processing: The server stores the received video data in a cloud storage service such as Amazon S3. When storing the data, metadata such as the user ID, shooting date and time, and location information is also stored.

[0524] Output: Compressed video data and its metadata stored in cloud storage.

[0525] Step 4:

[0526] The server analyzes the video data using a machine learning model

[0527] Input: Video data stored in cloud storage.

[0528] Processing: The server uses machine learning libraries such as TensorFlow and PyTorch to apply the YOLO algorithm to recognize objects and actions in the video. For example, it identifies "littering" and "smoking" in the video.

[0529] Output: JSON formatted analysis result data containing the analyzed actions.

[0530] Step 5:

[0531] The server compares the analysis results with a legal database to determine whether they are illegal.

[0532] Input: Analysis result data generated by the machine learning model and legal database.

[0533] Processing: The server retrieves legal information from a MySQL or PostgreSQL database and checks whether the resulting behavior violates specific laws or regulations. For example, does it violate local smoking bans?

[0534] Output: Data containing the result of the illegality determination.

[0535] Step 6:

[0536] The server generates a feedback message

[0537] Input: Illegality determination result data.

[0538] Processing: If the server determines that the action is illegal, it generates a feedback message based on the relevant legal provisions and ordinances. For example, it creates a message stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act."

[0539] Output: The generated feedback message.

[0540] Step 7:

[0541] The server sends a feedback message to the device.

[0542] Input: The generated feedback message.

[0543] Processing: The server sends a feedback message to the terminal using the HTTPS protocol. The terminal notifies the user of the received message.

[0544] Output: The feedback message displayed on the user's terminal.

[0545] Step 8:

[0546] Users can report and seek legal advice based on feedback messages

[0547] Input: The feedback message displayed on the terminal.

[0548] Action: The user can report the incident to the appropriate authorities by pressing the report button within the app, and if necessary, contact a lawyer or legal advice center using the links and contact information in the feedback message.

[0549] Output: The relevant authorities have been notified or legal advice has been provided.

[0550] (Application example 1)

[0551] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0552] Current security systems have the challenge of detecting illegal or suspicious behavior in real time and notifying users immediately. They also lack the feedback needed to respond appropriately to the situation they encounter. As a result, local safety management may not be adequately managed, making real-time responses necessary.

[0553] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0554] In this invention, the server includes means for acquiring video, means for transmitting the acquired video to the server, means for analyzing the video received by the server, means for comparing the analysis result with a legal database to determine illegality, means for generating feedback and transmitting it to the terminal if illegality is determined, means for capturing video in real time using smart glasses, means for transmitting compressed data via the Internet, and means for displaying feedback to the user on a display, thereby making it possible to detect illegal or suspicious behavior in real time and provide immediate feedback to the user.

[0555] "Means for acquiring video" refers to the means by which a user uses a photographic device to record a specific scene or situation as video data.

[0556] The "means for transmitting the acquired video to the server" refers to a means for transmitting the captured video data via a network to a server for temporary storage or analysis.

[0557] "Means for analyzing video received by the server" refers to a means for analyzing the content of received video data using AI or machine learning technology.

[0558] "Means for comparing the analysis results with a legal database and determining illegality" refers to means for comparing the analyzed actions with a legal database and evaluating whether the actions are legally problematic.

[0559] "Means for generating feedback and sending it to the terminal when illegality is confirmed" refers to means for generating and sending a feedback message to notify the user of the result when illegal activity is confirmed.

[0560] "Means for capturing video in real time using smart glasses" refers to a means for recording video of the outside world in real time using the camera of smart glasses worn by the user.

[0561] "Means for transmitting compressed data via the Internet" refers to a means for compressing video data using highly efficient data compression technology and transferring the data to a server via an Internet line.

[0562] The "means for displaying feedback to the user on a display" refers to a means for visually displaying the analysis results and legal feedback on the display of the smart glasses or terminal used by the user.

[0563] The present invention is a system that uses AI technology to detect illegal activities based on video footage taken by users, compares the results with the law, and provides feedback. The specific configuration and procedures for implementing this system are described in detail below.

[0564] Video acquisition and transmission

[0565] The user wears the smart glasses and captures specific scenes or situations in real time. The smart glasses' built-in camera captures images of the outside world and temporarily stores them in the glasses' internal memory. The stored video data is then compressed using highly efficient data compression technology such as H.264 and sent to a server via the Internet.

[0566] Video analysis

[0567] The server stores the video data received from the smart glasses in a database. The stored video data is then analyzed using deep learning frameworks such as TensorFlow and PyTorch. A pre-trained machine learning model identifies activities in the video (e.g., acts of violence, theft, graffiti, etc.).

[0568] Legal Compliance

[0569] The analyzed behavior is compared with a legal database on the server (specifically, MongoDB or Elasticsearch). The legal database stores various laws and regulations, and the illegality of the analysis results is determined based on this. For example, if smoking in a public place is detected, the system determines whether the behavior violates the local government's smoking ban.

[0570] Generate and send feedback

[0571] If the action is deemed illegal, the server generates a feedback message and displays it on the smart glasses' display. The feedback message includes the relevant legal provisions and ordinances, providing the user with the necessary information immediately. For example, a specific message such as "This action violates Article 5 of the smoking ban ordinance of X City" may be displayed.

[0572] Reporting and legal consultation support

[0573] Users can operate the smart glasses interface to report or seek legal advice based on the feedback message. By pressing the report button within the application, users can directly report to the relevant authorities (police station or local government). Furthermore, contact details for experts and a free consultation service are provided for legal advice, if necessary.

[0574] Specific examples

[0575] 1. A user is wearing smart glasses while walking around town.

[0576] 2. Smart glasses capture scenes of street violence in real time.

[0577] 3. The server analyzes the video and detects violent acts.

[0578] 4. The server checks a legal database and determines that the activity violates criminal law.

[0579] 5. The server generates a feedback message saying "This act violates Article XX of the Criminal Code" and displays it on the smart glasses display.

[0580] 6. The user presses the report button on the smart glasses to call the police.

[0581] Prompt Sentence Examples

[0582] "Detect illegal or suspicious activity in the current footage and match it with applicable legal provisions."

[0583] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0584] Step 1: Capturing video with smart glasses

[0585] What happens: The camera in the smart glasses worn by the user captures images of the outside world in real time.

[0586] Input: Video of the user's field of view (environment video).

[0587] Output: Video data (real-time video frames).

[0588] How it works: The smart glasses' camera captures video frames at 30fps and temporarily stores them in its internal memory.

[0589] Step 2: Compress and transmit video data

[0590] What it does: Captured video data is compressed into a highly efficient format (such as H.264) and sent to a server over the Internet.

[0591] Input: Video data.

[0592] Output: Compressed video data.

[0593] How it works: The encoder inside the smart glasses compresses the video data and sends it to the server via Wi-Fi.

[0594] Step 3: Receiving and storing video data on the server

[0595] Processing details: The server decompresses the compressed video data received from the smart glasses and stores it in a database.

[0596] Input: Compressed video data.

[0597] Output: Decompressed video data.

[0598] What happens: The server's decoder decompresses the data and stores it in a database (e.g., MongoDB).

[0599] Step 4: Analyzing the video data

[0600] What it does: The server analyzes the stored video data using machine learning models (e.g., TensorFlow or PyTorch) to identify actions in the video.

[0601] Input: Decompressed video data.

[0602] Output: Analysis results (behavior classification).

[0603] How it works: An AI model running on a server analyzes video frames to detect suspicious or illegal behavior.

[0604] Step 5: Legal database matching

[0605] Processing details: The analyzed behavior is compared with a legal database to determine whether it is illegal.

[0606] Input: Analysis results.

[0607] Output: Legal decision result.

[0608] Specific operation: The server queries a legal database (e.g., Elasticsearch) with the analysis results and compares them with applicable laws and regulations.

[0609] Step 6: Generate and send a feedback message

[0610] What it does: Generates a feedback message and displays it on the smart glasses display.

[0611] Input: Legal decision result.

[0612] Output: The feedback message.

[0613] Specific operation: The server creates a feedback message citing the appropriate legal provisions for illegal activities and sends it to the smart glasses via the Internet.

[0614] Step 7: User Reports and Legal Advice

[0615] What it does: Users use the smart glasses interface to report or seek legal advice based on their feedback.

[0616] Input: The feedback message.

[0617] Output: Report or request for legal advice.

[0618] How it works: When the user presses the report button on the smart glasses, the device will directly report the incident to the police or local government. If necessary, contact details for legal advice will also be displayed on the screen.

[0619] Through these steps, users can obtain video footage in real time, instantly identify illegal or suspicious activity, and receive appropriate feedback to report or seek legal advice.

[0620] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0621] This system uses AI technology and an emotion engine to detect illegal activity based on video footage taken by users, compares the results with the law, and provides feedback. The system includes a video capture device, a server, a legal database, an emotion engine, and a communication network connecting them.

[0622] Video acquisition and transmission

[0623] Users use their smartphones or tablets to capture specific scenes, and the captured data is recorded as video or still images. The device then compresses the captured video and sends it to a server via the Internet.

[0624] Video analysis

[0625] The server stores the video data received from the device in a database. The stored data is then analyzed using AI technology. Specifically, machine learning models and deep learning are used to identify actions within the video (e.g., illegal dumping, illegal parking, smoking, etc.).

[0626] Legal Compliance

[0627] The analyzed behavior is compared with a legal database on the server. The legal database stores various laws and ordinances, and determines whether they are illegal. For example, if smoking is detected in a no-smoking area, the system determines whether the behavior violates the local government's smoking ban ordinance.

[0628] Combining Emotion Engines

[0629] This system incorporates an emotion engine and has been added with the ability to recognize the user's emotional state. The device collects emotional data from the user's voice, facial expressions, text input, etc., and the server analyzes this data using the emotion engine. Specifically, if the user is in a high stress state, feedback reflecting that state is provided.

[0630] Generate and send feedback

[0631] If illegality is found, the server generates a feedback message to reflect the result. The tone of the message and recommended actions change depending on the user's emotional state. For example, if the user is in a state of high stress, a more reassuring message (such as "Don't push yourself, take appropriate action") will be generated and sent to the device.

[0632] Reporting and legal consultation support

[0633] Users can report or seek legal advice based on the feedback messages displayed on their device. For example, by pressing the report button within the app, they can directly report to the relevant authorities (police station or local government). In addition, based on the analysis results of the emotion engine, the app will provide recommended actions to reduce psychological stress and the contact information for lawyers for legal advice.

[0634] Specific examples

[0635] 1. The user takes a picture of someone throwing away trash in a park and says, "What should I do about this?"

[0636] 2. The device compresses the video and audio data and sends it to the server.

[0637] 3. The server analyzes the video to detect illegal dumping, and simultaneously analyzes the audio data with an emotion engine to recognize high stress levels.

[0638] 4. The server determines that the act constitutes illegal dumping based on local government ordinances.

[0639] 5. The server generates feedback stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not force yourself and take appropriate action," and sends it to the device.

[0640] 6. The user reports the message.

[0641] This allows users to instantly identify nuisance or illegal behavior and take appropriate action, and by utilizing the emotion engine, they can deal with the situation while reducing the mental burden.

[0642] The processing flow will be explained below.

[0643] Step 1:

[0644] The user launches the app and switches to camera mode by tapping the app icon on their smartphone or tablet and then pressing the camera button on the screen.

[0645] Step 2:

[0646] The user captures a specific scene, for example, someone throwing out trash, and the camera records the footage within its coverage area.

[0647] Step 3:

[0648] The device also records the user's voice and facial expressions. For example, if the user says, "What should I do about this?", that voice data will also be recorded.

[0649] Step 4:

[0650] The device temporarily stores the video and audio data it captures. The video and audio data are stored in the device's memory.

[0651] Step 5:

[0652] The device compresses the captured video and audio data and prepares it for transmission to the server, converting the captured data into an optimized format and establishing an Internet connection.

[0653] Step 6:

[0654] The device sends the video and audio data to the server, where it is encrypted using the HTTPS protocol and securely uploaded to the server.

[0655] Step 7:

[0656] The server receives the video and audio data sent from the terminal, and saves the received data in a specified directory on the server.

[0657] Step 8:

[0658] The server analyzes the received video, specifically using machine learning models and deep learning algorithms to detect specific behaviors within the video, in this case illegal dumping of trash.

[0659] Step 9:

[0660] The server analyzes the voice and facial expression data it receives using an emotion engine, specifically recognizing emotional states such as high stress or anger.

[0661] Step 10:

[0662] The server compares the video analysis results with a legal database, which stores various laws and regulations, and determines whether the video is illegal based on the analysis results.

[0663] Step 11:

[0664] If the server determines that something is illegal, it generates a feedback message that reflects the results of the emotion engine. For example, if a high stress state is recognized, a message such as "This behavior violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not push yourself and take appropriate action" will be created.

[0665] Step 12:

[0666] The server sends the generated feedback message to the terminal, specifically, sends the generated message back to the user's terminal for display.

[0667] Step 13:

[0668] The device receives the feedback message and displays it to the user as a notification, which can be a pop-up notification or an in-app message.

[0669] Step 14:

[0670] Based on the feedback messages received, users can report or seek legal advice. Specifically, by tapping the "Report" or "Legal Advice" button within the app, an interface will appear for contacting the relevant authorities.

[0671] This allows users to quickly identify illegal activity occurring in front of them and take appropriate action, while also reducing the mental burden by utilizing the emotion engine.

[0672] Example 2

[0673] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0674] Conventional illegal activity detection systems have the problem of low accuracy in analyzing video data, which makes it difficult to quickly determine illegal activity. In addition, users often feel emotional stress when reporting illegal activity, and appropriate feedback may not be provided. This makes it difficult for users to take appropriate action.

[0675] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0676] In this invention, the server includes a means for storing and analyzing data in a database, a means for classifying the analyzed results as specific acts using a machine learning model, and a means for comparing the results with a legal database to determine illegality. This enables highly accurate analysis of video data and rapid determination of illegality. Furthermore, the server includes a means for generating feedback based on the user's emotional data, which can provide appropriate feedback suited to the user's emotional state, reducing stress for the user and encouraging appropriate responses.

[0677] "Means for acquiring images" refers to the ability of a user to capture a specific scene using a device such as a smartphone or tablet.

[0678] "Means for the terminal to compress the captured video and send it to the server" is a function that compresses the captured video data and transfers it to the server via the Internet.

[0679] The "means for storing in a database and analyzing" is a mechanism for storing the video data received by the server and later analyzing the data.

[0680] "Means of using a machine learning model to classify the analyzed results as specific actions" refers to a system that uses machine learning technology to recognize and classify specific actions (e.g., illegal dumping, illegal parking, etc.) from analyzed video data.

[0681] "Means for comparing with a legal database and determining illegality" is a function that compares the analyzed act with the laws and ordinances in the legal database and determines whether the act is illegal.

[0682] "Means for collecting user emotional data" refers to devices and software for collecting data such as the user's voice, facial expressions, and text input.

[0683] The "means for analyzing emotional data" is a system that analyzes collected emotional data using an emotional engine and recognizes the user's emotional state (e.g., high stress level).

[0684] "Means for generating feedback and sending it to the device" refers to a function that generates a feedback message based on the user's emotional state when illegality is recognized, and sends and displays that message on the device.

[0685] "Means for reporting or seeking legal advice" means a system that allows users to report to relevant authorities based on feedback messages or provides contact information for assistance in seeking legal advice.

[0686] This invention is a system that uses AI technology and an emotion engine to detect illegal activity based on video footage taken by users, compares the results with a legal database, and provides feedback.The system is primarily composed of devices such as smartphones and tablets, a server that processes data, a legal database, an emotion engine, and a communication network that connects them.

[0687] Video acquisition and transmission

[0688] Users use their smartphones or tablets to capture specific scenes. The captured data is recorded as video or still images. The device then compresses the video data using algorithms such as the H.264 compression algorithm and sends it to a server via the Internet. For example, if a user captures someone throwing away trash in a park, the video is compressed on the device and sent to the server.

[0689] Video analysis

[0690] The server stores the video data received from the device in a dedicated database (e.g., MongoDB). The server then analyzes the stored video data using a deep learning library such as TensorFlow. This analysis allows it to recognize specific behaviors (e.g., illegal dumping, illegal parking, etc.). For example, the received video data can be analyzed using a deep learning model to identify illegal dumping of garbage in a park.

[0691] Legal Compliance

[0692] The analyzed actions are compared with a legal database (e.g., PostgreSQL) on the server. The legal database stores various laws and ordinances, and determines whether a particular action is illegal. For example, it determines whether illegal dumping of garbage violates local government ordinances.

[0693] Combining Emotion Engines

[0694] This system is equipped with an emotion engine, and the device collects emotional data from the user's voice, facial expressions, text input, etc. The server analyzes this emotional data using an emotion engine (e.g., Microsoft's Emotion API) to recognize the user's emotional state (e.g., high stress). For example, analyzing voice data in which the user mutters, "What should I do about this?", the system recognizes that the user is in a high stress state.

[0695] Generate and send feedback

[0696] If illegality is found, the server generates a feedback message to that effect. The tone of the message and recommended actions change based on the user's emotional state. For example, if the user is in a state of high stress, a reassuring message will be generated and sent to the device. For example, a message such as, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please take appropriate action without forcing yourself" will be generated.

[0697] Reporting and legal consultation support

[0698] Based on the feedback messages displayed on the device, users can report the incident or seek legal advice. For example, by pressing the report button within the app, users can directly report the incident to the relevant authority (e.g., the local government's environmental protection department). In addition, based on the analysis results of the emotion engine, recommended actions to reduce psychological stress and contact information for legal advice are provided. For example, users can press the report button to report the incident to the police station.

[0699] Prompt Sentence Examples

[0700] An example of a prompt that can be input to a generative AI model is, "Detect illegal dumping of garbage in a park and generate a feedback message based on the user's emotional state."

[0701] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0702] Step 1:

[0703] A user uses a smartphone or tablet to capture a specific scene. The input is the captured video data (video or still images). A specific operation is for the user to capture a scene of someone throwing away trash in a park. The output is the video data stored on the device.

[0704] Step 2:

[0705] The device compresses the captured video data and sends it to the server via the Internet. The input is the video data generated in step 1. Specifically, the captured data is compressed using the H.264 compression algorithm (for example, reducing a 100MB video to 5MB). The output is the compressed video data being sent to the server.

[0706] Step 3:

[0707] The server stores compressed video data received from the device in a database. The input is compressed video data. Specifically, the video data is stored in a database such as MongoDB. The output is video data stored in the database.

[0708] Step 4:

[0709] The server uses a deep learning model (e.g., TensorFlow) to analyze the video data. The input is the video data stored in a database. Specifically, the video data is input into the deep learning model to detect specific behaviors, such as illegal dumping of garbage. The output is specific behavior data as the analysis result.

[0710] Step 5:

[0711] The server compares the analysis results with a legal database to determine whether the act is illegal. The input is specific act data. Specifically, it accesses a legal database such as PostgreSQL and compares the analysis results with various laws and ordinances. The output is a determination of whether the act is illegal.

[0712] Step 6:

[0713] The device collects emotion data from the user's voice, facial expressions, text input, etc. The input is the user's emotion-related data (voice, facial expressions, text). Specifically, it acquires voice data in which the user mutters, "What should I do about this?" The output is emotion data.

[0714] Step 7:

[0715] The server uses an emotion engine (e.g., Microsoft's Emotion API) to analyze emotion data. The input is the collected emotion data. Specific operations include inputting data into the emotion engine and recognizing the user's emotional state (e.g., high stress). The output is the emotional state data resulting from the analysis.

[0716] Step 8:

[0717] The server generates a feedback message based on the illegality determination result and the user's emotional state data. The input is the illegality determination result and the emotional state data. Specific operations include appropriately adjusting the tone and content of the feedback message. A message is generated stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not force yourself and take appropriate action." The output is the generated feedback message.

[0718] Step 9:

[0719] The server sends the generated feedback message to the terminal. The input is the feedback message generated in step 8. As a specific operation, the message is sent to the terminal via the Internet. The output is the feedback message displayed on the terminal.

[0720] Step 10:

[0721] The user can report or seek legal advice based on the feedback message. The input is the feedback message displayed on the terminal. Specific actions include the user pressing the report button to report to the police station or the local government's environmental protection department. The output is the completion of the report to the relevant agency or the contact for legal advice.

[0722] (Application example 2)

[0723] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0724] Previously, systems existed to detect illegal and nuisance behavior, but they simply detected and reported illegal behavior without taking into account the user's emotional state. This made it difficult for them to take appropriate action when users were experiencing high levels of stress or anxiety. Furthermore, they lacked the accuracy of recognizing illegal behavior and the ability to provide appropriate feedback in real time. As a result, users were unsure of how to act, leading to a problem of low enforcement rates.

[0725] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video, means for comparing the analysis results with a legal database to determine illegality, and means for recognizing the user's emotional state and generating an appropriate feedback message. This makes it possible to quickly detect illegal acts from video captured by the user and provide appropriate feedback based on the emotional analysis without burdening the user. The user can take appropriate action based on the feedback message, and can report the act or seek legal advice in real time. This realizes a quick and effective response to illegal acts while reducing the mental burden on the user.

[0726] "Means for acquiring video footage" refers to a function that allows users to record videos and images using devices such as smartphones and cameras.

[0727] The "means for transmitting to a server" is a function for transmitting the acquired video data to a server via the Internet or other communication network.

[0728] "Means for analyzing video" refers to a function that utilizes machine learning models and deep learning technology to analyze received video data and identify specific actions and objects within it.

[0729] The "means of comparing with a legal database to determine illegality" is a function that compares the content of the analyzed video data with pre-registered legal and ordinance data to determine whether it violates laws and regulations.

[0730] "Means for recognizing the user's emotional state" refers to a function that analyzes the user's voice, facial expression, or text input to measure their current emotional state and stress level.

[0731] The "means for generating appropriate feedback messages" is a function for creating messages that reassure users or encourage them to take appropriate action based on the recognized emotional state of the user.

[0732] The "means for sending a feedback message to a user terminal" is a function for sending the generated feedback message to a user terminal such as a smartphone or tablet.

[0733] An "emotion engine" is a technology that includes algorithms and models for analyzing a user's emotions from voice and text data and identifying their emotional state.

[0734] A "machine learning model" is a collection of algorithms that learn from data and recognize patterns to make predictions or classifications for a specific task.

[0735] "Real-time" refers to a process in which data is processed and analyzed within a very short time after it is acquired, and the results are fed back immediately.

[0736] The present invention is a system that analyzes videos taken by users, detects illegal activities using AI technology and an emotion engine, compares the results with the law, and provides feedback. Specific embodiments of the system are described in detail below.

[0737] System Configuration

[0738] This system mainly consists of a user terminal, a server, a legal database, an emotion engine, and a communication network that connects them.

[0739] 1. User Device

[0740] Users use devices such as smartphones or cameras to capture images of the target. The captured image data is generated using the device's camera function. The emotion engine also collects the user's voice and text inputs as needed.

[0741] 2. Server

[0742] The video data captured by the device is sent to a server via a communication network. The server has the following functions:

[0743] 1. Video analysis: Video data is received and analyzed using machine learning models and deep learning techniques to identify actions and objects in the video. This process uses libraries such as TensorFlow and OpenCV.

[0744] 2. Legal matching: The analyzed behavior is compared with a legal database containing various laws and regulations to determine whether it constitutes an illegal act.

[0745] 3. Emotion analysis: Analyze the user's voice and text data using an emotion engine (e.g., TextBlob) to recognize the user's emotional state.

[0746] 4. Feedback Generation: If illegality is detected, an appropriate feedback message is generated taking into account the user's emotional state. This message is sent to the device.

[0747] 3. Communication Network

[0748] Communication between the user device and the server is carried out via a communication network such as the Internet, and video data, analysis results, and feedback messages are sent and received in a timely manner.

[0749] Specific examples

[0750] 1. A user spots someone illegally littering in a park and takes a photo of the incident with their smartphone.

[0751] 2. When taking a photo, the user mutters out loud, "What should I do with this?"

[0752] 3. The smartphone compresses the video and audio data and sends it to the server.

[0753] 4. The server analyzes the video to detect illegal dumping, and simultaneously analyzes the audio data with an emotion engine to recognize high stress levels.

[0754] 5. The server checks the legal database and determines that the illegal dumping violates Article 16 of the Waste Disposal and Public Cleansing Act.

[0755] 6. The server generates a feedback message stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not force yourself and take appropriate action," and sends it to the user's device.

[0756] 7. The user checks the message and reports it to local authorities using the reporting function within the app.

[0757] Example prompts for generative AI models

[0758] "A user's smartphone was used to record the act of littering in a park. Based on the video, AI technology was used to detect illegal dumping, and an emotion engine was used to identify the user's high stress level. The system then generated a feedback message saying, 'Don't push yourself, take appropriate action,' and reported the incident to local authorities."

[0759] In this way, the present invention makes it possible to deal with illegal activities quickly and appropriately while reducing the burden on users.

[0760] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0761] Step 1:

[0762] The user shoots video and inputs audio using a smartphone.

[0763] Input: Video and audio data obtained from the smartphone camera and microphone.

[0764] Output: Recorded video files (video or still images) and audio files.

[0765] Specific operation: The smartphone camera records the target's actions, and the microphone simultaneously records the user's voice.

[0766] Step 2:

[0767] The video and audio data captured by the device is compressed and sent to a server via the Internet.

[0768] Input: Recorded video and audio files.

[0769] Output: Compressed video and audio data.

[0770] Specific operation: Video and audio files are reduced in size using a compression algorithm and sent to the server's API endpoint.

[0771] Step 3:

[0772] The server analyzes the received video data and identifies the actions in the video.

[0773] Input: Compressed video data.

[0774] Output: Analysis results (identification of actions).

[0775] How it works: The server decodes the video data and uses machine learning models (e.g., TensorFlow) to analyze the actions in the video, such as identifying illegal dumping and illegal parking.

[0776] Step 4:

[0777] The server compares the analysis results with a legal database to determine whether they are illegal.

[0778] Input: Analysis results (identification of actions).

[0779] Output: Illegality determination result.

[0780] Specific operation: The analysis results are compared with ordinances and laws registered in a legal database to determine whether the behavior is illegal. For example, if smoking is detected in a local government no-smoking zone, the behavior will be determined to be a violation of the ordinance.

[0781] Step 5:

[0782] The server analyzes the received voice data using an emotion engine to recognize the user's emotional state.

[0783] Input: Audio data.

[0784] Output: Emotional state judgment result.

[0785] What it does: Analyzes voice data and recognizes the user's emotional state. It uses a sentiment analysis model (e.g., TextBlob) to determine emotions (e.g., high stress, anger, relief, etc.) from the user's voice.

[0786] Step 6:

[0787] The server generates an appropriate feedback message based on the results of the illegal act determination and the emotional state determination.

[0788] Input: Illegality judgment results and emotional state judgment results.

[0789] Output: Feedback message.

[0790] Specific operation: The results of the illegality judgment and the emotional state judgment are combined to generate a feedback message for the user. For example, if illegal dumping is detected and the user is in a high stress state, a message such as "This behavior is illegal. Please do not push yourself and take appropriate action" is generated.

[0791] Step 7:

[0792] The server sends the generated feedback message to the terminal.

[0793] Input: Feedback message.

[0794] Output: Messages that are displayed on the user's terminal.

[0795] Specific operation: The generated feedback message is sent to the user's device and displayed on the screen, allowing the user to check the message.

[0796] Step 8:

[0797] The user reports or seeks legal advice based on the feedback message.

[0798] Input: Feedback message.

[0799] Output: Report or provide legal advice.

[0800] Specific actions: The user checks the feedback message displayed on the device and, if necessary, reports the issue to the appropriate authorities using the in-app reporting function, or contacts a lawyer or other professional using the legal consultation function.

[0801] By taking these steps, the system of the present invention can detect illegal activity from video footage taken by users, provide appropriate feedback, and take prompt action.

[0802] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0803] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0804] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0805] [Third embodiment]

[0806] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0807] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0808] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0809] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0810] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0811] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0812] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0813] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0814] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0815] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0816] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0817] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0818] This system uses AI technology to detect illegal activity based on video footage taken by users, compares the results with the law, and provides feedback. The system includes a video capture device, a server, a legal database, and a communication network connecting them.

[0819] Video acquisition and transmission

[0820] Users use their smartphones or tablets to capture specific scenes, and the captured data is recorded as video or still images. The device then compresses the captured video and sends it to a server via the Internet.

[0821] Video analysis

[0822] The server stores the video data received from the device in a database. The stored data is then analyzed using AI technology. Specifically, machine learning models and deep learning are used to identify actions within the video (e.g., illegal dumping, illegal parking, smoking, etc.).

[0823] Legal Compliance

[0824] The analyzed behavior is compared with a legal database on the server. The legal database stores various laws and ordinances, and determines whether they are illegal. For example, if smoking is detected in a no-smoking area, the system determines whether the behavior violates the local government's smoking ban ordinance.

[0825] Generate and send feedback

[0826] If the action is found to be illegal, the server generates a feedback message containing the relevant legal provisions and ordinances and sends it to the device. For example, a specific message such as "This action violates Article 5 of the smoking ban ordinance of X City" may be displayed.

[0827] Reporting and legal consultation support

[0828] Users can report or seek legal advice based on the feedback messages displayed on their device. For example, by pressing the report button within the app, they can report directly to the relevant authorities (police station or local government). Furthermore, if necessary, the app will provide contact information for lawyers and free legal advice.

[0829] Specific examples

[0830] 1. A user takes a photo of someone littering in a park.

[0831] 2. The device compresses the video and sends it to the server.

[0832] 3. The server analyzes the video and detects illegal dumping of garbage.

[0833] 4. The server determines that the act constitutes illegal dumping based on local government ordinances.

[0834] 5. The server creates feedback stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act," and sends it to the device.

[0835] 6. The user reports the message.

[0836] This will enable users to immediately identify nuisance or illegal behavior and take appropriate action, contributing to improving public safety in the local area.

[0837] This system can solve the problems that traditional reporting systems have by providing real-time analysis and notification, as well as support for reporting and legal consultation.

[0838] The processing flow will be explained below.

[0839] Step 1:

[0840] The user launches the app and switches to camera mode by tapping the app icon to open it and then pressing the camera button on the screen to switch to camera mode.

[0841] Step 2:

[0842] When a user captures a specific scene, for example, recording someone throwing trash in a park, the camera will record footage within its coverage area.

[0843] Step 3:

[0844] The device temporarily stores the video it has taken. The video data is stored in the device's memory.

[0845] Step 4:

[0846] The device compresses the captured video and prepares it for transmission to the server, converting the video data into an optimized format and establishing an Internet connection.

[0847] Step 5:

[0848] The device sends the video data to the server. Specifically, the data is encrypted using the HTTPS protocol and securely uploaded to the server.

[0849] Step 6:

[0850] The server receives the video data sent from the device and saves it in a specified directory on the server.

[0851] Step 7:

[0852] The server analyzes the received video, specifically using machine learning models and deep learning algorithms to detect specific behaviors within the video, in this case illegal dumping of trash.

[0853] Step 8:

[0854] The server compares the analysis results with a legal database, which stores various laws and regulations, and determines whether the information is illegal based on the analysis results.

[0855] Step 9:

[0856] If the server determines that the action is illegal, it generates a feedback message indicating the result, such as "This action violates Article 16 of the Waste Disposal and Public Cleansing Act."

[0857] Step 10:

[0858] The server sends a feedback message to the terminal, specifically, a generated message sent back to the user's terminal for display.

[0859] Step 11:

[0860] The device receives the feedback message and displays it to the user as a notification, which can be a pop-up notification or an in-app message.

[0861] Step 12:

[0862] Based on the feedback messages received, users can report or seek legal advice. Specifically, by tapping the "Report" or "Legal Advice" button within the app, an interface will appear for contacting the relevant authorities.

[0863] By following these steps, users can quickly identify any illegal activity occurring in front of them and take appropriate action.

[0864] Example 1

[0865] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0866] In modern society, there is a need for real-time identification and response to illegal and nuisance behavior. Conventional reporting systems require users to manually report illegal behavior, making it difficult to respond quickly and accurately. In addition, determining illegality takes time, which makes it difficult to respond quickly even when illegal behavior occurs.

[0867] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0868] In this invention, the server includes a means for a user to acquire video, a means for a terminal to compress the acquired video and transmit it to the server, and a means for the server to store the video received. This allows video captured by a user to be quickly and efficiently transmitted to the server and stored. The server also includes a means for analyzing the stored video using a machine learning model, a means for identifying the analyzed behavior and comparing it with a legal database to determine illegality, and a means for generating a feedback message and transmitting it to the terminal if illegality is determined. This makes it possible to quickly determine illegality based on the analysis results and provide feedback to the user, thereby realizing a rapid response to illegal or nuisance behavior.

[0869] "User" means any person or entity that uses the system to obtain footage and detect or report illegal activity.

[0870] A "terminal" is a hardware device used by a user to capture video, and includes devices such as smartphones and tablets.

[0871] A "server" is a computer system that receives, stores, and analyzes video data sent from a terminal.

[0872] "Video data" refers to digital information such as videos and still images taken by a user using a device.

[0873] "Compression" is a technical process for reducing the file size of video data and is a means of making data transfer more efficient.

[0874] A "machine learning model" is a collection of algorithms used for data analysis and pattern recognition, using artificial intelligence techniques to extract features in video.

[0875] A "legal database" is a database that stores various laws and ordinances in digital format and contains standard information for determining whether an analyzed act is illegal.

[0876] A "feedback message" is a notification generated based on the results of the analysis, and is a message that provides users with a determination of illegality and information on related legal provisions.

[0877] "Real-time" refers to near-instant processing, meaning results are returned within a short time of a user's action.

[0878] "Reporting" refers to the act of a user notifying the relevant authorities of illegal activity that they have discovered, and is done through the functions within the application.

[0879] The present invention is a system that uses AI technology to detect illegal activity based on video footage taken by users, compares the results with the law, and provides feedback. This system includes a video capture device, a server, a legal database, and a communication network connecting them. A specific embodiment of this system is described below.

[0880] Users use a camera device such as a smartphone or tablet to capture a specific scene. For example, they can capture a video of someone throwing away trash in a park. The device records the video data in high quality. Still images can also be captured as needed.

[0881] The device then compresses the captured video data, for example using software such as FFmpeg to reduce the size of the video. This compression process makes data transfer more efficient, and the compressed video data is sent to a server over the internet using the HTTPS protocol.

[0882] When the server receives video data from a device, it first stores it in a database (for example, cloud storage such as Amazon S3). The stored data is also accompanied by metadata such as the user ID and shooting date and time.

[0883] The server then analyzes the stored video data. For this analysis, it uses machine learning models (e.g., TensorFlow or PyTorch). Specifically, it uses object detection algorithms such as YOLO (You Only Look Once) to detect different objects and actions within the video. For example, it can identify "littering" within the video.

[0884] The analyzed behavior is compared with a legal database (e.g., MySQL or PostgreSQL) on the server. The legal database stores various laws and regulations, and determines which law the analyzed behavior violates. For example, it checks whether the behavior violates a specific local smoking ban.

[0885] If the action is found to be illegal, the server generates a feedback message based on the results, including information about the relevant legal provisions and regulations. For example, the generated text might read, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act."

[0886] The generated feedback message is sent by the server to the device, again using the HTTPS protocol to ensure data security, and the device notifies the user of the received message.

[0887] Based on the feedback message, users can report the incident to the relevant authorities by pressing the report button in the app. For example, there is a function to report directly to local governments or police. If necessary, the app also provides legal advice by displaying contact information for lawyers and free consultation services.

[0888] Examples and prompts

[0889] Example 1:

[0890] A user takes a photo of someone throwing trash in a park.

[0891] The device compresses the video and sends it to the server.

[0892] The server analyzes the video and detects illegal dumping of garbage.

[0893] The server determines that the act constitutes illegal dumping based on the local government ordinance.

[0894] The server creates a feedback message saying "This action violates Article 16 of the Waste Disposal and Public Cleansing Act" and sends it to the terminal.

[0895] Based on the message, the user reports it to the appropriate authorities.

[0896] Example prompt sentence:

[0897] A user filmed illegal dumping in a park. The video was sent from the device to the server, which analyzed the footage and identified the illegal dumping. Based on the results, the server checked the relevant legal database to confirm its illegality. A feedback message was then generated and sent to the device. The user received the message and reported it to the relevant authorities.

[0898] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0899] Step 1:

[0900] The user acquires the video

[0901] Input: The user captures a specific scene as a video or still image using a device (smartphone or tablet).

[0902] Processing: The user uses a camera application, presses the camera button, and records the subject's actions. The data is then saved in a high-quality format.

[0903] Output: Video files (video or still images) saved on the device.

[0904] Step 2:

[0905] The device compresses the video data and sends it to the server.

[0906] Input: User-shot video files.

[0907] Processing: The device compresses the video data using video compression software such as FFmpeg. This reduces the data size and makes transmission more efficient. After compression, the data is sent to the server using the HTTPS protocol.

[0908] Output: Compressed video data sent to the server.

[0909] Step 3:

[0910] The server receives and stores the video data.

[0911] Input: Compressed video data sent from the terminal.

[0912] Processing: The server stores the received video data in a cloud storage service such as Amazon S3. When storing the data, metadata such as the user ID, shooting date and time, and location information is also stored.

[0913] Output: Compressed video data and its metadata stored in cloud storage.

[0914] Step 4:

[0915] The server analyzes the video data using a machine learning model

[0916] Input: Video data stored in cloud storage.

[0917] Processing: The server uses machine learning libraries such as TensorFlow and PyTorch to apply the YOLO algorithm to recognize objects and actions in the video. For example, it identifies "littering" and "smoking" in the video.

[0918] Output: JSON formatted analysis result data containing the analyzed actions.

[0919] Step 5:

[0920] The server compares the analysis results with a legal database to determine whether they are illegal.

[0921] Input: Analysis result data generated by the machine learning model and legal database.

[0922] Processing: The server retrieves legal information from a MySQL or PostgreSQL database and checks whether the resulting behavior violates specific laws or regulations. For example, does it violate local smoking bans?

[0923] Output: Data containing the result of the illegality determination.

[0924] Step 6:

[0925] The server generates a feedback message

[0926] Input: Illegality determination result data.

[0927] Processing: If the server determines that the action is illegal, it generates a feedback message based on the relevant legal provisions and ordinances. For example, it creates a message stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act."

[0928] Output: The generated feedback message.

[0929] Step 7:

[0930] The server sends a feedback message to the device.

[0931] Input: The generated feedback message.

[0932] Processing: The server sends a feedback message to the terminal using the HTTPS protocol. The terminal notifies the user of the received message.

[0933] Output: The feedback message displayed on the user's terminal.

[0934] Step 8:

[0935] Users can report and seek legal advice based on feedback messages

[0936] Input: The feedback message displayed on the terminal.

[0937] Action: The user can report the incident to the appropriate authorities by pressing the report button within the app, and if necessary, contact a lawyer or legal advice center using the links and contact information in the feedback message.

[0938] Output: The relevant authorities have been notified or legal advice has been provided.

[0939] (Application example 1)

[0940] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0941] Current security systems have the challenge of detecting illegal or suspicious behavior in real time and notifying users immediately. They also lack the feedback needed to respond appropriately to the situation they encounter. As a result, local safety management may not be adequately managed, making real-time responses necessary.

[0942] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0943] In this invention, the server includes means for acquiring video, means for transmitting the acquired video to the server, means for analyzing the video received by the server, means for comparing the analysis result with a legal database to determine illegality, means for generating feedback and transmitting it to the terminal if illegality is determined, means for capturing video in real time using smart glasses, means for transmitting compressed data via the Internet, and means for displaying feedback to the user on a display, thereby making it possible to detect illegal or suspicious behavior in real time and provide immediate feedback to the user.

[0944] "Means for acquiring video" refers to the means by which a user uses a photographic device to record a specific scene or situation as video data.

[0945] The "means for transmitting the acquired video to the server" refers to a means for transmitting the captured video data via a network to a server for temporary storage or analysis.

[0946] "Means for analyzing video received by the server" refers to a means for analyzing the content of received video data using AI or machine learning technology.

[0947] "Means for comparing the analysis results with a legal database and determining illegality" refers to means for comparing the analyzed actions with a legal database and evaluating whether the actions are legally problematic.

[0948] "Means for generating feedback and sending it to the terminal when illegality is confirmed" refers to means for generating and sending a feedback message to notify the user of the result when illegal activity is confirmed.

[0949] "Means for capturing video in real time using smart glasses" refers to a means for recording video of the outside world in real time using the camera of smart glasses worn by the user.

[0950] "Means for transmitting compressed data via the Internet" refers to a means for compressing video data using highly efficient data compression technology and transferring the data to a server via an Internet line.

[0951] The "means for displaying feedback to the user on a display" refers to a means for visually displaying the analysis results and legal feedback on the display of the smart glasses or terminal used by the user.

[0952] The present invention is a system that uses AI technology to detect illegal activities based on video footage taken by users, compares the results with the law, and provides feedback. The specific configuration and procedures for implementing this system are described in detail below.

[0953] Video acquisition and transmission

[0954] The user wears the smart glasses and captures specific scenes or situations in real time. The smart glasses' built-in camera captures images of the outside world and temporarily stores them in the glasses' internal memory. The stored video data is then compressed using highly efficient data compression technology such as H.264 and sent to a server via the Internet.

[0955] Video analysis

[0956] The server stores the video data received from the smart glasses in a database. The stored video data is then analyzed using deep learning frameworks such as TensorFlow and PyTorch. A pre-trained machine learning model identifies activities in the video (e.g., acts of violence, theft, graffiti, etc.).

[0957] Legal Compliance

[0958] The analyzed behavior is compared with a legal database on the server (specifically, MongoDB or Elasticsearch). The legal database stores various laws and regulations, and the illegality of the analysis results is determined based on this. For example, if smoking in a public place is detected, the system determines whether the behavior violates the local government's smoking ban.

[0959] Generate and send feedback

[0960] If the action is deemed illegal, the server generates a feedback message and displays it on the smart glasses' display. The feedback message includes the relevant legal provisions and ordinances, providing the user with the necessary information immediately. For example, a specific message such as "This action violates Article 5 of the smoking ban ordinance of X City" may be displayed.

[0961] Reporting and legal consultation support

[0962] Users can operate the smart glasses interface to report or seek legal advice based on the feedback message. By pressing the report button within the application, users can directly report to the relevant authorities (police station or local government). Furthermore, contact details for experts and a free consultation service are provided for legal advice, if necessary.

[0963] Specific examples

[0964] 1. A user is wearing smart glasses while walking around town.

[0965] 2. Smart glasses capture scenes of street violence in real time.

[0966] 3. The server analyzes the video and detects violent acts.

[0967] 4. The server checks a legal database and determines that the activity violates criminal law.

[0968] 5. The server generates a feedback message saying "This act violates Article XX of the Criminal Code" and displays it on the smart glasses display.

[0969] 6. The user presses the report button on the smart glasses to call the police.

[0970] Prompt Sentence Examples

[0971] "Detect illegal or suspicious activity in the current footage and match it with applicable legal provisions."

[0972] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0973] Step 1: Capturing video with smart glasses

[0974] What happens: The camera in the smart glasses worn by the user captures images of the outside world in real time.

[0975] Input: Video of the user's field of view (environment video).

[0976] Output: Video data (real-time video frames).

[0977] How it works: The smart glasses' camera captures video frames at 30fps and temporarily stores them in its internal memory.

[0978] Step 2: Compress and transmit video data

[0979] What it does: Captured video data is compressed into a highly efficient format (such as H.264) and sent to a server over the Internet.

[0980] Input: Video data.

[0981] Output: Compressed video data.

[0982] How it works: The encoder inside the smart glasses compresses the video data and sends it to the server via Wi-Fi.

[0983] Step 3: Receiving and storing video data on the server

[0984] Processing details: The server decompresses the compressed video data received from the smart glasses and stores it in a database.

[0985] Input: Compressed video data.

[0986] Output: Decompressed video data.

[0987] What happens: The server's decoder decompresses the data and stores it in a database (e.g., MongoDB).

[0988] Step 4: Analyzing the video data

[0989] What it does: The server analyzes the stored video data using machine learning models (e.g., TensorFlow or PyTorch) to identify actions in the video.

[0990] Input: Decompressed video data.

[0991] Output: Analysis results (behavior classification).

[0992] How it works: An AI model running on a server analyzes video frames to detect suspicious or illegal behavior.

[0993] Step 5: Legal database matching

[0994] Processing details: The analyzed behavior is compared with a legal database to determine whether it is illegal.

[0995] Input: Analysis results.

[0996] Output: Legal decision result.

[0997] Specific operation: The server queries a legal database (e.g., Elasticsearch) with the analysis results and compares them with applicable laws and regulations.

[0998] Step 6: Generate and send a feedback message

[0999] What it does: Generates a feedback message and displays it on the smart glasses display.

[1000] Input: Legal decision result.

[1001] Output: The feedback message.

[1002] Specific operation: The server creates a feedback message citing the appropriate legal provisions for illegal activities and sends it to the smart glasses via the Internet.

[1003] Step 7: User Reports and Legal Advice

[1004] What it does: Users use the smart glasses interface to report or seek legal advice based on their feedback.

[1005] Input: The feedback message.

[1006] Output: Report or request for legal advice.

[1007] How it works: When the user presses the report button on the smart glasses, the device will directly report the incident to the police or local government. If necessary, contact details for legal advice will also be displayed on the screen.

[1008] Through these steps, users can obtain video footage in real time, instantly identify illegal or suspicious activity, and receive appropriate feedback to report or seek legal advice.

[1009] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1010] This system uses AI technology and an emotion engine to detect illegal activity based on video footage taken by users, compares the results with the law, and provides feedback. The system includes a video capture device, a server, a legal database, an emotion engine, and a communication network connecting them.

[1011] Video acquisition and transmission

[1012] Users use their smartphones or tablets to capture specific scenes, and the captured data is recorded as video or still images. The device then compresses the captured video and sends it to a server via the Internet.

[1013] Video analysis

[1014] The server stores the video data received from the device in a database. The stored data is then analyzed using AI technology. Specifically, machine learning models and deep learning are used to identify actions within the video (e.g., illegal dumping, illegal parking, smoking, etc.).

[1015] Legal Compliance

[1016] The analyzed behavior is compared with a legal database on the server. The legal database stores various laws and ordinances, and determines whether they are illegal. For example, if smoking is detected in a no-smoking area, the system determines whether the behavior violates the local government's smoking ban ordinance.

[1017] Combining Emotion Engines

[1018] This system incorporates an emotion engine and has been added with the ability to recognize the user's emotional state. The device collects emotional data from the user's voice, facial expressions, text input, etc., and the server analyzes this data using the emotion engine. Specifically, if the user is in a high stress state, feedback reflecting that state is provided.

[1019] Generate and send feedback

[1020] If illegality is found, the server generates a feedback message to reflect the result. The tone of the message and recommended actions change depending on the user's emotional state. For example, if the user is in a state of high stress, a more reassuring message (such as "Don't push yourself, take appropriate action") will be generated and sent to the device.

[1021] Reporting and legal consultation support

[1022] Users can report or seek legal advice based on the feedback messages displayed on their device. For example, by pressing the report button within the app, they can directly report to the relevant authorities (police station or local government). In addition, based on the analysis results of the emotion engine, the app will provide recommended actions to reduce psychological stress and the contact information for lawyers for legal advice.

[1023] Specific examples

[1024] 1. The user takes a picture of someone throwing away trash in a park and says, "What should I do about this?"

[1025] 2. The device compresses the video and audio data and sends it to the server.

[1026] 3. The server analyzes the video to detect illegal dumping, and simultaneously analyzes the audio data with an emotion engine to recognize high stress levels.

[1027] 4. The server determines that the act constitutes illegal dumping based on local government ordinances.

[1028] 5. The server generates feedback stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not force yourself and take appropriate action," and sends it to the device.

[1029] 6. The user reports the message.

[1030] This allows users to instantly identify nuisance or illegal behavior and take appropriate action, and by utilizing the emotion engine, they can deal with the situation while reducing the mental burden.

[1031] The processing flow will be explained below.

[1032] Step 1:

[1033] The user launches the app and switches to camera mode by tapping the app icon on their smartphone or tablet and then pressing the camera button on the screen.

[1034] Step 2:

[1035] The user captures a specific scene, for example, someone throwing out trash, and the camera records the footage within its coverage area.

[1036] Step 3:

[1037] The device also records the user's voice and facial expressions. For example, if the user says, "What should I do about this?", that voice data will also be recorded.

[1038] Step 4:

[1039] The device temporarily stores the video and audio data it captures. The video and audio data are stored in the device's memory.

[1040] Step 5:

[1041] The device compresses the captured video and audio data and prepares it for transmission to the server, converting the captured data into an optimized format and establishing an Internet connection.

[1042] Step 6:

[1043] The device sends the video and audio data to the server, where it is encrypted using the HTTPS protocol and securely uploaded to the server.

[1044] Step 7:

[1045] The server receives the video and audio data sent from the terminal, and saves the received data in a specified directory on the server.

[1046] Step 8:

[1047] The server analyzes the received video, specifically using machine learning models and deep learning algorithms to detect specific behaviors within the video, in this case illegal dumping of trash.

[1048] Step 9:

[1049] The server analyzes the voice and facial expression data it receives using an emotion engine, specifically recognizing emotional states such as high stress or anger.

[1050] Step 10:

[1051] The server compares the video analysis results with a legal database, which stores various laws and regulations, and determines whether the video is illegal based on the analysis results.

[1052] Step 11:

[1053] If the server determines that something is illegal, it generates a feedback message that reflects the results of the emotion engine. For example, if a high stress state is recognized, a message such as "This behavior violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not push yourself and take appropriate action" will be created.

[1054] Step 12:

[1055] The server sends the generated feedback message to the terminal, specifically, sends the generated message back to the user's terminal for display.

[1056] Step 13:

[1057] The device receives the feedback message and displays it to the user as a notification, which can be a pop-up notification or an in-app message.

[1058] Step 14:

[1059] Based on the feedback messages received, users can report or seek legal advice. Specifically, by tapping the "Report" or "Legal Advice" button within the app, an interface will appear for contacting the relevant authorities.

[1060] This allows users to quickly identify illegal activity occurring in front of them and take appropriate action, while also reducing the mental burden by utilizing the emotion engine.

[1061] Example 2

[1062] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1063] Conventional illegal activity detection systems have the problem of low accuracy in analyzing video data, which makes it difficult to quickly determine illegal activity. In addition, users often feel emotional stress when reporting illegal activity, and appropriate feedback may not be provided. This makes it difficult for users to take appropriate action.

[1064] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1065] In this invention, the server includes a means for storing and analyzing data in a database, a means for classifying the analyzed results as specific acts using a machine learning model, and a means for comparing the results with a legal database to determine illegality. This enables highly accurate analysis of video data and rapid determination of illegality. Furthermore, the server includes a means for generating feedback based on the user's emotional data, which can provide appropriate feedback suited to the user's emotional state, reducing stress for the user and encouraging appropriate responses.

[1066] "Means for acquiring images" refers to the ability of a user to capture a specific scene using a device such as a smartphone or tablet.

[1067] "Means for the terminal to compress the captured video and send it to the server" is a function that compresses the captured video data and transfers it to the server via the Internet.

[1068] The "means for storing in a database and analyzing" is a mechanism for storing the video data received by the server and later analyzing the data.

[1069] "Means of using a machine learning model to classify the analyzed results as specific actions" refers to a system that uses machine learning technology to recognize and classify specific actions (e.g., illegal dumping, illegal parking, etc.) from analyzed video data.

[1070] "Means for comparing with a legal database and determining illegality" is a function that compares the analyzed act with the laws and ordinances in the legal database and determines whether the act is illegal.

[1071] "Means for collecting user emotional data" refers to devices and software for collecting data such as the user's voice, facial expressions, and text input.

[1072] The "means for analyzing emotional data" is a system that analyzes collected emotional data using an emotional engine and recognizes the user's emotional state (e.g., high stress level).

[1073] "Means for generating feedback and sending it to the device" refers to a function that generates a feedback message based on the user's emotional state when illegality is recognized, and sends and displays that message on the device.

[1074] "Means for reporting or seeking legal advice" means a system that allows users to report to relevant authorities based on feedback messages or provides contact information for assistance in seeking legal advice.

[1075] This invention is a system that uses AI technology and an emotion engine to detect illegal activity based on video footage taken by users, compares the results with a legal database, and provides feedback.The system is primarily composed of devices such as smartphones and tablets, a server that processes data, a legal database, an emotion engine, and a communication network that connects them.

[1076] Video acquisition and transmission

[1077] Users use their smartphones or tablets to capture specific scenes. The captured data is recorded as video or still images. The device then compresses the video data using algorithms such as the H.264 compression algorithm and sends it to a server via the Internet. For example, if a user captures someone throwing away trash in a park, the video is compressed on the device and sent to the server.

[1078] Video analysis

[1079] The server stores the video data received from the device in a dedicated database (e.g., MongoDB). The server then analyzes the stored video data using a deep learning library such as TensorFlow. This analysis allows it to recognize specific behaviors (e.g., illegal dumping, illegal parking, etc.). For example, the received video data can be analyzed using a deep learning model to identify illegal dumping of garbage in a park.

[1080] Legal Compliance

[1081] The analyzed actions are compared with a legal database (e.g., PostgreSQL) on the server. The legal database stores various laws and ordinances, and determines whether a particular action is illegal. For example, it determines whether illegal dumping of garbage violates local government ordinances.

[1082] Combining Emotion Engines

[1083] This system is equipped with an emotion engine, and the device collects emotional data from the user's voice, facial expressions, text input, etc. The server analyzes this emotional data using an emotion engine (e.g., Microsoft's Emotion API) to recognize the user's emotional state (e.g., high stress). For example, analyzing voice data in which the user mutters, "What should I do about this?", the system recognizes that the user is in a high stress state.

[1084] Generate and send feedback

[1085] If illegality is found, the server generates a feedback message to that effect. The tone of the message and recommended actions change based on the user's emotional state. For example, if the user is in a state of high stress, a reassuring message will be generated and sent to the device. For example, a message such as, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please take appropriate action without forcing yourself" will be generated.

[1086] Reporting and legal consultation support

[1087] Based on the feedback messages displayed on the device, users can report the incident or seek legal advice. For example, by pressing the report button within the app, users can directly report the incident to the relevant authority (e.g., the local government's environmental protection department). In addition, based on the analysis results of the emotion engine, recommended actions to reduce psychological stress and contact information for legal advice are provided. For example, users can press the report button to report the incident to the police station.

[1088] Prompt Sentence Examples

[1089] An example of a prompt that can be input to a generative AI model is, "Detect illegal dumping of garbage in a park and generate a feedback message based on the user's emotional state."

[1090] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1091] Step 1:

[1092] A user uses a smartphone or tablet to capture a specific scene. The input is the captured video data (video or still images). A specific operation is for the user to capture a scene of someone throwing away trash in a park. The output is the video data stored on the device.

[1093] Step 2:

[1094] The device compresses the captured video data and sends it to the server via the Internet. The input is the video data generated in step 1. Specifically, the captured data is compressed using the H.264 compression algorithm (for example, reducing a 100MB video to 5MB). The output is the compressed video data being sent to the server.

[1095] Step 3:

[1096] The server stores compressed video data received from the device in a database. The input is compressed video data. Specifically, the video data is stored in a database such as MongoDB. The output is video data stored in the database.

[1097] Step 4:

[1098] The server uses a deep learning model (e.g., TensorFlow) to analyze the video data. The input is the video data stored in a database. Specifically, the video data is input into the deep learning model to detect specific behaviors, such as illegal dumping of garbage. The output is specific behavior data as the analysis result.

[1099] Step 5:

[1100] The server compares the analysis results with a legal database to determine whether the act is illegal. The input is specific act data. Specifically, it accesses a legal database such as PostgreSQL and compares the analysis results with various laws and ordinances. The output is a determination of whether the act is illegal.

[1101] Step 6:

[1102] The device collects emotion data from the user's voice, facial expressions, text input, etc. The input is the user's emotion-related data (voice, facial expressions, text). Specifically, it acquires voice data in which the user mutters, "What should I do about this?" The output is emotion data.

[1103] Step 7:

[1104] The server uses an emotion engine (e.g., Microsoft's Emotion API) to analyze emotion data. The input is the collected emotion data. Specific operations include inputting data into the emotion engine and recognizing the user's emotional state (e.g., high stress). The output is the emotional state data resulting from the analysis.

[1105] Step 8:

[1106] The server generates a feedback message based on the illegality determination result and the user's emotional state data. The input is the illegality determination result and the emotional state data. Specific operations include appropriately adjusting the tone and content of the feedback message. A message is generated stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not force yourself and take appropriate action." The output is the generated feedback message.

[1107] Step 9:

[1108] The server sends the generated feedback message to the terminal. The input is the feedback message generated in step 8. As a specific operation, the message is sent to the terminal via the Internet. The output is the feedback message displayed on the terminal.

[1109] Step 10:

[1110] The user can report or seek legal advice based on the feedback message. The input is the feedback message displayed on the terminal. Specific actions include the user pressing the report button to report to the police station or the local government's environmental protection department. The output is the completion of the report to the relevant agency or the contact for legal advice.

[1111] (Application example 2)

[1112] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1113] Previously, systems existed to detect illegal and nuisance behavior, but they simply detected and reported illegal behavior without taking into account the user's emotional state. This made it difficult for them to take appropriate action when users were experiencing high levels of stress or anxiety. Furthermore, they lacked the accuracy of recognizing illegal behavior and the ability to provide appropriate feedback in real time. As a result, users were unsure of how to act, leading to a problem of low enforcement rates.

[1114] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video, means for comparing the analysis results with a legal database to determine illegality, and means for recognizing the user's emotional state and generating an appropriate feedback message. This makes it possible to quickly detect illegal acts from video captured by the user and provide appropriate feedback based on the emotional analysis without burdening the user. The user can take appropriate action based on the feedback message, and can report the act or seek legal advice in real time. This realizes a quick and effective response to illegal acts while reducing the mental burden on the user.

[1115] "Means for acquiring video footage" refers to a function that allows users to record videos and images using devices such as smartphones and cameras.

[1116] The "means for transmitting to a server" is a function for transmitting the acquired video data to a server via the Internet or other communication network.

[1117] "Means for analyzing video" refers to a function that utilizes machine learning models and deep learning technology to analyze received video data and identify specific actions and objects within it.

[1118] The "means of comparing with a legal database to determine illegality" is a function that compares the content of the analyzed video data with pre-registered legal and ordinance data to determine whether it violates laws and regulations.

[1119] "Means for recognizing the user's emotional state" refers to a function that analyzes the user's voice, facial expression, or text input to measure their current emotional state and stress level.

[1120] The "means for generating appropriate feedback messages" is a function for creating messages that reassure users or encourage them to take appropriate action based on the recognized emotional state of the user.

[1121] The "means for sending a feedback message to a user terminal" is a function for sending the generated feedback message to a user terminal such as a smartphone or tablet.

[1122] An "emotion engine" is a technology that includes algorithms and models for analyzing a user's emotions from voice and text data and identifying their emotional state.

[1123] A "machine learning model" is a collection of algorithms that learn from data and recognize patterns to make predictions or classifications for a specific task.

[1124] "Real-time" refers to a process in which data is processed and analyzed within a very short time after it is acquired, and the results are fed back immediately.

[1125] The present invention is a system that analyzes videos taken by users, detects illegal activities using AI technology and an emotion engine, compares the results with the law, and provides feedback. Specific embodiments of the system are described in detail below.

[1126] System Configuration

[1127] This system mainly consists of a user terminal, a server, a legal database, an emotion engine, and a communication network that connects them.

[1128] 1. User Device

[1129] Users use devices such as smartphones or cameras to capture images of the target. The captured image data is generated using the device's camera function. The emotion engine also collects the user's voice and text inputs as needed.

[1130] 2. Server

[1131] The video data captured by the device is sent to a server via a communication network. The server has the following functions:

[1132] 1. Video analysis: Video data is received and analyzed using machine learning models and deep learning techniques to identify actions and objects in the video. This process uses libraries such as TensorFlow and OpenCV.

[1133] 2. Legal matching: The analyzed behavior is compared with a legal database containing various laws and regulations to determine whether it constitutes an illegal act.

[1134] 3. Emotion analysis: Analyze the user's voice and text data using an emotion engine (e.g., TextBlob) to recognize the user's emotional state.

[1135] 4. Feedback Generation: If illegality is detected, an appropriate feedback message is generated taking into account the user's emotional state. This message is sent to the device.

[1136] 3. Communication Network

[1137] Communication between the user device and the server is carried out via a communication network such as the Internet, and video data, analysis results, and feedback messages are sent and received in a timely manner.

[1138] Specific examples

[1139] 1. A user spots someone illegally littering in a park and takes a photo of the incident with their smartphone.

[1140] 2. When taking a photo, the user mutters out loud, "What should I do with this?"

[1141] 3. The smartphone compresses the video and audio data and sends it to the server.

[1142] 4. The server analyzes the video to detect illegal dumping, and simultaneously analyzes the audio data with an emotion engine to recognize high stress levels.

[1143] 5. The server checks the legal database and determines that the illegal dumping violates Article 16 of the Waste Disposal and Public Cleansing Act.

[1144] 6. The server generates a feedback message stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not force yourself and take appropriate action," and sends it to the user's device.

[1145] 7. The user checks the message and reports it to local authorities using the reporting function within the app.

[1146] Example prompts for generative AI models

[1147] "A user's smartphone was used to record the act of littering in a park. Based on the video, AI technology was used to detect illegal dumping, and an emotion engine was used to identify the user's high stress level. The system then generated a feedback message saying, 'Don't push yourself, take appropriate action,' and reported the incident to local authorities."

[1148] In this way, the present invention makes it possible to deal with illegal activities quickly and appropriately while reducing the burden on users.

[1149] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1150] Step 1:

[1151] The user shoots video and inputs audio using a smartphone.

[1152] Input: Video and audio data obtained from the smartphone camera and microphone.

[1153] Output: Recorded video files (video or still images) and audio files.

[1154] Specific operation: The smartphone camera records the target's actions, and the microphone simultaneously records the user's voice.

[1155] Step 2:

[1156] The video and audio data captured by the device is compressed and sent to a server via the Internet.

[1157] Input: Recorded video and audio files.

[1158] Output: Compressed video and audio data.

[1159] Specific operation: Video and audio files are reduced in size using a compression algorithm and sent to the server's API endpoint.

[1160] Step 3:

[1161] The server analyzes the received video data and identifies the actions in the video.

[1162] Input: Compressed video data.

[1163] Output: Analysis results (identification of actions).

[1164] How it works: The server decodes the video data and uses machine learning models (e.g., TensorFlow) to analyze the actions in the video, such as identifying illegal dumping and illegal parking.

[1165] Step 4:

[1166] The server compares the analysis results with a legal database to determine whether they are illegal.

[1167] Input: Analysis results (identification of actions).

[1168] Output: Illegality determination result.

[1169] Specific operation: The analysis results are compared with ordinances and laws registered in a legal database to determine whether the behavior is illegal. For example, if smoking is detected in a local government no-smoking zone, the behavior will be determined to be a violation of the ordinance.

[1170] Step 5:

[1171] The server analyzes the received voice data using an emotion engine to recognize the user's emotional state.

[1172] Input: Audio data.

[1173] Output: Emotional state judgment result.

[1174] What it does: Analyzes voice data and recognizes the user's emotional state. It uses a sentiment analysis model (e.g., TextBlob) to determine emotions (e.g., high stress, anger, relief, etc.) from the user's voice.

[1175] Step 6:

[1176] The server generates an appropriate feedback message based on the results of the illegal act determination and the emotional state determination.

[1177] Input: Illegality judgment results and emotional state judgment results.

[1178] Output: Feedback message.

[1179] Specific operation: The results of the illegality judgment and the emotional state judgment are combined to generate a feedback message for the user. For example, if illegal dumping is detected and the user is in a high stress state, a message such as "This behavior is illegal. Please do not push yourself and take appropriate action" is generated.

[1180] Step 7:

[1181] The server sends the generated feedback message to the terminal.

[1182] Input: Feedback message.

[1183] Output: Messages that are displayed on the user's terminal.

[1184] Specific operation: The generated feedback message is sent to the user's device and displayed on the screen, allowing the user to check the message.

[1185] Step 8:

[1186] The user reports or seeks legal advice based on the feedback message.

[1187] Input: Feedback message.

[1188] Output: Report or provide legal advice.

[1189] Specific actions: The user checks the feedback message displayed on the device and, if necessary, reports the issue to the appropriate authorities using the in-app reporting function, or contacts a lawyer or other professional using the legal consultation function.

[1190] By taking these steps, the system of the present invention can detect illegal activity from video footage taken by users, provide appropriate feedback, and take prompt action.

[1191] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1192] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1193] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1194] [Fourth embodiment]

[1195] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1196] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1197] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1198] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1199] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1200] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1201] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1202] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1203] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1204] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1205] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1206] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1207] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1208] This system uses AI technology to detect illegal activity based on video footage taken by users, compares the results with the law, and provides feedback. The system includes a video capture device, a server, a legal database, and a communication network connecting them.

[1209] Video acquisition and transmission

[1210] Users use their smartphones or tablets to capture specific scenes, and the captured data is recorded as video or still images. The device then compresses the captured video and sends it to a server via the Internet.

[1211] Video analysis

[1212] The server stores the video data received from the device in a database. The stored data is then analyzed using AI technology. Specifically, machine learning models and deep learning are used to identify actions within the video (e.g., illegal dumping, illegal parking, smoking, etc.).

[1213] Legal Compliance

[1214] The analyzed behavior is compared with a legal database on the server. The legal database stores various laws and ordinances, and determines whether they are illegal. For example, if smoking is detected in a no-smoking area, the system determines whether the behavior violates the local government's smoking ban ordinance.

[1215] Generate and send feedback

[1216] If the action is found to be illegal, the server generates a feedback message containing the relevant legal provisions and ordinances and sends it to the device. For example, a specific message such as "This action violates Article 5 of the smoking ban ordinance of X City" may be displayed.

[1217] Reporting and legal consultation support

[1218] Users can report or seek legal advice based on the feedback messages displayed on their device. For example, by pressing the report button within the app, they can report directly to the relevant authorities (police station or local government). Furthermore, if necessary, the app will provide contact information for lawyers and free legal advice.

[1219] Specific examples

[1220] 1. A user takes a photo of someone littering in a park.

[1221] 2. The device compresses the video and sends it to the server.

[1222] 3. The server analyzes the video and detects illegal dumping of garbage.

[1223] 4. The server determines that the act constitutes illegal dumping based on local government ordinances.

[1224] 5. The server creates feedback stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act," and sends it to the device.

[1225] 6. The user reports the message.

[1226] This will enable users to immediately identify nuisance or illegal behavior and take appropriate action, contributing to improving public safety in the local area.

[1227] This system can solve the problems that traditional reporting systems have by providing real-time analysis and notification, as well as support for reporting and legal consultation.

[1228] The processing flow will be explained below.

[1229] Step 1:

[1230] The user launches the app and switches to camera mode by tapping the app icon to open it and then pressing the camera button on the screen to switch to camera mode.

[1231] Step 2:

[1232] When a user captures a specific scene, for example, recording someone throwing trash in a park, the camera will record footage within its coverage area.

[1233] Step 3:

[1234] The device temporarily stores the video it has taken. The video data is stored in the device's memory.

[1235] Step 4:

[1236] The device compresses the captured video and prepares it for transmission to the server, converting the video data into an optimized format and establishing an Internet connection.

[1237] Step 5:

[1238] The device sends the video data to the server. Specifically, the data is encrypted using the HTTPS protocol and securely uploaded to the server.

[1239] Step 6:

[1240] The server receives the video data sent from the device and saves it in a specified directory on the server.

[1241] Step 7:

[1242] The server analyzes the received video, specifically using machine learning models and deep learning algorithms to detect specific behaviors within the video, in this case illegal dumping of trash.

[1243] Step 8:

[1244] The server compares the analysis results with a legal database, which stores various laws and regulations, and determines whether the information is illegal based on the analysis results.

[1245] Step 9:

[1246] If the server determines that the action is illegal, it generates a feedback message indicating the result, such as "This action violates Article 16 of the Waste Disposal and Public Cleansing Act."

[1247] Step 10:

[1248] The server sends a feedback message to the terminal, specifically, a generated message sent back to the user's terminal for display.

[1249] Step 11:

[1250] The device receives the feedback message and displays it to the user as a notification, which can be a pop-up notification or an in-app message.

[1251] Step 12:

[1252] Based on the feedback messages received, users can report or seek legal advice. Specifically, by tapping the "Report" or "Legal Advice" button within the app, an interface will appear for contacting the relevant authorities.

[1253] By following these steps, users can quickly identify any illegal activity occurring in front of them and take appropriate action.

[1254] Example 1

[1255] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1256] In modern society, there is a need for real-time identification and response to illegal and nuisance behavior. Conventional reporting systems require users to manually report illegal behavior, making it difficult to respond quickly and accurately. In addition, determining illegality takes time, which makes it difficult to respond quickly even when illegal behavior occurs.

[1257] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1258] In this invention, the server includes a means for a user to acquire video, a means for a terminal to compress the acquired video and transmit it to the server, and a means for the server to store the video received. This allows video captured by a user to be quickly and efficiently transmitted to the server and stored. The server also includes a means for analyzing the stored video using a machine learning model, a means for identifying the analyzed behavior and comparing it with a legal database to determine illegality, and a means for generating a feedback message and transmitting it to the terminal if illegality is determined. This makes it possible to quickly determine illegality based on the analysis results and provide feedback to the user, thereby realizing a rapid response to illegal or nuisance behavior.

[1259] "User" means any person or entity that uses the system to obtain footage and detect or report illegal activity.

[1260] A "terminal" is a hardware device used by a user to capture video, and includes devices such as smartphones and tablets.

[1261] A "server" is a computer system that receives, stores, and analyzes video data sent from a terminal.

[1262] "Video data" refers to digital information such as videos and still images taken by a user using a device.

[1263] "Compression" is a technical process for reducing the file size of video data and is a means of making data transfer more efficient.

[1264] A "machine learning model" is a collection of algorithms used for data analysis and pattern recognition, using artificial intelligence techniques to extract features in video.

[1265] A "legal database" is a database that stores various laws and ordinances in digital format and contains standard information for determining whether an analyzed act is illegal.

[1266] A "feedback message" is a notification generated based on the results of the analysis, and is a message that provides users with a determination of illegality and information on related legal provisions.

[1267] "Real-time" refers to near-instant processing, meaning results are returned within a short time of a user's action.

[1268] "Reporting" refers to the act of a user notifying the relevant authorities of illegal activity that they have discovered, and is done through the functions within the application.

[1269] The present invention is a system that uses AI technology to detect illegal activity based on video footage taken by users, compares the results with the law, and provides feedback. This system includes a video capture device, a server, a legal database, and a communication network connecting them. A specific embodiment of this system is described below.

[1270] Users use a camera device such as a smartphone or tablet to capture a specific scene. For example, they can capture a video of someone throwing away trash in a park. The device records the video data in high quality. Still images can also be captured as needed.

[1271] The device then compresses the captured video data, for example using software such as FFmpeg to reduce the size of the video. This compression process makes data transfer more efficient, and the compressed video data is sent to a server over the internet using the HTTPS protocol.

[1272] When the server receives video data from a device, it first stores it in a database (for example, cloud storage such as Amazon S3). The stored data is also accompanied by metadata such as the user ID and shooting date and time.

[1273] The server then analyzes the stored video data. For this analysis, it uses machine learning models (e.g., TensorFlow or PyTorch). Specifically, it uses object detection algorithms such as YOLO (You Only Look Once) to detect different objects and actions within the video. For example, it can identify "littering" within the video.

[1274] The analyzed behavior is compared with a legal database (e.g., MySQL or PostgreSQL) on the server. The legal database stores various laws and regulations, and determines which law the analyzed behavior violates. For example, it checks whether the behavior violates a specific local smoking ban.

[1275] If the action is found to be illegal, the server generates a feedback message based on the results, including information about the relevant legal provisions and regulations. For example, the generated text might read, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act."

[1276] The generated feedback message is sent by the server to the device, again using the HTTPS protocol to ensure data security, and the device notifies the user of the received message.

[1277] Based on the feedback message, users can report the incident to the relevant authorities by pressing the report button in the app. For example, there is a function to report directly to local governments or police. If necessary, the app also provides legal advice by displaying contact information for lawyers and free consultation services.

[1278] Examples and prompts

[1279] Example 1:

[1280] A user takes a photo of someone throwing trash in a park.

[1281] The device compresses the video and sends it to the server.

[1282] The server analyzes the video and detects illegal dumping of garbage.

[1283] The server determines that the act constitutes illegal dumping based on the local government ordinance.

[1284] The server creates a feedback message saying "This action violates Article 16 of the Waste Disposal and Public Cleansing Act" and sends it to the terminal.

[1285] Based on the message, the user reports it to the appropriate authorities.

[1286] Example prompt sentence:

[1287] A user filmed illegal dumping in a park. The video was sent from the device to the server, which analyzed the footage and identified the illegal dumping. Based on the results, the server checked the relevant legal database to confirm its illegality. A feedback message was then generated and sent to the device. The user received the message and reported it to the relevant authorities.

[1288] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1289] Step 1:

[1290] The user acquires the video

[1291] Input: The user captures a specific scene as a video or still image using a device (smartphone or tablet).

[1292] Processing: The user uses a camera application, presses the camera button, and records the subject's actions. The data is then saved in a high-quality format.

[1293] Output: Video files (video or still images) saved on the device.

[1294] Step 2:

[1295] The device compresses the video data and sends it to the server.

[1296] Input: User-shot video files.

[1297] Processing: The device compresses the video data using video compression software such as FFmpeg. This reduces the data size and makes transmission more efficient. After compression, the data is sent to the server using the HTTPS protocol.

[1298] Output: Compressed video data sent to the server.

[1299] Step 3:

[1300] The server receives and stores the video data.

[1301] Input: Compressed video data sent from the terminal.

[1302] Processing: The server stores the received video data in a cloud storage service such as Amazon S3. When storing the data, metadata such as the user ID, shooting date and time, and location information is also stored.

[1303] Output: Compressed video data and its metadata stored in cloud storage.

[1304] Step 4:

[1305] The server analyzes the video data using a machine learning model

[1306] Input: Video data stored in cloud storage.

[1307] Processing: The server uses machine learning libraries such as TensorFlow and PyTorch to apply the YOLO algorithm to recognize objects and actions in the video. For example, it identifies "littering" and "smoking" in the video.

[1308] Output: JSON formatted analysis result data containing the analyzed actions.

[1309] Step 5:

[1310] The server compares the analysis results with a legal database to determine whether they are illegal.

[1311] Input: Analysis result data generated by the machine learning model and legal database.

[1312] Processing: The server retrieves legal information from a MySQL or PostgreSQL database and checks whether the resulting behavior violates specific laws or regulations. For example, does it violate local smoking bans?

[1313] Output: Data containing the result of the illegality determination.

[1314] Step 6:

[1315] The server generates a feedback message

[1316] Input: Illegality determination result data.

[1317] Processing: If the server determines that the action is illegal, it generates a feedback message based on the relevant legal provisions and ordinances. For example, it creates a message stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act."

[1318] Output: The generated feedback message.

[1319] Step 7:

[1320] The server sends a feedback message to the device.

[1321] Input: The generated feedback message.

[1322] Processing: The server sends a feedback message to the terminal using the HTTPS protocol. The terminal notifies the user of the received message.

[1323] Output: The feedback message displayed on the user's terminal.

[1324] Step 8:

[1325] Users can report and seek legal advice based on feedback messages

[1326] Input: The feedback message displayed on the terminal.

[1327] Action: The user can report the incident to the appropriate authorities by pressing the report button within the app, and if necessary, contact a lawyer or legal advice center using the links and contact information in the feedback message.

[1328] Output: The relevant authorities have been notified or legal advice has been provided.

[1329] (Application example 1)

[1330] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1331] Current security systems have the challenge of detecting illegal or suspicious behavior in real time and notifying users immediately. They also lack the feedback needed to respond appropriately to the situation they encounter. As a result, local safety management may not be adequately managed, making real-time responses necessary.

[1332] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1333] In this invention, the server includes means for acquiring video, means for transmitting the acquired video to the server, means for analyzing the video received by the server, means for comparing the analysis result with a legal database to determine illegality, means for generating feedback and transmitting it to the terminal if illegality is determined, means for capturing video in real time using smart glasses, means for transmitting compressed data via the Internet, and means for displaying feedback to the user on a display, thereby making it possible to detect illegal or suspicious behavior in real time and provide immediate feedback to the user.

[1334] "Means for acquiring video" refers to the means by which a user uses a photographic device to record a specific scene or situation as video data.

[1335] The "means for transmitting the acquired video to the server" refers to a means for transmitting the captured video data via a network to a server for temporary storage or analysis.

[1336] "Means for analyzing video received by the server" refers to a means for analyzing the content of received video data using AI or machine learning technology.

[1337] "Means for comparing the analysis results with a legal database and determining illegality" refers to means for comparing the analyzed actions with a legal database and evaluating whether the actions are legally problematic.

[1338] "Means for generating feedback and sending it to the terminal when illegality is confirmed" refers to means for generating and sending a feedback message to notify the user of the result when illegal activity is confirmed.

[1339] "Means for capturing video in real time using smart glasses" refers to a means for recording video of the outside world in real time using the camera of smart glasses worn by the user.

[1340] "Means for transmitting compressed data via the Internet" refers to a means for compressing video data using highly efficient data compression technology and transferring the data to a server via an Internet line.

[1341] The "means for displaying feedback to the user on a display" refers to a means for visually displaying the analysis results and legal feedback on the display of the smart glasses or terminal used by the user.

[1342] The present invention is a system that uses AI technology to detect illegal activities based on video footage taken by users, compares the results with the law, and provides feedback. The specific configuration and procedures for implementing this system are described in detail below.

[1343] Video acquisition and transmission

[1344] The user wears the smart glasses and captures specific scenes or situations in real time. The smart glasses' built-in camera captures images of the outside world and temporarily stores them in the glasses' internal memory. The stored video data is then compressed using highly efficient data compression technology such as H.264 and sent to a server via the Internet.

[1345] Video analysis

[1346] The server stores the video data received from the smart glasses in a database. The stored video data is then analyzed using deep learning frameworks such as TensorFlow and PyTorch. A pre-trained machine learning model identifies activities in the video (e.g., acts of violence, theft, graffiti, etc.).

[1347] Legal Compliance

[1348] The analyzed behavior is compared with a legal database on the server (specifically, MongoDB or Elasticsearch). The legal database stores various laws and regulations, and the illegality of the analysis results is determined based on this. For example, if smoking in a public place is detected, the system determines whether the behavior violates the local government's smoking ban.

[1349] Generate and send feedback

[1350] If the action is deemed illegal, the server generates a feedback message and displays it on the smart glasses' display. The feedback message includes the relevant legal provisions and ordinances, providing the user with the necessary information immediately. For example, a specific message such as "This action violates Article 5 of the smoking ban ordinance of X City" may be displayed.

[1351] Reporting and legal consultation support

[1352] Users can operate the smart glasses interface to report or seek legal advice based on the feedback message. By pressing the report button within the application, users can directly report to the relevant authorities (police station or local government). Furthermore, contact details for experts and a free consultation service are provided for legal advice, if necessary.

[1353] Specific examples

[1354] 1. A user is wearing smart glasses while walking around town.

[1355] 2. Smart glasses capture scenes of street violence in real time.

[1356] 3. The server analyzes the video and detects violent acts.

[1357] 4. The server checks a legal database and determines that the activity violates criminal law.

[1358] 5. The server generates a feedback message saying "This act violates Article XX of the Criminal Code" and displays it on the smart glasses display.

[1359] 6. The user presses the report button on the smart glasses to call the police.

[1360] Prompt Sentence Examples

[1361] "Detect illegal or suspicious activity in the current footage and match it with applicable legal provisions."

[1362] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1363] Step 1: Capturing video with smart glasses

[1364] What happens: The camera in the smart glasses worn by the user captures images of the outside world in real time.

[1365] Input: Video of the user's field of view (environment video).

[1366] Output: Video data (real-time video frames).

[1367] How it works: The smart glasses' camera captures video frames at 30fps and temporarily stores them in its internal memory.

[1368] Step 2: Compress and transmit video data

[1369] What it does: Captured video data is compressed into a highly efficient format (such as H.264) and sent to a server over the Internet.

[1370] Input: Video data.

[1371] Output: Compressed video data.

[1372] How it works: The encoder inside the smart glasses compresses the video data and sends it to the server via Wi-Fi.

[1373] Step 3: Receiving and storing video data on the server

[1374] Processing details: The server decompresses the compressed video data received from the smart glasses and stores it in a database.

[1375] Input: Compressed video data.

[1376] Output: Decompressed video data.

[1377] What happens: The server's decoder decompresses the data and stores it in a database (e.g., MongoDB).

[1378] Step 4: Analyzing the video data

[1379] What it does: The server analyzes the stored video data using machine learning models (e.g., TensorFlow or PyTorch) to identify actions in the video.

[1380] Input: Decompressed video data.

[1381] Output: Analysis results (behavior classification).

[1382] How it works: An AI model running on a server analyzes video frames to detect suspicious or illegal behavior.

[1383] Step 5: Legal database matching

[1384] Processing details: The analyzed behavior is compared with a legal database to determine whether it is illegal.

[1385] Input: Analysis results.

[1386] Output: Legal decision result.

[1387] Specific operation: The server queries a legal database (e.g., Elasticsearch) with the analysis results and compares them with applicable laws and regulations.

[1388] Step 6: Generate and send a feedback message

[1389] What it does: Generates a feedback message and displays it on the smart glasses display.

[1390] Input: Legal decision result.

[1391] Output: The feedback message.

[1392] Specific operation: The server creates a feedback message citing the appropriate legal provisions for illegal activities and sends it to the smart glasses via the Internet.

[1393] Step 7: User Reports and Legal Advice

[1394] What it does: Users use the smart glasses interface to report or seek legal advice based on their feedback.

[1395] Input: The feedback message.

[1396] Output: Report or request for legal advice.

[1397] How it works: When the user presses the report button on the smart glasses, the device will directly report the incident to the police or local government. If necessary, contact details for legal advice will also be displayed on the screen.

[1398] Through these steps, users can obtain video footage in real time, instantly identify illegal or suspicious activity, and receive appropriate feedback to report or seek legal advice.

[1399] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1400] This system uses AI technology and an emotion engine to detect illegal activity based on video footage taken by users, compares the results with the law, and provides feedback. The system includes a video capture device, a server, a legal database, an emotion engine, and a communication network connecting them.

[1401] Video acquisition and transmission

[1402] Users use their smartphones or tablets to capture specific scenes, and the captured data is recorded as video or still images. The device then compresses the captured video and sends it to a server via the Internet.

[1403] Video analysis

[1404] The server stores the video data received from the device in a database. The stored data is then analyzed using AI technology. Specifically, machine learning models and deep learning are used to identify actions within the video (e.g., illegal dumping, illegal parking, smoking, etc.).

[1405] Legal Compliance

[1406] The analyzed behavior is compared with a legal database on the server. The legal database stores various laws and ordinances, and determines whether they are illegal. For example, if smoking is detected in a no-smoking area, the system determines whether the behavior violates the local government's smoking ban ordinance.

[1407] Combining Emotion Engines

[1408] This system incorporates an emotion engine and has been added with the ability to recognize the user's emotional state. The device collects emotional data from the user's voice, facial expressions, text input, etc., and the server analyzes this data using the emotion engine. Specifically, if the user is in a high stress state, feedback reflecting that state is provided.

[1409] Generate and send feedback

[1410] If illegality is found, the server generates a feedback message to reflect the result. The tone of the message and recommended actions change depending on the user's emotional state. For example, if the user is in a state of high stress, a more reassuring message (such as "Don't push yourself, take appropriate action") will be generated and sent to the device.

[1411] Reporting and legal consultation support

[1412] Users can report or seek legal advice based on the feedback messages displayed on their device. For example, by pressing the report button within the app, they can directly report to the relevant authorities (police station or local government). In addition, based on the analysis results of the emotion engine, the app will provide recommended actions to reduce psychological stress and the contact information for lawyers for legal advice.

[1413] Specific examples

[1414] 1. The user takes a picture of someone throwing away trash in a park and says, "What should I do about this?"

[1415] 2. The device compresses the video and audio data and sends it to the server.

[1416] 3. The server analyzes the video to detect illegal dumping, and simultaneously analyzes the audio data with an emotion engine to recognize high stress levels.

[1417] 4. The server determines that the act constitutes illegal dumping based on local government ordinances.

[1418] 5. The server generates feedback stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not force yourself and take appropriate action," and sends it to the device.

[1419] 6. The user reports the message.

[1420] This allows users to instantly identify nuisance or illegal behavior and take appropriate action, and by utilizing the emotion engine, they can deal with the situation while reducing the mental burden.

[1421] The processing flow will be explained below.

[1422] Step 1:

[1423] The user launches the app and switches to camera mode by tapping the app icon on their smartphone or tablet and then pressing the camera button on the screen.

[1424] Step 2:

[1425] The user captures a specific scene, for example, someone throwing out trash, and the camera records the footage within its coverage area.

[1426] Step 3:

[1427] The device also records the user's voice and facial expressions. For example, if the user says, "What should I do about this?", that voice data will also be recorded.

[1428] Step 4:

[1429] The device temporarily stores the video and audio data it captures. The video and audio data are stored in the device's memory.

[1430] Step 5:

[1431] The device compresses the captured video and audio data and prepares it for transmission to the server, converting the captured data into an optimized format and establishing an Internet connection.

[1432] Step 6:

[1433] The device sends the video and audio data to the server, where it is encrypted using the HTTPS protocol and securely uploaded to the server.

[1434] Step 7:

[1435] The server receives the video and audio data sent from the terminal, and saves the received data in a specified directory on the server.

[1436] Step 8:

[1437] The server analyzes the received video, specifically using machine learning models and deep learning algorithms to detect specific behaviors within the video, in this case illegal dumping of trash.

[1438] Step 9:

[1439] The server analyzes the voice and facial expression data it receives using an emotion engine, specifically recognizing emotional states such as high stress or anger.

[1440] Step 10:

[1441] The server compares the video analysis results with a legal database, which stores various laws and regulations, and determines whether the video is illegal based on the analysis results.

[1442] Step 11:

[1443] If the server determines that something is illegal, it generates a feedback message that reflects the results of the emotion engine. For example, if a high stress state is recognized, a message such as "This behavior violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not push yourself and take appropriate action" will be created.

[1444] Step 12:

[1445] The server sends the generated feedback message to the terminal, specifically, sends the generated message back to the user's terminal for display.

[1446] Step 13:

[1447] The device receives the feedback message and displays it to the user as a notification, which can be a pop-up notification or an in-app message.

[1448] Step 14:

[1449] Based on the feedback messages received, users can report or seek legal advice. Specifically, by tapping the "Report" or "Legal Advice" button within the app, an interface will appear for contacting the relevant authorities.

[1450] This allows users to quickly identify illegal activity occurring in front of them and take appropriate action, while also reducing the mental burden by utilizing the emotion engine.

[1451] Example 2

[1452] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1453] Conventional illegal activity detection systems have the problem of low accuracy in analyzing video data, which makes it difficult to quickly determine illegal activity. In addition, users often feel emotional stress when reporting illegal activity, and appropriate feedback may not be provided. This makes it difficult for users to take appropriate action.

[1454] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1455] In this invention, the server includes a means for storing and analyzing data in a database, a means for classifying the analyzed results as specific acts using a machine learning model, and a means for comparing the results with a legal database to determine illegality. This enables highly accurate analysis of video data and rapid determination of illegality. Furthermore, the server includes a means for generating feedback based on the user's emotional data, which can provide appropriate feedback suited to the user's emotional state, reducing stress for the user and encouraging appropriate responses.

[1456] "Means for acquiring images" refers to the ability of a user to capture a specific scene using a device such as a smartphone or tablet.

[1457] "Means for the terminal to compress the captured video and send it to the server" is a function that compresses the captured video data and transfers it to the server via the Internet.

[1458] The "means for storing in a database and analyzing" is a mechanism for storing the video data received by the server and later analyzing the data.

[1459] "Means of using a machine learning model to classify the analyzed results as specific actions" refers to a system that uses machine learning technology to recognize and classify specific actions (e.g., illegal dumping, illegal parking, etc.) from analyzed video data.

[1460] "Means for comparing with a legal database and determining illegality" is a function that compares the analyzed act with the laws and ordinances in the legal database and determines whether the act is illegal.

[1461] "Means for collecting user emotional data" refers to devices and software for collecting data such as the user's voice, facial expressions, and text input.

[1462] The "means for analyzing emotional data" is a system that analyzes collected emotional data using an emotional engine and recognizes the user's emotional state (e.g., high stress level).

[1463] "Means for generating feedback and sending it to the device" refers to a function that generates a feedback message based on the user's emotional state when illegality is recognized, and sends and displays that message on the device.

[1464] "Means for reporting or seeking legal advice" means a system that allows users to report to relevant authorities based on feedback messages or provides contact information for assistance in seeking legal advice.

[1465] This invention is a system that uses AI technology and an emotion engine to detect illegal activity based on video footage taken by users, compares the results with a legal database, and provides feedback.The system is primarily composed of devices such as smartphones and tablets, a server that processes data, a legal database, an emotion engine, and a communication network that connects them.

[1466] Video acquisition and transmission

[1467] Users use their smartphones or tablets to capture specific scenes. The captured data is recorded as video or still images. The device then compresses the video data using algorithms such as the H.264 compression algorithm and sends it to a server via the Internet. For example, if a user captures someone throwing away trash in a park, the video is compressed on the device and sent to the server.

[1468] Video analysis

[1469] The server stores the video data received from the device in a dedicated database (e.g., MongoDB). The server then analyzes the stored video data using a deep learning library such as TensorFlow. This analysis allows it to recognize specific behaviors (e.g., illegal dumping, illegal parking, etc.). For example, the received video data can be analyzed using a deep learning model to identify illegal dumping of garbage in a park.

[1470] Legal Compliance

[1471] The analyzed actions are compared with a legal database (e.g., PostgreSQL) on the server. The legal database stores various laws and ordinances, and determines whether a particular action is illegal. For example, it determines whether illegal dumping of garbage violates local government ordinances.

[1472] Combining Emotion Engines

[1473] This system is equipped with an emotion engine, and the device collects emotional data from the user's voice, facial expressions, text input, etc. The server analyzes this emotional data using an emotion engine (e.g., Microsoft's Emotion API) to recognize the user's emotional state (e.g., high stress). For example, analyzing voice data in which the user mutters, "What should I do about this?", the system recognizes that the user is in a high stress state.

[1474] Generate and send feedback

[1475] If illegality is found, the server generates a feedback message to that effect. The tone of the message and recommended actions change based on the user's emotional state. For example, if the user is in a state of high stress, a reassuring message will be generated and sent to the device. For example, a message such as, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please take appropriate action without forcing yourself" will be generated.

[1476] Reporting and legal consultation support

[1477] Based on the feedback messages displayed on the device, users can report the incident or seek legal advice. For example, by pressing the report button within the app, users can directly report the incident to the relevant authority (e.g., the local government's environmental protection department). In addition, based on the analysis results of the emotion engine, recommended actions to reduce psychological stress and contact information for legal advice are provided. For example, users can press the report button to report the incident to the police station.

[1478] Prompt Sentence Examples

[1479] An example of a prompt that can be input to a generative AI model is, "Detect illegal dumping of garbage in a park and generate a feedback message based on the user's emotional state."

[1480] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1481] Step 1:

[1482] A user uses a smartphone or tablet to capture a specific scene. The input is the captured video data (video or still images). A specific operation is for the user to capture a scene of someone throwing away trash in a park. The output is the video data stored on the device.

[1483] Step 2:

[1484] The device compresses the captured video data and sends it to the server via the Internet. The input is the video data generated in step 1. Specifically, the captured data is compressed using the H.264 compression algorithm (for example, reducing a 100MB video to 5MB). The output is the compressed video data being sent to the server.

[1485] Step 3:

[1486] The server stores compressed video data received from the device in a database. The input is compressed video data. Specifically, the video data is stored in a database such as MongoDB. The output is video data stored in the database.

[1487] Step 4:

[1488] The server uses a deep learning model (e.g., TensorFlow) to analyze the video data. The input is the video data stored in a database. Specifically, the video data is input into the deep learning model to detect specific behaviors, such as illegal dumping of garbage. The output is specific behavior data as the analysis result.

[1489] Step 5:

[1490] The server compares the analysis results with a legal database to determine whether the act is illegal. The input is specific act data. Specifically, it accesses a legal database such as PostgreSQL and compares the analysis results with various laws and ordinances. The output is a determination of whether the act is illegal.

[1491] Step 6:

[1492] The device collects emotion data from the user's voice, facial expressions, text input, etc. The input is the user's emotion-related data (voice, facial expressions, text). Specifically, it acquires voice data in which the user mutters, "What should I do about this?" The output is emotion data.

[1493] Step 7:

[1494] The server uses an emotion engine (e.g., Microsoft's Emotion API) to analyze emotion data. The input is the collected emotion data. Specific operations include inputting data into the emotion engine and recognizing the user's emotional state (e.g., high stress). The output is the emotional state data resulting from the analysis.

[1495] Step 8:

[1496] The server generates a feedback message based on the illegality determination result and the user's emotional state data. The input is the illegality determination result and the emotional state data. Specific operations include appropriately adjusting the tone and content of the feedback message. A message is generated stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not force yourself and take appropriate action." The output is the generated feedback message.

[1497] Step 9:

[1498] The server sends the generated feedback message to the terminal. The input is the feedback message generated in step 8. As a specific operation, the message is sent to the terminal via the Internet. The output is the feedback message displayed on the terminal.

[1499] Step 10:

[1500] The user can report or seek legal advice based on the feedback message. The input is the feedback message displayed on the terminal. Specific actions include the user pressing the report button to report to the police station or the local government's environmental protection department. The output is the completion of the report to the relevant agency or the contact for legal advice.

[1501] (Application example 2)

[1502] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1503] Previously, systems existed to detect illegal and nuisance behavior, but they simply detected and reported illegal behavior without taking into account the user's emotional state. This made it difficult for them to take appropriate action when users were experiencing high levels of stress or anxiety. Furthermore, they lacked the accuracy of recognizing illegal behavior and the ability to provide appropriate feedback in real time. As a result, users were unsure of how to act, leading to a problem of low enforcement rates.

[1504] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video, means for comparing the analysis results with a legal database to determine illegality, and means for recognizing the user's emotional state and generating an appropriate feedback message. This makes it possible to quickly detect illegal acts from video captured by the user and provide appropriate feedback based on the emotional analysis without burdening the user. The user can take appropriate action based on the feedback message, and can report the act or seek legal advice in real time. This realizes a quick and effective response to illegal acts while reducing the mental burden on the user.

[1505] "Means for acquiring video footage" refers to a function that allows users to record videos and images using devices such as smartphones and cameras.

[1506] The "means for transmitting to a server" is a function for transmitting the acquired video data to a server via the Internet or other communication network.

[1507] "Means for analyzing video" refers to a function that utilizes machine learning models and deep learning technology to analyze received video data and identify specific actions and objects within it.

[1508] The "means of comparing with a legal database to determine illegality" is a function that compares the content of the analyzed video data with pre-registered legal and ordinance data to determine whether it violates laws and regulations.

[1509] "Means for recognizing the user's emotional state" refers to a function that analyzes the user's voice, facial expression, or text input to measure their current emotional state and stress level.

[1510] The "means for generating appropriate feedback messages" is a function for creating messages that reassure users or encourage them to take appropriate action based on the recognized emotional state of the user.

[1511] The "means for sending a feedback message to a user terminal" is a function for sending the generated feedback message to a user terminal such as a smartphone or tablet.

[1512] An "emotion engine" is a technology that includes algorithms and models for analyzing a user's emotions from voice and text data and identifying their emotional state.

[1513] A "machine learning model" is a collection of algorithms that learn from data and recognize patterns to make predictions or classifications for a specific task.

[1514] "Real-time" refers to a process in which data is processed and analyzed within a very short time after it is acquired, and the results are fed back immediately.

[1515] The present invention is a system that analyzes videos taken by users, detects illegal activities using AI technology and an emotion engine, compares the results with the law, and provides feedback. Specific embodiments of the system are described in detail below.

[1516] System Configuration

[1517] This system mainly consists of a user terminal, a server, a legal database, an emotion engine, and a communication network that connects them.

[1518] 1. User Device

[1519] Users use devices such as smartphones or cameras to capture images of the target. The captured image data is generated using the device's camera function. The emotion engine also collects the user's voice and text inputs as needed.

[1520] 2. Server

[1521] The video data captured by the device is sent to a server via a communication network. The server has the following functions:

[1522] 1. Video analysis: Video data is received and analyzed using machine learning models and deep learning techniques to identify actions and objects in the video. This process uses libraries such as TensorFlow and OpenCV.

[1523] 2. Legal matching: The analyzed behavior is compared with a legal database containing various laws and regulations to determine whether it constitutes an illegal act.

[1524] 3. Emotion analysis: Analyze the user's voice and text data using an emotion engine (e.g., TextBlob) to recognize the user's emotional state.

[1525] 4. Feedback Generation: If illegality is detected, an appropriate feedback message is generated taking into account the user's emotional state. This message is sent to the device.

[1526] 3. Communication Network

[1527] Communication between the user device and the server is carried out via a communication network such as the Internet, and video data, analysis results, and feedback messages are sent and received in a timely manner.

[1528] Specific examples

[1529] 1. A user spots someone illegally littering in a park and takes a photo of the incident with their smartphone.

[1530] 2. When taking a photo, the user mutters out loud, "What should I do with this?"

[1531] 3. The smartphone compresses the video and audio data and sends it to the server.

[1532] 4. The server analyzes the video to detect illegal dumping, and simultaneously analyzes the audio data with an emotion engine to recognize high stress levels.

[1533] 5. The server checks the legal database and determines that the illegal dumping violates Article 16 of the Waste Disposal and Public Cleansing Act.

[1534] 6. The server generates a feedback message stating, "This action violates Article 16 of the Waste Disposal and Public Cleansing Act. Please do not force yourself and take appropriate action," and sends it to the user's device.

[1535] 7. The user checks the message and reports it to local authorities using the reporting function within the app.

[1536] Example prompts for generative AI models

[1537] "A user's smartphone was used to record the act of littering in a park. Based on the video, AI technology was used to detect illegal dumping, and an emotion engine was used to identify the user's high stress level. The system then generated a feedback message saying, 'Don't push yourself, take appropriate action,' and reported the incident to local authorities."

[1538] In this way, the present invention makes it possible to deal with illegal activities quickly and appropriately while reducing the burden on users.

[1539] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1540] Step 1:

[1541] The user shoots video and inputs audio using a smartphone.

[1542] Input: Video and audio data obtained from the smartphone camera and microphone.

[1543] Output: Recorded video files (video or still images) and audio files.

[1544] Specific operation: The smartphone camera records the target's actions, and the microphone simultaneously records the user's voice.

[1545] Step 2:

[1546] The video and audio data captured by the device is compressed and sent to a server via the Internet.

[1547] Input: Recorded video and audio files.

[1548] Output: Compressed video and audio data.

[1549] Specific operation: Video and audio files are reduced in size using a compression algorithm and sent to the server's API endpoint.

[1550] Step 3:

[1551] The server analyzes the received video data and identifies the actions in the video.

[1552] Input: Compressed video data.

[1553] Output: Analysis results (identification of actions).

[1554] How it works: The server decodes the video data and uses machine learning models (e.g., TensorFlow) to analyze the actions in the video, such as identifying illegal dumping and illegal parking.

[1555] Step 4:

[1556] The server compares the analysis results with a legal database to determine whether they are illegal.

[1557] Input: Analysis results (identification of actions).

[1558] Output: Illegality determination result.

[1559] Specific operation: The analysis results are compared with ordinances and laws registered in a legal database to determine whether the behavior is illegal. For example, if smoking is detected in a local government no-smoking zone, the behavior will be determined to be a violation of the ordinance.

[1560] Step 5:

[1561] The server analyzes the received voice data using an emotion engine to recognize the user's emotional state.

[1562] Input: Audio data.

[1563] Output: Emotional state judgment result.

[1564] What it does: Analyzes voice data and recognizes the user's emotional state. It uses a sentiment analysis model (e.g., TextBlob) to determine emotions (e.g., high stress, anger, relief, etc.) from the user's voice.

[1565] Step 6:

[1566] The server generates an appropriate feedback message based on the results of the illegal act determination and the emotional state determination.

[1567] Input: Illegality judgment results and emotional state judgment results.

[1568] Output: Feedback message.

[1569] Specific operation: The results of the illegality judgment and the emotional state judgment are combined to generate a feedback message for the user. For example, if illegal dumping is detected and the user is in a high stress state, a message such as "This behavior is illegal. Please do not push yourself and take appropriate action" is generated.

[1570] Step 7:

[1571] The server sends the generated feedback message to the terminal.

[1572] Input: Feedback message.

[1573] Output: Messages that are displayed on the user's terminal.

[1574] Specific operation: The generated feedback message is sent to the user's device and displayed on the screen, allowing the user to check the message.

[1575] Step 8:

[1576] The user reports or seeks legal advice based on the feedback message.

[1577] Input: Feedback message.

[1578] Output: Report or provide legal advice.

[1579] Specific actions: The user checks the feedback message displayed on the device and, if necessary, reports the issue to the appropriate authorities using the in-app reporting function, or contacts a lawyer or other professional using the legal consultation function.

[1580] By taking these steps, the system of the present invention can detect illegal activity from video footage taken by users, provide appropriate feedback, and take prompt action.

[1581] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1582] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1583] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1584] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1585] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1586] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1587] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1588] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1589] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1590] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1591] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1592] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1593] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1594] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1595] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1596] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1597] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1598] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1599] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1600] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1601] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1602] The following is further disclosed regarding the above embodiment.

[1603] (Claim 1)

[1604] a means for acquiring the image;

[1605] means for transmitting the acquired video to a server;

[1606] A means for analyzing the video received by the server;

[1607] A means of comparing the analysis results with legal databases to determine illegality;

[1608] means for generating and transmitting feedback to the device if illegality is found;

[1609] A system including:

[1610] (Claim 2)

[1611] A means of analyzing the video provided by the user and determining whether it is illegal under applicable law; and

[1612] A means of notifying users of the judgment results in real time,

[1613] The means by which a notice can be filed or legal advice can be obtained;

[1614] 10. The system of claim 1, comprising:

[1615] (Claim 3)

[1616] a means for using machine learning models in video analysis;

[1617] A means for classifying the analyzed results as a specific action;

[1618] a means for matching the classification results with a local legal database;

[1619] 10. The system of claim 1, comprising:

[1620] "Example 1"

[1621] (Claim 1)

[1622] a means by which the user obtains the footage;

[1623] A means for compressing the acquired video by the terminal and transmitting it to a server;

[1624] A means for storing the video received by the server;

[1625] A means for analyzing the stored video using a machine learning model;

[1626] A means for identifying the analyzed acts and checking them against a legal database to determine whether they are illegal;

[1627] means for generating and transmitting a feedback message to the terminal when illegality is detected;

[1628] A system including:

[1629] (Claim 2)

[1630] The system according to claim 1, further comprising a means for notifying the user of the judgment result in real time and enabling the user to report or seek legal advice based on the notification.

[1631] (Claim 3)

[1632] The system according to claim 1, further comprising means for using a machine learning model for video analysis, classifying the analyzed results as specific acts, and associating the classified results with a legal database.

[1633] "Application Example 1"

[1634] (Claim 1)

[1635] a means for acquiring the image;

[1636] means for transmitting the acquired video to a server;

[1637] A means for analyzing the video received by the server;

[1638] A means of comparing the analysis results with legal databases to determine illegality;

[1639] means for generating and transmitting feedback to the device if illegality is found;

[1640] a means for capturing video in real time using smart glasses;

[1641] means for transmitting the compressed data over the Internet;

[1642] means for displaying feedback to the user on the display;

[1643] A system including:

[1644] (Claim 2)

[1645] A means of analyzing the video provided by the user and determining whether it is illegal under applicable law; and

[1646] A means of notifying users of the judgment results in real time,

[1647] The means by which a notice can be filed or legal advice can be obtained;

[1648] A means of analysis using generative AI models,

[1649] 10. The system of claim 1.

[1650] (Claim 3)

[1651] a means for using machine learning models in video analysis;

[1652] A means for classifying the analyzed results as a specific action;

[1653] a means for matching the classification results with a local legal database;

[1654] A means of performing real-time analysis and prompting the user as needed;

[1655] 10. The system of claim 1.

[1656] "Example 2: Combining Emotion Engines"

[1657] (Claim 1)

[1658] a means for acquiring the image;

[1659] A means for compressing the acquired video by the terminal and transmitting it to the server;

[1660] A means for storing and analyzing the video received by the server in a database;

[1661] A machine learning model is used for video analysis, and the results are classified as specific actions.

[1662] A means of comparing the analysis results with legal databases to determine illegality;

[1663] A means for the device to collect user emotional data;

[1664] A means for the server to analyze the emotion data;

[1665] means for generating and transmitting feedback to the terminal based on the user's emotional state if illegality is detected;

[1666] A system including:

[1667] (Claim 2)

[1668] A means of analyzing the video provided by the user and determining whether it is illegal under applicable law; and

[1669] A means of notifying users of the judgment results in real time,

[1670] The means by which a notice can be filed or legal advice can be obtained;

[1671] 10. The system of claim 1, comprising:

[1672] (Claim 3)

[1673] A means for classifying the analyzed results as a specific action;

[1674] a means for matching the classification results with a local legal database;

[1675] A means for taking into account user emotional data when generating feedback messages;

[1676] 10. The system of claim 1, comprising:

[1677] "Application example 2 when combining emotion engines"

[1678] (Claim 1)

[1679] a means for acquiring the image;

[1680] means for transmitting the acquired video to a server;

[1681] A means for analyzing the video received by the server;

[1682] A means of comparing the analysis results with legal databases to determine illegality;

[1683] means for recognizing the emotional state of the user and generating appropriate feedback messages;

[1684] means for transmitting a feedback message to a user terminal;

[1685] A system including:

[1686] (Claim 2)

[1687] A means of analyzing the video provided by the user and determining whether it is illegal under applicable law; and

[1688] A means for generating appropriate feedback messages based on the judgment results and emotion analysis results and notifying the user in real time;

[1689] The means by which a notice can be filed or legal advice can be obtained;

[1690] 10. The system of claim 1, comprising:

[1691] (Claim 3)

[1692] a means for using machine learning models in video analysis;

[1693] A means for classifying the analyzed results as a specific action;

[1694] a means for matching the classification results with a local legal database;

[1695] A means of analyzing the user's voice and text with an emotion engine,

[1696] 10. The system of claim 1, comprising: [Explanation of symbols]

[1697] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring the image; means for transmitting the acquired video to a server; A means for analyzing the video received by the server; A means of comparing the analysis results with legal databases to determine illegality; means for generating and transmitting feedback to the device if illegality is found; A system including:

2. A means of analyzing the video provided by the user and determining whether it is illegal under applicable law; and A means of notifying users of the judgment results in real time, The means by which a notice can be filed or legal advice can be obtained; The system of claim 1 , comprising:

3. a means for using machine learning models in video analysis; A means for classifying the analyzed results as a specific action; a means for matching the classification results with a local legal database; The system of claim 1 , comprising:

Images