System

Abstract

A system is provided.SOLUTION: A system comprising: means for detecting dangerous animals, harmful insects, and foreign organisms; means for generating a countermeasure using a generation AI based on a type and an invasion place of the detected organisms; and means for notifying a user of the generated countermeasure and notifying a public institution of the countermeasure as necessary.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the introduction of dangerous animals, pests, and invasive species has caused a great deal of damage to human living environments. To address these issues, security companies and public institutions are being asked to take effective measures to prevent damage before it occurs, but at present it is difficult to respond quickly and appropriately, and there is also a problem of a shortage of security guards.

[0005] Conventional surveillance systems simply record and store camera footage and audio data, but lack the functionality to detect intrusions in real time and provide immediate, specific countermeasures according to the situation. This makes it extremely difficult to detect the intrusion of dangerous animals or pests early and respond appropriately.

[0006] Therefore, the present invention aims to solve these problems by collecting data in real time from surveillance cameras and sound-collecting microphones, analyzing that data to immediately detect the intrusion of dangerous organisms, and proposing appropriate countermeasures using generative AI. [Means for solving the problem]

[0007] In order to solve the above problems, the present invention provides the following means.

[0008] The system includes a means for acquiring video data from surveillance cameras and audio data from sound-collecting microphones in real time. It also includes a means for analyzing the acquired video and audio data to detect dangerous animals, pests, and invasive species. It also includes a means for generating countermeasures using generative AI based on the type of organism detected and the location of its intrusion. The system also includes a means for notifying the user of the generated countermeasures and, if necessary, reporting them to public authorities.

[0009] Additionally, the system of the present invention includes a means for continuously collecting video and audio data using a surveillance camera and a sound-collecting microphone, temporarily storing the data in a buffer, and transmitting the data to a server. It also includes a means for evaluating the analyzed data based on specific criteria and issuing an alarm if an abnormality is detected.

[0010] Furthermore, the system of the present invention includes a means for re-verifying the countermeasures proposed by the generation AI, adding supplemental information, and then transmitting the countermeasures to the terminal for display. It also includes a means for the system to automatically execute some actions (e.g., activating an audio alarm) when the user acts based on the countermeasures.

[0011] A "surveillance camera" is a device that monitors a specific area and captures and records video.

[0012] A "sound collection microphone" is a device that collects surrounding sounds, converts them into electrical signals, and provides them as audio data.

[0013] "Video data" refers to data that digitally represents video footage captured by a surveillance camera.

[0014] "Audio data" is data that represents in digital form the sound picked up by a sound-collecting microphone.

[0015] "Real-time" means that the entire process, from data collection to processing and analysis, is carried out immediately without delay.

[0016] "Means for acquiring" refers to a method or device for collecting video data and audio data from a surveillance camera or a sound-collecting microphone.

[0017] "Means for analyzing" refers to a method or device for analyzing collected video and audio data to detect specific patterns or anomalies.

[0018] "Generative AI" is a technology that uses artificial intelligence to generate appropriate responses and countermeasures from specific input data.

[0019] "Means for generating countermeasures" refers to methods or devices for devising and proposing appropriate countermeasures for detected abnormalities.

[0020] "User" refers to the person or entity that operates and uses the system.

[0021] "Notification means" refers to a method or device for notifying a user of generated countermeasures or alerts.

[0022] "Means of reporting" means a method or device for reporting the situation to public authorities or related agencies in an emergency.

[0023] A "buffer" refers to a memory space that temporarily stores data.

[0024] A "server" is a computer system used to collect, analyze, store, and distribute data.

[0025] "Criteria" refers to specific conditions or rules used in evaluating data.

[0026] "When an abnormality is detected" refers to when the analyzed data exceeds a pre-set standard.

[0027] An "alert" refers to an alert or warning issued when an abnormality is detected.

[0028] "Means for re-verification" refers to a method or device for reconfirming the appropriateness of the generated countermeasures.

[0029] "Supplementary information" refers to information added to strengthen or complement the original countermeasure proposal.

[0030] An "automatic execution means" is a method or device that allows the system to perform a particular action without user intervention.

[0031] An "audio alarm" is a device or system that issues a warning by playing an alarm sound or voice message. [Brief explanation of the drawings]

[0032] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0033] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0034] First, the terms used in the following description will be explained.

[0035] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0036] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0037] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0038] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0039] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0040] [First embodiment]

[0041] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0042] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0043] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0044] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0045] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0046] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0047] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0048] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0049] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0050] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0051] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0052] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0053] This invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of dangerous animals, pests, and invasive species in real time, and then uses AI to propose countermeasures appropriate to the situation. The specific form and operation of this system are described below.

[0054] System configuration

[0055] 1. Surveillance cameras

[0056] The surveillance cameras monitor designated areas and capture high-resolution video data. All-weather cameras that can be used day and night are recommended.

[0057] 2. Sound collection microphone

[0058] The sound-collecting microphone collects sounds within a designated area and records them as audio data, allowing you to capture the sounds of animals and pests.

[0059] 3. Terminal

[0060] The terminal temporarily stores data from the surveillance camera and sound-collecting microphone and transmits it to the server.

[0061] 4. Server

[0062] The server analyzes the received video and audio data in real time to detect anomalies, using machine learning models.

[0063] 5. Generation AI

[0064] The generative AI generates appropriate countermeasures based on the detected anomalies, notifies the user of the countermeasures, and automatically takes action as needed.

[0065] 6. Notification System

[0066] Terminal applications and browsers will be provided to notify users of the measures, and a function to notify public authorities in the event of an emergency will also be included.

[0067] Program processing explanation

[0068] Data collection and transmission

[0069] The server periodically acquires video and audio data in real time from the surveillance cameras and microphones.

[0070] The device stores the acquired data in a buffer and prepares it for transmission to the server, after which the data is sent to the server.

[0071] Data analysis

[0072] The server immediately analyzes the received video and audio data: video data is analyzed frame by frame, and audio data is analyzed based on the sampling rate.

[0073] The server uses machine learning models to identify patterns of specific animals, pests and invasive species to detect intrusions.

[0074] Real-time detection and notification

[0075] The server uses the analysis results to detect anomalies, evaluates specific criteria, and issues an alert if an anomaly is detected.

[0076] The terminal receives the alarm information from the server and notifies the user.

[0077] Countermeasure generation

[0078] The server uses a generation AI to generate appropriate countermeasures, including specific instructions and precautions.

[0079] The terminal notifies the user of the generated measures and displays them on the screen.

[0080] Reporting and Response

[0081] Users are then advised to take action in accordance with the measures they are notified of. In addition, if a serious abnormality occurs, the server will automatically notify public authorities.

[0082] The device supports user actions and performs some actions automatically (e.g., sound alarm activation).

[0083] Specific examples

[0084] Example 1: Bear invasion

[0085] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[0086] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[0087] The server uses a generation AI to generate and notify the user of countermeasures such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated."

[0088] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[0089] Example 2: Mosquito outbreak

[0090] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[0091] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[0092] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[0093] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[0094] With the above configuration and processing, the present invention can detect the intrusion of dangerous animals, pests, and invasive species in real time and provide quick and effective countermeasures, thereby preventing damage and improving the efficiency of security operations.

[0095] The processing flow will be explained below.

[0096] Step 1: Data collection

[0097] The server receives video and audio data from the surveillance cameras and microphones in real time, allowing it to record all events that occur within the surveillance area.

[0098] The device temporarily stores the captured video and audio data in a buffer, minimizing the risk of data loss.

[0099] Step 2: Send the data

[0100] The device prepares the buffered data for transmission to the server. The data is converted to a certain format (e.g. MP4, WAV, etc.).

[0101] The device sends the prepared data to the server via the network, where it is encrypted and transmitted to ensure security.

[0102] Step 3: Begin data analysis

[0103] The server immediately begins the process of analyzing the received video and audio data.

[0104] The server analyzes and compares the video data frame by frame to identify animals and pests.

[0105] The server analyzes the audio data based on the sampling rate to see if certain audio patterns are present.

[0106] Step 4: Identify specific organisms

[0107] The server uses pre-trained machine learning models to identify specific animals, pests, and invasive species from the data it receives with a high degree of accuracy.

[0108] The server determines the species, location, and behavioral patterns of the identified creatures.

[0109] Step 5: Real-time detection

[0110] The server uses the analysis results to evaluate certain criteria (e.g., animal size, behavior, location of intrusion, etc.) and determine whether an anomaly has been detected.

[0111] If the server detects an anomaly, it will immediately issue an alert, which will be filtered and different actions will be taken depending on the severity.

[0112] Step 6: Notification

[0113] The server sends the alarm information to the terminal so that the user can check it. The alarm includes information on the location, time, and target organism.

[0114] When the terminal receives an alert, it notifies the user with a sound or a pop-up message.

[0115] Step 7: Countermeasure generation

[0116] The server uses generative AI to generate appropriate countermeasures based on the detected anomalies, including specific instructions and precautions.

[0117] The server sends the generated countermeasures to the terminal in a format that is easy for the user to understand.

[0118] Step 8: Review and implement measures

[0119] The device displays the generated countermeasure information to the user, who can then check the countermeasure proposal and prepare to implement it if necessary.

[0120] The user implements the instructed measures and takes the actions recommended by the system, and the system automatically performs some actions (e.g., sounding an audio alarm).

[0121] Step 9: Report

[0122] If a serious abnormality occurs, the server automatically reports the situation to security companies and public institutions, including the type of organism detected, its location, and a summary of the countermeasures taken.

[0123] The server monitors the progress of the notification and provides additional information as needed.

[0124] Through the above processing steps, the system of the present invention can quickly detect abnormalities within the monitored area and propose and implement appropriate countermeasures, thereby preventing damage from dangerous animals, pests, and invasive species.

[0125] Example 1

[0126] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0127] In modern society, the invasion of dangerous animals, pests, and invasive species is increasingly threatening people's daily lives. It is necessary to quickly and effectively detect the invasion of such organisms and promptly implement appropriate countermeasures. However, conventional systems have had difficulty detecting anomalies in real time or automatically generating and implementing appropriate countermeasures. Furthermore, they lacked sufficient support functions to help users take appropriate measures, making it difficult to prevent damage before it occurs.

[0128] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0129] In this invention, the server includes: means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time; means for analyzing the acquired video data and audio data to detect dangerous organisms; means for generating countermeasures using a generation AI based on the type of organism detected and the location of the intrusion; means for notifying the user of the generated countermeasures and, if necessary, notifying public authorities; and means for automatically executing action as necessary to support the user's actions. This makes it possible to detect the intrusion of dangerous organisms in real time and provide quick and effective countermeasures. Support functions for users to take appropriate actions are also enhanced, preventing damage before it occurs.

[0130] A "surveillance camera" is a device that continuously monitors a specific area and captures high-resolution video data.

[0131] A "sound collection microphone" is a device that collects surrounding sounds in real time and records and transmits them as audio data.

[0132] "Video data" refers to digital data of visual information captured by a surveillance camera.

[0133] "Audio data" is digital data of auditory information acquired by a sound-collecting microphone.

[0134] A "server" is a computer system that executes a series of processes such as data collection, analysis, countermeasure generation, and notification.

[0135] "Generative AI" is an artificial intelligence model that automatically generates appropriate countermeasures for users.

[0136] "Countermeasures" are specific measures such as instructions for actions or precautions to be taken in response to detected abnormalities.

[0137] "User" means an individual or organization that uses the system and acts on the measures provided.

[0138] "Public institutions" are organizations such as governments and administrative agencies that are required to report any abnormalities detected.

[0139] A "buffer" is a memory area for temporarily storing data.

[0140] An "alarm" is a warning signal such as sound or light that notifies the user of an abnormality.

[0141] An "action" is an operation or behavior that the system automatically performs (e.g., activating an audio alarm).

[0142] The present invention is a system that uses a surveillance camera and a sound-collecting microphone to detect the intrusion of dangerous organisms in real time, and uses a generating AI to propose countermeasures according to the situation. Detailed embodiments for carrying out the present invention will be described below.

[0143] System configuration

[0144] 1. Surveillance cameras

[0145] The surveillance cameras continuously monitor designated areas and capture high-resolution video data. All-weather cameras are recommended, and include high-resolution CCD cameras and infrared cameras.

[0146] 2. Sound collection microphone

[0147] Sound-collecting microphones collect surrounding sounds and record them as audio data. This makes it possible to capture the sounds of animals and pests. Highly directional microphones and dynamic microphones that are resistant to environmental sounds are used.

[0148] 3. Terminal

[0149] The device temporarily stores data from the surveillance camera and microphone, stores it in a buffer, and then transmits it to the server. The device is preferably a computer or dedicated device with a high-speed processor and sufficient memory.

[0150] 4. Server

[0151] The server analyzes the received video and audio data in real time to detect anomalies. This analysis is performed using machine learning models, such as deep learning frameworks like TensorFlow and PyTorch.

[0152] 5. Generation AI

[0153] The generative AI generates appropriate countermeasures based on the detected anomalies. A natural language generation model (e.g., GPT-4) is used for generation. The generated countermeasures are notified to the user, and automatic action is taken if necessary.

[0154] 6. Notification System

[0155] The notification system will include a terminal application and browser to notify users of countermeasures, and will also have a function to notify public authorities in the event of an emergency.

[0156] Specific examples

[0157] Example 1: Bear invasion

[0158] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[0159] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[0160] The server uses a generation AI to generate and notify the user of countermeasures such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated."

[0161] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[0162] Example 2: Mosquito outbreak

[0163] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[0164] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[0165] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[0166] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[0167] Prompt Sentence Examples

[0168] Example 1: Bear invasion

[0169] "A bear has been detected in security camera footage. Please generate a measure to notify the user."

[0170] Example 2: Mosquito outbreak

[0171] "A large number of mosquito sounds have been detected from the microphone. Please generate a measure to notify the user."

[0172] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0173] Program processing flow

[0174] Step 1: Collect data

[0175] Input: Real-time video and audio data from surveillance cameras and microphones

[0176] Output: Video and audio data temporarily stored in the buffer

[0177] How it works: The server collects real-time data from a specified area via surveillance cameras and microphones. The day and night, all-weather cameras capture high-resolution images, while the microphones collect ambient sounds. This data is first temporarily stored in a buffer on the device.

[0178] Step 2: Sending data

[0179] Input: Video and audio data stored in the buffer

[0180] Output: Video and audio data sent to the server

[0181] What it does: The device periodically checks the data stored in the buffer, and if new data is available, it sends it to the server using a secure protocol (e.g., HTTPS).

[0182] Step 3: Analyze the data

[0183] Input: Video and audio data sent to the server

[0184] Output: Anomaly detection information as analysis results

[0185] Specific operation: The server analyzes the received video and audio data using a machine learning model. Feature extraction and recognition are performed on each frame of the video data using OpenCV, TensorFlow, etc. Audio data is analyzed based on the sampling rate, and the frequency spectrum is analyzed using FFT.

[0186] Step 4: Real-time anomaly detection

[0187] Input: Parsed data

[0188] Output: Issue an alert

[0189] Specific operation: The server detects anomalies based on the analysis results and issues an alert if certain criteria are met. For example, if a specific animal is seen in the video or a sound of a specific frequency is detected, it will recognize it as an anomaly and issue an alert. These criteria include thresholds and the results of pattern recognition.

[0190] Step 5: Generate countermeasures

[0191] Input: Anomaly detection information

[0192] Output: Generated countermeasure statement

[0193] Specific operation: The server uses a generative AI model (e.g., GPT-4) to generate appropriate countermeasures based on the anomaly detection information. The prompt contains detailed information about the current abnormal situation, and the generative AI model generates countermeasures based on this.

[0194] Step 6: Notification and response support

[0195] Input: Generated countermeasures

[0196] Output: User notification and automatic actions

[0197] Specific operations: The device notifies the user of the alert from the server and the generated countermeasures. The notification is displayed via a smartphone app or web browser. The notification contains specific response instructions that the user can act on. Furthermore, if a serious abnormality is detected, the server automatically notifies public authorities. Some actions (such as sounding an audio alarm) are also performed automatically, helping the user to take immediate action.

[0198] Specific examples

[0199] Bear invasion

[0200] The server analyzes video data from surveillance cameras, and if it detects a large animal, it identifies it as a bear based on its characteristics.

[0201] The server issues a bear intrusion warning and sends a countermeasure message to the device.

[0202] The generative AI model is given a prompt message: "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated.", and a countermeasure message is generated.

[0203] The terminal notifies the user of the generated countermeasure statement, and the system automatically activates an audio alarm if necessary.

[0204] Massive mosquito outbreaks

[0205] The server analyzes the audio data from the microphone and detects the sounds of large numbers of mosquitoes.

[0206] The server issues a mosquito outbreak warning and sends a countermeasure message to the terminal.

[0207] The generative AI model is given a prompt message: "There is a mosquito infestation. Please use insect repellent and close windows and doors. Also, contact a professional if necessary.", and a countermeasure message is generated.

[0208] The terminal notifies the user of the generated countermeasure statement and assists the user in carrying out the countermeasure based on the instruction.

[0209] (Application example 1)

[0210] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0211] Conventional security systems have difficulty detecting the intrusion of dangerous animals, pests, and invasive species in real time, often resulting in delayed implementation of appropriate countermeasures. Furthermore, they lack the means to provide users with quick and specific instructions in emergencies, increasing the risk of damage spreading. The present invention aims to solve these issues by providing a system that uses surveillance cameras and sound-collecting microphones to efficiently detect the intrusion of living organisms and uses generative AI to propose appropriate countermeasures in real time.

[0212] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0213] In this invention, the server includes means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time, means for analyzing the acquired video data and audio data to detect dangerous animals, pests, and invasive species, means for generating countermeasures using a generation AI based on the type of organism detected and the location of its intrusion, means for notifying a user of the generated countermeasures and reporting them to a public institution as necessary, and means for a user to check and implement the countermeasures using a smartphone, thereby enabling efficient and specific countermeasures to be provided in real time.

[0214] A "surveillance camera" is a device used to acquire high-resolution video data in real time and analyze that data.

[0215] A "sound collection microphone" is a device that collects sound data within a designated area and captures the sounds of animals and pests.

[0216] A "server" is a computer system that implements a machine learning model to analyze received video and audio data in real time and detect anomalies.

[0217] "Generative AI" is an artificial intelligence model that generates appropriate countermeasures based on detected anomalies.

[0218] A "buffer" is a memory system that contains an area for temporarily storing data and sending it on to the next processing step.

[0219] "Alert" is an alert system that issues a warning to the user when an abnormality is detected.

[0220] A "smartphone" is a mobile information terminal used by a user to receive notifications and to check and take countermeasures.

[0221] A "prompt" is textual input data that gives specific instructions or questions to a generative AI model.

[0222] The present invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of dangerous animals, pests, and invasive species in real time, generates countermeasures using generation AI, and notifies the user. This system is composed of surveillance cameras, sound-collecting microphones, a server, and a smartphone app. By combining these, it is possible to safely and quickly detect abnormalities and take appropriate countermeasures. Specific embodiments are described below.

[0223] Data collection and transmission

[0224] First, a surveillance camera and a sound-collecting microphone monitor a designated area, capturing high-resolution video and audio data in real time. The surveillance camera must be high-resolution and usable day and night, while the sound-collecting microphone must be highly sensitive.

[0225] The acquired data is temporarily stored in a terminal (e.g., a server) and then transmitted to the server. The terminal also includes a buffer for temporarily storing the data.

[0226] Data analysis

[0227] The server uses machine learning models to analyze the received video and audio data in real time: TensorFlow is used to analyze the video data, and the Librosa library is used to analyze the audio data.

[0228] The analyzed data is evaluated based on specific criteria, and if an anomaly is detected, an alert is issued, which is sent directly to the user via a smartphone app.

[0229] Real-time detection and notification

[0230] The smartphone app receives the alert information from the server and notifies the user. This notification system uses Firebase Cloud Messaging. The user can check the alert details and take action on the smartphone app.

[0231] Countermeasure generation

[0232] Generative AI is used to generate appropriate countermeasures. Specific countermeasure proposals for detected anomalies are generated using generative AI models such as GPT-4. The generated countermeasure proposals are provided to users via a smartphone app.

[0233] For example, if a snake is detected, the AI ​​will generate a countermeasure suggestion such as, "A snake has been found. Please keep as far away as possible and contact a specialist." An example of a prompt would be, "A snake has been detected through analysis of surveillance camera footage. Please generate appropriate response procedures."

[0234] Communication and Action

[0235] When the user acts in accordance with the generated countermeasures, the smartphone app also has the function of automatically executing some actions, such as activating an audio alarm.

[0236] As described above, this system combines various sensors and advanced analysis technology to detect danger in real time and provide prompt and appropriate countermeasures, thereby ensuring the safety of users and preventing damage before it occurs.

[0237] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0238] Step 1:

[0239] Surveillance cameras and microphones monitor designated areas, capturing high-resolution video and audio data in real time.

[0240] Input: Video data from surveillance cameras, audio data from microphones

[0241] Output: High-resolution video and audio data

[0242] How it works: The surveillance camera captures an entire area and records the footage in high resolution, while the sound pickup microphone collects surrounding sounds and stores them as digital audio data.

[0243] Step 2:

[0244] The video and audio data acquired by the terminal is temporarily stored in a buffer.

[0245] Input: High-resolution video and audio data

[0246] Output: Temporary data stored in a buffer

[0247] Specific operation: The terminal stores the data collected in real time in a buffer (temporary storage area) and prepares it for subsequent processing.

[0248] Step 3:

[0249] Send data from the terminal to the server.

[0250] Input: Temporary data stored in a buffer

[0251] Output: Video and audio data sent to the server

[0252] Specific operation: The device executes a process to send data to a server via the Internet.

[0253] Step 4:

[0254] The video data received by the server is analyzed using TensorFlow, and the audio data is analyzed using the Librosa library.

[0255] Input: Video and audio data sent to the server

[0256] Output: Analysis results (detected anomalies and patterns)

[0257] How it works: The server breaks down the video data into frames and performs image analysis using TensorFlow. The audio data is analyzed using the Librosa library based on the sampling rate to detect abnormal audio patterns.

[0258] Step 5:

[0259] The server detects anomalies based on the analysis results and issues an alert if certain criteria are met.

[0260] Input: Analysis results

[0261] Output: Warning information

[0262] Specific operation: The server evaluates the analysis results and generates an alert if dangerous animals, pests, or invasive species are detected.

[0263] Step 6:

[0264] The server uses the generated AI model to generate specific countermeasures for the detected anomalies.

[0265] Input: Alarm information

[0266] Output: Generated countermeasures

[0267] Specific operation: The generative AI model generates appropriate countermeasures based on the prompt. For example, a prompt such as "A snake has been detected during surveillance camera video analysis. Please generate appropriate response procedures."

[0268] Step 7:

[0269] The server notifies the user of the generated countermeasures via a smartphone app.

[0270] Input: Generated countermeasures

[0271] Output: Notification displayed on smartphone app

[0272] Specific operation: Using Firebase Cloud Messaging, the generated countermeasures are sent to the smartphone app and the user is notified.

[0273] Step 8:

[0274] Users can check the countermeasures and take action using their smartphones.

[0275] Input: Notification displayed on smartphone

[0276] Output: User action

[0277] Specific action: The user acknowledges the notification and takes appropriate action, for example, following the instructions, "A snake has been found. Move away as far away as possible and contact a professional."

[0278] Step 9:

[0279] The system will automatically take some action (e.g. activate an audio alarm) if necessary.

[0280] Input: User confirmation and warning information

[0281] Output: Actions taken

[0282] Specific operation: The system automatically activates an audio alarm to warn those around.

[0283] The above steps realize a real-time danger detection system using a surveillance camera and a sound-collecting microphone.

[0284] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0285] This invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of dangerous animals, pests, and invasive species in real time, and uses AI to generate and propose countermeasures appropriate to the situation. It also incorporates an emotion engine that recognizes the user's emotions. The system aims to optimize the countermeasures based on the user's emotional state and reduce the user's psychological burden.

[0286] System configuration

[0287] 1. Surveillance cameras

[0288] The surveillance cameras monitor designated areas and capture high-resolution video data, allowing for day and night surveillance.

[0289] 2. Sound collection microphone

[0290] The sound-collecting microphone collects sounds within a designated area and records them as audio data, allowing you to capture the sounds of animals and pests.

[0291] 3. Terminal

[0292] The terminal temporarily stores data from the surveillance camera and microphone and transmits it to the server. Data can be transmitted in real time.

[0293] 4. Server

[0294] The server analyzes the received video and audio data in real time to detect anomalies, using a pre-trained machine learning model.

[0295] 5. Generation AI

[0296] The generative AI generates appropriate countermeasures based on the detected anomalies, notifies the user of the countermeasures, and automatically takes action as needed.

[0297] 6. Emotion Engine

[0298] The emotion engine analyzes the user's voice and facial expressions to recognize their emotions and adjusts the content of the countermeasures based on that information. The urgency and level of detail of the countermeasures can be changed based on the recognized emotional state.

[0299] 7. Notification System

[0300] Terminal applications and browsers will be provided to notify users of the measures, and a function to notify public authorities in the event of an emergency will also be included.

[0301] Program processing explanation

[0302] Data collection and transmission

[0303] The server periodically acquires real-time video and audio data from the surveillance cameras and microphones, thereby recording all events that occur within the surveillance area.

[0304] The device stores the captured video and audio data in a buffer and prepares it for transmission to the server, where it is then encrypted and sent to the server.

[0305] Start of data analysis

[0306] The server immediately begins the process of analyzing the received video and audio data: video data is analyzed frame by frame, and audio data is analyzed based on the sampling rate.

[0307] The server uses pre-trained machine learning models to identify specific animals, pests, and invasive species from the data it receives with a high degree of accuracy.

[0308] Real-time detection and notification

[0309] The server evaluates certain criteria based on the analysis results and issues alerts if anomalies are detected. Alerts are filtered and different actions are taken depending on their severity.

[0310] The terminal receives the alarm information from the server and notifies the user. The alarm includes information on the location and time of occurrence, and the target organism.

[0311] Countermeasure generation and adjustment

[0312] The server uses generative AI to generate appropriate countermeasures based on the detected anomalies, including specific instructions and precautions.

[0313] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state. For example, if the user is in a high stress state, the generative AI will adjust the urgency and level of detail of the countermeasures.

[0314] Check and implement measures

[0315] The device displays the generated countermeasure information to the user, who can then check the countermeasure proposal and prepare to implement it if necessary.

[0316] The user implements the instructed measures and takes the actions recommended by the system, and the system automatically performs some actions (e.g., sounding an audio alarm).

[0317] Report

[0318] If a serious abnormality occurs, the server automatically reports the situation to security companies and public institutions, including the type of organism detected, its location, and a summary of the countermeasures taken.

[0319] The server monitors the progress of the notification and provides additional information as needed.

[0320] Specific examples

[0321] Example 1: Bear invasion

[0322] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[0323] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[0324] The emotion engine analyzes the user's facial expressions and tone of voice to recognize when the user is in a state of high stress.

[0325] The server uses a generation AI to generate and notify the user of a countermeasure such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated." The server also emphasizes the urgency of the countermeasure, as the user is in a high-stress state.

[0326] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[0327] Example 2: Mosquito outbreak

[0328] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[0329] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[0330] The emotion engine analyzes the user's emotional state and recognizes that they are not particularly stressed.

[0331] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[0332] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[0333] With the above configuration and processing, the present invention can quickly detect abnormalities within a monitored area and provide and implement appropriate countermeasures that are adapted to the user's emotional state, thereby preventing damage from dangerous animals, pests, and invasive species and reducing the psychological burden on the user.

[0334] The processing flow will be explained below.

[0335] Step 1: Data collection

[0336] The server receives video and audio data from the surveillance cameras and microphones in real time, ensuring that all events occurring within the surveillance area are accurately recorded.

[0337] The device temporarily stores the acquired video and audio data in a buffer, and the data is periodically accumulated in preparation for the next processing.

[0338] Step 2: Send the data

[0339] The device prepares the data stored in the buffer for transmission to the server. The data is converted into a predetermined format.

[0340] The device encrypts the prepared data and sends it over the network to the server, ensuring data security.

[0341] Step 3: Begin data analysis

[0342] The server starts analyzing the received video and audio data, where the video data is analyzed frame by frame and the audio data is analyzed based on the sampling rate.

[0343] The server uses pre-trained machine learning models to identify specific animals, pests and invasive species from the data it receives.

[0344] Step 4: Identify specific organisms

[0345] The server uses machine learning models to identify animal and pest patterns in the data, for example detecting the outlines of specific animals in video data and recognizing specific animal sounds in audio data.

[0346] The server determines the type, location, and behavioral patterns of the identified organisms and evaluates whether there are any abnormalities.

[0347] Step 5: Real-time detection

[0348] The server evaluates the analysis results and issues an alert if an abnormality is detected, based on specific criteria such as the size, behavioral patterns, and location of the animal.

[0349] Based on the detected abnormality information, the server determines the importance of the alert and selects the appropriate response.

[0350] Step 6: Notification

[0351] The server sends the alert information to the terminal for the user to view. The alert includes details of the location and time of the alert and the detected organism, allowing the user to take prompt action.

[0352] The terminal notifies the user of the received warning information, and the user receives the warning through voice or pop-up message.

[0353] Step 7: Countermeasure generation and sentiment analysis

[0354] The server uses a generative AI to generate appropriate countermeasures for the detected anomalies, creating countermeasure proposals that include specific instructions for action and precautions.

[0355] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state, for example, determining their stress level based on their voice tone and facial expression.

[0356] The server adjusts the countermeasures generated by the generative AI based on the recognized emotional state, providing more detailed and urgent countermeasures to users in high stress states.

[0357] Step 8: Review and implement measures

[0358] The device displays the adjusted countermeasure information to the user, who then checks the countermeasure proposal and takes necessary measures.

[0359] The user initiates action based on the displayed countermeasures. If the system automatically performs some action (e.g., activates an audio alarm), the user confirms this and takes the necessary steps.

[0360] Step 9: Report

[0361] If a serious abnormality occurs, the server automatically reports the situation to security companies and public institutions, including the type of organism detected, its location, and a summary of the countermeasures taken.

[0362] The server monitors the progress of the report and provides additional information as needed, enabling rapid coordination with relevant authorities.

[0363] Specific examples

[0364] Example 1: Bear invasion

[0365] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[0366] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[0367] The emotion engine analyzes the user's facial expressions and tone of voice to recognize when the user is in a state of high stress.

[0368] The server uses a generation AI to generate and notify the user of a countermeasure such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated." The server also emphasizes the urgency of the countermeasure, as the user is in a high-stress state.

[0369] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[0370] Example 2: Mosquito outbreak

[0371] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[0372] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[0373] The emotion engine analyzes the user's emotional state and recognizes that they are not particularly stressed.

[0374] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[0375] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[0376] With the above specific configuration and processing, the present invention can quickly detect abnormalities within a monitored area and provide and implement appropriate countermeasures that are adapted to the user's emotional state, thereby preventing damage from dangerous animals, pests, and invasive species and reducing the psychological burden on the user.

[0377] Example 2

[0378] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0379] In recent years, the intrusion of dangerous animals, pests, and invasive species has been increasing, and the resulting damage has become more serious. There is a need for a system that can detect such intrusions in real time and respond quickly. However, current systems do not provide countermeasures that take the user's emotional state into consideration, which increases the psychological burden on the user. Therefore, the present invention aims to optimize countermeasures based on the user's emotional state and reduce the psychological burden.

[0380] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0381] In this invention, the server includes means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time, means for analyzing the acquired video and audio data to detect dangerous animals, pests, and invasive species, and an emotion engine for analyzing the user's voice and facial expressions to recognize emotions and adjust countermeasures based on the user's emotional state. This makes it possible to quickly detect abnormalities and provide appropriate countermeasures while reducing the psychological burden on the user.

[0382] A "surveillance camera" is a device that monitors a designated area and captures high-resolution video data.

[0383] A "sound collection microphone" is a device that collects sounds within a specified area and records them as audio data.

[0384] "Video data" refers to a collection of video frames captured from a surveillance camera, which record events occurring within a surveillance area.

[0385] "Audio data" refers to a collection of sound sampling data acquired from a sound-collecting microphone, and is data that records sounds occurring within a specified area.

[0386] The "server" is a central processing unit that receives video data and audio data, analyzes them, and detects abnormalities.

[0387] The "emotion engine" is an engine that analyzes the user's voice and facial expressions to recognize their emotions, and adjusts the content of countermeasures based on that information.

[0388] "Generative AI" is an artificial intelligence model that generates appropriate countermeasures based on detected anomalies.

[0389] "Countermeasure generation" refers to the process of using generative AI to design and provide appropriate countermeasures based on the analysis results.

[0390] A "prompt" is an instruction sentence input to a generation AI, and is text containing instructions for generating a specific countermeasure.

[0391] The "notification means" is a mechanism for notifying the user of the generated countermeasures, and includes a terminal application and a browser.

[0392] "User" refers to the person who uses the system, who receives the proposed measures and implements them.

[0393] This invention is a system that detects the intrusion of dangerous animals, pests, and invasive species in real time and provides optimized countermeasures based on the user's emotional state. This system is composed of surveillance cameras, sound-collecting microphones, a server, terminals, a generation AI, an emotion engine, and a notification system.

[0394] Hardware and Software Details

[0395] Surveillance cameras and microphones

[0396] A surveillance camera is a device that monitors a designated area and captures high-resolution video data, allowing for surveillance day and night.

[0397] A sound collecting microphone is a device that collects sounds within a specified area and records them as audio data, allowing you to capture the sounds of animals and pests.

[0398] server

[0399] The server is a central processing unit that receives and analyzes data from surveillance cameras and microphones in real time. The server has the following functions:

[0400] Data collection function: Video and audio data is acquired in real time and temporarily stored.

[0401] Data analysis function: Analyzes received video and audio data to detect anomalies. This analysis is performed using a pre-trained machine learning model.

[0402] Emotion analysis function: Analyzes the user's voice and facial expressions to recognize their emotional state. An emotion prediction model is used to assess whether the user is in a high-stress or calm state.

[0403] Countermeasure generation function: Using a generation AI, a countermeasure is generated based on the detected anomaly. A prompt sentence is input to the generation AI, and the optimal countermeasure is output.

[0404] Terminal

[0405] The terminal is a device that temporarily stores data from the surveillance camera and microphone and transmits it to the server. The terminal has the following functions:

[0406] Data transmission function: Stored data is encrypted and sent to the server using a secure communication protocol, ensuring data safety.

[0407] Notification function: Notifies users of countermeasure information and warnings from the server. Notification methods include terminal applications, browsers, and audio alarms.

[0408] Generative AI model and prompts

[0409] A generative AI model is an artificial intelligence model that generates appropriate countermeasures based on detected anomalies. A prompt sentence is input to the generative AI model, and the optimal countermeasure is output.

[0410] Example prompt: There's a bear infestation. What precautions should be taken?

[0411] Specific examples

[0412] Example 1: Bear invasion

[0413] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[0414] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[0415] The emotion engine analyzes the user's facial expressions and tone of voice to recognize when the user is in a state of high stress.

[0416] The server uses a generation AI to generate and notify the user of a countermeasure such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated." The server also emphasizes the urgency of the countermeasures, as the user is in a high-stress state.

[0417] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[0418] Example 2: Mosquito outbreak

[0419] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[0420] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[0421] The emotion engine analyzes the user's emotional state and recognizes that they are not particularly stressed.

[0422] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[0423] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[0424] In this way, the system can detect abnormalities in real time and provide countermeasures that adapt to the user's emotional state, thereby preventing damage from dangerous animals, pests, and invasive species and reducing the psychological burden on the user.

[0425] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0426] Step 1: Data collection

[0427] The server acquires video data from the surveillance cameras in real time. Specifically, it continuously acquires video data frame by frame and stores it in high resolution. The input is the video data from the surveillance cameras, and the output is the video data temporarily stored in the buffer.

[0428] The server acquires audio data from a microphone in real time based on a sampling rate. The audio data includes environmental sounds within a specified area and the sounds of specific animals. The input is the audio data from the microphone, and the output is the audio data temporarily stored in a buffer.

[0429] Step 2: Send data

[0430] The terminal temporarily stores the acquired video and audio data in a buffer. The input is the data stored in the buffer from the server, and the output is encrypted data.

[0431] The device sends encrypted data to the server using a secure communication protocol (e.g., HTTPS). This ensures data security. The input is encrypted video and audio data, and the output is the data sent to the server.

[0432] Step 3: Data analysis

[0433] The server uses a pre-trained machine learning model to analyze the received video data frame by frame. The input is the video data sent to the server, and the output is the analysis results (e.g., the presence of specific animals, pests, or invasive species).

[0434] The server samples and analyzes the audio data, similarly using pre-trained models to identify specific frequencies and patterns. The input is the audio data sent to the server, and the output is the analysis results (e.g., detection of the sounds of specific animals, pests, or invasive species).

[0435] Step 4: Anomaly detection

[0436] The server evaluates the presence of specific animals, pests, and invasive species based on the results of data analysis, and issues an alert if an abnormality is detected. The input is the data analysis results, and the output is alert information (e.g., location, time, and information about the target species).

[0437] The server filters the alerts and determines the action to be taken (e.g., prioritizing notifications, reporting) depending on the severity. The input is the alert information, and the output is the filtered alert information.

[0438] Step 5: Sentiment Analysis

[0439] The server then analyzes the received video and audio data again and uses an emotion engine to recognize the user's emotional state from their voice and facial expressions. The input is real-time video and audio data, and the output is the user's emotional state (e.g., high stress, calm).

[0440] The server makes adjustments based on the user's emotional state that affect the countermeasures generated by the generation AI. The input is the user's emotional state, and the output is the adjusted countermeasures.

[0441] Step 6: Countermeasure Generation

[0442] The server uses a generative AI to generate countermeasures based on the detected anomalies and the user's emotional state. The generative AI inputs a prompt statement and outputs an optimal countermeasure. The inputs are the prompt statement and the generative AI model, and the output is a countermeasure statement.

[0443] Example: Enter the prompt text "Bears are appearing. What measures are needed?" into the generation AI.

[0444] Step 7: Notification of measures

[0445] The terminal receives countermeasure information from the server and notifies the user. Notification methods include terminal applications, browsers, and audio alarms. The input is countermeasure information from the server, and the output is a notification to the user.

[0446] The device notifies the user of the details of the countermeasures and the implementation procedure, and provides specific instructions for the user to take action. The input is the countermeasure proposal, and the output is specific instructions to the user.

[0447] Step 8: Take action

[0448] The user checks the notified countermeasures and takes action according to the instructions, such as evacuating, reporting, or installing physical barriers. The input is the countermeasure proposal, and the output is the user's action.

[0449] The terminal executes automated countermeasures. For example, the system automatically activates an audio alarm to prevent bears from entering. The input is the countermeasure proposal, and the output is the automatically executed action.

[0450] Step 9: Report Processing

[0451] If a serious abnormality occurs, the server automatically notifies security companies and public institutions. The input is the alarm information of the serious abnormality, and the output is the content of the notification (e.g., the type of organism detected, its location, and an overview of countermeasures).

[0452] The server monitors the progress of the notification and provides additional information as needed. The input is progress feedback and the output is additional information.

[0453] Each processing step of this system detects abnormalities in real time, such as dangerous animals, pests, and invasive species, and generates and implements appropriate countermeasures tailored to the user's emotional state, thereby ensuring safety while reducing the user's psychological burden.

[0454] (Application example 2)

[0455] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0456] Conventional security and surveillance systems have limited capabilities for detecting the intrusion of dangerous animals, pests, and invasive species, making it difficult to quickly implement appropriate countermeasures. Furthermore, they provide countermeasures without taking into account the user's emotional state, which increases the psychological burden on the user. Furthermore, even when an abnormality is detected, the user may not receive an appropriate notification, or the reporting to public authorities may be delayed.

[0457] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time, means for analyzing the acquired video data and audio data to detect dangerous animals, pests, and invasive species, means for generating countermeasures using a generative model based on the type and intrusion location of the detected species, means for notifying the user of the generated countermeasures and, if necessary, reporting them to public institutions, means for analyzing the user's emotional state, and means for adjusting the generated countermeasures based on the user's emotions. This makes it possible to quickly detect the intrusion of dangerous animals, pests, and invasive species, provide appropriate countermeasures adapted to the user's emotional state, and reduce psychological burden. Furthermore, since anomalies are immediately notified and public institutions are promptly notified when they are detected, greater safety can be achieved.

[0458] A "surveillance camera" is a device that monitors a designated area and captures high-resolution video data.

[0459] A "sound collection microphone" is a device that collects sounds within a specified area and records them as audio data.

[0460] "Real-time" refers to acquisition and analysis occurring immediately, without delay.

[0461] An "anomaly" refers to an unusual phenomenon detected by the system, specifically the intrusion of dangerous animals, pests, or invasive species.

[0462] "Generative model" refers to the process of using pre-trained machine learning algorithms to analyze data and generate appropriate countermeasures.

[0463] "Countermeasures" are specific instructions for actions or precautions that can be taken in response to detected abnormalities.

[0464] "Notification" is the act of informing the user of a detected anomaly and the countermeasures that have been created.

[0465] "Public institutions" refers to public safety agencies such as police and fire departments.

[0466] "Emotional state" refers to the user's psychological usage status, particularly stress and sense of security.

[0467] "Adjustment" is the process of changing the content and urgency of the generated measures based on the user's emotional state.

[0468] A "buffer" is a memory area that temporarily stores data.

[0469] An "alarm" is a warning signal issued when an abnormality is detected.

[0470] A "terminal" is a device that temporarily stores collected data and transmits it to a server.

[0471] The present invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of animals, pests, and invasive species in real time, and then uses AI to generate and propose countermeasures appropriate to the situation. Furthermore, this system incorporates an emotion engine that analyzes the user's emotional state and optimizes the countermeasures. A specific embodiment of this system is described below.

[0472] System configuration

[0473] 1. Surveillance cameras

[0474] The surveillance cameras periodically monitor designated areas, capturing high-resolution video data 24 hours a day, day or night.

[0475] 2. Sound collection microphone

[0476] The microphones collect sounds within a designated area and record them as audio data, including animal calls and the buzzing of pests.

[0477] 3. Terminal

[0478] The device temporarily stores data from the surveillance camera and microphone in a buffer and then transmits it to the server. The data transmission is encrypted and secure.

[0479] 4. Server

[0480] The server receives and analyzes the video and audio data sent from the device. A pre-trained machine learning model is used for the analysis. For example, a deep learning model such as TensorFlow or Keras is used.

[0481] 5. Generation AI

[0482] The generative AI generates appropriate countermeasures based on the anomalies detected by the server. The countermeasures are proposed as specific instructions for action or precautions. The generative AI model generates countermeasures based on the prompt sentences entered by the user into the system.

[0483] 6. Emotion Engine

[0484] The emotion engine analyzes the user's voice and facial expressions to recognize the user's emotional state in real time. Based on this information, the generative AI adjusts countermeasures to reduce the user's psychological burden.

[0485] 7. Notification System

[0486] Users will be notified of the measures via their smartphone or head-mounted display, and in the event of an emergency, public authorities will also be notified automatically.

[0487] Program processing explanation

[0488] Data collection:

[0489] Video data captured by the surveillance camera and audio data collected by the microphone are temporarily stored in a buffer on the device, and then encrypted and securely transmitted to the server.

[0490] Data Analysis:

[0491] The server analyzes the received video and audio data. The required hardware is a server equipped with a high-performance GPU (e.g., an NVIDIA GPU). Deep learning frameworks such as TensorFlow and Keras are used for analysis. The machine learning model identifies the characteristics of animals, pests, and invasive species with high accuracy.

[0492] Countermeasures generated:

[0493] Based on the analysis results, the AI ​​generates countermeasures using prompt sentences. For example, if the server detects a bear, it will suggest a countermeasure such as "A bear has been spotted. Please evacuate immediately."

[0494] Emotion analysis:

[0495] The emotion engine recognizes the user's tone of voice and facial expressions to gauge stress levels, for example, by using the EmotionRecognition library to detect emotional changes in real time.

[0496] notification:

[0497] The derived countermeasures are notified to the user via a smartphone or head-mounted display. If the user is in a high-stress state, a more urgent notification method is selected. If necessary, a report is also sent to public institutions.

[0498] Examples of concrete examples and prompts

[0499] Bear Intrusion Detection:

[0500] If a bear is detected on a surveillance camera, the AI ​​will generate a countermeasure such as, "A bear has been detected. Please evacuate immediately and notify public authorities." The emotion engine analyzes the user's voice and facial expression, and if it determines that the user is under high stress, it emphasizes the urgency of the countermeasure.

[0501] Example prompt sentence:

[0502] "A large animal is visible on the camera footage, is it a bear?"

[0503] "A bear has been detected. The user's emotional state is high stress. What is the best course of action in this case?"

[0504] "We have a huge population of mosquitoes and they don't seem to be stressed. What mitigation measures should we take in this case?"

[0505] By implementing this invention, it is possible to detect the invasion of abnormal animals, pests, and invasive species at an early stage, provide appropriate countermeasures according to the user's emotional state, and create an environment where users can feel safe.

[0506] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0507] Step 1:

[0508] Data is collected from surveillance cameras and microphones.

[0509] Input: Video data from a surveillance camera and audio data from a microphone.

[0510] Output: The raw data collected.

[0511] Specific operation: Surveillance cameras and microphones monitor designated areas and capture video and audio data in real time.

[0512] Step 2:

[0513] The data is temporarily stored in a buffer on the device and then sent to the server.

[0514] Input: Raw data collected.

[0515] Output: The encrypted data sent to the server.

[0516] Specific operation: The terminal temporarily stores the video and audio data acquired in real time in a buffer, encrypts the data, and sends it to the server.

[0517] Step 3:

[0518] The data received by the server is analyzed.

[0519] Input: Video and audio data sent to the server.

[0520] Output: Analyzed data (anomaly detection results).

[0521] Specific operation: The server analyzes the received video data frame by frame using OpenCV and the audio data using PyAudio. It then uses machine learning models such as TensorFlow and Keras to detect anomalies (animals, pests, invasive species) from the data.

[0522] Step 4:

[0523] Generative AI generates countermeasures based on detected anomalies.

[0524] Input: Analyzed data (anomaly detection results).

[0525] Output: Generated countermeasures.

[0526] Specific operation: The server's generation AI generates appropriate countermeasures based on the anomaly detection results. For example, if a bear is detected, it will suggest a countermeasure such as "A bear has been detected. Please evacuate immediately."

[0527] Step 5:

[0528] An emotion engine analyzes the user's emotional state.

[0529] Input: User's voice or facial video data.

[0530] Output: User emotional state analysis results.

[0531] Specific operation: The emotion engine uses the EmotionRecognition library to analyze the user's tone of voice and facial expressions in real time, and measures psychological states such as stress and relief.

[0532] Step 6:

[0533] The generated countermeasures are adjusted based on the user's emotional state.

[0534] Input: Generated countermeasures, analysis results of the user's emotional state.

[0535] Output: Adjusted countermeasures.

[0536] Specific operation: The generation AI adjusts the content and urgency of the proposed measures according to the user's emotional state. For example, if the user is in a high-stress state, the notification will be more urgent.

[0537] Step 7:

[0538] Inform users of the adjusted measures and, if necessary, notify public authorities.

[0539] Input: Adjusted countermeasures.

[0540] Output: Notify user, notify public authorities.

[0541] Specific operation: The notification system notifies the user of the adjusted measures via smartphone or head-mounted display. In addition, if the abnormality level is high, an automatic report will be sent to public authorities.

[0542] Step 8:

[0543] The user acts on the notified measures.

[0544] Input: The proposed solution communicated to the user.

[0545] Output: User actions.

[0546] Specific actions: The user takes immediate action based on the notified countermeasures, such as evacuating, closing windows and doors, or contacting a professional.

[0547] Step 9:

[0548] The server collects feedback on the actions taken and uses this to improve the system.

[0549] Input: User behavior data, results of countermeasure implementation.

[0550] Output: Improved system parameters.

[0551] Specific operation: The server collects user behavior data and the results of countermeasure implementation, and uses the feedback data to improve the accuracy and optimization of the system.

[0552] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0553] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0554] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0555] [Second embodiment]

[0556] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0557] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0558] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0559] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0560] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0561] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0562] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0563] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0564] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0565] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0566] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0567] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0568] This invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of dangerous animals, pests, and invasive species in real time, and then uses AI to propose countermeasures appropriate to the situation. The specific form and operation of this system are described below.

[0569] System configuration

[0570] 1. Surveillance cameras

[0571] The surveillance cameras monitor designated areas and capture high-resolution video data. All-weather cameras that can be used day and night are recommended.

[0572] 2. Sound collection microphone

[0573] The sound-collecting microphone collects sounds within a designated area and records them as audio data, allowing you to capture the sounds of animals and pests.

[0574] 3. Terminal

[0575] The terminal temporarily stores data from the surveillance camera and sound-collecting microphone and transmits it to the server.

[0576] 4. Server

[0577] The server analyzes the received video and audio data in real time to detect anomalies, using machine learning models.

[0578] 5. Generation AI

[0579] The generative AI generates appropriate countermeasures based on the detected anomalies, notifies the user of the countermeasures, and automatically takes action as needed.

[0580] 6. Notification System

[0581] Terminal applications and browsers will be provided to notify users of the measures, and a function to notify public authorities in the event of an emergency will also be included.

[0582] Program processing explanation

[0583] Data collection and transmission

[0584] The server periodically acquires video and audio data in real time from the surveillance cameras and microphones.

[0585] The device stores the acquired data in a buffer and prepares it for transmission to the server, after which the data is sent to the server.

[0586] Data analysis

[0587] The server immediately analyzes the received video and audio data: video data is analyzed frame by frame, and audio data is analyzed based on the sampling rate.

[0588] The server uses machine learning models to identify patterns of specific animals, pests and invasive species to detect intrusions.

[0589] Real-time detection and notification

[0590] The server uses the analysis results to detect anomalies, evaluates specific criteria, and issues an alert if an anomaly is detected.

[0591] The terminal receives the alarm information from the server and notifies the user.

[0592] Countermeasure generation

[0593] The server uses a generation AI to generate appropriate countermeasures, including specific instructions and precautions.

[0594] The terminal notifies the user of the generated measures and displays them on the screen.

[0595] Reporting and Response

[0596] Users are then advised to take action in accordance with the measures they are notified of. In addition, if a serious abnormality occurs, the server will automatically notify public authorities.

[0597] The device supports user actions and performs some actions automatically (e.g., sound alarm activation).

[0598] Specific examples

[0599] Example 1: Bear invasion

[0600] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[0601] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[0602] The server uses a generation AI to generate and notify the user of countermeasures such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated."

[0603] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[0604] Example 2: Mosquito outbreak

[0605] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[0606] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[0607] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[0608] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[0609] With the above configuration and processing, the present invention can detect the intrusion of dangerous animals, pests, and invasive species in real time and provide quick and effective countermeasures, thereby preventing damage and improving the efficiency of security operations.

[0610] The processing flow will be explained below.

[0611] Step 1: Data collection

[0612] The server receives video and audio data from the surveillance cameras and microphones in real time, allowing it to record all events that occur within the surveillance area.

[0613] The device temporarily stores the captured video and audio data in a buffer, minimizing the risk of data loss.

[0614] Step 2: Send the data

[0615] The device prepares the buffered data for transmission to the server. The data is converted to a certain format (e.g. MP4, WAV, etc.).

[0616] The device sends the prepared data to the server via the network, where it is encrypted and transmitted to ensure security.

[0617] Step 3: Begin data analysis

[0618] The server immediately begins the process of analyzing the received video and audio data.

[0619] The server analyzes and compares the video data frame by frame to identify animals and pests.

[0620] The server analyzes the audio data based on the sampling rate to see if certain audio patterns are present.

[0621] Step 4: Identify specific organisms

[0622] The server uses pre-trained machine learning models to identify specific animals, pests, and invasive species from the data it receives with a high degree of accuracy.

[0623] The server determines the species, location, and behavioral patterns of the identified creatures.

[0624] Step 5: Real-time detection

[0625] The server uses the analysis results to evaluate certain criteria (e.g., animal size, behavior, location of intrusion, etc.) and determine whether an anomaly has been detected.

[0626] If the server detects an anomaly, it will immediately issue an alert, which will be filtered and different actions will be taken depending on the severity.

[0627] Step 6: Notification

[0628] The server sends the alarm information to the terminal so that the user can check it. The alarm includes information on the location, time, and target organism.

[0629] When the terminal receives an alert, it notifies the user with a sound or a pop-up message.

[0630] Step 7: Countermeasure generation

[0631] The server uses generative AI to generate appropriate countermeasures based on the detected anomalies, including specific instructions and precautions.

[0632] The server sends the generated countermeasures to the terminal in a format that is easy for the user to understand.

[0633] Step 8: Review and implement measures

[0634] The device displays the generated countermeasure information to the user, who can then check the countermeasure proposal and prepare to implement it if necessary.

[0635] The user implements the instructed measures and takes the actions recommended by the system, and the system automatically performs some actions (e.g., sounding an audio alarm).

[0636] Step 9: Report

[0637] If a serious abnormality occurs, the server automatically reports the situation to security companies and public institutions, including the type of organism detected, its location, and a summary of the countermeasures taken.

[0638] The server monitors the progress of the notification and provides additional information as needed.

[0639] Through the above processing steps, the system of the present invention can quickly detect abnormalities within the monitored area and propose and implement appropriate countermeasures, thereby preventing damage from dangerous animals, pests, and invasive species.

[0640] Example 1

[0641] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0642] In modern society, the invasion of dangerous animals, pests, and invasive species is increasingly threatening people's daily lives. It is necessary to quickly and effectively detect the invasion of such organisms and promptly implement appropriate countermeasures. However, conventional systems have had difficulty detecting anomalies in real time or automatically generating and implementing appropriate countermeasures. Furthermore, they lacked sufficient support functions to help users take appropriate measures, making it difficult to prevent damage before it occurs.

[0643] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0644] In this invention, the server includes: means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time; means for analyzing the acquired video data and audio data to detect dangerous organisms; means for generating countermeasures using a generation AI based on the type of organism detected and the location of the intrusion; means for notifying the user of the generated countermeasures and, if necessary, notifying public authorities; and means for automatically executing action as necessary to support the user's actions. This makes it possible to detect the intrusion of dangerous organisms in real time and provide quick and effective countermeasures. Support functions for users to take appropriate actions are also enhanced, preventing damage before it occurs.

[0645] A "surveillance camera" is a device that continuously monitors a specific area and captures high-resolution video data.

[0646] A "sound collection microphone" is a device that collects surrounding sounds in real time and records and transmits them as audio data.

[0647] "Video data" refers to digital data of visual information captured by a surveillance camera.

[0648] "Audio data" is digital data of auditory information acquired by a sound-collecting microphone.

[0649] A "server" is a computer system that executes a series of processes such as data collection, analysis, countermeasure generation, and notification.

[0650] "Generative AI" is an artificial intelligence model that automatically generates appropriate countermeasures for users.

[0651] "Countermeasures" are specific measures such as instructions for actions or precautions to be taken in response to detected abnormalities.

[0652] "User" means an individual or organization that uses the system and acts on the measures provided.

[0653] "Public institutions" are organizations such as governments and administrative agencies that are required to report any abnormalities detected.

[0654] A "buffer" is a memory area for temporarily storing data.

[0655] An "alarm" is a warning signal such as sound or light that notifies the user of an abnormality.

[0656] An "action" is an operation or behavior that the system automatically performs (e.g., activating an audio alarm).

[0657] The present invention is a system that uses a surveillance camera and a sound-collecting microphone to detect the intrusion of dangerous organisms in real time, and uses a generating AI to propose countermeasures according to the situation. Detailed embodiments for carrying out the present invention will be described below.

[0658] System configuration

[0659] 1. Surveillance cameras

[0660] The surveillance cameras continuously monitor designated areas and capture high-resolution video data. All-weather cameras are recommended, and include high-resolution CCD cameras and infrared cameras.

[0661] 2. Sound collection microphone

[0662] Sound-collecting microphones collect surrounding sounds and record them as audio data. This makes it possible to capture the sounds of animals and pests. Highly directional microphones and dynamic microphones that are resistant to environmental sounds are used.

[0663] 3. Terminal

[0664] The device temporarily stores data from the surveillance camera and microphone, stores it in a buffer, and then transmits it to the server. The device is preferably a computer or dedicated device with a high-speed processor and sufficient memory.

[0665] 4. Server

[0666] The server analyzes the received video and audio data in real time to detect anomalies. This analysis is performed using machine learning models, such as deep learning frameworks like TensorFlow and PyTorch.

[0667] 5. Generation AI

[0668] The generative AI generates appropriate countermeasures based on the detected anomalies. A natural language generation model (e.g., GPT-4) is used for generation. The generated countermeasures are notified to the user, and automatic action is taken if necessary.

[0669] 6. Notification System

[0670] The notification system will include a terminal application and browser to notify users of countermeasures, and will also have a function to notify public authorities in the event of an emergency.

[0671] Specific examples

[0672] Example 1: Bear invasion

[0673] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[0674] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[0675] The server uses a generation AI to generate and notify the user of countermeasures such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated."

[0676] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[0677] Example 2: Mosquito outbreak

[0678] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[0679] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[0680] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[0681] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[0682] Prompt Sentence Examples

[0683] Example 1: Bear invasion

[0684] "A bear has been detected in security camera footage. Please generate a measure to notify the user."

[0685] Example 2: Mosquito outbreak

[0686] "A large number of mosquito sounds have been detected from the microphone. Please generate a measure to notify the user."

[0687] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0688] Program processing flow

[0689] Step 1: Collect data

[0690] Input: Real-time video and audio data from surveillance cameras and microphones

[0691] Output: Video and audio data temporarily stored in the buffer

[0692] How it works: The server collects real-time data from a specified area via surveillance cameras and microphones. The day and night, all-weather cameras capture high-resolution images, while the microphones collect ambient sounds. This data is first temporarily stored in a buffer on the device.

[0693] Step 2: Sending data

[0694] Input: Video and audio data stored in the buffer

[0695] Output: Video and audio data sent to the server

[0696] What it does: The device periodically checks the data stored in the buffer, and if new data is available, it sends it to the server using a secure protocol (e.g., HTTPS).

[0697] Step 3: Analyze the data

[0698] Input: Video and audio data sent to the server

[0699] Output: Anomaly detection information as analysis results

[0700] Specific operation: The server analyzes the received video and audio data using a machine learning model. Feature extraction and recognition are performed on each frame of the video data using OpenCV, TensorFlow, etc. Audio data is analyzed based on the sampling rate, and the frequency spectrum is analyzed using FFT.

[0701] Step 4: Real-time anomaly detection

[0702] Input: Parsed data

[0703] Output: Issue an alert

[0704] Specific operation: The server detects anomalies based on the analysis results and issues an alert if certain criteria are met. For example, if a specific animal is seen in the video or a sound of a specific frequency is detected, it will recognize it as an anomaly and issue an alert. These criteria include thresholds and the results of pattern recognition.

[0705] Step 5: Generate countermeasures

[0706] Input: Anomaly detection information

[0707] Output: Generated countermeasure statement

[0708] Specific operation: The server uses a generative AI model (e.g., GPT-4) to generate appropriate countermeasures based on the anomaly detection information. The prompt contains detailed information about the current abnormal situation, and the generative AI model generates countermeasures based on this.

[0709] Step 6: Notification and response support

[0710] Input: Generated countermeasures

[0711] Output: User notification and automatic actions

[0712] Specific operations: The device notifies the user of the alert from the server and the generated countermeasures. The notification is displayed via a smartphone app or web browser. The notification contains specific response instructions that the user can act on. Furthermore, if a serious abnormality is detected, the server automatically notifies public authorities. Some actions (such as sounding an audio alarm) are also performed automatically, helping the user to take immediate action.

[0713] Specific examples

[0714] Bear invasion

[0715] The server analyzes video data from surveillance cameras, and if it detects a large animal, it identifies it as a bear based on its characteristics.

[0716] The server issues a bear intrusion warning and sends a countermeasure message to the device.

[0717] The generative AI model is given a prompt message: "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated.", and a countermeasure message is generated.

[0718] The terminal notifies the user of the generated countermeasure statement, and the system automatically activates an audio alarm if necessary.

[0719] Massive mosquito outbreaks

[0720] The server analyzes the audio data from the microphone and detects the sounds of large numbers of mosquitoes.

[0721] The server issues a mosquito outbreak warning and sends a countermeasure message to the terminal.

[0722] The generative AI model is given a prompt message: "There is a mosquito infestation. Please use insect repellent and close windows and doors. Also, contact a professional if necessary.", and a countermeasure message is generated.

[0723] The terminal notifies the user of the generated countermeasure statement and assists the user in carrying out the countermeasure based on the instruction.

[0724] (Application example 1)

[0725] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0726] Conventional security systems have difficulty detecting the intrusion of dangerous animals, pests, and invasive species in real time, often resulting in delayed implementation of appropriate countermeasures. Furthermore, they lack the means to provide users with quick and specific instructions in emergencies, increasing the risk of damage spreading. The present invention aims to solve these issues by providing a system that uses surveillance cameras and sound-collecting microphones to efficiently detect the intrusion of living organisms and uses generative AI to propose appropriate countermeasures in real time.

[0727] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0728] In this invention, the server includes means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time, means for analyzing the acquired video data and audio data to detect dangerous animals, pests, and invasive species, means for generating countermeasures using a generation AI based on the type of organism detected and the location of its intrusion, means for notifying a user of the generated countermeasures and reporting them to a public institution as necessary, and means for a user to check and implement the countermeasures using a smartphone, thereby enabling efficient and specific countermeasures to be provided in real time.

[0729] A "surveillance camera" is a device used to acquire high-resolution video data in real time and analyze that data.

[0730] A "sound collection microphone" is a device that collects sound data within a designated area and captures the sounds of animals and pests.

[0731] A "server" is a computer system that implements a machine learning model to analyze received video and audio data in real time and detect anomalies.

[0732] "Generative AI" is an artificial intelligence model that generates appropriate countermeasures based on detected anomalies.

[0733] A "buffer" is a memory system that contains an area for temporarily storing data and sending it on to the next processing step.

[0734] "Alert" is an alert system that issues a warning to the user when an abnormality is detected.

[0735] A "smartphone" is a mobile information terminal used by a user to receive notifications and to check and take countermeasures.

[0736] A "prompt" is textual input data that gives specific instructions or questions to a generative AI model.

[0737] The present invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of dangerous animals, pests, and invasive species in real time, generates countermeasures using generation AI, and notifies the user. This system is composed of surveillance cameras, sound-collecting microphones, a server, and a smartphone app. By combining these, it is possible to safely and quickly detect abnormalities and take appropriate countermeasures. Specific embodiments are described below.

[0738] Data collection and transmission

[0739] First, a surveillance camera and a sound-collecting microphone monitor a designated area, capturing high-resolution video and audio data in real time. The surveillance camera must be high-resolution and usable day and night, while the sound-collecting microphone must be highly sensitive.

[0740] The acquired data is temporarily stored in a terminal (e.g., a server) and then transmitted to the server. The terminal also includes a buffer for temporarily storing the data.

[0741] Data analysis

[0742] The server uses machine learning models to analyze the received video and audio data in real time: TensorFlow is used to analyze the video data, and the Librosa library is used to analyze the audio data.

[0743] The analyzed data is evaluated based on specific criteria, and if an anomaly is detected, an alert is issued, which is sent directly to the user via a smartphone app.

[0744] Real-time detection and notification

[0745] The smartphone app receives the alert information from the server and notifies the user. This notification system uses Firebase Cloud Messaging. The user can check the alert details and take action on the smartphone app.

[0746] Countermeasure generation

[0747] Generative AI is used to generate appropriate countermeasures. Specific countermeasure proposals for detected anomalies are generated using generative AI models such as GPT-4. The generated countermeasure proposals are provided to users via a smartphone app.

[0748] For example, if a snake is detected, the AI ​​will generate a countermeasure suggestion such as, "A snake has been found. Please keep as far away as possible and contact a specialist." An example of a prompt would be, "A snake has been detected through analysis of surveillance camera footage. Please generate appropriate response procedures."

[0749] Communication and Action

[0750] When the user acts in accordance with the generated countermeasures, the smartphone app also has the function of automatically executing some actions, such as activating an audio alarm.

[0751] As described above, this system combines various sensors and advanced analysis technology to detect danger in real time and provide prompt and appropriate countermeasures, thereby ensuring the safety of users and preventing damage before it occurs.

[0752] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0753] Step 1:

[0754] Surveillance cameras and microphones monitor designated areas, capturing high-resolution video and audio data in real time.

[0755] Input: Video data from surveillance cameras, audio data from microphones

[0756] Output: High-resolution video and audio data

[0757] How it works: The surveillance camera captures an entire area and records the footage in high resolution, while the sound pickup microphone collects surrounding sounds and stores them as digital audio data.

[0758] Step 2:

[0759] The video and audio data acquired by the terminal is temporarily stored in a buffer.

[0760] Input: High-resolution video and audio data

[0761] Output: Temporary data stored in a buffer

[0762] Specific operation: The terminal stores the data collected in real time in a buffer (temporary storage area) and prepares it for subsequent processing.

[0763] Step 3:

[0764] Send data from the terminal to the server.

[0765] Input: Temporary data stored in a buffer

[0766] Output: Video and audio data sent to the server

[0767] Specific operation: The device executes a process to send data to a server via the Internet.

[0768] Step 4:

[0769] The video data received by the server is analyzed using TensorFlow, and the audio data is analyzed using the Librosa library.

[0770] Input: Video and audio data sent to the server

[0771] Output: Analysis results (detected anomalies and patterns)

[0772] How it works: The server breaks down the video data into frames and performs image analysis using TensorFlow. The audio data is analyzed using the Librosa library based on the sampling rate to detect abnormal audio patterns.

[0773] Step 5:

[0774] The server detects anomalies based on the analysis results and issues an alert if certain criteria are met.

[0775] Input: Analysis results

[0776] Output: Warning information

[0777] Specific operation: The server evaluates the analysis results and generates an alert if dangerous animals, pests, or invasive species are detected.

[0778] Step 6:

[0779] The server uses the generated AI model to generate specific countermeasures for the detected anomalies.

[0780] Input: Alarm information

[0781] Output: Generated countermeasures

[0782] Specific operation: The generative AI model generates appropriate countermeasures based on the prompt. For example, a prompt such as "A snake has been detected during surveillance camera video analysis. Please generate appropriate response procedures."

[0783] Step 7:

[0784] The server notifies the user of the generated countermeasures via a smartphone app.

[0785] Input: Generated countermeasures

[0786] Output: Notification displayed on smartphone app

[0787] Specific operation: Using Firebase Cloud Messaging, the generated countermeasures are sent to the smartphone app and the user is notified.

[0788] Step 8:

[0789] Users can check the countermeasures and take action using their smartphones.

[0790] Input: Notification displayed on smartphone

[0791] Output: User action

[0792] Specific action: The user acknowledges the notification and takes appropriate action, for example, following the instructions, "A snake has been found. Move away as far away as possible and contact a professional."

[0793] Step 9:

[0794] The system will automatically take some action (e.g. activate an audio alarm) if necessary.

[0795] Input: User confirmation and warning information

[0796] Output: Actions taken

[0797] Specific operation: The system automatically activates an audio alarm to warn those around.

[0798] The above steps realize a real-time danger detection system using a surveillance camera and a sound-collecting microphone.

[0799] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0800] This invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of dangerous animals, pests, and invasive species in real time, and uses AI to generate and propose countermeasures appropriate to the situation. It also incorporates an emotion engine that recognizes the user's emotions. The system aims to optimize the countermeasures based on the user's emotional state and reduce the user's psychological burden.

[0801] System configuration

[0802] 1. Surveillance cameras

[0803] The surveillance cameras monitor designated areas and capture high-resolution video data, allowing for day and night surveillance.

[0804] 2. Sound collection microphone

[0805] The sound-collecting microphone collects sounds within a designated area and records them as audio data, allowing you to capture the sounds of animals and pests.

[0806] 3. Terminal

[0807] The terminal temporarily stores data from the surveillance camera and microphone and transmits it to the server. Data can be transmitted in real time.

[0808] 4. Server

[0809] The server analyzes the received video and audio data in real time to detect anomalies, using a pre-trained machine learning model.

[0810] 5. Generation AI

[0811] The generative AI generates appropriate countermeasures based on the detected anomalies, notifies the user of the countermeasures, and automatically takes action as needed.

[0812] 6. Emotion Engine

[0813] The emotion engine analyzes the user's voice and facial expressions to recognize their emotions and adjusts the content of the countermeasures based on that information. The urgency and level of detail of the countermeasures can be changed based on the recognized emotional state.

[0814] 7. Notification System

[0815] Terminal applications and browsers will be provided to notify users of the measures, and a function to notify public authorities in the event of an emergency will also be included.

[0816] Program processing explanation

[0817] Data collection and transmission

[0818] The server periodically acquires real-time video and audio data from the surveillance cameras and microphones, thereby recording all events that occur within the surveillance area.

[0819] The device stores the captured video and audio data in a buffer and prepares it for transmission to the server, where it is then encrypted and sent to the server.

[0820] Start of data analysis

[0821] The server immediately begins the process of analyzing the received video and audio data: video data is analyzed frame by frame, and audio data is analyzed based on the sampling rate.

[0822] The server uses pre-trained machine learning models to identify specific animals, pests, and invasive species from the data it receives with a high degree of accuracy.

[0823] Real-time detection and notification

[0824] The server evaluates certain criteria based on the analysis results and issues alerts if anomalies are detected. Alerts are filtered and different actions are taken depending on their severity.

[0825] The terminal receives the alarm information from the server and notifies the user. The alarm includes information on the location and time of occurrence, and the target organism.

[0826] Countermeasure generation and adjustment

[0827] The server uses generative AI to generate appropriate countermeasures based on the detected anomalies, including specific instructions and precautions.

[0828] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state. For example, if the user is in a high stress state, the generative AI will adjust the urgency and level of detail of the countermeasures.

[0829] Check and implement measures

[0830] The device displays the generated countermeasure information to the user, who can then check the countermeasure proposal and prepare to implement it if necessary.

[0831] The user implements the instructed measures and takes the actions recommended by the system, and the system automatically performs some actions (e.g., sounding an audio alarm).

[0832] Report

[0833] If a serious abnormality occurs, the server automatically reports the situation to security companies and public institutions, including the type of organism detected, its location, and a summary of the countermeasures taken.

[0834] The server monitors the progress of the notification and provides additional information as needed.

[0835] Specific examples

[0836] Example 1: Bear invasion

[0837] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[0838] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[0839] The emotion engine analyzes the user's facial expressions and tone of voice to recognize when the user is in a state of high stress.

[0840] The server uses a generation AI to generate and notify the user of a countermeasure such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated." The server also emphasizes the urgency of the countermeasure, as the user is in a high-stress state.

[0841] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[0842] Example 2: Mosquito outbreak

[0843] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[0844] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[0845] The emotion engine analyzes the user's emotional state and recognizes that they are not particularly stressed.

[0846] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[0847] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[0848] With the above configuration and processing, the present invention can quickly detect abnormalities within a monitored area and provide and implement appropriate countermeasures that are adapted to the user's emotional state, thereby preventing damage from dangerous animals, pests, and invasive species and reducing the psychological burden on the user.

[0849] The processing flow will be explained below.

[0850] Step 1: Data collection

[0851] The server receives video and audio data from the surveillance cameras and microphones in real time, ensuring that all events occurring within the surveillance area are accurately recorded.

[0852] The device temporarily stores the acquired video and audio data in a buffer, and the data is periodically accumulated in preparation for the next processing.

[0853] Step 2: Send the data

[0854] The device prepares the data stored in the buffer for transmission to the server. The data is converted into a predetermined format.

[0855] The device encrypts the prepared data and sends it over the network to the server, ensuring data security.

[0856] Step 3: Begin data analysis

[0857] The server starts analyzing the received video and audio data, where the video data is analyzed frame by frame and the audio data is analyzed based on the sampling rate.

[0858] The server uses pre-trained machine learning models to identify specific animals, pests and invasive species from the data it receives.

[0859] Step 4: Identify specific organisms

[0860] The server uses machine learning models to identify animal and pest patterns in the data, for example detecting the outlines of specific animals in video data and recognizing specific animal sounds in audio data.

[0861] The server determines the type, location, and behavioral patterns of the identified organisms and evaluates whether there are any abnormalities.

[0862] Step 5: Real-time detection

[0863] The server evaluates the analysis results and issues an alert if an abnormality is detected, based on specific criteria such as the size, behavioral patterns, and location of the animal.

[0864] Based on the detected abnormality information, the server determines the importance of the alert and selects the appropriate response.

[0865] Step 6: Notification

[0866] The server sends the alert information to the terminal for the user to view. The alert includes details of the location and time of the alert and the detected organism, allowing the user to take prompt action.

[0867] The terminal notifies the user of the received warning information, and the user receives the warning through voice or pop-up message.

[0868] Step 7: Countermeasure generation and sentiment analysis

[0869] The server uses a generative AI to generate appropriate countermeasures for the detected anomalies, creating countermeasure proposals that include specific instructions for action and precautions.

[0870] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state, for example, determining their stress level based on their voice tone and facial expression.

[0871] The server adjusts the countermeasures generated by the generative AI based on the recognized emotional state, providing more detailed and urgent countermeasures to users in high stress states.

[0872] Step 8: Review and implement measures

[0873] The device displays the adjusted countermeasure information to the user, who then checks the countermeasure proposal and takes necessary measures.

[0874] The user initiates action based on the displayed countermeasures. If the system automatically performs some action (e.g., activates an audio alarm), the user confirms this and takes the necessary steps.

[0875] Step 9: Report

[0876] If a serious abnormality occurs, the server automatically reports the situation to security companies and public institutions, including the type of organism detected, its location, and a summary of the countermeasures taken.

[0877] The server monitors the progress of the report and provides additional information as needed, enabling rapid coordination with relevant authorities.

[0878] Specific examples

[0879] Example 1: Bear invasion

[0880] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[0881] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[0882] The emotion engine analyzes the user's facial expressions and tone of voice to recognize when the user is in a state of high stress.

[0883] The server uses a generation AI to generate and notify the user of a countermeasure such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated." The server also emphasizes the urgency of the countermeasure, as the user is in a high-stress state.

[0884] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[0885] Example 2: Mosquito outbreak

[0886] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[0887] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[0888] The emotion engine analyzes the user's emotional state and recognizes that they are not particularly stressed.

[0889] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[0890] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[0891] With the above specific configuration and processing, the present invention can quickly detect abnormalities within a monitored area and provide and implement appropriate countermeasures that are adapted to the user's emotional state, thereby preventing damage from dangerous animals, pests, and invasive species and reducing the psychological burden on the user.

[0892] Example 2

[0893] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0894] In recent years, the intrusion of dangerous animals, pests, and invasive species has been increasing, and the resulting damage has become more serious. There is a need for a system that can detect such intrusions in real time and respond quickly. However, current systems do not provide countermeasures that take the user's emotional state into consideration, which increases the psychological burden on the user. Therefore, the present invention aims to optimize countermeasures based on the user's emotional state and reduce the psychological burden.

[0895] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0896] In this invention, the server includes means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time, means for analyzing the acquired video and audio data to detect dangerous animals, pests, and invasive species, and an emotion engine for analyzing the user's voice and facial expressions to recognize emotions and adjust countermeasures based on the user's emotional state. This makes it possible to quickly detect abnormalities and provide appropriate countermeasures while reducing the psychological burden on the user.

[0897] A "surveillance camera" is a device that monitors a designated area and captures high-resolution video data.

[0898] A "sound collection microphone" is a device that collects sounds within a specified area and records them as audio data.

[0899] "Video data" refers to a collection of video frames captured from a surveillance camera, which record events occurring within a surveillance area.

[0900] "Audio data" refers to a collection of sound sampling data acquired from a sound-collecting microphone, and is data that records sounds occurring within a specified area.

[0901] The "server" is a central processing unit that receives video data and audio data, analyzes them, and detects abnormalities.

[0902] The "emotion engine" is an engine that analyzes the user's voice and facial expressions to recognize their emotions, and adjusts the content of countermeasures based on that information.

[0903] "Generative AI" is an artificial intelligence model that generates appropriate countermeasures based on detected anomalies.

[0904] "Countermeasure generation" refers to the process of using generative AI to design and provide appropriate countermeasures based on the analysis results.

[0905] A "prompt" is an instruction sentence input to a generation AI, and is text containing instructions for generating a specific countermeasure.

[0906] The "notification means" is a mechanism for notifying the user of the generated countermeasures, and includes a terminal application and a browser.

[0907] "User" refers to the person who uses the system, who receives the proposed measures and implements them.

[0908] This invention is a system that detects the intrusion of dangerous animals, pests, and invasive species in real time and provides optimized countermeasures based on the user's emotional state. This system is composed of surveillance cameras, sound-collecting microphones, a server, terminals, a generation AI, an emotion engine, and a notification system.

[0909] Hardware and Software Details

[0910] Surveillance cameras and microphones

[0911] A surveillance camera is a device that monitors a designated area and captures high-resolution video data, allowing for surveillance day and night.

[0912] A sound collecting microphone is a device that collects sounds within a specified area and records them as audio data, allowing you to capture the sounds of animals and pests.

[0913] server

[0914] The server is a central processing unit that receives and analyzes data from surveillance cameras and microphones in real time. The server has the following functions:

[0915] Data collection function: Video and audio data is acquired in real time and temporarily stored.

[0916] Data analysis function: Analyzes received video and audio data to detect anomalies. This analysis is performed using a pre-trained machine learning model.

[0917] Emotion analysis function: Analyzes the user's voice and facial expressions to recognize their emotional state. An emotion prediction model is used to assess whether the user is in a high-stress or calm state.

[0918] Countermeasure generation function: Using a generation AI, a countermeasure is generated based on the detected anomaly. A prompt sentence is input to the generation AI, and the optimal countermeasure is output.

[0919] Terminal

[0920] The terminal is a device that temporarily stores data from the surveillance camera and microphone and transmits it to the server. The terminal has the following functions:

[0921] Data transmission function: Stored data is encrypted and sent to the server using a secure communication protocol, ensuring data safety.

[0922] Notification function: Notifies users of countermeasure information and warnings from the server. Notification methods include terminal applications, browsers, and audio alarms.

[0923] Generative AI model and prompts

[0924] A generative AI model is an artificial intelligence model that generates appropriate countermeasures based on detected anomalies. A prompt sentence is input to the generative AI model, and the optimal countermeasure is output.

[0925] Example prompt: There's a bear infestation. What precautions should be taken?

[0926] Specific examples

[0927] Example 1: Bear invasion

[0928] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[0929] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[0930] The emotion engine analyzes the user's facial expressions and tone of voice to recognize when the user is in a state of high stress.

[0931] The server uses a generation AI to generate and notify the user of a countermeasure such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated." The server also emphasizes the urgency of the countermeasures, as the user is in a high-stress state.

[0932] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[0933] Example 2: Mosquito outbreak

[0934] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[0935] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[0936] The emotion engine analyzes the user's emotional state and recognizes that they are not particularly stressed.

[0937] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[0938] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[0939] In this way, the system can detect abnormalities in real time and provide countermeasures that adapt to the user's emotional state, thereby preventing damage from dangerous animals, pests, and invasive species and reducing the psychological burden on the user.

[0940] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0941] Step 1: Data collection

[0942] The server acquires video data from the surveillance cameras in real time. Specifically, it continuously acquires video data frame by frame and stores it in high resolution. The input is the video data from the surveillance cameras, and the output is the video data temporarily stored in the buffer.

[0943] The server acquires audio data from a microphone in real time based on a sampling rate. The audio data includes environmental sounds within a specified area and the sounds of specific animals. The input is the audio data from the microphone, and the output is the audio data temporarily stored in a buffer.

[0944] Step 2: Send data

[0945] The terminal temporarily stores the acquired video and audio data in a buffer. The input is the data stored in the buffer from the server, and the output is encrypted data.

[0946] The device sends encrypted data to the server using a secure communication protocol (e.g., HTTPS). This ensures data security. The input is encrypted video and audio data, and the output is the data sent to the server.

[0947] Step 3: Data analysis

[0948] The server uses a pre-trained machine learning model to analyze the received video data frame by frame. The input is the video data sent to the server, and the output is the analysis results (e.g., the presence of specific animals, pests, or invasive species).

[0949] The server samples and analyzes the audio data, similarly using pre-trained models to identify specific frequencies and patterns. The input is the audio data sent to the server, and the output is the analysis results (e.g., detection of the sounds of specific animals, pests, or invasive species).

[0950] Step 4: Anomaly detection

[0951] The server evaluates the presence of specific animals, pests, and invasive species based on the results of data analysis, and issues an alert if an abnormality is detected. The input is the data analysis results, and the output is alert information (e.g., location, time, and information about the target species).

[0952] The server filters the alerts and determines the action to be taken (e.g., prioritizing notifications, reporting) depending on the severity. The input is the alert information, and the output is the filtered alert information.

[0953] Step 5: Sentiment Analysis

[0954] The server then analyzes the received video and audio data again and uses an emotion engine to recognize the user's emotional state from their voice and facial expressions. The input is real-time video and audio data, and the output is the user's emotional state (e.g., high stress, calm).

[0955] The server makes adjustments based on the user's emotional state that affect the countermeasures generated by the generation AI. The input is the user's emotional state, and the output is the adjusted countermeasures.

[0956] Step 6: Countermeasure Generation

[0957] The server uses a generative AI to generate countermeasures based on the detected anomalies and the user's emotional state. The generative AI inputs a prompt statement and outputs an optimal countermeasure. The inputs are the prompt statement and the generative AI model, and the output is a countermeasure statement.

[0958] Example: Enter the prompt text "Bears are appearing. What measures are needed?" into the generation AI.

[0959] Step 7: Notification of measures

[0960] The terminal receives countermeasure information from the server and notifies the user. Notification methods include terminal applications, browsers, and audio alarms. The input is countermeasure information from the server, and the output is a notification to the user.

[0961] The device notifies the user of the details of the countermeasures and the implementation procedure, and provides specific instructions for the user to take action. The input is the countermeasure proposal, and the output is specific instructions to the user.

[0962] Step 8: Take action

[0963] The user checks the notified countermeasures and takes action according to the instructions, such as evacuating, reporting, or installing physical barriers. The input is the countermeasure proposal, and the output is the user's action.

[0964] The terminal executes automated countermeasures. For example, the system automatically activates an audio alarm to prevent bears from entering. The input is the countermeasure proposal, and the output is the automatically executed action.

[0965] Step 9: Report Processing

[0966] If a serious abnormality occurs, the server automatically notifies security companies and public institutions. The input is the alarm information of the serious abnormality, and the output is the content of the notification (e.g., the type of organism detected, its location, and an overview of countermeasures).

[0967] The server monitors the progress of the notification and provides additional information as needed. The input is progress feedback and the output is additional information.

[0968] Each processing step of this system detects abnormalities in real time, such as dangerous animals, pests, and invasive species, and generates and implements appropriate countermeasures tailored to the user's emotional state, thereby ensuring safety while reducing the user's psychological burden.

[0969] (Application example 2)

[0970] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0971] Conventional security and surveillance systems have limited capabilities for detecting the intrusion of dangerous animals, pests, and invasive species, making it difficult to quickly implement appropriate countermeasures. Furthermore, they provide countermeasures without taking into account the user's emotional state, which increases the psychological burden on the user. Furthermore, even when an abnormality is detected, the user may not receive an appropriate notification, or the reporting to public authorities may be delayed.

[0972] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time, means for analyzing the acquired video data and audio data to detect dangerous animals, pests, and invasive species, means for generating countermeasures using a generative model based on the type and intrusion location of the detected species, means for notifying the user of the generated countermeasures and, if necessary, reporting them to public institutions, means for analyzing the user's emotional state, and means for adjusting the generated countermeasures based on the user's emotions. This makes it possible to quickly detect the intrusion of dangerous animals, pests, and invasive species, provide appropriate countermeasures adapted to the user's emotional state, and reduce psychological burden. Furthermore, since anomalies are immediately notified and public institutions are promptly notified when they are detected, greater safety can be achieved.

[0973] A "surveillance camera" is a device that monitors a designated area and captures high-resolution video data.

[0974] A "sound collection microphone" is a device that collects sounds within a specified area and records them as audio data.

[0975] "Real-time" refers to acquisition and analysis occurring immediately, without delay.

[0976] An "anomaly" refers to an unusual phenomenon detected by the system, specifically the intrusion of dangerous animals, pests, or invasive species.

[0977] "Generative model" refers to the process of using pre-trained machine learning algorithms to analyze data and generate appropriate countermeasures.

[0978] "Countermeasures" are specific instructions for actions or precautions that can be taken in response to detected abnormalities.

[0979] "Notification" is the act of informing the user of a detected anomaly and the countermeasures that have been created.

[0980] "Public institutions" refers to public safety agencies such as police and fire departments.

[0981] "Emotional state" refers to the user's psychological usage status, particularly stress and sense of security.

[0982] "Adjustment" is the process of changing the content and urgency of the generated measures based on the user's emotional state.

[0983] A "buffer" is a memory area that temporarily stores data.

[0984] An "alarm" is a warning signal issued when an abnormality is detected.

[0985] A "terminal" is a device that temporarily stores collected data and transmits it to a server.

[0986] The present invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of animals, pests, and invasive species in real time, and then uses AI to generate and propose countermeasures appropriate to the situation. Furthermore, this system incorporates an emotion engine that analyzes the user's emotional state and optimizes the countermeasures. A specific embodiment of this system is described below.

[0987] System configuration

[0988] 1. Surveillance cameras

[0989] The surveillance cameras periodically monitor designated areas, capturing high-resolution video data 24 hours a day, day or night.

[0990] 2. Sound collection microphone

[0991] The microphones collect sounds within a designated area and record them as audio data, including animal calls and the buzzing of pests.

[0992] 3. Terminal

[0993] The device temporarily stores data from the surveillance camera and microphone in a buffer and then transmits it to the server. The data transmission is encrypted and secure.

[0994] 4. Server

[0995] The server receives and analyzes the video and audio data sent from the device. A pre-trained machine learning model is used for the analysis. For example, a deep learning model such as TensorFlow or Keras is used.

[0996] 5. Generation AI

[0997] The generative AI generates appropriate countermeasures based on the anomalies detected by the server. The countermeasures are proposed as specific instructions for action or precautions. The generative AI model generates countermeasures based on the prompt sentences entered by the user into the system.

[0998] 6. Emotion Engine

[0999] The emotion engine analyzes the user's voice and facial expressions to recognize the user's emotional state in real time. Based on this information, the generative AI adjusts countermeasures to reduce the user's psychological burden.

[1000] 7. Notification System

[1001] Users will be notified of the measures via their smartphone or head-mounted display, and in the event of an emergency, public authorities will also be notified automatically.

[1002] Program processing explanation

[1003] Data collection:

[1004] Video data captured by the surveillance camera and audio data collected by the microphone are temporarily stored in a buffer on the device, and then encrypted and securely transmitted to the server.

[1005] Data Analysis:

[1006] The server analyzes the received video and audio data. The required hardware is a server equipped with a high-performance GPU (e.g., an NVIDIA GPU). Deep learning frameworks such as TensorFlow and Keras are used for analysis. The machine learning model identifies the characteristics of animals, pests, and invasive species with high accuracy.

[1007] Countermeasures generated:

[1008] Based on the analysis results, the AI ​​generates countermeasures using prompt sentences. For example, if the server detects a bear, it will suggest a countermeasure such as "A bear has been spotted. Please evacuate immediately."

[1009] Emotion analysis:

[1010] The emotion engine recognizes the user's tone of voice and facial expressions to gauge stress levels, for example, by using the EmotionRecognition library to detect emotional changes in real time.

[1011] notification:

[1012] The derived countermeasures are notified to the user via a smartphone or head-mounted display. If the user is in a high-stress state, a more urgent notification method is selected. If necessary, a report is also sent to public institutions.

[1013] Examples of concrete examples and prompts

[1014] Bear Intrusion Detection:

[1015] If a bear is detected on a surveillance camera, the AI ​​will generate a countermeasure such as, "A bear has been detected. Please evacuate immediately and notify public authorities." The emotion engine analyzes the user's voice and facial expression, and if it determines that the user is under high stress, it emphasizes the urgency of the countermeasure.

[1016] Example prompt sentence:

[1017] "A large animal is visible on the camera footage, is it a bear?"

[1018] "A bear has been detected. The user's emotional state is high stress. What is the best course of action in this case?"

[1019] "We have a huge population of mosquitoes and they don't seem to be stressed. What mitigation measures should we take in this case?"

[1020] By implementing this invention, it is possible to detect the invasion of abnormal animals, pests, and invasive species at an early stage, provide appropriate countermeasures according to the user's emotional state, and create an environment where users can feel safe.

[1021] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1022] Step 1:

[1023] Data is collected from surveillance cameras and microphones.

[1024] Input: Video data from a surveillance camera and audio data from a microphone.

[1025] Output: The raw data collected.

[1026] Specific operation: Surveillance cameras and microphones monitor designated areas and capture video and audio data in real time.

[1027] Step 2:

[1028] The data is temporarily stored in a buffer on the device and then sent to the server.

[1029] Input: Raw data collected.

[1030] Output: The encrypted data sent to the server.

[1031] Specific operation: The terminal temporarily stores the video and audio data acquired in real time in a buffer, encrypts the data, and sends it to the server.

[1032] Step 3:

[1033] The data received by the server is analyzed.

[1034] Input: Video and audio data sent to the server.

[1035] Output: Analyzed data (anomaly detection results).

[1036] Specific operation: The server analyzes the received video data frame by frame using OpenCV and the audio data using PyAudio. It then uses machine learning models such as TensorFlow and Keras to detect anomalies (animals, pests, invasive species) from the data.

[1037] Step 4:

[1038] Generative AI generates countermeasures based on detected anomalies.

[1039] Input: Analyzed data (anomaly detection results).

[1040] Output: Generated countermeasures.

[1041] Specific operation: The server's generation AI generates appropriate countermeasures based on the anomaly detection results. For example, if a bear is detected, it will suggest a countermeasure such as "A bear has been detected. Please evacuate immediately."

[1042] Step 5:

[1043] An emotion engine analyzes the user's emotional state.

[1044] Input: User's voice or facial video data.

[1045] Output: User emotional state analysis results.

[1046] Specific operation: The emotion engine uses the EmotionRecognition library to analyze the user's tone of voice and facial expressions in real time, and measures psychological states such as stress and relief.

[1047] Step 6:

[1048] The generated countermeasures are adjusted based on the user's emotional state.

[1049] Input: Generated countermeasures, analysis results of the user's emotional state.

[1050] Output: Adjusted countermeasures.

[1051] Specific operation: The generation AI adjusts the content and urgency of the proposed measures according to the user's emotional state. For example, if the user is in a high-stress state, the notification will be more urgent.

[1052] Step 7:

[1053] Inform users of the adjusted measures and, if necessary, notify public authorities.

[1054] Input: Adjusted countermeasures.

[1055] Output: Notify user, notify public authorities.

[1056] Specific operation: The notification system notifies the user of the adjusted measures via smartphone or head-mounted display. In addition, if the abnormality level is high, an automatic report will be sent to public authorities.

[1057] Step 8:

[1058] The user acts on the notified measures.

[1059] Input: The proposed solution communicated to the user.

[1060] Output: User actions.

[1061] Specific actions: The user takes immediate action based on the notified countermeasures, such as evacuating, closing windows and doors, or contacting a professional.

[1062] Step 9:

[1063] The server collects feedback on the actions taken and uses this to improve the system.

[1064] Input: User behavior data, results of countermeasure implementation.

[1065] Output: Improved system parameters.

[1066] Specific operation: The server collects user behavior data and the results of countermeasure implementation, and uses the feedback data to improve the accuracy and optimization of the system.

[1067] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1068] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1069] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1070] [Third embodiment]

[1071] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1072] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1073] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1074] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1075] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1076] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1077] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1078] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1079] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1080] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1081] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1082] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1083] This invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of dangerous animals, pests, and invasive species in real time, and then uses AI to propose countermeasures appropriate to the situation. The specific form and operation of this system are described below.

[1084] System configuration

[1085] 1. Surveillance cameras

[1086] The surveillance cameras monitor designated areas and capture high-resolution video data. All-weather cameras that can be used day and night are recommended.

[1087] 2. Sound collection microphone

[1088] The sound-collecting microphone collects sounds within a designated area and records them as audio data, allowing you to capture the sounds of animals and pests.

[1089] 3. Terminal

[1090] The terminal temporarily stores data from the surveillance camera and sound-collecting microphone and transmits it to the server.

[1091] 4. Server

[1092] The server analyzes the received video and audio data in real time to detect anomalies, using machine learning models.

[1093] 5. Generation AI

[1094] The generative AI generates appropriate countermeasures based on the detected anomalies, notifies the user of the countermeasures, and automatically takes action as needed.

[1095] 6. Notification System

[1096] Terminal applications and browsers will be provided to notify users of the measures, and a function to notify public authorities in the event of an emergency will also be included.

[1097] Program processing explanation

[1098] Data collection and transmission

[1099] The server periodically acquires video and audio data in real time from the surveillance cameras and microphones.

[1100] The device stores the acquired data in a buffer and prepares it for transmission to the server, after which the data is sent to the server.

[1101] Data analysis

[1102] The server immediately analyzes the received video and audio data: video data is analyzed frame by frame, and audio data is analyzed based on the sampling rate.

[1103] The server uses machine learning models to identify patterns of specific animals, pests and invasive species to detect intrusions.

[1104] Real-time detection and notification

[1105] The server uses the analysis results to detect anomalies, evaluates specific criteria, and issues an alert if an anomaly is detected.

[1106] The terminal receives the alarm information from the server and notifies the user.

[1107] Countermeasure generation

[1108] The server uses a generation AI to generate appropriate countermeasures, including specific instructions and precautions.

[1109] The terminal notifies the user of the generated measures and displays them on the screen.

[1110] Reporting and Response

[1111] Users are then advised to take action in accordance with the measures they are notified of. In addition, if a serious abnormality occurs, the server will automatically notify public authorities.

[1112] The device supports user actions and performs some actions automatically (e.g., sound alarm activation).

[1113] Specific examples

[1114] Example 1: Bear invasion

[1115] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[1116] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[1117] The server uses a generation AI to generate and notify the user of countermeasures such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated."

[1118] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[1119] Example 2: Mosquito outbreak

[1120] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[1121] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[1122] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[1123] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[1124] With the above configuration and processing, the present invention can detect the intrusion of dangerous animals, pests, and invasive species in real time and provide quick and effective countermeasures, thereby preventing damage and improving the efficiency of security operations.

[1125] The processing flow will be explained below.

[1126] Step 1: Data collection

[1127] The server receives video and audio data from the surveillance cameras and microphones in real time, allowing it to record all events that occur within the surveillance area.

[1128] The device temporarily stores the captured video and audio data in a buffer, minimizing the risk of data loss.

[1129] Step 2: Send the data

[1130] The device prepares the buffered data for transmission to the server. The data is converted to a certain format (e.g. MP4, WAV, etc.).

[1131] The device sends the prepared data to the server via the network, where it is encrypted and transmitted to ensure security.

[1132] Step 3: Begin data analysis

[1133] The server immediately begins the process of analyzing the received video and audio data.

[1134] The server analyzes and compares the video data frame by frame to identify animals and pests.

[1135] The server analyzes the audio data based on the sampling rate to see if certain audio patterns are present.

[1136] Step 4: Identify specific organisms

[1137] The server uses pre-trained machine learning models to identify specific animals, pests, and invasive species from the data it receives with a high degree of accuracy.

[1138] The server determines the species, location, and behavioral patterns of the identified creatures.

[1139] Step 5: Real-time detection

[1140] The server uses the analysis results to evaluate certain criteria (e.g., animal size, behavior, location of intrusion, etc.) and determine whether an anomaly has been detected.

[1141] If the server detects an anomaly, it will immediately issue an alert, which will be filtered and different actions will be taken depending on the severity.

[1142] Step 6: Notification

[1143] The server sends the alarm information to the terminal so that the user can check it. The alarm includes information on the location, time, and target organism.

[1144] When the terminal receives an alert, it notifies the user with a sound or a pop-up message.

[1145] Step 7: Countermeasure generation

[1146] The server uses generative AI to generate appropriate countermeasures based on the detected anomalies, including specific instructions and precautions.

[1147] The server sends the generated countermeasures to the terminal in a format that is easy for the user to understand.

[1148] Step 8: Review and implement measures

[1149] The device displays the generated countermeasure information to the user, who can then check the countermeasure proposal and prepare to implement it if necessary.

[1150] The user implements the instructed measures and takes the actions recommended by the system, and the system automatically performs some actions (e.g., sounding an audio alarm).

[1151] Step 9: Report

[1152] If a serious abnormality occurs, the server automatically reports the situation to security companies and public institutions, including the type of organism detected, its location, and a summary of the countermeasures taken.

[1153] The server monitors the progress of the notification and provides additional information as needed.

[1154] Through the above processing steps, the system of the present invention can quickly detect abnormalities within the monitored area and propose and implement appropriate countermeasures, thereby preventing damage from dangerous animals, pests, and invasive species.

[1155] Example 1

[1156] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1157] In modern society, the invasion of dangerous animals, pests, and invasive species is increasingly threatening people's daily lives. It is necessary to quickly and effectively detect the invasion of such organisms and promptly implement appropriate countermeasures. However, conventional systems have had difficulty detecting anomalies in real time or automatically generating and implementing appropriate countermeasures. Furthermore, they lacked sufficient support functions to help users take appropriate measures, making it difficult to prevent damage before it occurs.

[1158] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1159] In this invention, the server includes: means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time; means for analyzing the acquired video data and audio data to detect dangerous organisms; means for generating countermeasures using a generation AI based on the type of organism detected and the location of the intrusion; means for notifying the user of the generated countermeasures and, if necessary, notifying public authorities; and means for automatically executing action as necessary to support the user's actions. This makes it possible to detect the intrusion of dangerous organisms in real time and provide quick and effective countermeasures. Support functions for users to take appropriate actions are also enhanced, preventing damage before it occurs.

[1160] A "surveillance camera" is a device that continuously monitors a specific area and captures high-resolution video data.

[1161] A "sound collection microphone" is a device that collects surrounding sounds in real time and records and transmits them as audio data.

[1162] "Video data" refers to digital data of visual information captured by a surveillance camera.

[1163] "Audio data" is digital data of auditory information acquired by a sound-collecting microphone.

[1164] A "server" is a computer system that executes a series of processes such as data collection, analysis, countermeasure generation, and notification.

[1165] "Generative AI" is an artificial intelligence model that automatically generates appropriate countermeasures for users.

[1166] "Countermeasures" are specific measures such as instructions for actions or precautions to be taken in response to detected abnormalities.

[1167] "User" means an individual or organization that uses the system and acts on the measures provided.

[1168] "Public institutions" are organizations such as governments and administrative agencies that are required to report any abnormalities detected.

[1169] A "buffer" is a memory area for temporarily storing data.

[1170] An "alarm" is a warning signal such as sound or light that notifies the user of an abnormality.

[1171] An "action" is an operation or behavior that the system automatically performs (e.g., activating an audio alarm).

[1172] The present invention is a system that uses a surveillance camera and a sound-collecting microphone to detect the intrusion of dangerous organisms in real time, and uses a generating AI to propose countermeasures according to the situation. Detailed embodiments for carrying out the present invention will be described below.

[1173] System configuration

[1174] 1. Surveillance cameras

[1175] The surveillance cameras continuously monitor designated areas and capture high-resolution video data. All-weather cameras are recommended, and include high-resolution CCD cameras and infrared cameras.

[1176] 2. Sound collection microphone

[1177] Sound-collecting microphones collect surrounding sounds and record them as audio data. This makes it possible to capture the sounds of animals and pests. Highly directional microphones and dynamic microphones that are resistant to environmental sounds are used.

[1178] 3. Terminal

[1179] The device temporarily stores data from the surveillance camera and microphone, stores it in a buffer, and then transmits it to the server. The device is preferably a computer or dedicated device with a high-speed processor and sufficient memory.

[1180] 4. Server

[1181] The server analyzes the received video and audio data in real time to detect anomalies. This analysis is performed using machine learning models, such as deep learning frameworks like TensorFlow and PyTorch.

[1182] 5. Generation AI

[1183] The generative AI generates appropriate countermeasures based on the detected anomalies. A natural language generation model (e.g., GPT-4) is used for generation. The generated countermeasures are notified to the user, and automatic action is taken if necessary.

[1184] 6. Notification System

[1185] The notification system will include a terminal application and browser to notify users of countermeasures, and will also have a function to notify public authorities in the event of an emergency.

[1186] Specific examples

[1187] Example 1: Bear invasion

[1188] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[1189] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[1190] The server uses a generation AI to generate and notify the user of countermeasures such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated."

[1191] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[1192] Example 2: Mosquito outbreak

[1193] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[1194] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[1195] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[1196] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[1197] Prompt Sentence Examples

[1198] Example 1: Bear invasion

[1199] "A bear has been detected in security camera footage. Please generate a measure to notify the user."

[1200] Example 2: Mosquito outbreak

[1201] "A large number of mosquito sounds have been detected from the microphone. Please generate a measure to notify the user."

[1202] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1203] Program processing flow

[1204] Step 1: Collect data

[1205] Input: Real-time video and audio data from surveillance cameras and microphones

[1206] Output: Video and audio data temporarily stored in the buffer

[1207] How it works: The server collects real-time data from a specified area via surveillance cameras and microphones. The day and night, all-weather cameras capture high-resolution images, while the microphones collect ambient sounds. This data is first temporarily stored in a buffer on the device.

[1208] Step 2: Sending data

[1209] Input: Video and audio data stored in the buffer

[1210] Output: Video and audio data sent to the server

[1211] What it does: The device periodically checks the data stored in the buffer, and if new data is available, it sends it to the server using a secure protocol (e.g., HTTPS).

[1212] Step 3: Analyze the data

[1213] Input: Video and audio data sent to the server

[1214] Output: Anomaly detection information as analysis results

[1215] Specific operation: The server analyzes the received video and audio data using a machine learning model. Feature extraction and recognition are performed on each frame of the video data using OpenCV, TensorFlow, etc. Audio data is analyzed based on the sampling rate, and the frequency spectrum is analyzed using FFT.

[1216] Step 4: Real-time anomaly detection

[1217] Input: Parsed data

[1218] Output: Issue an alert

[1219] Specific operation: The server detects anomalies based on the analysis results and issues an alert if certain criteria are met. For example, if a specific animal is seen in the video or a sound of a specific frequency is detected, it will recognize it as an anomaly and issue an alert. These criteria include thresholds and the results of pattern recognition.

[1220] Step 5: Generate countermeasures

[1221] Input: Anomaly detection information

[1222] Output: Generated countermeasure statement

[1223] Specific operation: The server uses a generative AI model (e.g., GPT-4) to generate appropriate countermeasures based on the anomaly detection information. The prompt contains detailed information about the current abnormal situation, and the generative AI model generates countermeasures based on this.

[1224] Step 6: Notification and response support

[1225] Input: Generated countermeasures

[1226] Output: User notification and automatic actions

[1227] Specific operations: The device notifies the user of the alert from the server and the generated countermeasures. The notification is displayed via a smartphone app or web browser. The notification contains specific response instructions that the user can act on. Furthermore, if a serious abnormality is detected, the server automatically notifies public authorities. Some actions (such as sounding an audio alarm) are also performed automatically, helping the user to take immediate action.

[1228] Specific examples

[1229] Bear invasion

[1230] The server analyzes video data from surveillance cameras, and if it detects a large animal, it identifies it as a bear based on its characteristics.

[1231] The server issues a bear intrusion warning and sends a countermeasure message to the device.

[1232] The generative AI model is given a prompt message: "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated.", and a countermeasure message is generated.

[1233] The terminal notifies the user of the generated countermeasure statement, and the system automatically activates an audio alarm if necessary.

[1234] Massive mosquito outbreaks

[1235] The server analyzes the audio data from the microphone and detects the sounds of large numbers of mosquitoes.

[1236] The server issues a mosquito outbreak warning and sends a countermeasure message to the terminal.

[1237] The generative AI model is given a prompt message: "There is a mosquito infestation. Please use insect repellent and close windows and doors. Also, contact a professional if necessary.", and a countermeasure message is generated.

[1238] The terminal notifies the user of the generated countermeasure statement and assists the user in carrying out the countermeasure based on the instruction.

[1239] (Application example 1)

[1240] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1241] Conventional security systems have difficulty detecting the intrusion of dangerous animals, pests, and invasive species in real time, often resulting in delayed implementation of appropriate countermeasures. Furthermore, they lack the means to provide users with quick and specific instructions in emergencies, increasing the risk of damage spreading. The present invention aims to solve these issues by providing a system that uses surveillance cameras and sound-collecting microphones to efficiently detect the intrusion of living organisms and uses generative AI to propose appropriate countermeasures in real time.

[1242] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1243] In this invention, the server includes means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time, means for analyzing the acquired video data and audio data to detect dangerous animals, pests, and invasive species, means for generating countermeasures using a generation AI based on the type of organism detected and the location of its intrusion, means for notifying a user of the generated countermeasures and reporting them to a public institution as necessary, and means for a user to check and implement the countermeasures using a smartphone, thereby enabling efficient and specific countermeasures to be provided in real time.

[1244] A "surveillance camera" is a device used to acquire high-resolution video data in real time and analyze that data.

[1245] A "sound collection microphone" is a device that collects sound data within a designated area and captures the sounds of animals and pests.

[1246] A "server" is a computer system that implements a machine learning model to analyze received video and audio data in real time and detect anomalies.

[1247] "Generative AI" is an artificial intelligence model that generates appropriate countermeasures based on detected anomalies.

[1248] A "buffer" is a memory system that contains an area for temporarily storing data and sending it on to the next processing step.

[1249] "Alert" is an alert system that issues a warning to the user when an abnormality is detected.

[1250] A "smartphone" is a mobile information terminal used by a user to receive notifications and to check and take countermeasures.

[1251] A "prompt" is textual input data that gives specific instructions or questions to a generative AI model.

[1252] The present invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of dangerous animals, pests, and invasive species in real time, generates countermeasures using generation AI, and notifies the user. This system is composed of surveillance cameras, sound-collecting microphones, a server, and a smartphone app. By combining these, it is possible to safely and quickly detect abnormalities and take appropriate countermeasures. Specific embodiments are described below.

[1253] Data collection and transmission

[1254] First, a surveillance camera and a sound-collecting microphone monitor a designated area, capturing high-resolution video and audio data in real time. The surveillance camera must be high-resolution and usable day and night, while the sound-collecting microphone must be highly sensitive.

[1255] The acquired data is temporarily stored in a terminal (e.g., a server) and then transmitted to the server. The terminal also includes a buffer for temporarily storing the data.

[1256] Data analysis

[1257] The server uses machine learning models to analyze the received video and audio data in real time: TensorFlow is used to analyze the video data, and the Librosa library is used to analyze the audio data.

[1258] The analyzed data is evaluated based on specific criteria, and if an anomaly is detected, an alert is issued, which is sent directly to the user via a smartphone app.

[1259] Real-time detection and notification

[1260] The smartphone app receives the alert information from the server and notifies the user. This notification system uses Firebase Cloud Messaging. The user can check the alert details and take action on the smartphone app.

[1261] Countermeasure generation

[1262] Generative AI is used to generate appropriate countermeasures. Specific countermeasure proposals for detected anomalies are generated using generative AI models such as GPT-4. The generated countermeasure proposals are provided to users via a smartphone app.

[1263] For example, if a snake is detected, the AI ​​will generate a countermeasure suggestion such as, "A snake has been found. Please keep as far away as possible and contact a specialist." An example of a prompt would be, "A snake has been detected through analysis of surveillance camera footage. Please generate appropriate response procedures."

[1264] Communication and Action

[1265] When the user acts in accordance with the generated countermeasures, the smartphone app also has the function of automatically executing some actions, such as activating an audio alarm.

[1266] As described above, this system combines various sensors and advanced analysis technology to detect danger in real time and provide prompt and appropriate countermeasures, thereby ensuring the safety of users and preventing damage before it occurs.

[1267] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1268] Step 1:

[1269] Surveillance cameras and microphones monitor designated areas, capturing high-resolution video and audio data in real time.

[1270] Input: Video data from surveillance cameras, audio data from microphones

[1271] Output: High-resolution video and audio data

[1272] How it works: The surveillance camera captures an entire area and records the footage in high resolution, while the sound pickup microphone collects surrounding sounds and stores them as digital audio data.

[1273] Step 2:

[1274] The video and audio data acquired by the terminal is temporarily stored in a buffer.

[1275] Input: High-resolution video and audio data

[1276] Output: Temporary data stored in a buffer

[1277] Specific operation: The terminal stores the data collected in real time in a buffer (temporary storage area) and prepares it for subsequent processing.

[1278] Step 3:

[1279] Send data from the terminal to the server.

[1280] Input: Temporary data stored in a buffer

[1281] Output: Video and audio data sent to the server

[1282] Specific operation: The device executes a process to send data to a server via the Internet.

[1283] Step 4:

[1284] The video data received by the server is analyzed using TensorFlow, and the audio data is analyzed using the Librosa library.

[1285] Input: Video and audio data sent to the server

[1286] Output: Analysis results (detected anomalies and patterns)

[1287] How it works: The server breaks down the video data into frames and performs image analysis using TensorFlow. The audio data is analyzed using the Librosa library based on the sampling rate to detect abnormal audio patterns.

[1288] Step 5:

[1289] The server detects anomalies based on the analysis results and issues an alert if certain criteria are met.

[1290] Input: Analysis results

[1291] Output: Warning information

[1292] Specific operation: The server evaluates the analysis results and generates an alert if dangerous animals, pests, or invasive species are detected.

[1293] Step 6:

[1294] The server uses the generated AI model to generate specific countermeasures for the detected anomalies.

[1295] Input: Alarm information

[1296] Output: Generated countermeasures

[1297] Specific operation: The generative AI model generates appropriate countermeasures based on the prompt. For example, a prompt such as "A snake has been detected during surveillance camera video analysis. Please generate appropriate response procedures."

[1298] Step 7:

[1299] The server notifies the user of the generated countermeasures via a smartphone app.

[1300] Input: Generated countermeasures

[1301] Output: Notification displayed on smartphone app

[1302] Specific operation: Using Firebase Cloud Messaging, the generated countermeasures are sent to the smartphone app and the user is notified.

[1303] Step 8:

[1304] Users can check the countermeasures and take action using their smartphones.

[1305] Input: Notification displayed on smartphone

[1306] Output: User action

[1307] Specific action: The user acknowledges the notification and takes appropriate action, for example, following the instructions, "A snake has been found. Move away as far away as possible and contact a professional."

[1308] Step 9:

[1309] The system will automatically take some action (e.g. activate an audio alarm) if necessary.

[1310] Input: User confirmation and warning information

[1311] Output: Actions taken

[1312] Specific operation: The system automatically activates an audio alarm to warn those around.

[1313] The above steps realize a real-time danger detection system using a surveillance camera and a sound-collecting microphone.

[1314] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1315] This invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of dangerous animals, pests, and invasive species in real time, and uses AI to generate and propose countermeasures appropriate to the situation. It also incorporates an emotion engine that recognizes the user's emotions. The system aims to optimize the countermeasures based on the user's emotional state and reduce the user's psychological burden.

[1316] System configuration

[1317] 1. Surveillance cameras

[1318] The surveillance cameras monitor designated areas and capture high-resolution video data, allowing for day and night surveillance.

[1319] 2. Sound collection microphone

[1320] The sound-collecting microphone collects sounds within a designated area and records them as audio data, allowing you to capture the sounds of animals and pests.

[1321] 3. Terminal

[1322] The terminal temporarily stores data from the surveillance camera and microphone and transmits it to the server. Data can be transmitted in real time.

[1323] 4. Server

[1324] The server analyzes the received video and audio data in real time to detect anomalies, using a pre-trained machine learning model.

[1325] 5. Generation AI

[1326] The generative AI generates appropriate countermeasures based on the detected anomalies, notifies the user of the countermeasures, and automatically takes action as needed.

[1327] 6. Emotion Engine

[1328] The emotion engine analyzes the user's voice and facial expressions to recognize their emotions and adjusts the content of the countermeasures based on that information. The urgency and level of detail of the countermeasures can be changed based on the recognized emotional state.

[1329] 7. Notification System

[1330] Terminal applications and browsers will be provided to notify users of the measures, and a function to notify public authorities in the event of an emergency will also be included.

[1331] Program processing explanation

[1332] Data collection and transmission

[1333] The server periodically acquires real-time video and audio data from the surveillance cameras and microphones, thereby recording all events that occur within the surveillance area.

[1334] The device stores the captured video and audio data in a buffer and prepares it for transmission to the server, where it is then encrypted and sent to the server.

[1335] Start of data analysis

[1336] The server immediately begins the process of analyzing the received video and audio data: video data is analyzed frame by frame, and audio data is analyzed based on the sampling rate.

[1337] The server uses pre-trained machine learning models to identify specific animals, pests, and invasive species from the data it receives with a high degree of accuracy.

[1338] Real-time detection and notification

[1339] The server evaluates certain criteria based on the analysis results and issues alerts if anomalies are detected. Alerts are filtered and different actions are taken depending on their severity.

[1340] The terminal receives the alarm information from the server and notifies the user. The alarm includes information on the location and time of occurrence, and the target organism.

[1341] Countermeasure generation and adjustment

[1342] The server uses generative AI to generate appropriate countermeasures based on the detected anomalies, including specific instructions and precautions.

[1343] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state. For example, if the user is in a high stress state, the generative AI will adjust the urgency and level of detail of the countermeasures.

[1344] Check and implement measures

[1345] The device displays the generated countermeasure information to the user, who can then check the countermeasure proposal and prepare to implement it if necessary.

[1346] The user implements the instructed measures and takes the actions recommended by the system, and the system automatically performs some actions (e.g., sounding an audio alarm).

[1347] Report

[1348] If a serious abnormality occurs, the server automatically reports the situation to security companies and public institutions, including the type of organism detected, its location, and a summary of the countermeasures taken.

[1349] The server monitors the progress of the notification and provides additional information as needed.

[1350] Specific examples

[1351] Example 1: Bear invasion

[1352] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[1353] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[1354] The emotion engine analyzes the user's facial expressions and tone of voice to recognize when the user is in a state of high stress.

[1355] The server uses a generation AI to generate and notify the user of a countermeasure such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated." The server also emphasizes the urgency of the countermeasure, as the user is in a high-stress state.

[1356] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[1357] Example 2: Mosquito outbreak

[1358] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[1359] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[1360] The emotion engine analyzes the user's emotional state and recognizes that they are not particularly stressed.

[1361] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[1362] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[1363] With the above configuration and processing, the present invention can quickly detect abnormalities within a monitored area and provide and implement appropriate countermeasures that are adapted to the user's emotional state, thereby preventing damage from dangerous animals, pests, and invasive species and reducing the psychological burden on the user.

[1364] The processing flow will be explained below.

[1365] Step 1: Data collection

[1366] The server receives video and audio data from the surveillance cameras and microphones in real time, ensuring that all events occurring within the surveillance area are accurately recorded.

[1367] The device temporarily stores the acquired video and audio data in a buffer, and the data is periodically accumulated in preparation for the next processing.

[1368] Step 2: Send the data

[1369] The device prepares the data stored in the buffer for transmission to the server. The data is converted into a predetermined format.

[1370] The device encrypts the prepared data and sends it over the network to the server, ensuring data security.

[1371] Step 3: Begin data analysis

[1372] The server starts analyzing the received video and audio data, where the video data is analyzed frame by frame and the audio data is analyzed based on the sampling rate.

[1373] The server uses pre-trained machine learning models to identify specific animals, pests and invasive species from the data it receives.

[1374] Step 4: Identify specific organisms

[1375] The server uses machine learning models to identify animal and pest patterns in the data, for example detecting the outlines of specific animals in video data and recognizing specific animal sounds in audio data.

[1376] The server determines the type, location, and behavioral patterns of the identified organisms and evaluates whether there are any abnormalities.

[1377] Step 5: Real-time detection

[1378] The server evaluates the analysis results and issues an alert if an abnormality is detected, based on specific criteria such as the size, behavioral patterns, and location of the animal.

[1379] Based on the detected abnormality information, the server determines the importance of the alert and selects the appropriate response.

[1380] Step 6: Notification

[1381] The server sends the alert information to the terminal for the user to view. The alert includes details of the location and time of the alert and the detected organism, allowing the user to take prompt action.

[1382] The terminal notifies the user of the received warning information, and the user receives the warning through voice or pop-up message.

[1383] Step 7: Countermeasure generation and sentiment analysis

[1384] The server uses a generative AI to generate appropriate countermeasures for the detected anomalies, creating countermeasure proposals that include specific instructions for action and precautions.

[1385] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state, for example, determining their stress level based on their voice tone and facial expression.

[1386] The server adjusts the countermeasures generated by the generative AI based on the recognized emotional state, providing more detailed and urgent countermeasures to users in high stress states.

[1387] Step 8: Review and implement measures

[1388] The device displays the adjusted countermeasure information to the user, who then checks the countermeasure proposal and takes necessary measures.

[1389] The user initiates action based on the displayed countermeasures. If the system automatically performs some action (e.g., activates an audio alarm), the user confirms this and takes the necessary steps.

[1390] Step 9: Report

[1391] If a serious abnormality occurs, the server automatically reports the situation to security companies and public institutions, including the type of organism detected, its location, and a summary of the countermeasures taken.

[1392] The server monitors the progress of the report and provides additional information as needed, enabling rapid coordination with relevant authorities.

[1393] Specific examples

[1394] Example 1: Bear invasion

[1395] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[1396] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[1397] The emotion engine analyzes the user's facial expressions and tone of voice to recognize when the user is in a state of high stress.

[1398] The server uses a generation AI to generate and notify the user of a countermeasure such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated." The server also emphasizes the urgency of the countermeasure, as the user is in a high-stress state.

[1399] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[1400] Example 2: Mosquito outbreak

[1401] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[1402] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[1403] The emotion engine analyzes the user's emotional state and recognizes that they are not particularly stressed.

[1404] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[1405] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[1406] With the above specific configuration and processing, the present invention can quickly detect abnormalities within a monitored area and provide and implement appropriate countermeasures that are adapted to the user's emotional state, thereby preventing damage from dangerous animals, pests, and invasive species and reducing the psychological burden on the user.

[1407] Example 2

[1408] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1409] In recent years, the intrusion of dangerous animals, pests, and invasive species has been increasing, and the resulting damage has become more serious. There is a need for a system that can detect such intrusions in real time and respond quickly. However, current systems do not provide countermeasures that take the user's emotional state into consideration, which increases the psychological burden on the user. Therefore, the present invention aims to optimize countermeasures based on the user's emotional state and reduce the psychological burden.

[1410] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1411] In this invention, the server includes means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time, means for analyzing the acquired video and audio data to detect dangerous animals, pests, and invasive species, and an emotion engine for analyzing the user's voice and facial expressions to recognize emotions and adjust countermeasures based on the user's emotional state. This makes it possible to quickly detect abnormalities and provide appropriate countermeasures while reducing the psychological burden on the user.

[1412] A "surveillance camera" is a device that monitors a designated area and captures high-resolution video data.

[1413] A "sound collection microphone" is a device that collects sounds within a specified area and records them as audio data.

[1414] "Video data" refers to a collection of video frames captured from a surveillance camera, which record events occurring within a surveillance area.

[1415] "Audio data" refers to a collection of sound sampling data acquired from a sound-collecting microphone, and is data that records sounds occurring within a specified area.

[1416] The "server" is a central processing unit that receives video data and audio data, analyzes them, and detects abnormalities.

[1417] The "emotion engine" is an engine that analyzes the user's voice and facial expressions to recognize their emotions, and adjusts the content of countermeasures based on that information.

[1418] "Generative AI" is an artificial intelligence model that generates appropriate countermeasures based on detected anomalies.

[1419] "Countermeasure generation" refers to the process of using generative AI to design and provide appropriate countermeasures based on the analysis results.

[1420] A "prompt" is an instruction sentence input to a generation AI, and is text containing instructions for generating a specific countermeasure.

[1421] The "notification means" is a mechanism for notifying the user of the generated countermeasures, and includes a terminal application and a browser.

[1422] "User" refers to the person who uses the system, who receives the proposed measures and implements them.

[1423] This invention is a system that detects the intrusion of dangerous animals, pests, and invasive species in real time and provides optimized countermeasures based on the user's emotional state. This system is composed of surveillance cameras, sound-collecting microphones, a server, terminals, a generation AI, an emotion engine, and a notification system.

[1424] Hardware and Software Details

[1425] Surveillance cameras and microphones

[1426] A surveillance camera is a device that monitors a designated area and captures high-resolution video data, allowing for surveillance day and night.

[1427] A sound collecting microphone is a device that collects sounds within a specified area and records them as audio data, allowing you to capture the sounds of animals and pests.

[1428] server

[1429] The server is a central processing unit that receives and analyzes data from surveillance cameras and microphones in real time. The server has the following functions:

[1430] Data collection function: Video and audio data is acquired in real time and temporarily stored.

[1431] Data analysis function: Analyzes received video and audio data to detect anomalies. This analysis is performed using a pre-trained machine learning model.

[1432] Emotion analysis function: Analyzes the user's voice and facial expressions to recognize their emotional state. An emotion prediction model is used to assess whether the user is in a high-stress or calm state.

[1433] Countermeasure generation function: Using a generation AI, a countermeasure is generated based on the detected anomaly. A prompt sentence is input to the generation AI, and the optimal countermeasure is output.

[1434] Terminal

[1435] The terminal is a device that temporarily stores data from the surveillance camera and microphone and transmits it to the server. The terminal has the following functions:

[1436] Data transmission function: Stored data is encrypted and sent to the server using a secure communication protocol, ensuring data safety.

[1437] Notification function: Notifies users of countermeasure information and warnings from the server. Notification methods include terminal applications, browsers, and audio alarms.

[1438] Generative AI model and prompts

[1439] A generative AI model is an artificial intelligence model that generates appropriate countermeasures based on detected anomalies. A prompt sentence is input to the generative AI model, and the optimal countermeasure is output.

[1440] Example prompt: There's a bear infestation. What precautions should be taken?

[1441] Specific examples

[1442] Example 1: Bear invasion

[1443] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[1444] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[1445] The emotion engine analyzes the user's facial expressions and tone of voice to recognize when the user is in a state of high stress.

[1446] The server uses a generation AI to generate and notify the user of a countermeasure such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated." The server also emphasizes the urgency of the countermeasures, as the user is in a high-stress state.

[1447] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[1448] Example 2: Mosquito outbreak

[1449] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[1450] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[1451] The emotion engine analyzes the user's emotional state and recognizes that they are not particularly stressed.

[1452] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[1453] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[1454] In this way, the system can detect abnormalities in real time and provide countermeasures that adapt to the user's emotional state, thereby preventing damage from dangerous animals, pests, and invasive species and reducing the psychological burden on the user.

[1455] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1456] Step 1: Data collection

[1457] The server acquires video data from the surveillance cameras in real time. Specifically, it continuously acquires video data frame by frame and stores it in high resolution. The input is the video data from the surveillance cameras, and the output is the video data temporarily stored in the buffer.

[1458] The server acquires audio data from a microphone in real time based on a sampling rate. The audio data includes environmental sounds within a specified area and the sounds of specific animals. The input is the audio data from the microphone, and the output is the audio data temporarily stored in a buffer.

[1459] Step 2: Send data

[1460] The terminal temporarily stores the acquired video and audio data in a buffer. The input is the data stored in the buffer from the server, and the output is encrypted data.

[1461] The device sends encrypted data to the server using a secure communication protocol (e.g., HTTPS). This ensures data security. The input is encrypted video and audio data, and the output is the data sent to the server.

[1462] Step 3: Data analysis

[1463] The server uses a pre-trained machine learning model to analyze the received video data frame by frame. The input is the video data sent to the server, and the output is the analysis results (e.g., the presence of specific animals, pests, or invasive species).

[1464] The server samples and analyzes the audio data, similarly using pre-trained models to identify specific frequencies and patterns. The input is the audio data sent to the server, and the output is the analysis results (e.g., detection of the sounds of specific animals, pests, or invasive species).

[1465] Step 4: Anomaly detection

[1466] The server evaluates the presence of specific animals, pests, and invasive species based on the results of data analysis, and issues an alert if an abnormality is detected. The input is the data analysis results, and the output is alert information (e.g., location, time, and information about the target species).

[1467] The server filters the alerts and determines the action to be taken (e.g., prioritizing notifications, reporting) depending on the severity. The input is the alert information, and the output is the filtered alert information.

[1468] Step 5: Sentiment Analysis

[1469] The server then analyzes the received video and audio data again and uses an emotion engine to recognize the user's emotional state from their voice and facial expressions. The input is real-time video and audio data, and the output is the user's emotional state (e.g., high stress, calm).

[1470] The server makes adjustments based on the user's emotional state that affect the countermeasures generated by the generation AI. The input is the user's emotional state, and the output is the adjusted countermeasures.

[1471] Step 6: Countermeasure Generation

[1472] The server uses a generative AI to generate countermeasures based on the detected anomalies and the user's emotional state. The generative AI inputs a prompt statement and outputs an optimal countermeasure. The inputs are the prompt statement and the generative AI model, and the output is a countermeasure statement.

[1473] Example: Enter the prompt text "Bears are appearing. What measures are needed?" into the generation AI.

[1474] Step 7: Notification of measures

[1475] The terminal receives countermeasure information from the server and notifies the user. Notification methods include terminal applications, browsers, and audio alarms. The input is countermeasure information from the server, and the output is a notification to the user.

[1476] The device notifies the user of the details of the countermeasures and the implementation procedure, and provides specific instructions for the user to take action. The input is the countermeasure proposal, and the output is specific instructions to the user.

[1477] Step 8: Take action

[1478] The user checks the notified countermeasures and takes action according to the instructions, such as evacuating, reporting, or installing physical barriers. The input is the countermeasure proposal, and the output is the user's action.

[1479] The terminal executes automated countermeasures. For example, the system automatically activates an audio alarm to prevent bears from entering. The input is the countermeasure proposal, and the output is the automatically executed action.

[1480] Step 9: Report Processing

[1481] If a serious abnormality occurs, the server automatically notifies security companies and public institutions. The input is the alarm information of the serious abnormality, and the output is the content of the notification (e.g., the type of organism detected, its location, and an overview of countermeasures).

[1482] The server monitors the progress of the notification and provides additional information as needed. The input is progress feedback and the output is additional information.

[1483] Each processing step of this system detects abnormalities in real time, such as dangerous animals, pests, and invasive species, and generates and implements appropriate countermeasures tailored to the user's emotional state, thereby ensuring safety while reducing the user's psychological burden.

[1484] (Application example 2)

[1485] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1486] Conventional security and surveillance systems have limited capabilities for detecting the intrusion of dangerous animals, pests, and invasive species, making it difficult to quickly implement appropriate countermeasures. Furthermore, they provide countermeasures without taking into account the user's emotional state, which increases the psychological burden on the user. Furthermore, even when an abnormality is detected, the user may not receive an appropriate notification, or the reporting to public authorities may be delayed.

[1487] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time, means for analyzing the acquired video data and audio data to detect dangerous animals, pests, and invasive species, means for generating countermeasures using a generative model based on the type and intrusion location of the detected species, means for notifying the user of the generated countermeasures and, if necessary, reporting them to public institutions, means for analyzing the user's emotional state, and means for adjusting the generated countermeasures based on the user's emotions. This makes it possible to quickly detect the intrusion of dangerous animals, pests, and invasive species, provide appropriate countermeasures adapted to the user's emotional state, and reduce psychological burden. Furthermore, since anomalies are immediately notified and public institutions are promptly notified when they are detected, greater safety can be achieved.

[1488] A "surveillance camera" is a device that monitors a designated area and captures high-resolution video data.

[1489] A "sound collection microphone" is a device that collects sounds within a specified area and records them as audio data.

[1490] "Real-time" refers to acquisition and analysis occurring immediately, without delay.

[1491] An "anomaly" refers to an unusual phenomenon detected by the system, specifically the intrusion of dangerous animals, pests, or invasive species.

[1492] "Generative model" refers to the process of using pre-trained machine learning algorithms to analyze data and generate appropriate countermeasures.

[1493] "Countermeasures" are specific instructions for actions or precautions that can be taken in response to detected abnormalities.

[1494] "Notification" is the act of informing the user of a detected anomaly and the countermeasures that have been created.

[1495] "Public institutions" refers to public safety agencies such as police and fire departments.

[1496] "Emotional state" refers to the user's psychological usage status, particularly stress and sense of security.

[1497] "Adjustment" is the process of changing the content and urgency of the generated measures based on the user's emotional state.

[1498] A "buffer" is a memory area that temporarily stores data.

[1499] An "alarm" is a warning signal issued when an abnormality is detected.

[1500] A "terminal" is a device that temporarily stores collected data and transmits it to a server.

[1501] The present invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of animals, pests, and invasive species in real time, and then uses AI to generate and propose countermeasures appropriate to the situation. Furthermore, this system incorporates an emotion engine that analyzes the user's emotional state and optimizes the countermeasures. A specific embodiment of this system is described below.

[1502] System configuration

[1503] 1. Surveillance cameras

[1504] The surveillance cameras periodically monitor designated areas, capturing high-resolution video data 24 hours a day, day or night.

[1505] 2. Sound collection microphone

[1506] The microphones collect sounds within a designated area and record them as audio data, including animal calls and the buzzing of pests.

[1507] 3. Terminal

[1508] The device temporarily stores data from the surveillance camera and microphone in a buffer and then transmits it to the server. The data transmission is encrypted and secure.

[1509] 4. Server

[1510] The server receives and analyzes the video and audio data sent from the device. A pre-trained machine learning model is used for the analysis. For example, a deep learning model such as TensorFlow or Keras is used.

[1511] 5. Generation AI

[1512] The generative AI generates appropriate countermeasures based on the anomalies detected by the server. The countermeasures are proposed as specific instructions for action or precautions. The generative AI model generates countermeasures based on the prompt sentences entered by the user into the system.

[1513] 6. Emotion Engine

[1514] The emotion engine analyzes the user's voice and facial expressions to recognize the user's emotional state in real time. Based on this information, the generative AI adjusts countermeasures to reduce the user's psychological burden.

[1515] 7. Notification System

[1516] Users will be notified of the measures via their smartphone or head-mounted display, and in the event of an emergency, public authorities will also be notified automatically.

[1517] Program processing explanation

[1518] Data collection:

[1519] Video data captured by the surveillance camera and audio data collected by the microphone are temporarily stored in a buffer on the device, and then encrypted and securely transmitted to the server.

[1520] Data Analysis:

[1521] The server analyzes the received video and audio data. The required hardware is a server equipped with a high-performance GPU (e.g., an NVIDIA GPU). Deep learning frameworks such as TensorFlow and Keras are used for analysis. The machine learning model identifies the characteristics of animals, pests, and invasive species with high accuracy.

[1522] Countermeasures generated:

[1523] Based on the analysis results, the AI ​​generates countermeasures using prompt sentences. For example, if the server detects a bear, it will suggest a countermeasure such as "A bear has been spotted. Please evacuate immediately."

[1524] Emotion analysis:

[1525] The emotion engine recognizes the user's tone of voice and facial expressions to gauge stress levels, for example, by using the EmotionRecognition library to detect emotional changes in real time.

[1526] notification:

[1527] The derived countermeasures are notified to the user via a smartphone or head-mounted display. If the user is in a high-stress state, a more urgent notification method is selected. If necessary, a report is also sent to public institutions.

[1528] Examples of concrete examples and prompts

[1529] Bear Intrusion Detection:

[1530] If a bear is detected on a surveillance camera, the AI ​​will generate a countermeasure such as, "A bear has been detected. Please evacuate immediately and notify public authorities." The emotion engine analyzes the user's voice and facial expression, and if it determines that the user is under high stress, it emphasizes the urgency of the countermeasure.

[1531] Example prompt sentence:

[1532] "A large animal is visible on the camera footage, is it a bear?"

[1533] "A bear has been detected. The user's emotional state is high stress. What is the best course of action in this case?"

[1534] "We have a huge population of mosquitoes and they don't seem to be stressed. What mitigation measures should we take in this case?"

[1535] By implementing this invention, it is possible to detect the invasion of abnormal animals, pests, and invasive species at an early stage, provide appropriate countermeasures according to the user's emotional state, and create an environment where users can feel safe.

[1536] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1537] Step 1:

[1538] Data is collected from surveillance cameras and microphones.

[1539] Input: Video data from a surveillance camera and audio data from a microphone.

[1540] Output: The raw data collected.

[1541] Specific operation: Surveillance cameras and microphones monitor designated areas and capture video and audio data in real time.

[1542] Step 2:

[1543] The data is temporarily stored in a buffer on the device and then sent to the server.

[1544] Input: Raw data collected.

[1545] Output: The encrypted data sent to the server.

[1546] Specific operation: The terminal temporarily stores the video and audio data acquired in real time in a buffer, encrypts the data, and sends it to the server.

[1547] Step 3:

[1548] The data received by the server is analyzed.

[1549] Input: Video and audio data sent to the server.

[1550] Output: Analyzed data (anomaly detection results).

[1551] Specific operation: The server analyzes the received video data frame by frame using OpenCV and the audio data using PyAudio. It then uses machine learning models such as TensorFlow and Keras to detect anomalies (animals, pests, invasive species) from the data.

[1552] Step 4:

[1553] Generative AI generates countermeasures based on detected anomalies.

[1554] Input: Analyzed data (anomaly detection results).

[1555] Output: Generated countermeasures.

[1556] Specific operation: The server's generation AI generates appropriate countermeasures based on the anomaly detection results. For example, if a bear is detected, it will suggest a countermeasure such as "A bear has been detected. Please evacuate immediately."

[1557] Step 5:

[1558] An emotion engine analyzes the user's emotional state.

[1559] Input: User's voice or facial video data.

[1560] Output: User emotional state analysis results.

[1561] Specific operation: The emotion engine uses the EmotionRecognition library to analyze the user's tone of voice and facial expressions in real time, and measures psychological states such as stress and relief.

[1562] Step 6:

[1563] The generated countermeasures are adjusted based on the user's emotional state.

[1564] Input: Generated countermeasures, analysis results of the user's emotional state.

[1565] Output: Adjusted countermeasures.

[1566] Specific operation: The generation AI adjusts the content and urgency of the proposed measures according to the user's emotional state. For example, if the user is in a high-stress state, the notification will be more urgent.

[1567] Step 7:

[1568] Inform users of the adjusted measures and, if necessary, notify public authorities.

[1569] Input: Adjusted countermeasures.

[1570] Output: Notify user, notify public authorities.

[1571] Specific operation: The notification system notifies the user of the adjusted measures via smartphone or head-mounted display. In addition, if the abnormality level is high, an automatic report will be sent to public authorities.

[1572] Step 8:

[1573] The user acts on the notified measures.

[1574] Input: The proposed solution communicated to the user.

[1575] Output: User actions.

[1576] Specific actions: The user takes immediate action based on the notified countermeasures, such as evacuating, closing windows and doors, or contacting a professional.

[1577] Step 9:

[1578] The server collects feedback on the actions taken and uses this to improve the system.

[1579] Input: User behavior data, results of countermeasure implementation.

[1580] Output: Improved system parameters.

[1581] Specific operation: The server collects user behavior data and the results of countermeasure implementation, and uses the feedback data to improve the accuracy and optimization of the system.

[1582] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1583] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1584] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1585] [Fourth embodiment]

[1586] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1587] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1588] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1589] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1590] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1591] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1592] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1593] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1594] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1595] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1596] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1597] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1598] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1599] This invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of dangerous animals, pests, and invasive species in real time, and then uses AI to propose countermeasures appropriate to the situation. The specific form and operation of this system are described below.

[1600] System configuration

[1601] 1. Surveillance cameras

[1602] The surveillance cameras monitor designated areas and capture high-resolution video data. All-weather cameras that can be used day and night are recommended.

[1603] 2. Sound collection microphone

[1604] The sound-collecting microphone collects sounds within a designated area and records them as audio data, allowing you to capture the sounds of animals and pests.

[1605] 3. Terminal

[1606] The terminal temporarily stores data from the surveillance camera and sound-collecting microphone and transmits it to the server.

[1607] 4. Server

[1608] The server analyzes the received video and audio data in real time to detect anomalies, using machine learning models.

[1609] 5. Generation AI

[1610] The generative AI generates appropriate countermeasures based on the detected anomalies, notifies the user of the countermeasures, and automatically takes action as needed.

[1611] 6. Notification System

[1612] Terminal applications and browsers will be provided to notify users of the measures, and a function to notify public authorities in the event of an emergency will also be included.

[1613] Program processing explanation

[1614] Data collection and transmission

[1615] The server periodically acquires video and audio data in real time from the surveillance cameras and microphones.

[1616] The device stores the acquired data in a buffer and prepares it for transmission to the server, after which the data is sent to the server.

[1617] Data analysis

[1618] The server immediately analyzes the received video and audio data: video data is analyzed frame by frame, and audio data is analyzed based on the sampling rate.

[1619] The server uses machine learning models to identify patterns of specific animals, pests and invasive species to detect intrusions.

[1620] Real-time detection and notification

[1621] The server uses the analysis results to detect anomalies, evaluates specific criteria, and issues an alert if an anomaly is detected.

[1622] The terminal receives the alarm information from the server and notifies the user.

[1623] Countermeasure generation

[1624] The server uses a generation AI to generate appropriate countermeasures, including specific instructions and precautions.

[1625] The terminal notifies the user of the generated measures and displays them on the screen.

[1626] Reporting and Response

[1627] Users are then advised to take action in accordance with the measures they are notified of. In addition, if a serious abnormality occurs, the server will automatically notify public authorities.

[1628] The device supports user actions and performs some actions automatically (e.g., sound alarm activation).

[1629] Specific examples

[1630] Example 1: Bear invasion

[1631] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[1632] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[1633] The server uses a generation AI to generate and notify the user of countermeasures such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated."

[1634] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[1635] Example 2: Mosquito outbreak

[1636] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[1637] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[1638] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[1639] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[1640] With the above configuration and processing, the present invention can detect the intrusion of dangerous animals, pests, and invasive species in real time and provide quick and effective countermeasures, thereby preventing damage and improving the efficiency of security operations.

[1641] The processing flow will be explained below.

[1642] Step 1: Data collection

[1643] The server receives video and audio data from the surveillance cameras and microphones in real time, allowing it to record all events that occur within the surveillance area.

[1644] The device temporarily stores the captured video and audio data in a buffer, minimizing the risk of data loss.

[1645] Step 2: Send the data

[1646] The device prepares the buffered data for transmission to the server. The data is converted to a certain format (e.g. MP4, WAV, etc.).

[1647] The device sends the prepared data to the server via the network, where it is encrypted and transmitted to ensure security.

[1648] Step 3: Begin data analysis

[1649] The server immediately begins the process of analyzing the received video and audio data.

[1650] The server analyzes and compares the video data frame by frame to identify animals and pests.

[1651] The server analyzes the audio data based on the sampling rate to see if certain audio patterns are present.

[1652] Step 4: Identify specific organisms

[1653] The server uses pre-trained machine learning models to identify specific animals, pests, and invasive species from the data it receives with a high degree of accuracy.

[1654] The server determines the species, location, and behavioral patterns of the identified creatures.

[1655] Step 5: Real-time detection

[1656] The server uses the analysis results to evaluate certain criteria (e.g., animal size, behavior, location of intrusion, etc.) and determine whether an anomaly has been detected.

[1657] If the server detects an anomaly, it will immediately issue an alert, which will be filtered and different actions will be taken depending on the severity.

[1658] Step 6: Notification

[1659] The server sends the alarm information to the terminal so that the user can check it. The alarm includes information on the location, time, and target organism.

[1660] When the terminal receives an alert, it notifies the user with a sound or a pop-up message.

[1661] Step 7: Countermeasure generation

[1662] The server uses generative AI to generate appropriate countermeasures based on the detected anomalies, including specific instructions and precautions.

[1663] The server sends the generated countermeasures to the terminal in a format that is easy for the user to understand.

[1664] Step 8: Review and implement measures

[1665] The device displays the generated countermeasure information to the user, who can then check the countermeasure proposal and prepare to implement it if necessary.

[1666] The user implements the instructed measures and takes the actions recommended by the system, and the system automatically performs some actions (e.g., sounding an audio alarm).

[1667] Step 9: Report

[1668] If a serious abnormality occurs, the server automatically reports the situation to security companies and public institutions, including the type of organism detected, its location, and a summary of the countermeasures taken.

[1669] The server monitors the progress of the notification and provides additional information as needed.

[1670] Through the above processing steps, the system of the present invention can quickly detect abnormalities within the monitored area and propose and implement appropriate countermeasures, thereby preventing damage from dangerous animals, pests, and invasive species.

[1671] Example 1

[1672] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1673] In modern society, the invasion of dangerous animals, pests, and invasive species is increasingly threatening people's daily lives. It is necessary to quickly and effectively detect the invasion of such organisms and promptly implement appropriate countermeasures. However, conventional systems have had difficulty detecting anomalies in real time or automatically generating and implementing appropriate countermeasures. Furthermore, they lacked sufficient support functions to help users take appropriate measures, making it difficult to prevent damage before it occurs.

[1674] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1675] In this invention, the server includes: means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time; means for analyzing the acquired video data and audio data to detect dangerous organisms; means for generating countermeasures using a generation AI based on the type of organism detected and the location of the intrusion; means for notifying the user of the generated countermeasures and, if necessary, notifying public authorities; and means for automatically executing action as necessary to support the user's actions. This makes it possible to detect the intrusion of dangerous organisms in real time and provide quick and effective countermeasures. Support functions for users to take appropriate actions are also enhanced, preventing damage before it occurs.

[1676] A "surveillance camera" is a device that continuously monitors a specific area and captures high-resolution video data.

[1677] A "sound collection microphone" is a device that collects surrounding sounds in real time and records and transmits them as audio data.

[1678] "Video data" refers to digital data of visual information captured by a surveillance camera.

[1679] "Audio data" is digital data of auditory information acquired by a sound-collecting microphone.

[1680] A "server" is a computer system that executes a series of processes such as data collection, analysis, countermeasure generation, and notification.

[1681] "Generative AI" is an artificial intelligence model that automatically generates appropriate countermeasures for users.

[1682] "Countermeasures" are specific measures such as instructions for actions or precautions to be taken in response to detected abnormalities.

[1683] "User" means an individual or organization that uses the system and acts on the measures provided.

[1684] "Public institutions" are organizations such as governments and administrative agencies that are required to report any abnormalities detected.

[1685] A "buffer" is a memory area for temporarily storing data.

[1686] An "alarm" is a warning signal such as sound or light that notifies the user of an abnormality.

[1687] An "action" is an operation or behavior that the system automatically performs (e.g., activating an audio alarm).

[1688] The present invention is a system that uses a surveillance camera and a sound-collecting microphone to detect the intrusion of dangerous organisms in real time, and uses a generating AI to propose countermeasures according to the situation. Detailed embodiments for carrying out the present invention will be described below.

[1689] System configuration

[1690] 1. Surveillance cameras

[1691] The surveillance cameras continuously monitor designated areas and capture high-resolution video data. All-weather cameras are recommended, and include high-resolution CCD cameras and infrared cameras.

[1692] 2. Sound collection microphone

[1693] Sound-collecting microphones collect surrounding sounds and record them as audio data. This makes it possible to capture the sounds of animals and pests. Highly directional microphones and dynamic microphones that are resistant to environmental sounds are used.

[1694] 3. Terminal

[1695] The device temporarily stores data from the surveillance camera and microphone, stores it in a buffer, and then transmits it to the server. The device is preferably a computer or dedicated device with a high-speed processor and sufficient memory.

[1696] 4. Server

[1697] The server analyzes the received video and audio data in real time to detect anomalies. This analysis is performed using machine learning models, such as deep learning frameworks like TensorFlow and PyTorch.

[1698] 5. Generation AI

[1699] The generative AI generates appropriate countermeasures based on the detected anomalies. A natural language generation model (e.g., GPT-4) is used for generation. The generated countermeasures are notified to the user, and automatic action is taken if necessary.

[1700] 6. Notification System

[1701] The notification system will include a terminal application and browser to notify users of countermeasures, and will also have a function to notify public authorities in the event of an emergency.

[1702] Specific examples

[1703] Example 1: Bear invasion

[1704] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[1705] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[1706] The server uses a generation AI to generate and notify the user of countermeasures such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated."

[1707] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[1708] Example 2: Mosquito outbreak

[1709] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[1710] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[1711] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[1712] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[1713] Prompt Sentence Examples

[1714] Example 1: Bear invasion

[1715] "A bear has been detected in security camera footage. Please generate a measure to notify the user."

[1716] Example 2: Mosquito outbreak

[1717] "A large number of mosquito sounds have been detected from the microphone. Please generate a measure to notify the user."

[1718] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1719] Program processing flow

[1720] Step 1: Collect data

[1721] Input: Real-time video and audio data from surveillance cameras and microphones

[1722] Output: Video and audio data temporarily stored in the buffer

[1723] How it works: The server collects real-time data from a specified area via surveillance cameras and microphones. The day and night, all-weather cameras capture high-resolution images, while the microphones collect ambient sounds. This data is first temporarily stored in a buffer on the device.

[1724] Step 2: Sending data

[1725] Input: Video and audio data stored in the buffer

[1726] Output: Video and audio data sent to the server

[1727] What it does: The device periodically checks the data stored in the buffer, and if new data is available, it sends it to the server using a secure protocol (e.g., HTTPS).

[1728] Step 3: Analyze the data

[1729] Input: Video and audio data sent to the server

[1730] Output: Anomaly detection information as analysis results

[1731] Specific operation: The server analyzes the received video and audio data using a machine learning model. Feature extraction and recognition are performed on each frame of the video data using OpenCV, TensorFlow, etc. Audio data is analyzed based on the sampling rate, and the frequency spectrum is analyzed using FFT.

[1732] Step 4: Real-time anomaly detection

[1733] Input: Parsed data

[1734] Output: Issue an alert

[1735] Specific operation: The server detects anomalies based on the analysis results and issues an alert if certain criteria are met. For example, if a specific animal is seen in the video or a sound of a specific frequency is detected, it will recognize it as an anomaly and issue an alert. These criteria include thresholds and the results of pattern recognition.

[1736] Step 5: Generate countermeasures

[1737] Input: Anomaly detection information

[1738] Output: Generated countermeasure statement

[1739] Specific operation: The server uses a generative AI model (e.g., GPT-4) to generate appropriate countermeasures based on the anomaly detection information. The prompt contains detailed information about the current abnormal situation, and the generative AI model generates countermeasures based on this.

[1740] Step 6: Notification and response support

[1741] Input: Generated countermeasures

[1742] Output: User notification and automatic actions

[1743] Specific operations: The device notifies the user of the alert from the server and the generated countermeasures. The notification is displayed via a smartphone app or web browser. The notification contains specific response instructions that the user can act on. Furthermore, if a serious abnormality is detected, the server automatically notifies public authorities. Some actions (such as sounding an audio alarm) are also performed automatically, helping the user to take immediate action.

[1744] Specific examples

[1745] Bear invasion

[1746] The server analyzes video data from surveillance cameras, and if it detects a large animal, it identifies it as a bear based on its characteristics.

[1747] The server issues a bear intrusion warning and sends a countermeasure message to the device.

[1748] The generative AI model is given a prompt message: "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated.", and a countermeasure message is generated.

[1749] The terminal notifies the user of the generated countermeasure statement, and the system automatically activates an audio alarm if necessary.

[1750] Massive mosquito outbreaks

[1751] The server analyzes the audio data from the microphone and detects the sounds of large numbers of mosquitoes.

[1752] The server issues a mosquito outbreak warning and sends a countermeasure message to the terminal.

[1753] The generative AI model is given a prompt message: "There is a mosquito infestation. Please use insect repellent and close windows and doors. Also, contact a professional if necessary.", and a countermeasure message is generated.

[1754] The terminal notifies the user of the generated countermeasure statement and assists the user in carrying out the countermeasure based on the instruction.

[1755] (Application example 1)

[1756] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1757] Conventional security systems have difficulty detecting the intrusion of dangerous animals, pests, and invasive species in real time, often resulting in delayed implementation of appropriate countermeasures. Furthermore, they lack the means to provide users with quick and specific instructions in emergencies, increasing the risk of damage spreading. The present invention aims to solve these issues by providing a system that uses surveillance cameras and sound-collecting microphones to efficiently detect the intrusion of living organisms and uses generative AI to propose appropriate countermeasures in real time.

[1758] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1759] In this invention, the server includes means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time, means for analyzing the acquired video data and audio data to detect dangerous animals, pests, and invasive species, means for generating countermeasures using a generation AI based on the type of organism detected and the location of its intrusion, means for notifying a user of the generated countermeasures and reporting them to a public institution as necessary, and means for a user to check and implement the countermeasures using a smartphone, thereby enabling efficient and specific countermeasures to be provided in real time.

[1760] A "surveillance camera" is a device used to acquire high-resolution video data in real time and analyze that data.

[1761] A "sound collection microphone" is a device that collects sound data within a designated area and captures the sounds of animals and pests.

[1762] A "server" is a computer system that implements a machine learning model to analyze received video and audio data in real time and detect anomalies.

[1763] "Generative AI" is an artificial intelligence model that generates appropriate countermeasures based on detected anomalies.

[1764] A "buffer" is a memory system that contains an area for temporarily storing data and sending it on to the next processing step.

[1765] "Alert" is an alert system that issues a warning to the user when an abnormality is detected.

[1766] A "smartphone" is a mobile information terminal used by a user to receive notifications and to check and take countermeasures.

[1767] A "prompt" is textual input data that gives specific instructions or questions to a generative AI model.

[1768] The present invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of dangerous animals, pests, and invasive species in real time, generates countermeasures using generation AI, and notifies the user. This system is composed of surveillance cameras, sound-collecting microphones, a server, and a smartphone app. By combining these, it is possible to safely and quickly detect abnormalities and take appropriate countermeasures. Specific embodiments are described below.

[1769] Data collection and transmission

[1770] First, a surveillance camera and a sound-collecting microphone monitor a designated area, capturing high-resolution video and audio data in real time. The surveillance camera must be high-resolution and usable day and night, while the sound-collecting microphone must be highly sensitive.

[1771] The acquired data is temporarily stored in a terminal (e.g., a server) and then transmitted to the server. The terminal also includes a buffer for temporarily storing the data.

[1772] Data analysis

[1773] The server uses machine learning models to analyze the received video and audio data in real time: TensorFlow is used to analyze the video data, and the Librosa library is used to analyze the audio data.

[1774] The analyzed data is evaluated based on specific criteria, and if an anomaly is detected, an alert is issued, which is sent directly to the user via a smartphone app.

[1775] Real-time detection and notification

[1776] The smartphone app receives the alert information from the server and notifies the user. This notification system uses Firebase Cloud Messaging. The user can check the alert details and take action on the smartphone app.

[1777] Countermeasure generation

[1778] Generative AI is used to generate appropriate countermeasures. Specific countermeasure proposals for detected anomalies are generated using generative AI models such as GPT-4. The generated countermeasure proposals are provided to users via a smartphone app.

[1779] For example, if a snake is detected, the AI ​​will generate a countermeasure suggestion such as, "A snake has been found. Please keep as far away as possible and contact a specialist." An example of a prompt would be, "A snake has been detected through analysis of surveillance camera footage. Please generate appropriate response procedures."

[1780] Communication and Action

[1781] When the user acts in accordance with the generated countermeasures, the smartphone app also has the function of automatically executing some actions, such as activating an audio alarm.

[1782] As described above, this system combines various sensors and advanced analysis technology to detect danger in real time and provide prompt and appropriate countermeasures, thereby ensuring the safety of users and preventing damage before it occurs.

[1783] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1784] Step 1:

[1785] Surveillance cameras and microphones monitor designated areas, capturing high-resolution video and audio data in real time.

[1786] Input: Video data from surveillance cameras, audio data from microphones

[1787] Output: High-resolution video and audio data

[1788] How it works: The surveillance camera captures an entire area and records the footage in high resolution, while the sound pickup microphone collects surrounding sounds and stores them as digital audio data.

[1789] Step 2:

[1790] The video and audio data acquired by the terminal is temporarily stored in a buffer.

[1791] Input: High-resolution video and audio data

[1792] Output: Temporary data stored in a buffer

[1793] Specific operation: The terminal stores the data collected in real time in a buffer (temporary storage area) and prepares it for subsequent processing.

[1794] Step 3:

[1795] Send data from the terminal to the server.

[1796] Input: Temporary data stored in a buffer

[1797] Output: Video and audio data sent to the server

[1798] Specific operation: The device executes a process to send data to a server via the Internet.

[1799] Step 4:

[1800] The video data received by the server is analyzed using TensorFlow, and the audio data is analyzed using the Librosa library.

[1801] Input: Video and audio data sent to the server

[1802] Output: Analysis results (detected anomalies and patterns)

[1803] How it works: The server breaks down the video data into frames and performs image analysis using TensorFlow. The audio data is analyzed using the Librosa library based on the sampling rate to detect abnormal audio patterns.

[1804] Step 5:

[1805] The server detects anomalies based on the analysis results and issues an alert if certain criteria are met.

[1806] Input: Analysis results

[1807] Output: Warning information

[1808] Specific operation: The server evaluates the analysis results and generates an alert if dangerous animals, pests, or invasive species are detected.

[1809] Step 6:

[1810] The server uses the generated AI model to generate specific countermeasures for the detected anomalies.

[1811] Input: Alarm information

[1812] Output: Generated countermeasures

[1813] Specific operation: The generative AI model generates appropriate countermeasures based on the prompt. For example, a prompt such as "A snake has been detected during surveillance camera video analysis. Please generate appropriate response procedures."

[1814] Step 7:

[1815] The server notifies the user of the generated countermeasures via a smartphone app.

[1816] Input: Generated countermeasures

[1817] Output: Notification displayed on smartphone app

[1818] Specific operation: Using Firebase Cloud Messaging, the generated countermeasures are sent to the smartphone app and the user is notified.

[1819] Step 8:

[1820] Users can check the countermeasures and take action using their smartphones.

[1821] Input: Notification displayed on smartphone

[1822] Output: User action

[1823] Specific action: The user acknowledges the notification and takes appropriate action, for example, following the instructions, "A snake has been found. Move away as far away as possible and contact a professional."

[1824] Step 9:

[1825] The system will automatically take some action (e.g. activate an audio alarm) if necessary.

[1826] Input: User confirmation and warning information

[1827] Output: Actions taken

[1828] Specific operation: The system automatically activates an audio alarm to warn those around.

[1829] The above steps realize a real-time danger detection system using a surveillance camera and a sound-collecting microphone.

[1830] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1831] This invention is a system that uses surveillance cameras and sound-collecting microphones to detect the intrusion of dangerous animals, pests, and invasive species in real time, and uses AI to generate and propose countermeasures appropriate to the situation. It also incorporates an emotion engine that recognizes the user's emotions. The system aims to optimize the countermeasures based on the user's emotional state and reduce the user's psychological burden.

[1832] System configuration

[1833] 1. Surveillance cameras

[1834] The surveillance cameras monitor designated areas and capture high-resolution video data, allowing for day and night surveillance.

[1835] 2. Sound collection microphone

[1836] The sound-collecting microphone collects sounds within a designated area and records them as audio data, allowing you to capture the sounds of animals and pests.

[1837] 3. Terminal

[1838] The terminal temporarily stores data from the surveillance camera and microphone and transmits it to the server. Data can be transmitted in real time.

[1839] 4. Server

[1840] The server analyzes the received video and audio data in real time to detect anomalies, using a pre-trained machine learning model.

[1841] 5. Generation AI

[1842] The generative AI generates appropriate countermeasures based on the detected anomalies, notifies the user of the countermeasures, and automatically takes action as needed.

[1843] 6. Emotion Engine

[1844] The emotion engine analyzes the user's voice and facial expressions to recognize their emotions and adjusts the content of the countermeasures based on that information. The urgency and level of detail of the countermeasures can be changed based on the recognized emotional state.

[1845] 7. Notification System

[1846] Terminal applications and browsers will be provided to notify users of the measures, and a function to notify public authorities in the event of an emergency will also be included.

[1847] Program processing explanation

[1848] Data collection and transmission

[1849] The server periodically acquires real-time video and audio data from the surveillance cameras and microphones, thereby recording all events that occur within the surveillance area.

[1850] The device stores the captured video and audio data in a buffer and prepares it for transmission to the server, where it is then encrypted and sent to the server.

[1851] Start of data analysis

[1852] The server immediately begins the process of analyzing the received video and audio data: video data is analyzed frame by frame, and audio data is analyzed based on the sampling rate.

[1853] The server uses pre-trained machine learning models to identify specific animals, pests, and invasive species from the data it receives with a high degree of accuracy.

[1854] Real-time detection and notification

[1855] The server evaluates certain criteria based on the analysis results and issues alerts if anomalies are detected. Alerts are filtered and different actions are taken depending on their severity.

[1856] The terminal receives the alarm information from the server and notifies the user. The alarm includes information on the location and time of occurrence, and the target organism.

[1857] Countermeasure generation and adjustment

[1858] The server uses generative AI to generate appropriate countermeasures based on the detected anomalies, including specific instructions and precautions.

[1859] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state. For example, if the user is in a high stress state, the generative AI will adjust the urgency and level of detail of the countermeasures.

[1860] Check and implement measures

[1861] The device displays the generated countermeasure information to the user, who can then check the countermeasure proposal and prepare to implement it if necessary.

[1862] The user implements the instructed measures and takes the actions recommended by the system, and the system automatically performs some actions (e.g., sounding an audio alarm).

[1863] Report

[1864] If a serious abnormality occurs, the server automatically reports the situation to security companies and public institutions, including the type of organism detected, its location, and a summary of the countermeasures taken.

[1865] The server monitors the progress of the notification and provides additional information as needed.

[1866] Specific examples

[1867] Example 1: Bear invasion

[1868] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[1869] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[1870] The emotion engine analyzes the user's facial expressions and tone of voice to recognize when the user is in a state of high stress.

[1871] The server uses a generation AI to generate and notify the user of a countermeasure such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated." The server also emphasizes the urgency of the countermeasure, as the user is in a high-stress state.

[1872] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[1873] Example 2: Mosquito outbreak

[1874] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[1875] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[1876] The emotion engine analyzes the user's emotional state and recognizes that they are not particularly stressed.

[1877] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[1878] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[1879] With the above configuration and processing, the present invention can quickly detect abnormalities within a monitored area and provide and implement appropriate countermeasures that are adapted to the user's emotional state, thereby preventing damage from dangerous animals, pests, and invasive species and reducing the psychological burden on the user.

[1880] The processing flow will be explained below.

[1881] Step 1: Data collection

[1882] The server receives video and audio data from the surveillance cameras and microphones in real time, ensuring that all events occurring within the surveillance area are accurately recorded.

[1883] The device temporarily stores the acquired video and audio data in a buffer, and the data is periodically accumulated in preparation for the next processing.

[1884] Step 2: Send the data

[1885] The device prepares the data stored in the buffer for transmission to the server. The data is converted into a predetermined format.

[1886] The device encrypts the prepared data and sends it over the network to the server, ensuring data security.

[1887] Step 3: Begin data analysis

[1888] The server starts analyzing the received video and audio data, where the video data is analyzed frame by frame and the audio data is analyzed based on the sampling rate.

[1889] The server uses pre-trained machine learning models to identify specific animals, pests and invasive species from the data it receives.

[1890] Step 4: Identify specific organisms

[1891] The server uses machine learning models to identify animal and pest patterns in the data, for example detecting the outlines of specific animals in video data and recognizing specific animal sounds in audio data.

[1892] The server determines the type, location, and behavioral patterns of the identified organisms and evaluates whether there are any abnormalities.

[1893] Step 5: Real-time detection

[1894] The server evaluates the analysis results and issues an alert if an abnormality is detected, based on specific criteria such as the size, behavioral patterns, and location of the animal.

[1895] Based on the detected abnormality information, the server determines the importance of the alert and selects the appropriate response.

[1896] Step 6: Notification

[1897] The server sends the alert information to the terminal for the user to view. The alert includes details of the location and time of the alert and the detected organism, allowing the user to take prompt action.

[1898] The terminal notifies the user of the received warning information, and the user receives the warning through voice or pop-up message.

[1899] Step 7: Countermeasure generation and sentiment analysis

[1900] The server uses a generative AI to generate appropriate countermeasures for the detected anomalies, creating countermeasure proposals that include specific instructions for action and precautions.

[1901] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state, for example, determining their stress level based on their voice tone and facial expression.

[1902] The server adjusts the countermeasures generated by the generative AI based on the recognized emotional state, providing more detailed and urgent countermeasures to users in high stress states.

[1903] Step 8: Review and implement measures

[1904] The device displays the adjusted countermeasure information to the user, who then checks the countermeasure proposal and takes necessary measures.

[1905] The user initiates action based on the displayed countermeasures. If the system automatically performs some action (e.g., activates an audio alarm), the user confirms this and takes the necessary steps.

[1906] Step 9: Report

[1907] If a serious abnormality occurs, the server automatically reports the situation to security companies and public institutions, including the type of organism detected, its location, and a summary of the countermeasures taken.

[1908] The server monitors the progress of the report and provides additional information as needed, enabling rapid coordination with relevant authorities.

[1909] Specific examples

[1910] Example 1: Bear invasion

[1911] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[1912] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[1913] The emotion engine analyzes the user's facial expressions and tone of voice to recognize when the user is in a state of high stress.

[1914] The server uses a generation AI to generate and notify the user of a countermeasure such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated." The server also emphasizes the urgency of the countermeasure, as the user is in a high-stress state.

[1915] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[1916] Example 2: Mosquito outbreak

[1917] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[1918] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[1919] The emotion engine analyzes the user's emotional state and recognizes that they are not particularly stressed.

[1920] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[1921] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[1922] With the above specific configuration and processing, the present invention can quickly detect abnormalities within a monitored area and provide and implement appropriate countermeasures that are adapted to the user's emotional state, thereby preventing damage from dangerous animals, pests, and invasive species and reducing the psychological burden on the user.

[1923] Example 2

[1924] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1925] In recent years, the intrusion of dangerous animals, pests, and invasive species has been increasing, and the resulting damage has become more serious. There is a need for a system that can detect such intrusions in real time and respond quickly. However, current systems do not provide countermeasures that take the user's emotional state into consideration, which increases the psychological burden on the user. Therefore, the present invention aims to optimize countermeasures based on the user's emotional state and reduce the psychological burden.

[1926] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1927] In this invention, the server includes means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time, means for analyzing the acquired video and audio data to detect dangerous animals, pests, and invasive species, and an emotion engine for analyzing the user's voice and facial expressions to recognize emotions and adjust countermeasures based on the user's emotional state. This makes it possible to quickly detect abnormalities and provide appropriate countermeasures while reducing the psychological burden on the user.

[1928] A "surveillance camera" is a device that monitors a designated area and captures high-resolution video data.

[1929] A "sound collection microphone" is a device that collects sounds within a specified area and records them as audio data.

[1930] "Video data" refers to a collection of video frames captured from a surveillance camera, which record events occurring within a surveillance area.

[1931] "Audio data" refers to a collection of sound sampling data acquired from a sound-collecting microphone, and is data that records sounds occurring within a specified area.

[1932] The "server" is a central processing unit that receives video data and audio data, analyzes them, and detects abnormalities.

[1933] The "emotion engine" is an engine that analyzes the user's voice and facial expressions to recognize their emotions, and adjusts the content of countermeasures based on that information.

[1934] "Generative AI" is an artificial intelligence model that generates appropriate countermeasures based on detected anomalies.

[1935] "Countermeasure generation" refers to the process of using generative AI to design and provide appropriate countermeasures based on the analysis results.

[1936] A "prompt" is an instruction sentence input to a generation AI, and is text containing instructions for generating a specific countermeasure.

[1937] The "notification means" is a mechanism for notifying the user of the generated countermeasures, and includes a terminal application and a browser.

[1938] "User" refers to the person who uses the system, who receives the proposed measures and implements them.

[1939] This invention is a system that detects the intrusion of dangerous animals, pests, and invasive species in real time and provides optimized countermeasures based on the user's emotional state. This system is composed of surveillance cameras, sound-collecting microphones, a server, terminals, a generation AI, an emotion engine, and a notification system.

[1940] Hardware and Software Details

[1941] Surveillance cameras and microphones

[1942] A surveillance camera is a device that monitors a designated area and captures high-resolution video data, allowing for surveillance day and night.

[1943] A sound collecting microphone is a device that collects sounds within a specified area and records them as audio data, allowing you to capture the sounds of animals and pests.

[1944] server

[1945] The server is a central processing unit that receives and analyzes data from surveillance cameras and microphones in real time. The server has the following functions:

[1946] Data collection function: Video and audio data is acquired in real time and temporarily stored.

[1947] Data analysis function: Analyzes received video and audio data to detect anomalies. This analysis is performed using a pre-trained machine learning model.

[1948] Emotion analysis function: Analyzes the user's voice and facial expressions to recognize their emotional state. An emotion prediction model is used to assess whether the user is in a high-stress or calm state.

[1949] Countermeasure generation function: Using a generation AI, a countermeasure is generated based on the detected anomaly. A prompt sentence is input to the generation AI, and the optimal countermeasure is output.

[1950] Terminal

[1951] The terminal is a device that temporarily stores data from the surveillance camera and microphone and transmits it to the server. The terminal has the following functions:

[1952] Data transmission function: Stored data is encrypted and sent to the server using a secure communication protocol, ensuring data safety.

[1953] Notification function: Notifies users of countermeasure information and warnings from the server. Notification methods include terminal applications, browsers, and audio alarms.

[1954] Generative AI model and prompts

[1955] A generative AI model is an artificial intelligence model that generates appropriate countermeasures based on detected anomalies. A prompt sentence is input to the generative AI model, and the optimal countermeasure is output.

[1956] Example prompt: There's a bear infestation. What precautions should be taken?

[1957] Specific examples

[1958] Example 1: Bear invasion

[1959] The server detects large animals from surveillance camera footage and identifies them as bears based on their characteristics.

[1960] The server issues an alarm and sends information about the bear's intrusion to the terminal.

[1961] The emotion engine analyzes the user's facial expressions and tone of voice to recognize when the user is in a state of high stress.

[1962] The server uses a generation AI to generate and notify the user of a countermeasure such as, "A bear has been spotted. Please evacuate immediately and notify public authorities. An audio alarm will be activated." The server also emphasizes the urgency of the countermeasures, as the user is in a high-stress state.

[1963] The user checks the notification and evacuates, and the system automatically activates an audio alarm.

[1964] Example 2: Mosquito outbreak

[1965] The server analyzes data from the microphone and detects the presence of a large number of mosquitoes.

[1966] The server issues an alarm and sends information about the mosquito outbreak to the terminal.

[1967] The emotion engine analyzes the user's emotional state and recognizes that they are not particularly stressed.

[1968] The server uses a generative AI to generate and notify users of countermeasures such as, "There is a massive mosquito infestation. Please use insect repellent spray and close windows and doors. Also, contact a specialist if necessary."

[1969] The user checks the notification and takes corrective action according to the instructions. The system also assists the user in contacting a specialist.

[1970] In this way, the system can detect abnormalities in real time and provide countermeasures that adapt to the user's emotional state, thereby preventing damage from dangerous animals, pests, and invasive species and reducing the psychological burden on the user.

[1971] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1972] Step 1: Data collection

[1973] The server acquires video data from the surveillance cameras in real time. Specifically, it continuously acquires video data frame by frame and stores it in high resolution. The input is the video data from the surveillance cameras, and the output is the video data temporarily stored in the buffer.

[1974] The server acquires audio data from a microphone in real time based on a sampling rate. The audio data includes environmental sounds within a specified area and the sounds of specific animals. The input is the audio data from the microphone, and the output is the audio data temporarily stored in a buffer.

[1975] Step 2: Send data

[1976] The terminal temporarily stores the acquired video and audio data in a buffer. The input is the data stored in the buffer from the server, and the output is encrypted data.

[1977] The device sends encrypted data to the server using a secure communication protocol (e.g., HTTPS). This ensures data security. The input is encrypted video and audio data, and the output is the data sent to the server.

[1978] Step 3: Data analysis

[1979] The server uses a pre-trained machine learning model to analyze the received video data frame by frame. The input is the video data sent to the server, and the output is the analysis results (e.g., the presence of specific animals, pests, or invasive species).

[1980] The server samples and analyzes the audio data, similarly using pre-trained models to identify specific frequencies and patterns. The input is the audio data sent to the server, and the output is the analysis results (e.g., detection of the sounds of specific animals, pests, or invasive species).

[1981] Step 4: Anomaly detection

[1982] The server evaluates the presence of specific animals, pests, and invasive species based on the results of data analysis, and issues an alert if an abnormality is detected. The input is the data analysis results, and the output is alert information (e.g., location, time, and information about the target species).

[1983] The server filters the alerts and determines the action to be taken (e.g., prioritizing notifications, reporting) depending on the severity. The input is the alert information, and the output is the filtered alert information.

[1984] Step 5: Sentiment Analysis

[1985] The server then analyzes the received video and audio data again and uses an emotion engine to recognize the user's emotional state from their voice and facial expressions. The input is real-time video and audio data, and the output is the user's emotional state (e.g., high stress, calm).

[1986] The server makes adjustments based on the user's emotional state that affect the countermeasures generated by the generation AI. The input is the user's emotional state, and the output is the adjusted countermeasures.

[1987] Step 6: Countermeasure Generation

[1988] The server uses a generative AI to generate countermeasures based on the detected anomalies and the user's emotional state. The generative AI inputs a prompt statement and outputs an optimal countermeasure. The inputs are the prompt statement and the generative AI model, and the output is a countermeasure statement.

[1989] Example: Enter the prompt text "Bears are appearing. What measures are needed?" into the generation AI.

[1990] Step 7: Notification of measures

[1991] The terminal receives countermeasure information from the server and notifies the user. Notification methods include terminal applications, browsers, and audio alarms. The input is countermeasure information from the server, and the output is a notification to the user.

[1992] The device notifies the user of the details of the countermeasures and the implementation procedure, and provides specific instructions for the user to take action. The input is the countermeasure proposal, and the output is specific instructions to the user.

[1993] Step 8: Take action

[1994] The user checks the notified countermeasures and takes action according to the instructions, such as evacuating, reporting, or installing physical barriers. The input is the countermeasure proposal, and the output is the user's action.

[1995] The terminal executes automated countermeasures. For example, the system automatically activates an audio alarm to prevent bears from entering. The input is the countermeasure proposal, and the output is the automatically executed action.

[1996] Step 9: Report Processing

[1997] If a serious abnormality occurs, the server automatically notifies security companies and public institutions. The input is the alarm information of the serious abnormality, and the output is the content of the notification (e.g., the type of organism detected, its location, and an overview of countermeasures).

[1998] The server monitors the progress of the notification and provides additional information as needed. The input is progress feedback and the output is additional information.

[1999] Each processing step of this system detects abnormalities in real time, such as dangerous animals, pests, and invasive species, and generates and implements appropriate countermeasures tailored to the user's emotional state, thereby ensuring safety while reducing the user's psychological burden.

[2000] (Application example 2)

[2001] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2002] Conventional security and surveillance systems have limited capabilities for detecting the intrusion of dangerous animals, pests, and invasive species, making it difficult to quickly implement appropriate countermeasures. Furthermore, they provide countermeasures without taking into account the user's emotional state, which increases the psychological burden on the user. Furthermore, even when an abnormality is detected, the user may not receive an appropriate notification, or the reporting to public authorities may be delayed.

[2003] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring video data from a surveillance camera and audio data from a sound-collecting microphone in real time, means for analyzing the acquired video data and audio data to detect dangerous animals, pests, and invasive species, means for generating countermeasures using a generative model based on the type and intrusion location of the detected species, means for notifying the user of the generated countermeasures and, if necessary, reporting them to public institutions, means for analyzing the user's emotional state, and means for adjusting the generated countermeasures based on the user's emotions. This makes it possible to quickly detect the intrusion of dangerous animals, pests, and invasive species, provide appropriate countermeasures adapted to the user's emotional state, and reduce psychological burden. Furthermore, since anomalies are immediately notified and public institutions are promptly notified when they are detected, greater safety can be achieved.

[2004] A "surveillance camer...

Claims

1. A means for acquiring video data from a surveillance camera and audio data from a sound collecting microphone in real time; A means of analyzing the acquired video and audio data to detect dangerous animals, pests, and invasive species; A means for generating countermeasures using a generation AI based on the type of organism detected and the location of intrusion; means for informing the user of the generated measures and, if necessary, notifying public authorities; A system including:

2. A means for continuously collecting video and audio data using a surveillance camera and a sound-collecting microphone, temporarily storing the data in a buffer, and transmitting the data to a server; means for evaluating the analyzed data based on specific criteria and issuing an alarm if an anomaly is detected; The system of claim 1 .

3. A means for re-verifying the countermeasures proposed by the generating AI, adding supplementary information, and then transmitting the countermeasures to the terminal for display; A means for the system to automatically perform some actions when the user acts on the proposed countermeasures; The system of claim 1 .

Images