system

Abstract

We provide the system. [Solution] An AI support device for disaster relief dogs, A waterproof device with a built-in camera, microphone, and various sensors is attached to the collar. A server equipped with an AI platform that analyzes information collected from the aforementioned camera and microphone in real time, A speaker that transmits instructions generated from the server to the dog, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a disaster rescue dog acts alone, it has limitations in vision and judgment. In particular, the problems of recognizing stationary objects and color vision are factors that reduce the efficiency of rescue activities. Also, it is difficult for a handler to grasp the actions of the rescue dog in real time from a remote location and give appropriate instructions. In order to overcome such problems, there is a need for a technology that complements the limitations of the vision and judgment of disaster rescue dogs and strengthens communication with the handler.

Means for Solving the Problems

[0005] The present invention relates to an AI support device for disaster relief dogs, and provides a system that includes a means for a waterproof device with a built-in camera, microphone, and various sensors to be attached to a collar, a server equipped with an AI infrastructure that analyzes information collected from the camera and microphone in real time, and a speaker that transmits instructions generated by the server to the dog. Furthermore, the aforementioned problems can be solved by providing means for a handler to check information and issue instructions in real time from a remote location, and means for the AI ​​infrastructure to analyze images captured by the camera and detect specific objects through object recognition.

[0006] A "disaster relief dog" is a dog specially trained to search for and rescue missing persons at disaster sites.

[0007] An "AI support device" is a device equipped with artificial intelligence that assists the actions of rescue dogs and compensates for the limitations of their vision and judgment.

[0008] A "camera" is an optical device used to capture image data and to visually record the surrounding environment of a rescue dog.

[0009] A "microphone" is a device used to collect audio data and is used to detect ambient sounds and human voices.

[0010] "Various sensors" is a general term for devices that measure physical data such as temperature, humidity, and acceleration, and are used to measure environmental conditions and the movements of dogs.

[0011] A "waterproof structure" refers to a design that prevents water from entering the interior, allowing the device to maintain its functionality even in harsh environments.

[0012] A "server" is a computer device used for analyzing data and generating instructions; it is used to process collected data.

[0013] A "speaker" is a device used to output voice commands and to communicate instructions to rescue dogs.

[0014] A "handler" refers to a person who operates, trains, and directs rescue dogs, and who oversees rescue operations.

[0015] "Object recognition" is a technology that analyzes images captured by a camera to detect and identify specific objects.

[0016] "Real-time analysis" is a function that processes and analyzes collected data instantly, allowing results to be obtained immediately.

[0017] "Remote location" refers to a place where direct contact with the rescue dog is not possible, but where the handler can still operate the dog from that location.

[0018] "Instructions" refer to messages or voice commands given to rescue dogs to perform actions or movements, and are intended to encourage appropriate behavior. [Brief explanation of the drawing]

[0019] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7]It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0021] First, the language used in the following description will be explained.

[0022] <0In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0023] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0024] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0025] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0027] [First Embodiment]

[0028] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0029] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0032] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0035] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0039] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0040] This invention relates to an AI support device for disaster relief dogs, and its basic configuration and functions will be described in detail. The actual operation method and processing flow will also be discussed.

[0041] Basic device configuration

[0042] This device consists of the following elements:

[0043] 1. Camera: This camera is attached to the rescue dog's collar to collect visual information. Because the camera is mounted close to the dog's line of sight, it can accurately capture the direction and objects the dog is looking at.

[0044] 2. Microphone: This device is used to collect ambient sounds and human voices. It is built into the collar and collects sounds in accordance with the dog's movements.

[0045] 3. Various sensors: These include temperature sensors, humidity sensors, and acceleration sensors. This allows for detailed monitoring of environmental conditions and the dog's movements.

[0046] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[0047] 5. Speaker: This device transmits handler instructions and AI-generated instructions to the dog. It is built into the collar, allowing voice commands to reach the dog.

[0048] Central system (server)

[0049] Data collected from the device is transmitted wirelessly to a central system (server). The server analyzes the received data and performs the following processing:

[0050] Real-time analysis: Instantly analyzes video data from cameras, audio data from microphones, and data from various sensors. It uses image analysis algorithms for object recognition and audio analysis algorithms to classify ambient sounds and human voices.

[0051] Behavioral command generation: Based on the analysis results, behavioral commands are generated for the dog. This supports rescue dogs in acting efficiently.

[0052] User (handler) operation

[0053] The handler can monitor the rescue dog's perspective and situation in real time through a user interface (such as a smartphone or tablet) connected to a central system. They can also issue commands remotely as needed.

[0054] Program processing

[0055] Data collection

[0056] Device: A camera, microphone, and various sensors attached to the collar collect data. For example, if a dog is exploring through rubble, the camera will continuously film its surroundings, and the microphone will record ambient sounds.

[0057] Data transmission

[0058] Terminal: The collected data is sent to the server at regular intervals. This allows the server to receive the data in real time.

[0059] Data Analysis

[0060] Server: Analyzes received data and performs object recognition and audio analysis. For example, it can recognize human faces and limbs from camera images and detect voices such as "help" from audio data.

[0061] Action instruction generation

[0062] Server: Based on the analysis results, it generates appropriate instructions. These include specific instructions such as "move right" or "stay put."

[0063] Sending instructions

[0064] Server: The generated instructions are converted into audio format and sent to the terminal. These instructions are then transmitted to the dog through the terminal's speaker.

[0065] Feedback Collection

[0066] Terminal: Monitors the dog's behavior again after the command is given and sends the collected data to the server. The server continuously analyzes the data and generates new commands as needed.

[0067] Handler operation

[0068] User (Handler): The handler can monitor the rescue dog's status in real time through the user interface and issue additional instructions as needed.

[0069] Specific example

[0070] For example, after a major earthquake, when a rescue dog searches through rubble, video footage and audio recordings from the device are sent to a server. The server analyzes the received data and detects cries for help or people in the rubble. The server then generates an instruction, such as "move to the right," and transmits it to the rescue dog via the device. The handler can monitor the dog's perspective in real time and issue additional instructions as needed, such as "search carefully in that spot."

[0071] In this way, the present invention effectively supports the actions of disaster relief dogs and enables faster and more accurate rescue operations.

[0072] The following describes the processing flow.

[0073] Step 1:

[0074] The device is attached to the dog's collar, and the camera, microphone, and various sensors begin to operate. The camera captures images of the surroundings in real time at a frequency of several frames per second, and the microphone records ambient sounds. Temperature, humidity, and acceleration sensors measure environmental conditions and the dog's movements.

[0075] Step 2:

[0076] The device collects data, packets it sends to the server via wireless communication (e.g., Wi-Fi or Bluetooth), and transmits the data at regular intervals (e.g., every 10 seconds).

[0077] Step 3:

[0078] The server analyzes the data it receives. Specifically, it uses image analysis algorithms to analyze image data captured by cameras and perform object recognition. It also uses voice analysis algorithms to analyze audio data recorded by microphones and classify sounds such as cries for help and ambient noises.

[0079] Step 4:

[0080] The server generates behavioral instructions for the dog based on the analysis results. For example, the analysis results might generate an instruction such as "move right." These instructions are generated in text format.

[0081] Step 5:

[0082] The server converts the generated behavioral instructions into audio format and sends them to the terminal. The audio instructions are played back in a way that is easy for the dog to understand.

[0083] Step 6:

[0084] The device receives instructions and transmits them to the dog as voice commands through its speaker. For example, the command "Go right" is played through the speaker.

[0085] Step 7:

[0086] The device monitors the dog's behavior after the command is given and collects data again. This data includes whether the dog followed the command.

[0087] Step 8:

[0088] The device then packets the collected data and sends it to the server. This allows the server to always have up-to-date information on the dog's current behavior and location.

[0089] Step 9:

[0090] The user (handler) monitors information from the server and terminal in real time and issues additional instructions as needed. The handler uses a smartphone or tablet to give remote instructions via touch panel or voice commands.

[0091] Step 10:

[0092] The server receives additional instructions from the handler, converts them into voice commands, and sends them to the terminal. The terminal then relays the voice commands back to the dog.

[0093] This series of processes uses AI to complement the actions and judgment of disaster relief dogs, supporting effective and rapid rescue operations.

[0094] (Example 1)

[0095] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0096] In disaster relief operations, the swift and precise actions of rescue dogs are crucial. However, traditional methods involve time-consuming information gathering and analysis, making it difficult to accurately communicate instructions to the dogs. Furthermore, it is challenging for handlers in remote locations to grasp the situation in real time and give appropriate instructions to the dogs. This leads to a decrease in the efficiency of rescue operations and a reduction in the success rate of saving lives.

[0097] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0098] In this invention, the server includes means for a waterproof device with a built-in camera, microphone, and various sensors attached to a collar; means for transmitting information collected from the camera and microphone to the server in real time via wireless communication; means for analyzing the information received by the server and performing object recognition and voice analysis; means for converting instructions generated by the server into voice format and transmitting them to a speaker on the collar via wireless communication; and means for transmitting instructions to the dog from the speaker. This makes it possible to significantly improve the efficiency and success rate of rescue operations by enabling the rescue dog to collect accurate information in real time and the server to quickly analyze and issue instructions.

[0099] A "camera" is an optical device attached to a rescue dog's collar to collect visual information, capturing images of the direction and objects the dog is looking at.

[0100] A "microphone" is an acoustic device used to collect ambient sounds and human voices, and it is built into the collar.

[0101] "Various sensors" refers to devices including temperature sensors, humidity sensors, and acceleration sensors, which are used to monitor environmental conditions and the dog's movements in detail.

[0102] "Waterproof construction" means that the entire device is designed to be resistant to water and will function normally even in harsh environments.

[0103] A "device" is a device that incorporates a camera, microphone, and various sensors and is attached to a collar.

[0104] A "server" is a central computing device that receives information transmitted from devices, performs analysis, and generates instructions.

[0105] "Wireless communication" refers to technologies that transmit data without physical media such as cables, and includes Wi-Fi and LTE.

[0106] "Analysis" is the process of processing received data to extract meaningful information, and includes object recognition and speech analysis.

[0107] "Object recognition" is a technology that analyzes video data acquired from a camera to identify specific objects or people.

[0108] "Speech analysis" is a technology that processes audio data to identify meaningful sounds and voices.

[0109] An "instruction" is a command generated by the server that instructs the rescue dog to take a specific action.

[0110] "Voice format" refers to the format in which commands are converted into audio data for use in communicating instructions to dogs.

[0111] A "speaker" is an acoustic device that transmits commands converted into voice format to the dog, and it is built into the collar.

[0112] A "user interface" is a screen that allows handlers to check the status of rescue dogs and issue commands using a smartphone or tablet.

[0113] A "handler" is a person who trains and controls a rescue dog.

[0114] This invention relates to an AI support device for disaster relief dogs, and describes its specific usage method and processing flow. The system consists of a waterproof device with a built-in camera, microphone, and various sensors, which is attached to the dog's collar. It monitors and analyzes the dog's movements and surrounding environment in detail and generates appropriate commands.

[0115] Device Configuration

[0116] hardware

[0117] 1. Camera: A high-resolution camera is attached to the dog's collar to collect the dog's visual information in real time. It includes infrared capabilities for shooting at night or in dimly lit environments.

[0118] 2. Microphone: A microphone is built into the collar to collect ambient sounds and human voices. This allows for an accurate understanding of the surrounding sound environment.

[0119] 3. Various sensors: Includes temperature sensors, humidity sensors, acceleration sensors, etc., to monitor environmental conditions and the dog's movements in detail.

[0120] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[0121] Central system (server)

[0122] The server receives data transmitted from the device via wireless communication (e.g., Wi-Fi or LTE) and performs the following processing.

[0123] 1. Data Analysis: The server receives video data from cameras, audio data from microphones, and various sensor data, and analyzes them in real time. For example, it utilizes libraries such as TENSORFLOW® and OpenCV to perform object recognition using image analysis algorithms. It also uses audio analysis algorithms to classify and recognize specific sounds and voices (e.g., "help").

[0124] 2. Behavioral Instruction Generation: Based on the analysis results, specific behavioral instructions are generated for the rescue dogs. A natural language generation (NLG) algorithm is used to ensure the instructions are specific and appropriate.

[0125] Communication of instructions and feedback

[0126] The behavioral commands generated by the server are converted into audio format and transmitted wirelessly to the device's speaker. For example, using the Google® Text-to-Speech API, text commands can be converted to audio and transmitted to the dog. The dog's movements and surrounding environment after the command are collected again by the device and continuously transmitted to the server. This allows the server to continuously analyze the data and generate new commands.

[0127] User actions

[0128] The user (handler) can view the rescue dog's perspective and status in real time through a user interface such as a smartphone or tablet. Using a dedicated application, they can, for example, check the dog's location on a map or view past search history. The handler can also issue additional commands remotely.

[0129] Specific example

[0130] For example, when rescue dogs search through rubble after a major earthquake, the following specific measures are taken.

[0131] 1. Data Collection: As the dog moves around in the rubble, the device (attached camera, microphone, and sensors) collects video and audio data in real time.

[0132] 2. Data transmission: The collected data is transmitted to the server via wireless communication.

[0133] 3. Data Analysis: The server analyzes the received video data to identify human figures and voices. For example, the server recognizes a "person asking for help" from camera footage and detects the voice saying "help" through audio analysis.

[0134] 4. Generation of action instructions: Based on the analysis results, action instructions such as "move to the right" are generated.

[0135] 5. Sending Instructions: The generated instructions are sent to the device and communicated to the dog via the speaker.

[0136] 6. Feedback Collection: The dog's behavior after receiving instructions is closely monitored by a camera and accelerometer, and the data is retransmitted to a server to collect further information that helps in giving appropriate instructions for subsequent actions.

[0137] 7. Handler Operation: Handlers can use a dedicated app to check the dog's current location on a map and view its past search history. For example, if they want to command the dog to search an additional area, they can tap on the map to input the command.

[0138] Examples of prompts to input into a generative AI model

[0139] "Following a major earthquake, rescue dogs are searching through the rubble. The collected video and audio data is being analyzed immediately to detect human figures and cries for help. Please generate instructions for the next course of action."

[0140] Thus, this invention effectively supports the actions of disaster relief dogs, enabling faster and more accurate rescue operations.

[0141] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0142] Step 1: Data Collection

[0143] Input: The device (camera, microphone, and various sensors attached to the collar) collects information about the dog's movements and surrounding environment in real time.

[0144] Specific operation: The camera captures the dog's field of view and obtains high-resolution video. The microphone records ambient sounds, and temperature, humidity, and acceleration sensors collect environmental information.

[0145] Output: Various collected data (video, audio, temperature, humidity, acceleration) are temporarily stored on the device.

[0146] Step 2: Data transmission

[0147] Input: Various data acquired in the data collection step (video, audio, temperature, humidity, acceleration).

[0148] Specific operation: The device uses wireless communication (e.g., Wi-Fi or LTE) to send the collected data to the server at regular intervals.

[0149] Output: Video data, audio data, and various sensor data transmitted to the server.

[0150] Step 3: Data Analysis

[0151] Input: Received video data, audio data, and various sensor data.

[0152] Specific operation: The server processes video data using image analysis algorithms (e.g., TensorFlow or OpenCV) to analyze each received data, recognizing human faces, limbs, and the state of rubble. It also processes audio data using speech analysis algorithms to detect specific phrases such as "help."

[0153] Output: Recognized objects and audio information as analysis results.

[0154] Step 4: Generate action instructions

[0155] Input: Analysis results obtained in the data analysis step.

[0156] Specific operation: Based on the analysis results, the server uses an AI model to generate behavioral instructions for the dog. For example, specific instructions such as "move right" or "stay put" are generated.

[0157] Output: Generated action instructions.

[0158] Step 5: Send Instructions

[0159] Input: Generated action instructions.

[0160] Specific operation: The server converts the generated action instructions into speech format (for example, using the Google Text-to-Speech API) and sends them to the terminal via wireless communication.

[0161] Output: Action instructions converted into voice format are sent to the terminal.

[0162] Step 6: Instructions and communication

[0163] Input: Action instructions converted into voice format.

[0164] Specific operation: Voice commands are played from a speaker attached to the device, conveying instructions to the dog such as "move to the right" or "stay in place."

[0165] Output: The dog receives and executes the command.

[0166] Step 7: Gathering Feedback

[0167] Input: Information about the dog's movements after receiving a command and its surrounding environment.

[0168] Specific operation: The device monitors the dog's behavior again after the command is given and sends the collected data to the server.

[0169] Output: Feedback data sent to the server (information about the dog's movements and surrounding environment).

[0170] Step 8: Handler Operation

[0171] Input: Information that the user (handler) checks in real time.

[0172] Specific actions: The handler uses a dedicated application to check the rescue dog's current location, video, and audio data via a smartphone or tablet. They can also issue additional instructions remotely as needed.

[0173] Output: Additional instructions from the handler are sent to the terminal via the server and then transmitted to the rescue dog.

[0174] Through the steps described above, this system effectively supports the activities of disaster relief dogs and enables rapid and accurate rescue operations.

[0175] (Application Example 1)

[0176] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0177] In disaster areas, there is a constant need for means to conduct rescue operations quickly and efficiently. Traditional methods primarily rely on human rescuers and rescue dogs, but these have limitations, and efficiency is particularly low in large areas and dangerous environments. Furthermore, there is a lack of systematic systems for rapid information gathering, analysis, and command, so these challenges need to be addressed. The aim is to build a general-purpose disaster rescue system, including autonomous vehicles, to achieve more efficient information gathering and faster analysis during disasters.

[0178] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0179] In this invention, the server includes means for performing object recognition based on video data collected by a camera and using data from an acceleration sensor and a temperature sensor for environmental analysis; means for analyzing data collected by a generative AI model and generating appropriate action instructions for an autonomous vehicle; and means for operating based on prompt statements based on the results of analysis from sensor and voice data. This makes it possible to collect and analyze information quickly and efficiently at disaster sites and optimize rescue operations.

[0180] An "AI support device for disaster relief dogs" is a device equipped with a camera, microphone, and various sensors, and featuring a waterproof design, to support the activities of rescue dogs at disaster sites.

[0181] An "AI platform" is a central processing unit that analyzes information collected from cameras, microphones, and other sources in real time and generates action instructions. It is typically installed on a cloud server or similar location.

[0182] "Real-time analysis" is a data processing method that processes collected data immediately and produces results quickly.

[0183] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates appropriate action instructions based on collected data.

[0184] A "prompt message" is a sentence in the form of an instruction or question used when inputting specific data into a generative AI model.

[0185] A "camera" is a device that collects optical information as video data, and is used to acquire visual information from rescue dogs.

[0186] A "microphone" is a device that collects audio data and is used to record ambient sounds and human voices.

[0187] "Various sensors" refers to devices used to collect environmental data, such as temperature sensors, humidity sensors, and acceleration sensors.

[0188] "Waterproof construction" refers to a design that makes a device resistant to water and moisture, allowing it to function properly even in harsh environments.

[0189] A "speaker" is a device that outputs audio data and is used to transmit generated instructions as sound.

[0190] An "autonomous vehicle" is a vehicle that uses artificial intelligence to operate autonomously, and is intended for use in rescue operations at disaster sites.

[0191] "Object recognition" is a technology that identifies and detects specific objects from video data acquired by a camera.

[0192] "Environmental analysis" is the process of evaluating the surrounding physical and environmental conditions based on data acquired from various sensors.

[0193] "Behavioral instructions" are specific instructions on what actions the subject should take, generated based on the analysis results.

[0194] This invention relates to an AI support system applicable to disaster relief dogs and autonomous vehicles, and specifically describes its basic configuration and functions. It also touches upon actual operation methods and processing flows.

[0195] Basic System Configuration

[0196] This system consists of the following elements:

[0197] 1. Cameras: These are devices mounted on the front and sides of autonomous vehicles that collect images of the surroundings in real time.

[0198] 2. Microphone: A device used to collect ambient sounds, and multiple microphones are placed in the vehicle.

[0199] 3. Various sensors: These include temperature sensors, humidity sensors, acceleration sensors, LiDAR, etc., and are devices for detailed monitoring of the environment and vehicle movement.

[0200] 4. Waterproof structure: The entire vehicle and device are designed to be waterproof, ensuring normal operation even in harsh environments.

[0201] 5. Speaker: A device used to transmit voice instructions to vehicles or rescue dogs.

[0202] Data collection and transmission

[0203] The autonomous vehicle, acting as the terminal, collects data through cameras, microphones, and various sensors. This data is transmitted to a server at regular intervals via a communication module.

[0204] Data Analysis

[0205] The server uses the following hardware and software to analyze the received data:

[0206] Hardware: Cloud servers (e.g., AWS®, Google Cloud)

[0207] Software: Image analysis (e.g., OpenCV, TensorFlow), audio analysis (e.g., Amazon Transcribe)

[0208] The server analyzes image data from the camera to detect specific objects. It analyzes audio data to identify ambient sounds and important phrases (such as "help"). Furthermore, it evaluates environmental conditions (temperature, humidity, motion) from sensor data.

[0209] Action instruction generation

[0210] Based on the analysis results, the server uses a generative AI model (for example, an algorithm implemented in Python) to generate appropriate action instructions for the autonomous vehicle. The generated instructions determine the specific action based on the prompt and are communicated to the vehicle.

[0211] Examples of prompt statements

[0212] For example, use the following prompt:

[0213] def generate_disaster_relief_instructions(camera_data, audio_data, sensor_data):

[0214] """

[0215] A function that analyzes camera footage, audio data, and sensor data to generate appropriate instructions for autonomous vehicles.

[0216] :param camera_data: np.ndarray, camera image data

[0217] :param audio_data: list, microphone audio data

[0218] :param sensor_data: dict, data from various sensors

[0219] :return: str, Instructions for the vehicle

[0220] """

[0221] Video data analysis

[0222] objects_detected = object_detection_model.predict(camera_data)

[0223] Audio data analysis

[0224] emergency_phrases = audio_analysis_model.detect_phrases(audio_data)

[0225] Comprehensive data analysis

[0226] if 'human' in objects_detected and 'help' in emergency_phrases:

[0227] Return "Continue moving forward. Human voice detected. Proceed to the right."

[0228] Return "Continue the search."

[0229] Feedback Collection

[0230] Based on the generated instructions, the autonomous vehicle operates, and subsequent data is collected again and sent to the server. The server continuously analyzes the data and generates new instructions as needed.

[0231] Handler operation

[0232] The user (handler) can monitor the perspective and status of the rescue dog or autonomous vehicle in real time through a user interface (such as a smartphone or tablet). They can also issue commands remotely as needed.

[0233] In this way, the present invention can effectively support the actions of disaster relief dogs and autonomous vehicles, enabling faster and more accurate rescue operations.

[0234] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0235] Step 1:

[0236] The terminal (autonomous vehicle) collects data using cameras, microphones, and various sensors. Specifically, it captures images of the surroundings with cameras and records ambient sounds and voices with microphones. It also acquires ambient environmental data using temperature sensors, acceleration sensors, and humidity sensors. This collected data is temporarily stored as raw data in the vehicle's internal storage.

[0237] Input: Environmental video, audio, temperature, acceleration, humidity

[0238] Output: Collected environmental data

[0239] Step 2:

[0240] The terminal transmits the collected data to the server via a communication module. The communication uses a 5G or 4G network and leverages a protocol (MQTT) to ensure data is transmitted in real time. The server temporarily stores the received data for analysis.

[0241] Input: Collected environmental data

[0242] Output: Data to send to the server

[0243] Step 3:

[0244] The server uses various analysis software to analyze the received data. Specifically, it performs object recognition on video data from the camera using OpenCV and TensorFlow to identify people and obstacles. Audio data is analyzed using Amazon Transcribe to detect important voice phrases (e.g., "Help"). Sensor data is used to detect anomalies in temperature and humidity, and acceleration data is used to evaluate the vehicle's operating state.

[0245] Input: Data sent to the server

[0246] Output: Analysis results (object recognition results, audio analysis results, sensor analysis results)

[0247] Step 4:

[0248] The server uses a generative AI model based on the analysis results to generate action instructions for the autonomous vehicle. These generated instructions are specifically constructed based on prompt statements. These prompt statements generate appropriate instructions (e.g., "Turn right," "Explore in place") based on the analysis results. The generative AI model's algorithm is implemented in Python.

[0249] Input: Analysis results

[0250] Output: Action Instructions

[0251] Step 5:

[0252] The server converts the generated action instructions into voice format and sends them back to the terminal. The terminal receives the instructions from the server via a communication module and outputs them through its speaker. These voice instructions prompt the rescue dog or autonomous vehicle to perform its specific action.

[0253] Input: Action Instructions

[0254] Output: Voice instructions

[0255] Step 6:

[0256] The terminal performs actions based on instructions. For example, if a vehicle receives the instruction "move right," it uses the vehicle's motion control algorithm (e.g., ROS - Robot Operating System) to perform the specific action. After the action is performed, the data is collected again and sent to the server.

[0257] Input: Voice Instructions

[0258] Output: Data after execution

[0259] Step 7:

[0260] The server receives the data sent from the terminal again and continues to analyze it. If necessary, it generates new action instructions and sends them to the terminal. By repeating this feedback loop, the rescue operation is optimized.

[0261] Input: Data after execution of the operation

[0262] Output: New analysis results and action instructions

[0263] As described above, this system enables efficient rescue operations by having the server, terminal, and user components work together and repeat a series of processes including data collection, analysis, instruction, and execution.

[0264] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0265] This invention relates to an AI support device for disaster relief dogs, and further describes a system that combines this with an emotion engine that recognizes the emotions of the user (handler). This invention consists of the following elements:

[0266] Basic device configuration

[0267] 1. Camera: This camera is used to collect visual information from rescue dogs. It is attached to the collar and captures images from a position close to the dog's line of sight.

[0268] 2. Microphone: This device is used to collect ambient sounds and the user's voice. It is built into the collar and collects voice data in accordance with the dog's movements.

[0269] 3. Various sensors: These include temperature sensors, humidity sensors, and acceleration sensors. This allows for detailed monitoring of environmental conditions and the dog's movements.

[0270] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[0271] 5. Speaker: This device transmits handler instructions and AI-generated instructions to the dog. It is built into the collar, allowing voice commands to reach the dog.

[0272] Central system (server)

[0273] Data collected from the device is transmitted wirelessly to a central system (server). The server analyzes the received data and performs the following processing:

[0274] Real-time analysis: Instantly analyzes video data from cameras, audio data from microphones, and data from various sensors. It uses image analysis algorithms for object recognition and audio analysis algorithms to classify ambient sounds and human voices.

[0275] Behavioral command generation: Based on the analysis results, behavioral commands are generated for the dog. This supports rescue dogs in acting efficiently.

[0276] Emotion Engine: Analyzes the user's (handler's) voice and video data to recognize their emotional state. For example, it can detect the user's tension or anxiety through voice analysis and detect changes in the user's facial expressions through facial expression analysis.

[0277] User (handler) operation

[0278] The handler can monitor the rescue dog's perspective and situation in real time through a user interface (such as a smartphone or tablet) connected to a central system. They can also issue commands remotely as needed. Furthermore, the system automatically generates more appropriate commands based on the emotional state analyzed by the emotion engine.

[0279] Program processing

[0280] Data collection

[0281] Terminal: Cameras, microphones, and various sensors attached to the collar collect data. For example, when a dog is searching through rubble, the camera continuously captures the surrounding situation, and the microphone records the ambient sound.

[0282] Data transmission

[0283] Terminal: The collected data is transmitted to the server at regular intervals. This enables the server to receive the data in real time.

[0284] Data analysis

[0285] Server: Analyze the received data and perform object recognition and voice analysis. For example, recognize human faces and limbs from camera images and detect voices such as "help" from voice data.

[0286] Action instruction generation

[0287] Server: Generate appropriate instructions based on the analysis results. This includes specific instructions such as "Go right" or "Stay there".

[0288] Instruction transmission

[0289] Server: The generated instructions are converted into voice format and transmitted to the terminal. The voice instructions are played back in an easy-to-understand form for the dog.

[0290] Execution of instructions

[0291] Terminal: Transmit the received instructions to the dog as voice instructions through the speaker. For example, the instruction "Go right" is played back from the speaker.

[0292] Feedback collection

[0293] Terminal: Monitors the dog's behavior again after giving instructions and sends the collected data to the server. This data includes whether the dog followed the instructions.

[0294] Emotion analysis

[0295] Server: Analyzes the user's (handler's) voice and video data and recognizes their emotional state using an emotion engine. For example, it analyzes the user's level of tension from the voice data and recognizes changes in the user's facial expressions from the video data.

[0296] Emotion-based instruction generation

[0297] Server: Generates appropriate behavioral instructions based on the analyzed emotional state. The emotion engine understands the user's feelings and communicates corresponding instructions to the dog.

[0298] Handler operation

[0299] User (Handler): The handler can monitor the rescue dog's status in real time through the user interface and issue additional instructions as needed. Emotional analysis results are also displayed, allowing for appropriate instructions based on the situation.

[0300] Specific example

[0301] For example, after a major earthquake, when a rescue dog searches through rubble, the video and audio captured by the terminal are sent to a server. The server analyzes the received data and detects cries for help or people in the rubble. The server then generates an instruction such as "move to the right" and transmits it to the rescue dog via the terminal. On the other hand, if the emotion engine determines that the handler's voice is tense, it generates an instruction to alleviate that tension (for example, "search slowly"). The handler monitors the dog's perspective in real time and gives additional instructions as needed, such as "search carefully in place."

[0302] In this way, the present invention effectively supports the actions of disaster rescue dogs and realizes more rapid and accurate rescue activities in consideration of the user's emotional state.

[0303] The following describes the processing flow.

[0304] Step 1:

[0305] The terminal is attached to the dog's collar, and the camera, microphone, and various sensors start operating. The camera takes real-time images of the surroundings from a position close to the dog's line of sight, and the microphone collects environmental sounds. The temperature sensor, humidity sensor, and acceleration sensor measure environmental conditions and the dog's movements.

[0306] Step 2:

[0307] The data (image data, audio data, sensor data) collected by the terminal is organized into packets and transmitted to the server via wireless communication (e.g., Wi-Fi or Bluetooth). This transmission is performed at regular intervals (e.g., every 10 seconds).

[0308] Step 3:

[0309] The server analyzes the received data. Specifically, object recognition is performed on the image data captured by the camera using an image analysis algorithm, and the audio data recorded by the microphone is analyzed using an audio analysis algorithm to classify environmental sounds and human voices.

[0310] Step 4:

[0311] Based on the analysis results, the server generates action instructions for the rescue dog. For example, specific instructions such as "Go right" and "Stay there" are generated and recorded in text format.

[0312] Step 5:

[0313] The server generates behavioral commands, converts them into voice commands, and sends them to the terminal. The voice commands are then converted into a format that is easy for the dog to understand.

[0314] Step 6:

[0315] The device receives voice commands and transmits them to the rescue dog through a speaker. For example, the speaker might play the command "Go to the right."

[0316] Step 7:

[0317] The device monitors the rescue dog's actions after the command is given and collects data again. The newly collected data includes whether the rescue dog followed the command.

[0318] Step 8:

[0319] The device repackages the collected data into packets and sends them back to the server. This allows the server to always have up-to-date information on the rescue dog's current actions and location.

[0320] Step 9:

[0321] The server analyzes the user's (handler's) voice and video data. Using an emotion engine, it recognizes emotional states such as tension and anxiety through voice analysis and detects changes in the user's facial expressions through facial expression analysis.

[0322] Step 10:

[0323] The server generates additional action instructions based on the analyzed emotional state. For example, it might generate an instruction like "explore slowly" to reduce the handler's tension.

[0324] Step 11:

[0325] The server sends instructions, including those generated by the emotion engine, to the terminal, which then transmits them to the rescue dog via its speaker.

[0326] Step 12:

[0327] The user (handler) monitors information from the server and terminal in real time and issues additional instructions as needed. The handler uses a smartphone or tablet to remotely control the system via screen touch or voice commands.

[0328] Through these processing steps, the present invention effectively supports the actions of disaster relief dogs and enables accurate rescue operations that take into account the user's emotional state.

[0329] (Example 2)

[0330] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0331] While a wide range of technologies are needed to support the effective deployment of rescue dogs in disaster relief operations, challenges remain in efficient and rapid information gathering and analysis, as well as in generating commands that take into account the handler's emotional state. In particular, it is essential that devices function correctly in harsh environments, analyze collected information in real time, and make appropriate decisions quickly. Furthermore, there is a need for means for handlers to remotely and appropriately direct and manage the actions of rescue dogs.

[0332] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0333] In this invention, the server includes means equipped with an AI platform that analyzes information collected from a camera and microphone in real time, a speaker that transmits instructions generated by the server to the dog, and means including an emotion engine that analyzes the user's voice and video data and recognizes their emotional state. This not only effectively supports the actions of disaster relief dogs but also enables the generation of flexible instructions that take into account the handler's emotional state. Furthermore, the waterproof device ensures normal operation even in harsh environments and enables real-time instructions from a remote location.

[0334] A "camera" is a device used to collect video data, and it is attached to the collar of a rescue dog to record visual information of its surroundings.

[0335] A "microphone" is a device used to collect audio data, and it is built into the dog's collar to record ambient sounds and the user's voice.

[0336] "Various sensors" refers to multiple sensor devices such as temperature sensors, humidity sensors, and acceleration sensors, used to monitor environmental conditions and the dog's movements in detail.

[0337] "Waterproof construction" refers to a design that protects the entire device from water and moisture, ensuring it functions properly even in harsh environments.

[0338] A "device" refers to a set of equipment that includes a camera, microphone, and various sensors, which are attached to the collar of a rescue dog.

[0339] "AI infrastructure" refers to an artificial intelligence system for analyzing collected video and audio data in real time.

[0340] A "server" refers to a central processing unit that receives data transmitted from devices via wireless communication, performs analysis, and generates instructions.

[0341] A "speaker" refers to a device that plays back instructions generated from a server in audio format and transmits them to a dog.

[0342] An "emotion engine" refers to an analysis system that analyzes the voice and video data of a user (handler) to recognize their emotional state.

[0343] A "user interface" refers to the control screen that allows a handler to remotely monitor the device's status in real time and issue commands.

[0344] This invention relates to an AI support device and system for disaster relief dogs. Detailed embodiments thereof are described below.

[0345] Basic device configuration

[0346] The device in this system is attached to the collar of a rescue dog. The set of devices includes the following elements:

[0347] 1. Camera: This device is used to collect visual information from rescue dogs. The camera is attached to the collar and captures images from a position close to the dog's line of sight.

[0348] 2. Microphone: This device is used to collect ambient sounds and the user's voice. It is also built into the collar and collects voice data in accordance with the dog's movements.

[0349] 3. Various sensors: These include temperature sensors, humidity sensors, and acceleration sensors, allowing for detailed monitoring of environmental conditions and the dog's movements.

[0350] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[0351] 5. Speaker: This device transmits commands generated from the server to the dog. It is built into the collar, allowing voice commands to reach the dog.

[0352] Server configuration and functionality

[0353] Data collected from the device is transmitted to a central system (server) via wireless communication (e.g., Wi-Fi, Bluetooth). The server then performs the following processing:

[0354] Real-time analysis: Instantly analyzes video data from cameras, audio data from microphones, and data from various sensors. For example, it uses software libraries such as "TensorFlow" and "OpenCV" to perform image and audio analysis. It can recognize objects and people from camera images and classify ambient sounds and human voices from audio data.

[0355] Behavioral command generation: Based on the analysis results, behavioral commands are generated for the dog. These include specific commands such as "Go right" or "Stay where you are." The commands are then spoken using speech synthesis software (e.g., "Google Text-to-Speech").

[0356] Emotion Engine: Analyzes the user's (handler's) voice and video data to recognize their emotional state. Emotion analysis uses algorithms that analyze voice tone and facial expression changes from the video. This allows the user's emotional state to be recognized, and appropriate action instructions to be generated.

[0357] User (handler) operation

[0358] The user (handler) can view the rescue dog's perspective and situation in real time through a user interface connected to the server (e.g., a smartphone or tablet). The user can issue additional instructions as needed. Furthermore, the system automatically generates more appropriate instructions based on the emotional state analyzed by the emotion engine.

[0359] Specific example

[0360] For example, after a major earthquake, when a rescue dog searches through rubble, the video and audio captured by the device are sent to a server. The server analyzes the received data using TensorFlow and OpenCV to detect cries for help and people in the rubble. The server then generates an instruction such as "move to the right," converts it to speech using Google Text-to-Speech, and transmits it to the rescue dog via the device. Meanwhile, if the emotion engine detects tension in the handler's voice, it generates an instruction to alleviate that tension (for example, "search slowly"). The user (handler) can view the dog's perspective through their smartphone and issue additional instructions in real time, such as "search carefully in place."

[0361] Example of a Generated AI Model Prompt

[0362] Based on natural language processing, please describe the following device and system configuration:

[0363] A collar device for disaster relief dogs equipped with a camera, microphone, and sensors.

[0364] Central server for data analysis and action instruction generation.

[0365] An emotion engine that analyzes the emotions of users (handlers) during rescue operations.

[0366] A system that allows users to check the situation and give instructions through an interface.

[0367] Please describe the system's processes in order, using the server, terminal, and user as the subjects.

[0368] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0369] Step 1:

[0370] Data collection

[0371] Device: A camera, microphone, and various sensors attached to the collar collect data. When the dog is exploring through rubble, the camera continuously captures images of the surroundings, and the microphone records ambient sounds in real time. An accelerometer detects the dog's movements, and temperature and humidity sensors record environmental conditions. This collects data to provide the user with detailed information about the situation.

[0372] Input: Visual information of the surroundings, ambient sounds, dog movements, temperature, humidity, etc.

[0373] Output: Collected visual data, audio data, and sensor data.

[0374] Step 2:

[0375] Data transmission

[0376] Terminal: The collected data is transmitted to the server at regular intervals via wireless communication. For example, camera video, audio data, and sensor data are transmitted to the server every 10 seconds. Possible communication methods include Wi-Fi and Bluetooth. The server receives this data in real time and uses it for analysis.

[0377] Input: Collected visual data, audio data, and sensor data.

[0378] Output: Data sent to the server.

[0379] Step 3:

[0380] Data Analysis

[0381] Server: Analyzes received data and performs object recognition and audio analysis. It recognizes human faces and limbs from camera footage and detects cries for help from audio data. Software libraries such as "TensorFlow" and "OpenCV" are used for this analysis.

[0382] Input: Visual data, audio data, and sensor data sent to the server.

[0383] Output: Analysis results (e.g., human face detection, audio analysis results).

[0384] Step 4:

[0385] Action instruction generation

[0386] Server: Based on the analysis results, it generates appropriate instructions. For example, if a person is detected in the camera footage, it generates specific action instructions such as "move to the right." These instructions are converted into speech using speech synthesis software (e.g., "Google Text-to-Speech").

[0387] Input: Analysis results.

[0388] Output: Action instructions (audio data).

[0389] Step 5:

[0390] Sending instructions

[0391] Server: The generated instructions are converted into audio format and then sent to the terminal. The server sends the audio data to the terminal to quickly communicate the generated behavioral instructions to the dog.

[0392] Input: Action instructions (voice data).

[0393] Output: Audio data sent to the terminal.

[0394] Step 6:

[0395] Executing instructions

[0396] Terminal: Receives commands and transmits them to the dog as voice commands through a speaker. For example, the command "Go right" is played from the speaker, and the dog acts according to that command.

[0397] Input: Audio data sent to the terminal.

[0398] Output: Behavioral commands (voice) transmitted to the dog.

[0399] Step 7:

[0400] Feedback Collection

[0401] Terminal: Monitors the dog's behavior again after giving instructions and simultaneously sends the collected data to the server. Feedback data includes whether the dog followed instructions and changes in the surrounding environment. This feedback data is used for the next analysis.

[0402] Input: Dog behavior data, environmental data.

[0403] Output: Feedback data sent to the server.

[0404] Step 8:

[0405] Emotion analysis

[0406] Server: Analyzes the user's (handler's) voice and video data and recognizes their emotional state using an emotion engine. For example, it analyzes the user's level of tension from their voice tone and recognizes changes in their facial expressions from the video. Voice analysis algorithms and facial expression analysis algorithms are used for this purpose.

[0407] Input: User's voice data and video data.

[0408] Output: Analysis results of the user's emotional state.

[0409] Step 9:

[0410] Emotion-based instruction generation

[0411] Server: Based on the analyzed emotional state, it generates further appropriate action instructions. For example, if the emotion engine detects user tension, it generates instructions to alleviate that state (e.g., "Explore slowly").

[0412] Input: Analysis results of the user's emotional state.

[0413] Output: Action instructions (audio data).

[0414] Step 10:

[0415] Handler operation

[0416] User (Handler): The user can view the rescue dog's perspective and situation in real time through the user interface and issue additional instructions as needed. For example, the user can input the instruction "Carefully search the area" through the user interface.

[0417] Input: Rescue dog's point-of-view video, feedback data.

[0418] Output: Additional action instructions.

[0419] (Application Example 2)

[0420] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0421] Traditional disaster relief dogs and security devices primarily generated behavioral instructions by simply analyzing collected data, without considering the emotional state of the handler or personnel. This resulted in problems such as the inability to respond appropriately when the handler or personnel were experiencing tension or stress, making effective support and instruction difficult. Furthermore, the difficulty in real-time monitoring and issuing commands meant that responses in situations requiring quick and accurate action could be delayed.

[0422] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means equipped with an AI platform that analyzes information collected from the camera and microphone in real time, means that analyzes emotions using voice data and video data as an emotion engine that analyzes human emotional states and generates behavioral instructions based on the analysis results, and means equipped with a speaker that transmits the generated instructions to the dog or security device. This enables real-time monitoring and appropriate behavioral instructions while taking into account the emotional state of the handler or security personnel.

[0423] An "AI support device for disaster relief dogs" is a device that incorporates cameras, microphones, and various sensors to support rescue dogs in disaster relief operations. It utilizes AI to analyze data in real time and generate appropriate action instructions.

[0424] A "camera" is a device used to collect video data, and it is attached to the dog's collar to capture images of the surroundings from a position close to the dog's line of sight.

[0425] A "microphone" is a device used to collect audio data, and is built into the collar to record ambient sounds and the user's voice.

[0426] "Various sensors" are devices that collect data such as temperature, humidity, and acceleration to monitor environmental conditions and user behavior.

[0427] A "waterproof design" is a design feature that protects internal electronic components from water and moisture, enabling the device to function properly even in harsh environments.

[0428] A "server" is a central system that analyzes data collected from cameras, microphones, and various sensors in real time and sends the generated instructions to the devices.

[0429] An "AI platform" is a platform that uses artificial intelligence technology to analyze collected data and generates action instructions based on the analysis results.

[0430] A "speaker" is a device that transmits voice commands generated by a server to a dog, and has the function of conveying behavioral commands by voice.

[0431] An "emotion engine" is a technology for analyzing human emotional states. It uses audio and video data to analyze emotions and generates appropriate instructions based on the analysis results.

[0432] A "handler" is the person responsible for operating and managing the rescue dog, and their role is to monitor the dog's perspective in real time from a remote location and give necessary instructions.

[0433] This invention relates to a system that applies an AI support device for disaster relief dogs to security services. Specifically, it involves a waterproof device with a built-in camera, microphone, and various sensors that is attached to a collar, security robot, or surveillance drone. The system also includes a server equipped with an AI platform that analyzes the information collected by the device in real time, and transmits instructions generated by the server through a speaker.

[0434] Hardware to use

[0435] 1. Camera:

[0436] It will be attached to security robots and surveillance drones to collect video footage of the monitored area.

[0437] Examples: Typical webcams and surveillance cameras.

[0438] 2. Mike:

[0439] It is used to collect ambient sounds and detect suspicious noises or voices.

[0440] Example: High-sensitivity microphone.

[0441] 3. Various sensors:

[0442] Temperature sensors, acceleration sensors, and other sensors monitor environmental changes and anomalies.

[0443] Example: Multifunctional sensor.

[0444] 4. Speakers:

[0445] A device that transmits instructions from a server via voice.

[0446] Example: Portable speaker.

[0447] 5. Security robots:

[0448] It is used as a portable monitoring device.

[0449] Example: Autonomous robot.

[0450] Software to use

[0451] 1. AI foundation:

[0452] Image analysis algorithms (e.g., OpenCV, YOLO)

[0453] Speech analysis algorithms (e.g., Google Speech-to-Text API)

[0454] Emotion engine (e.g., IBM Watson® Tone Analyzer)

[0455] Data collection and analysis process

[0456] The server collects data from cameras, microphones, and various sensors installed in the monitoring area and analyzes it in real time. Specifically, it analyzes video data captured by cameras using OpenCV and YOLO to detect specific objects (e.g., suspicious persons) through object recognition. It also analyzes collected audio data using the Google Speech-to-Text API to identify abnormal sounds and specific voices (e.g., voices calling for help).

[0457] Furthermore, IBM Watson Tone Analyzer is used as an emotion engine to understand the emotional state of security personnel. The system analyzes tension and stress levels from the personnel's voice data, and the server generates appropriate action instructions based on the results.

[0458] Generation and transmission of instructions

[0459] Based on the analysis results, the server generates specific action instructions. For example, if a suspicious person is detected, a warning such as "Please leave immediately" is played through the speaker. Also, if the person in charge is nervous, advice such as "Please relax" is conveyed via voice.

[0460] Specific example

[0461] For example, suppose a security robot is patrolling a surveillance area and its camera captures the movement of a suspicious person. This video data is immediately sent to a server and analyzed by OpenCV or YOLO. If the analysis reveals a suspicious person, the server generates an instruction such as "Please leave immediately" and issues a voice warning through the robot's speaker.

[0462] Example of a prompt

[0463] "Analyze the footage from this security camera to determine if there is a suspicious person present. If a suspicious person is detected, generate the voice message 'There is a suspicious person present, please leave immediately.'"

[0464] Thus, the present invention enables security services to grasp the situation in real time, take into account the emotional state of the person in charge, generate appropriate instructions, and respond quickly and accurately.

[0465] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0466] Step 1:

[0467] The terminal devices (security robots, surveillance drones) use mounted cameras, microphones, and various sensors to collect video, audio, and environmental data from the monitored area. Inputs include camera footage, microphone audio, and sensor readings, and this data is collected in real time.

[0468] Step 2:

[0469] The terminal transmits the collected data to the server via wireless communication. The input consists of video, audio, and sensor data acquired from the terminal, and by transmitting these to the server, the system is ready for real-time analysis.

[0470] Step 3:

[0471] The server analyzes the received video data using OpenCV and YOLO. Specifically, it performs image analysis to detect suspicious individuals and abnormal objects using object recognition algorithms. The input is video data, and the output is the detected object information as a result of the analysis.

[0472] Step 4:

[0473] The server converts the received audio data into text data using the Google Speech-to-Text API. The input is audio data, and the output is the converted text data. This text data is then analyzed using an algorithm that identifies abnormal sounds and specific voices.

[0474] Step 5:

[0475] The server analyzes the received and analyzed emotional data using IBM Watson Tone Analyzer. Input is audio and video data, and output is analyzed emotional state information. This is used to assess the tension and stress levels of security personnel.

[0476] Step 6:

[0477] Based on the analysis results from steps 3-5, the server generates instructions for the next course of action. Specifically, it generates warning instructions for suspicious individuals based on object recognition results, and advice for personnel based on emotion analysis results. The input is the analysis results, and the output is text data as action instructions.

[0478] Step 7:

[0479] The server converts the generated action instructions into audio data. This is done using text-to-speech (TTS) technology. The input is the text data of the action instructions, and the output is the audio data.

[0480] Step 8:

[0481] The server sends the generated audio data to the terminal. The input is audio data, and the output is the transmission of audio data to the terminal.

[0482] Step 9:

[0483] The device plays the received audio data through its speaker, transmitting instructions to the target (dog, suspicious person, security personnel). Input is audio data, and output is actual audio playback. For example, it can warn a suspicious person to "leave immediately."

[0484] Step 10:

[0485] The terminal monitors the situation again using cameras and sensors after the instruction is given, and sends the data to the server. The input is the re-collected video, audio, and sensor data, and the output is the transmission of data to the server. This step allows for the collection of feedback and further improvement of the accuracy of action instructions.

[0486] Through the steps outlined above, a cycle of information gathering, analysis, instruction generation and transmission, and feedback is realized, enabling the provision of effective security services.

[0487] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0488] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0489] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0490] [Second Embodiment]

[0491] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0492] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0493] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0494] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0495] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0496] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0497] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0498] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0499] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0500] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0501] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0502] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0503] This invention relates to an AI support device for disaster relief dogs, and its basic configuration and functions will be described in detail. The actual operation method and processing flow will also be discussed.

[0504] Basic device configuration

[0505] This device consists of the following elements:

[0506] 1. Camera: This camera is attached to the rescue dog's collar to collect visual information. Because the camera is mounted close to the dog's line of sight, it can accurately capture the direction and objects the dog is looking at.

[0507] 2. Microphone: This device is used to collect ambient sounds and human voices. It is built into the collar and collects sounds in accordance with the dog's movements.

[0508] 3. Various sensors: These include temperature sensors, humidity sensors, and acceleration sensors. This allows for detailed monitoring of environmental conditions and the dog's movements.

[0509] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[0510] 5. Speaker: This device transmits handler instructions and AI-generated instructions to the dog. It is built into the collar, allowing voice commands to reach the dog.

[0511] Central system (server)

[0512] Data collected from the device is transmitted wirelessly to a central system (server). The server analyzes the received data and performs the following processing:

[0513] Real-time analysis: Instantly analyzes video data from cameras, audio data from microphones, and data from various sensors. It uses image analysis algorithms for object recognition and audio analysis algorithms to classify ambient sounds and human voices.

[0514] Behavioral command generation: Based on the analysis results, behavioral commands are generated for the dog. This supports rescue dogs in acting efficiently.

[0515] User (handler) operation

[0516] The handler can monitor the rescue dog's perspective and situation in real time through a user interface (such as a smartphone or tablet) connected to a central system. They can also issue commands remotely as needed.

[0517] Program processing

[0518] Data collection

[0519] Device: A camera, microphone, and various sensors attached to the collar collect data. For example, if a dog is exploring through rubble, the camera will continuously film its surroundings, and the microphone will record ambient sounds.

[0520] Data transmission

[0521] Terminal: The collected data is sent to the server at regular intervals. This allows the server to receive the data in real time.

[0522] Data Analysis

[0523] Server: Analyzes received data and performs object recognition and audio analysis. For example, it can recognize human faces and limbs from camera images and detect voices such as "help" from audio data.

[0524] Action instruction generation

[0525] Server: Based on the analysis results, it generates appropriate instructions. These include specific instructions such as "move right" or "stay put."

[0526] Sending instructions

[0527] Server: The generated instructions are converted into audio format and sent to the terminal. These instructions are then transmitted to the dog through the terminal's speaker.

[0528] Feedback Collection

[0529] Terminal: Monitors the dog's behavior again after the command is given and sends the collected data to the server. The server continuously analyzes the data and generates new commands as needed.

[0530] Handler operation

[0531] User (Handler): The handler can monitor the rescue dog's status in real time through the user interface and issue additional instructions as needed.

[0532] Specific example

[0533] For example, after a major earthquake, when a rescue dog searches through rubble, video footage and audio recordings from the device are sent to a server. The server analyzes the received data and detects cries for help or people in the rubble. The server then generates an instruction, such as "move to the right," and transmits it to the rescue dog via the device. The handler can monitor the dog's perspective in real time and issue additional instructions as needed, such as "search carefully in that spot."

[0534] In this way, the present invention effectively supports the actions of disaster relief dogs and enables faster and more accurate rescue operations.

[0535] The following describes the processing flow.

[0536] Step 1:

[0537] The device is attached to the dog's collar, and the camera, microphone, and various sensors begin to operate. The camera captures images of the surroundings in real time at a frequency of several frames per second, and the microphone records ambient sounds. Temperature, humidity, and acceleration sensors measure environmental conditions and the dog's movements.

[0538] Step 2:

[0539] The device collects data, packets it sends to the server via wireless communication (e.g., Wi-Fi or Bluetooth), and transmits the data at regular intervals (e.g., every 10 seconds).

[0540] Step 3:

[0541] The server analyzes the data it receives. Specifically, it uses image analysis algorithms to analyze image data captured by cameras and perform object recognition. It also uses voice analysis algorithms to analyze audio data recorded by microphones and classify sounds such as cries for help and ambient noises.

[0542] Step 4:

[0543] The server generates behavioral instructions for the dog based on the analysis results. For example, the analysis results might generate an instruction such as "move right." These instructions are generated in text format.

[0544] Step 5:

[0545] The server converts the generated behavioral instructions into audio format and sends them to the terminal. The audio instructions are played back in a way that is easy for the dog to understand.

[0546] Step 6:

[0547] The device receives instructions and transmits them to the dog as voice commands through its speaker. For example, the command "Go right" is played through the speaker.

[0548] Step 7:

[0549] The device monitors the dog's behavior after the command is given and collects data again. This data includes whether the dog followed the command.

[0550] Step 8:

[0551] The device then packets the collected data and sends it to the server. This allows the server to always have up-to-date information on the dog's current behavior and location.

[0552] Step 9:

[0553] The user (handler) monitors information from the server and terminal in real time and issues additional instructions as needed. The handler uses a smartphone or tablet to give remote instructions via touch panel or voice commands.

[0554] Step 10:

[0555] The server receives additional instructions from the handler, converts them into voice commands, and sends them to the terminal. The terminal then relays the voice commands back to the dog.

[0556] This series of processes uses AI to complement the actions and judgment of disaster relief dogs, supporting effective and rapid rescue operations.

[0557] (Example 1)

[0558] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0559] In disaster relief operations, the swift and precise actions of rescue dogs are crucial. However, traditional methods involve time-consuming information gathering and analysis, making it difficult to accurately communicate instructions to the dogs. Furthermore, it is challenging for handlers in remote locations to grasp the situation in real time and give appropriate instructions to the dogs. This leads to a decrease in the efficiency of rescue operations and a reduction in the success rate of saving lives.

[0560] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0561] In this invention, the server includes means for a waterproof device with a built-in camera, microphone, and various sensors attached to a collar; means for transmitting information collected from the camera and microphone to the server in real time via wireless communication; means for analyzing the information received by the server and performing object recognition and voice analysis; means for converting instructions generated by the server into voice format and transmitting them to a speaker on the collar via wireless communication; and means for transmitting instructions to the dog from the speaker. This makes it possible to significantly improve the efficiency and success rate of rescue operations by enabling the rescue dog to collect accurate information in real time and the server to quickly analyze and issue instructions.

[0562] A "camera" is an optical device attached to a rescue dog's collar to collect visual information, capturing images of the direction and objects the dog is looking at.

[0563] A "microphone" is an acoustic device used to collect ambient sounds and human voices, and it is built into the collar.

[0564] "Various sensors" refers to devices including temperature sensors, humidity sensors, and acceleration sensors, which are used to monitor environmental conditions and the dog's movements in detail.

[0565] "Waterproof construction" means that the entire device is designed to be resistant to water and will function normally even in harsh environments.

[0566] A "device" is a device that incorporates a camera, microphone, and various sensors and is attached to a collar.

[0567] A "server" is a central computing device that receives information transmitted from devices, performs analysis, and generates instructions.

[0568] "Wireless communication" refers to technologies that transmit data without physical media such as cables, and includes Wi-Fi and LTE.

[0569] "Analysis" is the process of processing received data to extract meaningful information, and includes object recognition and speech analysis.

[0570] "Object recognition" is a technology that analyzes video data acquired from a camera to identify specific objects or people.

[0571] "Speech analysis" is a technology that processes audio data to identify meaningful sounds and voices.

[0572] An "instruction" is a command generated by the server that instructs the rescue dog to take a specific action.

[0573] "Voice format" refers to the format in which commands are converted into audio data for use in communicating instructions to dogs.

[0574] A "speaker" is an acoustic device that transmits commands converted into voice format to the dog, and it is built into the collar.

[0575] A "user interface" is a screen that allows handlers to check the status of rescue dogs and issue commands using a smartphone or tablet.

[0576] A "handler" is a person who trains and controls a rescue dog.

[0577] This invention relates to an AI support device for disaster relief dogs, and describes its specific usage method and processing flow. The system consists of a waterproof device with a built-in camera, microphone, and various sensors, which is attached to the dog's collar. It monitors and analyzes the dog's movements and surrounding environment in detail and generates appropriate commands.

[0578] Device Configuration

[0579] hardware

[0580] 1. Camera: A high-resolution camera is attached to the dog's collar to collect the dog's visual information in real time. It includes infrared capabilities for shooting at night or in dimly lit environments.

[0581] 2. Microphone: A microphone is built into the collar to collect ambient sounds and human voices. This allows for an accurate understanding of the surrounding sound environment.

[0582] 3. Various sensors: Includes temperature sensors, humidity sensors, acceleration sensors, etc., to monitor environmental conditions and the dog's movements in detail.

[0583] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[0584] Central system (server)

[0585] The server receives data transmitted from the device via wireless communication (e.g., Wi-Fi or LTE) and performs the following processing.

[0586] 1. Data Analysis: The server receives video data from cameras, audio data from microphones, and various sensor data, and analyzes them in real time. For example, it utilizes libraries such as TensorFlow and OpenCV to perform object recognition using image analysis algorithms. It also uses audio analysis algorithms to classify and recognize specific sounds and voices (e.g., "help").

[0587] 2. Behavioral Instruction Generation: Based on the analysis results, specific behavioral instructions are generated for the rescue dogs. A natural language generation (NLG) algorithm is used to ensure the instructions are specific and appropriate.

[0588] Communication of instructions and feedback

[0589] The behavioral commands generated by the server are converted into audio format and transmitted wirelessly to the device's speaker. For example, the Google Text-to-Speech API can be used to convert text commands into speech and transmit them to the dog. The dog's movements and surrounding environment after the command are collected again by the device and continuously transmitted to the server. This allows the server to continuously analyze the data and generate new commands.

[0590] User actions

[0591] The user (handler) can view the rescue dog's perspective and status in real time through a user interface such as a smartphone or tablet. Using a dedicated application, they can, for example, check the dog's location on a map or view past search history. The handler can also issue additional commands remotely.

[0592] Specific example

[0593] For example, when rescue dogs search through rubble after a major earthquake, the following specific measures are taken.

[0594] 1. Data Collection: As the dog moves around in the rubble, the device (attached camera, microphone, and sensors) collects video and audio data in real time.

[0595] 2. Data transmission: The collected data is transmitted to the server via wireless communication.

[0596] 3. Data Analysis: The server analyzes the received video data to identify human figures and voices. For example, the server recognizes a "person asking for help" from camera footage and detects the voice saying "help" through audio analysis.

[0597] 4. Generation of action instructions: Based on the analysis results, action instructions such as "move to the right" are generated.

[0598] 5. Sending Instructions: The generated instructions are sent to the device and communicated to the dog via the speaker.

[0599] 6. Feedback Collection: The dog's behavior after receiving instructions is closely monitored by a camera and accelerometer, and the data is retransmitted to a server to collect further information that helps in giving appropriate instructions for subsequent actions.

[0600] 7. Handler Operation: Handlers can use a dedicated app to check the dog's current location on a map and view its past search history. For example, if they want to command the dog to search an additional area, they can tap on the map to input the command.

[0601] Examples of prompts to input into a generative AI model

[0602] "Following a major earthquake, rescue dogs are searching through the rubble. The collected video and audio data is being analyzed immediately to detect human figures and cries for help. Please generate instructions for the next course of action."

[0603] Thus, this invention effectively supports the actions of disaster relief dogs, enabling faster and more accurate rescue operations.

[0604] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0605] Step 1: Data Collection

[0606] Input: The device (camera, microphone, and various sensors attached to the collar) collects information about the dog's movements and surrounding environment in real time.

[0607] Specific operation: The camera captures the dog's field of view and obtains high-resolution video. The microphone records ambient sounds, and temperature, humidity, and acceleration sensors collect environmental information.

[0608] Output: Various collected data (video, audio, temperature, humidity, acceleration) are temporarily stored on the device.

[0609] Step 2: Data transmission

[0610] Input: Various data acquired in the data collection step (video, audio, temperature, humidity, acceleration).

[0611] Specific operation: The device uses wireless communication (e.g., Wi-Fi or LTE) to send the collected data to the server at regular intervals.

[0612] Output: Video data, audio data, and various sensor data transmitted to the server.

[0613] Step 3: Data Analysis

[0614] Input: Received video data, audio data, and various sensor data.

[0615] Specific operation: The server processes video data using image analysis algorithms (e.g., TensorFlow or OpenCV) to analyze each received data, recognizing human faces, limbs, and the state of rubble. It also processes audio data using speech analysis algorithms to detect specific phrases such as "help."

[0616] Output: Recognized objects and audio information as analysis results.

[0617] Step 4: Generate action instructions

[0618] Input: Analysis results obtained in the data analysis step.

[0619] Specific operation: Based on the analysis results, the server uses an AI model to generate behavioral instructions for the dog. For example, specific instructions such as "move right" or "stay put" are generated.

[0620] Output: Generated action instructions.

[0621] Step 5: Send Instructions

[0622] Input: Generated action instructions.

[0623] Specific operation: The server converts the generated action instructions into speech format (for example, using the Google Text-to-Speech API) and sends them to the terminal via wireless communication.

[0624] Output: Action instructions converted into voice format are sent to the terminal.

[0625] Step 6: Instructions and communication

[0626] Input: Action instructions converted into voice format.

[0627] Specific operation: Voice commands are played from a speaker attached to the device, conveying instructions to the dog such as "move to the right" or "stay in place."

[0628] Output: The dog receives and executes the command.

[0629] Step 7: Gathering Feedback

[0630] Input: Information about the dog's movements after receiving a command and its surrounding environment.

[0631] Specific operation: The device monitors the dog's behavior again after the command is given and sends the collected data to the server.

[0632] Output: Feedback data sent to the server (information about the dog's movements and surrounding environment).

[0633] Step 8: Handler Operation

[0634] Input: Information that the user (handler) checks in real time.

[0635] Specific actions: The handler uses a dedicated application to check the rescue dog's current location, video, and audio data via a smartphone or tablet. They can also issue additional instructions remotely as needed.

[0636] Output: Additional instructions from the handler are sent to the terminal via the server and then transmitted to the rescue dog.

[0637] Through the steps described above, this system effectively supports the activities of disaster relief dogs and enables rapid and accurate rescue operations.

[0638] (Application Example 1)

[0639] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0640] In disaster areas, there is a constant need for means to conduct rescue operations quickly and efficiently. Traditional methods primarily rely on human rescuers and rescue dogs, but these have limitations, and efficiency is particularly low in large areas and dangerous environments. Furthermore, there is a lack of systematic systems for rapid information gathering, analysis, and command, so these challenges need to be addressed. The aim is to build a general-purpose disaster rescue system, including autonomous vehicles, to achieve more efficient information gathering and faster analysis during disasters.

[0641] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0642] In this invention, the server includes means for performing object recognition based on video data collected by a camera and using data from an acceleration sensor and a temperature sensor for environmental analysis; means for analyzing data collected by a generative AI model and generating appropriate action instructions for an autonomous vehicle; and means for operating based on prompt statements based on the results of analysis from sensor and voice data. This makes it possible to collect and analyze information quickly and efficiently at disaster sites and optimize rescue operations.

[0643] An "AI support device for disaster relief dogs" is a device equipped with a camera, microphone, and various sensors, and featuring a waterproof design, to support the activities of rescue dogs at disaster sites.

[0644] An "AI platform" is a central processing unit that analyzes information collected from cameras, microphones, and other sources in real time and generates action instructions. It is typically installed on a cloud server or similar location.

[0645] "Real-time analysis" is a data processing method that processes collected data immediately and produces results quickly.

[0646] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates appropriate action instructions based on collected data.

[0647] A "prompt message" is a sentence in the form of an instruction or question used when inputting specific data into a generative AI model.

[0648] A "camera" is a device that collects optical information as video data, and is used to acquire visual information from rescue dogs.

[0649] A "microphone" is a device that collects audio data and is used to record ambient sounds and human voices.

[0650] "Various sensors" refers to devices used to collect environmental data, such as temperature sensors, humidity sensors, and acceleration sensors.

[0651] "Waterproof construction" refers to a design that makes a device resistant to water and moisture, allowing it to function properly even in harsh environments.

[0652] A "speaker" is a device that outputs audio data and is used to transmit generated instructions as sound.

[0653] An "autonomous vehicle" is a vehicle that uses artificial intelligence to operate autonomously, and is intended for use in rescue operations at disaster sites.

[0654] "Object recognition" is a technology that identifies and detects specific objects from video data acquired by a camera.

[0655] "Environmental analysis" is the process of evaluating the surrounding physical and environmental conditions based on data acquired from various sensors.

[0656] "Behavioral instructions" are specific instructions on what actions the subject should take, generated based on the analysis results.

[0657] This invention relates to an AI support system applicable to disaster relief dogs and autonomous vehicles, and specifically describes its basic configuration and functions. It also touches upon actual operation methods and processing flows.

[0658] Basic System Configuration

[0659] This system consists of the following elements:

[0660] 1. Cameras: These are devices mounted on the front and sides of autonomous vehicles that collect images of the surroundings in real time.

[0661] 2. Microphone: A device used to collect ambient sounds, and multiple microphones are placed in the vehicle.

[0662] 3. Various sensors: These include temperature sensors, humidity sensors, acceleration sensors, LiDAR, etc., and are devices for detailed monitoring of the environment and vehicle movement.

[0663] 4. Waterproof structure: The entire vehicle and device are designed to be waterproof, ensuring normal operation even in harsh environments.

[0664] 5. Speaker: A device used to transmit voice instructions to vehicles or rescue dogs.

[0665] Data collection and transmission

[0666] The autonomous vehicle, acting as the terminal, collects data through cameras, microphones, and various sensors. This data is transmitted to a server at regular intervals via a communication module.

[0667] Data Analysis

[0668] The server uses the following hardware and software to analyze the received data:

[0669] Hardware: Cloud servers (e.g., AWS, Google Cloud)

[0670] Software: Image analysis (e.g., OpenCV, TensorFlow), audio analysis (e.g., Amazon Transcribe)

[0671] The server analyzes image data from the camera to detect specific objects. It analyzes audio data to identify ambient sounds and important phrases (such as "help"). Furthermore, it evaluates environmental conditions (temperature, humidity, motion) from sensor data.

[0672] Action instruction generation

[0673] Based on the analysis results, the server uses a generative AI model (for example, an algorithm implemented in Python) to generate appropriate action instructions for the autonomous vehicle. The generated instructions determine the specific action based on the prompt and are communicated to the vehicle.

[0674] Examples of prompt statements

[0675] For example, use the following prompt:

[0676] def generate_disaster_relief_instructions(camera_data, audio_data, sensor_data):

[0677] """

[0678] A function that analyzes camera footage, audio data, and sensor data to generate appropriate instructions for autonomous vehicles.

[0679] :param camera_data: np.ndarray, camera image data

[0680] :param audio_data: list, microphone audio data

[0681] :param sensor_data: dict, data from various sensors

[0682] :return: str, Instructions for the vehicle

[0683] """

[0684] Video data analysis

[0685] objects_detected = object_detection_model.predict(camera_data)

[0686] Audio data analysis

[0687] emergency_phrases = audio_analysis_model.detect_phrases(audio_data)

[0688] Comprehensive data analysis

[0689] if 'human' in objects_detected and 'help' in emergency_phrases:

[0690] Return "Continue moving forward. Human voice detected. Proceed to the right."

[0691] Return "Continue the search."

[0692] Feedback Collection

[0693] Based on the generated instructions, the autonomous vehicle operates, and subsequent data is collected again and sent to the server. The server continuously analyzes the data and generates new instructions as needed.

[0694] Handler operation

[0695] The user (handler) can monitor the perspective and status of the rescue dog or autonomous vehicle in real time through a user interface (such as a smartphone or tablet). They can also issue commands remotely as needed.

[0696] In this way, the present invention can effectively support the actions of disaster relief dogs and autonomous vehicles, enabling faster and more accurate rescue operations.

[0697] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0698] Step 1:

[0699] The terminal (autonomous vehicle) collects data using cameras, microphones, and various sensors. Specifically, it captures images of the surroundings with cameras and records ambient sounds and voices with microphones. It also acquires ambient environmental data using temperature sensors, acceleration sensors, and humidity sensors. This collected data is temporarily stored as raw data in the vehicle's internal storage.

[0700] Input: Environmental video, audio, temperature, acceleration, humidity

[0701] Output: Collected environmental data

[0702] Step 2:

[0703] The terminal transmits the collected data to the server via a communication module. The communication uses a 5G or 4G network and leverages a protocol (MQTT) to ensure data is transmitted in real time. The server temporarily stores the received data for analysis.

[0704] Input: Collected environmental data

[0705] Output: Data to send to the server

[0706] Step 3:

[0707] The server uses various analysis software to analyze the received data. Specifically, it performs object recognition on video data from the camera using OpenCV and TensorFlow to identify people and obstacles. Audio data is analyzed using Amazon Transcribe to detect important voice phrases (e.g., "Help"). Sensor data is used to detect anomalies in temperature and humidity, and acceleration data is used to evaluate the vehicle's operating state.

[0708] Input: Data sent to the server

[0709] Output: Analysis results (object recognition results, audio analysis results, sensor analysis results)

[0710] Step 4:

[0711] The server uses a generative AI model based on the analysis results to generate action instructions for the autonomous vehicle. These generated instructions are specifically constructed based on prompt statements. These prompt statements generate appropriate instructions (e.g., "Turn right," "Explore in place") based on the analysis results. The generative AI model's algorithm is implemented in Python.

[0712] Input: Analysis results

[0713] Output: Action Instructions

[0714] Step 5:

[0715] The server converts the generated action instructions into voice format and sends them back to the terminal. The terminal receives the instructions from the server via a communication module and outputs them through its speaker. These voice instructions prompt the rescue dog or autonomous vehicle to perform its specific action.

[0716] Input: Action Instructions

[0717] Output: Voice instructions

[0718] Step 6:

[0719] The terminal performs actions based on instructions. For example, if a vehicle receives the instruction "move right," it uses the vehicle's motion control algorithm (e.g., ROS - Robot Operating System) to perform the specific action. After the action is performed, the data is collected again and sent to the server.

[0720] Input: Voice Instructions

[0721] Output: Data after execution

[0722] Step 7:

[0723] The server receives the data sent from the terminal again and continues to analyze it. If necessary, it generates new action instructions and sends them to the terminal. By repeating this feedback loop, the rescue operation is optimized.

[0724] Input: Data after execution of the operation

[0725] Output: New analysis results and action instructions

[0726] As described above, this system enables efficient rescue operations by having the server, terminal, and user components work together and repeat a series of processes including data collection, analysis, instruction, and execution.

[0727] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0728] This invention relates to an AI support device for disaster relief dogs, and further describes a system that combines this with an emotion engine that recognizes the emotions of the user (handler). This invention consists of the following elements:

[0729] Basic device configuration

[0730] 1. Camera: This camera is used to collect visual information from rescue dogs. It is attached to the collar and captures images from a position close to the dog's line of sight.

[0731] 2. Microphone: This device is used to collect ambient sounds and the user's voice. It is built into the collar and collects voice data in accordance with the dog's movements.

[0732] 3. Various sensors: These include temperature sensors, humidity sensors, and acceleration sensors. This allows for detailed monitoring of environmental conditions and the dog's movements.

[0733] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[0734] 5. Speaker: This device transmits handler instructions and AI-generated instructions to the dog. It is built into the collar, allowing voice commands to reach the dog.

[0735] Central system (server)

[0736] Data collected from the device is transmitted wirelessly to a central system (server). The server analyzes the received data and performs the following processing:

[0737] Real-time analysis: Instantly analyzes video data from cameras, audio data from microphones, and data from various sensors. It uses image analysis algorithms for object recognition and audio analysis algorithms to classify ambient sounds and human voices.

[0738] Behavioral command generation: Based on the analysis results, behavioral commands are generated for the dog. This supports rescue dogs in acting efficiently.

[0739] Emotion Engine: Analyzes the user's (handler's) voice and video data to recognize their emotional state. For example, it can detect the user's tension or anxiety through voice analysis and detect changes in the user's facial expressions through facial expression analysis.

[0740] User (handler) operation

[0741] The handler can monitor the rescue dog's perspective and situation in real time through a user interface (such as a smartphone or tablet) connected to a central system. They can also issue commands remotely as needed. Furthermore, the system automatically generates more appropriate commands based on the emotional state analyzed by the emotion engine.

[0742] Program processing

[0743] Data collection

[0744] Device: A camera, microphone, and various sensors attached to the collar collect data. For example, if a dog is exploring through rubble, the camera will continuously film its surroundings, and the microphone will record ambient sounds.

[0745] Data transmission

[0746] Terminal: The collected data is sent to the server at regular intervals. This allows the server to receive the data in real time.

[0747] Data Analysis

[0748] Server: Analyzes received data and performs object recognition and audio analysis. For example, it can recognize human faces and limbs from camera images and detect voices such as "help" from audio data.

[0749] Action instruction generation

[0750] Server: Based on the analysis results, it generates appropriate instructions. These include specific instructions such as "move right" or "stay put."

[0751] Sending instructions

[0752] Server: The generated instructions are converted into audio format and sent to the terminal. The audio instructions are played back to the dog in a way that is easy for it to understand.

[0753] Executing instructions

[0754] Terminal: Receives commands and transmits them to the dog as voice commands through a speaker. For example, the command "Go right" is played through the speaker.

[0755] Feedback Collection

[0756] Terminal: Monitors the dog's behavior again after giving instructions and sends the collected data to the server. This data includes whether the dog followed the instructions.

[0757] Emotion analysis

[0758] Server: Analyzes the user's (handler's) voice and video data and recognizes their emotional state using an emotion engine. For example, it analyzes the user's level of tension from the voice data and recognizes changes in the user's facial expressions from the video data.

[0759] Emotion-based instruction generation

[0760] Server: Generates appropriate behavioral instructions based on the analyzed emotional state. The emotion engine understands the user's feelings and communicates corresponding instructions to the dog.

[0761] Handler operation

[0762] User (Handler): The handler can monitor the rescue dog's status in real time through the user interface and issue additional instructions as needed. Emotional analysis results are also displayed, allowing for appropriate instructions based on the situation.

[0763] Specific example

[0764] For example, after a major earthquake, when a rescue dog searches through rubble, the video and audio captured by the terminal are sent to a server. The server analyzes the received data and detects cries for help or people in the rubble. The server then generates an instruction such as "move to the right" and transmits it to the rescue dog via the terminal. On the other hand, if the emotion engine determines that the handler's voice is tense, it generates an instruction to alleviate that tension (for example, "search slowly"). The handler monitors the dog's perspective in real time and gives additional instructions as needed, such as "search carefully in place."

[0765] In this way, the present invention effectively supports the actions of disaster relief dogs and enables faster and more accurate rescue operations while taking into account the user's emotional state.

[0766] The following describes the processing flow.

[0767] Step 1:

[0768] The device is attached to the dog's collar, and the camera, microphone, and various sensors begin operating. The camera captures images of the surroundings in real time from a position close to the dog's line of sight, and the microphone collects ambient sounds. Temperature, humidity, and acceleration sensors measure environmental conditions and the dog's movements.

[0769] Step 2:

[0770] The device collects data (image data, audio data, sensor data), organizes it into packets, and sends them to a server via wireless communication (e.g., Wi-Fi or Bluetooth). This transmission occurs at regular intervals (e.g., every 10 seconds).

[0771] Step 3:

[0772] The server analyzes the received data. Specifically, it uses image analysis algorithms to perform object recognition from image data captured by the camera, and uses voice analysis algorithms to analyze audio data recorded by the microphone, classifying ambient sounds and human voices.

[0773] Step 4:

[0774] The server generates action commands for the rescue dogs based on the analysis results. For example, it generates specific commands such as "move to the right" or "stay put," and records them in text format.

[0775] Step 5:

[0776] The server generates behavioral commands, converts them into voice commands, and sends them to the terminal. The voice commands are then converted into a format that is easy for the dog to understand.

[0777] Step 6:

[0778] The device receives voice commands and transmits them to the rescue dog through a speaker. For example, the speaker might play the command "Go to the right."

[0779] Step 7:

[0780] The device monitors the rescue dog's actions after the command is given and collects data again. The newly collected data includes whether the rescue dog followed the command.

[0781] Step 8:

[0782] The device repackages the collected data into packets and sends them back to the server. This allows the server to always have up-to-date information on the rescue dog's current actions and location.

[0783] Step 9:

[0784] The server analyzes the user's (handler's) voice and video data. Using an emotion engine, it recognizes emotional states such as tension and anxiety through voice analysis and detects changes in the user's facial expressions through facial expression analysis.

[0785] Step 10:

[0786] The server generates additional action instructions based on the analyzed emotional state. For example, it might generate an instruction like "explore slowly" to reduce the handler's tension.

[0787] Step 11:

[0788] The server sends instructions, including those generated by the emotion engine, to the terminal, which then transmits them to the rescue dog via its speaker.

[0789] Step 12:

[0790] The user (handler) monitors information from the server and terminal in real time and issues additional instructions as needed. The handler uses a smartphone or tablet to remotely control the system via screen touch or voice commands.

[0791] Through these processing steps, the present invention effectively supports the actions of disaster relief dogs and enables accurate rescue operations that take into account the user's emotional state.

[0792] (Example 2)

[0793] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0794] While a wide range of technologies are needed to support the effective deployment of rescue dogs in disaster relief operations, challenges remain in efficient and rapid information gathering and analysis, as well as in generating commands that take into account the handler's emotional state. In particular, it is essential that devices function correctly in harsh environments, analyze collected information in real time, and make appropriate decisions quickly. Furthermore, there is a need for means for handlers to remotely and appropriately direct and manage the actions of rescue dogs.

[0795] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0796] In this invention, the server includes means equipped with an AI platform that analyzes information collected from a camera and microphone in real time, a speaker that transmits instructions generated by the server to the dog, and means including an emotion engine that analyzes the user's voice and video data and recognizes their emotional state. This not only effectively supports the actions of disaster relief dogs but also enables the generation of flexible instructions that take into account the handler's emotional state. Furthermore, the waterproof device ensures normal operation even in harsh environments and enables real-time instructions from a remote location.

[0797] A "camera" is a device used to collect video data, and it is attached to the collar of a rescue dog to record visual information of its surroundings.

[0798] A "microphone" is a device used to collect audio data, and it is built into the dog's collar to record ambient sounds and the user's voice.

[0799] "Various sensors" refers to multiple sensor devices such as temperature sensors, humidity sensors, and acceleration sensors, used to monitor environmental conditions and the dog's movements in detail.

[0800] "Waterproof construction" refers to a design that protects the entire device from water and moisture, ensuring it functions properly even in harsh environments.

[0801] A "device" refers to a set of equipment that includes a camera, microphone, and various sensors, which are attached to the collar of a rescue dog.

[0802] "AI infrastructure" refers to an artificial intelligence system for analyzing collected video and audio data in real time.

[0803] A "server" refers to a central processing unit that receives data transmitted from devices via wireless communication, performs analysis, and generates instructions.

[0804] A "speaker" refers to a device that plays back instructions generated from a server in audio format and transmits them to a dog.

[0805] An "emotion engine" refers to an analysis system that analyzes the voice and video data of a user (handler) to recognize their emotional state.

[0806] A "user interface" refers to the control screen that allows a handler to remotely monitor the device's status in real time and issue commands.

[0807] This invention relates to an AI support device and system for disaster relief dogs. Detailed embodiments thereof are described below.

[0808] Basic device configuration

[0809] The device in this system is attached to the collar of a rescue dog. The set of devices includes the following elements:

[0810] 1. Camera: This device is used to collect visual information from rescue dogs. The camera is attached to the collar and captures images from a position close to the dog's line of sight.

[0811] 2. Microphone: This device is used to collect ambient sounds and the user's voice. It is also built into the collar and collects voice data in accordance with the dog's movements.

[0812] 3. Various sensors: These include temperature sensors, humidity sensors, and acceleration sensors, allowing for detailed monitoring of environmental conditions and the dog's movements.

[0813] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[0814] 5. Speaker: This device transmits commands generated from the server to the dog. It is built into the collar, allowing voice commands to reach the dog.

[0815] Server configuration and functionality

[0816] Data collected from the device is transmitted to a central system (server) via wireless communication (e.g., Wi-Fi, Bluetooth). The server then performs the following processing:

[0817] Real-time analysis: Instantly analyzes video data from cameras, audio data from microphones, and data from various sensors. For example, it uses software libraries such as "TensorFlow" and "OpenCV" to perform image and audio analysis. It can recognize objects and people from camera images and classify ambient sounds and human voices from audio data.

[0818] Behavioral command generation: Based on the analysis results, behavioral commands are generated for the dog. These include specific commands such as "Go right" or "Stay where you are." The commands are then spoken using speech synthesis software (e.g., "Google Text-to-Speech").

[0819] Emotion Engine: Analyzes the user's (handler's) voice and video data to recognize their emotional state. Emotion analysis uses algorithms that analyze voice tone and facial expression changes from the video. This allows the user's emotional state to be recognized, and appropriate action instructions to be generated.

[0820] User (handler) operation

[0821] The user (handler) can view the rescue dog's perspective and situation in real time through a user interface connected to the server (e.g., a smartphone or tablet). The user can issue additional instructions as needed. Furthermore, the system automatically generates more appropriate instructions based on the emotional state analyzed by the emotion engine.

[0822] Specific example

[0823] For example, after a major earthquake, when a rescue dog searches through rubble, the video and audio captured by the device are sent to a server. The server analyzes the received data using TensorFlow and OpenCV to detect cries for help and people in the rubble. The server then generates an instruction such as "move to the right," converts it to speech using Google Text-to-Speech, and transmits it to the rescue dog via the device. Meanwhile, if the emotion engine detects tension in the handler's voice, it generates an instruction to alleviate that tension (for example, "search slowly"). The user (handler) can view the dog's perspective through their smartphone and issue additional instructions in real time, such as "search carefully in place."

[0824] Example of a Generated AI Model Prompt

[0825] Based on natural language processing, please describe the following device and system configuration:

[0826] A collar device for disaster relief dogs equipped with a camera, microphone, and sensors.

[0827] Central server for data analysis and action instruction generation.

[0828] An emotion engine that analyzes the emotions of users (handlers) during rescue operations.

[0829] A system that allows users to check the situation and give instructions through an interface.

[0830] Please describe the system's processes in order, using the server, terminal, and user as the subjects.

[0831] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0832] Step 1:

[0833] Data collection

[0834] Device: A camera, microphone, and various sensors attached to the collar collect data. When the dog is exploring through rubble, the camera continuously captures images of the surroundings, and the microphone records ambient sounds in real time. An accelerometer detects the dog's movements, and temperature and humidity sensors record environmental conditions. This collects data to provide the user with detailed information about the situation.

[0835] Input: Visual information of the surroundings, ambient sounds, dog movements, temperature, humidity, etc.

[0836] Output: Collected visual data, audio data, and sensor data.

[0837] Step 2:

[0838] Data transmission

[0839] Terminal: The collected data is transmitted to the server at regular intervals via wireless communication. For example, camera video, audio data, and sensor data are transmitted to the server every 10 seconds. Possible communication methods include Wi-Fi and Bluetooth. The server receives this data in real time and uses it for analysis.

[0840] Input: Collected visual data, audio data, and sensor data.

[0841] Output: Data sent to the server.

[0842] Step 3:

[0843] Data Analysis

[0844] Server: Analyzes received data and performs object recognition and audio analysis. It recognizes human faces and limbs from camera footage and detects cries for help from audio data. Software libraries such as "TensorFlow" and "OpenCV" are used for this analysis.

[0845] Input: Visual data, audio data, and sensor data sent to the server.

[0846] Output: Analysis results (e.g., human face detection, audio analysis results).

[0847] Step 4:

[0848] Action instruction generation

[0849] Server: Based on the analysis results, it generates appropriate instructions. For example, if a person is detected in the camera footage, it generates specific action instructions such as "move to the right." These instructions are converted into speech using speech synthesis software (e.g., "Google Text-to-Speech").

[0850] Input: Analysis results.

[0851] Output: Action instructions (audio data).

[0852] Step 5:

[0853] Sending instructions

[0854] Server: The generated instructions are converted into audio format and then sent to the terminal. The server sends the audio data to the terminal to quickly communicate the generated behavioral instructions to the dog.

[0855] Input: Action instructions (voice data).

[0856] Output: Audio data sent to the terminal.

[0857] Step 6:

[0858] Executing instructions

[0859] Terminal: Receives commands and transmits them to the dog as voice commands through a speaker. For example, the command "Go right" is played from the speaker, and the dog acts according to that command.

[0860] Input: Audio data sent to the terminal.

[0861] Output: Behavioral commands (voice) transmitted to the dog.

[0862] Step 7:

[0863] Feedback Collection

[0864] Terminal: Monitors the dog's behavior again after giving instructions and simultaneously sends the collected data to the server. Feedback data includes whether the dog followed instructions and changes in the surrounding environment. This feedback data is used for the next analysis.

[0865] Input: Dog behavior data, environmental data.

[0866] Output: Feedback data sent to the server.

[0867] Step 8:

[0868] Emotion analysis

[0869] Server: Analyzes the user's (handler's) voice and video data and recognizes their emotional state using an emotion engine. For example, it analyzes the user's level of tension from their voice tone and recognizes changes in their facial expressions from the video. Voice analysis algorithms and facial expression analysis algorithms are used for this purpose.

[0870] Input: User's voice data and video data.

[0871] Output: Analysis results of the user's emotional state.

[0872] Step 9:

[0873] Emotion-based instruction generation

[0874] Server: Based on the analyzed emotional state, it generates further appropriate action instructions. For example, if the emotion engine detects user tension, it generates instructions to alleviate that state (e.g., "Explore slowly").

[0875] Input: Analysis results of the user's emotional state.

[0876] Output: Action instructions (audio data).

[0877] Step 10:

[0878] Handler operation

[0879] User (Handler): The user can view the rescue dog's perspective and situation in real time through the user interface and issue additional instructions as needed. For example, the user can input the instruction "Carefully search the area" through the user interface.

[0880] Input: Rescue dog's point-of-view video, feedback data.

[0881] Output: Additional action instructions.

[0882] (Application Example 2)

[0883] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0884] Traditional disaster relief dogs and security devices primarily generated behavioral instructions by simply analyzing collected data, without considering the emotional state of the handler or personnel. This resulted in problems such as the inability to respond appropriately when the handler or personnel were experiencing tension or stress, making effective support and instruction difficult. Furthermore, the difficulty in real-time monitoring and issuing commands meant that responses in situations requiring quick and accurate action could be delayed.

[0885] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means equipped with an AI platform that analyzes information collected from the camera and microphone in real time, means that analyzes emotions using voice data and video data as an emotion engine that analyzes human emotional states and generates behavioral instructions based on the analysis results, and means equipped with a speaker that transmits the generated instructions to the dog or security device. This enables real-time monitoring and appropriate behavioral instructions while taking into account the emotional state of the handler or security personnel.

[0886] An "AI support device for disaster relief dogs" is a device that incorporates cameras, microphones, and various sensors to support rescue dogs in disaster relief operations. It utilizes AI to analyze data in real time and generate appropriate action instructions.

[0887] A "camera" is a device used to collect video data, and it is attached to the dog's collar to capture images of the surroundings from a position close to the dog's line of sight.

[0888] A "microphone" is a device used to collect audio data, and is built into the collar to record ambient sounds and the user's voice.

[0889] "Various sensors" are devices that collect data such as temperature, humidity, and acceleration to monitor environmental conditions and user behavior.

[0890] A "waterproof design" is a design feature that protects internal electronic components from water and moisture, enabling the device to function properly even in harsh environments.

[0891] A "server" is a central system that analyzes data collected from cameras, microphones, and various sensors in real time and sends the generated instructions to the devices.

[0892] An "AI platform" is a platform that uses artificial intelligence technology to analyze collected data and generates action instructions based on the analysis results.

[0893] A "speaker" is a device that transmits voice commands generated by a server to a dog, and has the function of conveying behavioral commands by voice.

[0894] An "emotion engine" is a technology for analyzing human emotional states. It uses audio and video data to analyze emotions and generates appropriate instructions based on the analysis results.

[0895] A "handler" is the person responsible for operating and managing the rescue dog, and their role is to monitor the dog's perspective in real time from a remote location and give necessary instructions.

[0896] This invention relates to a system that applies an AI support device for disaster relief dogs to security services. Specifically, it involves a waterproof device with a built-in camera, microphone, and various sensors that is attached to a collar, security robot, or surveillance drone. The system also includes a server equipped with an AI platform that analyzes the information collected by the device in real time, and transmits instructions generated by the server through a speaker.

[0897] Hardware to use

[0898] 1. Camera:

[0899] It will be attached to security robots and surveillance drones to collect video footage of the monitored area.

[0900] Examples: Typical webcams and surveillance cameras.

[0901] 2. Mike:

[0902] It is used to collect ambient sounds and detect suspicious noises or voices.

[0903] Example: High-sensitivity microphone.

[0904] 3. Various sensors:

[0905] Temperature sensors, acceleration sensors, and other sensors monitor environmental changes and anomalies.

[0906] Example: Multifunctional sensor.

[0907] 4. Speakers:

[0908] A device that transmits instructions from a server via voice.

[0909] Example: Portable speaker.

[0910] 5. Security robots:

[0911] It is used as a portable monitoring device.

[0912] Example: Autonomous robot.

[0913] Software to use

[0914] 1. AI foundation:

[0915] Image analysis algorithms (e.g., OpenCV, YOLO)

[0916] Speech analysis algorithms (e.g., Google Speech-to-Text API)

[0917] Emotion engine (e.g., IBM Watson Tone Analyzer)

[0918] Data collection and analysis process

[0919] The server collects data from cameras, microphones, and various sensors installed in the monitoring area and analyzes it in real time. Specifically, it analyzes video data captured by cameras using OpenCV and YOLO to detect specific objects (e.g., suspicious persons) through object recognition. It also analyzes collected audio data using the Google Speech-to-Text API to identify abnormal sounds and specific voices (e.g., voices calling for help).

[0920] Furthermore, IBM Watson Tone Analyzer is used as an emotion engine to understand the emotional state of security personnel. The system analyzes tension and stress levels from the personnel's voice data, and the server generates appropriate action instructions based on the results.

[0921] Generation and transmission of instructions

[0922] Based on the analysis results, the server generates specific action instructions. For example, if a suspicious person is detected, a warning such as "Please leave immediately" is played through the speaker. Also, if the person in charge is nervous, advice such as "Please relax" is conveyed via voice.

[0923] Specific example

[0924] For example, suppose a security robot is patrolling a surveillance area and its camera captures the movement of a suspicious person. This video data is immediately sent to a server and analyzed by OpenCV or YOLO. If the analysis reveals a suspicious person, the server generates an instruction such as "Please leave immediately" and issues a voice warning through the robot's speaker.

[0925] Example of a prompt

[0926] "Analyze the footage from this security camera to determine if there is a suspicious person present. If a suspicious person is detected, generate the voice message 'There is a suspicious person present, please leave immediately.'"

[0927] Thus, the present invention enables security services to grasp the situation in real time, take into account the emotional state of the person in charge, generate appropriate instructions, and respond quickly and accurately.

[0928] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0929] Step 1:

[0930] The terminal devices (security robots, surveillance drones) use mounted cameras, microphones, and various sensors to collect video, audio, and environmental data from the monitored area. Inputs include camera footage, microphone audio, and sensor readings, and this data is collected in real time.

[0931] Step 2:

[0932] The terminal transmits the collected data to the server via wireless communication. The input consists of video, audio, and sensor data acquired from the terminal, and by transmitting these to the server, the system is ready for real-time analysis.

[0933] Step 3:

[0934] The server analyzes the received video data using OpenCV and YOLO. Specifically, it performs image analysis to detect suspicious individuals and abnormal objects using object recognition algorithms. The input is video data, and the output is the detected object information as a result of the analysis.

[0935] Step 4:

[0936] The server converts the received audio data into text data using the Google Speech-to-Text API. The input is audio data, and the output is the converted text data. This text data is then analyzed using an algorithm that identifies abnormal sounds and specific voices.

[0937] Step 5:

[0938] The server analyzes the received and analyzed emotional data using IBM Watson Tone Analyzer. Input is audio and video data, and output is analyzed emotional state information. This is used to assess the tension and stress levels of security personnel.

[0939] Step 6:

[0940] Based on the analysis results from steps 3-5, the server generates instructions for the next course of action. Specifically, it generates warning instructions for suspicious individuals based on object recognition results, and advice for personnel based on emotion analysis results. The input is the analysis results, and the output is text data as action instructions.

[0941] Step 7:

[0942] The server converts the generated action instructions into audio data. This is done using text-to-speech (TTS) technology. The input is the text data of the action instructions, and the output is the audio data.

[0943] Step 8:

[0944] The server sends the generated audio data to the terminal. The input is audio data, and the output is the transmission of audio data to the terminal.

[0945] Step 9:

[0946] The device plays the received audio data through its speaker, transmitting instructions to the target (dog, suspicious person, security personnel). Input is audio data, and output is actual audio playback. For example, it can warn a suspicious person to "leave immediately."

[0947] Step 10:

[0948] The terminal monitors the situation again using cameras and sensors after the instruction is given, and sends the data to the server. The input is the re-collected video, audio, and sensor data, and the output is the transmission of data to the server. This step allows for the collection of feedback and further improvement of the accuracy of action instructions.

[0949] Through the steps outlined above, a cycle of information gathering, analysis, instruction generation and transmission, and feedback is realized, enabling the provision of effective security services.

[0950] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0951] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0952] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0953] [Third Embodiment]

[0954] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0955] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0956] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0957] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0958] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0959] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0960] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0961] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0962] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0963] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0964] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0965] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0966] This invention relates to an AI support device for disaster relief dogs, and its basic configuration and functions will be described in detail. The actual operation method and processing flow will also be discussed.

[0967] Basic device configuration

[0968] This device consists of the following elements:

[0969] 1. Camera: This camera is attached to the rescue dog's collar to collect visual information. Because the camera is mounted close to the dog's line of sight, it can accurately capture the direction and objects the dog is looking at.

[0970] 2. Microphone: This device is used to collect ambient sounds and human voices. It is built into the collar and collects sounds in accordance with the dog's movements.

[0971] 3. Various sensors: These include temperature sensors, humidity sensors, and acceleration sensors. This allows for detailed monitoring of environmental conditions and the dog's movements.

[0972] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[0973] 5. Speaker: This device transmits handler instructions and AI-generated instructions to the dog. It is built into the collar, allowing voice commands to reach the dog.

[0974] Central system (server)

[0975] Data collected from the device is transmitted wirelessly to a central system (server). The server analyzes the received data and performs the following processing:

[0976] Real-time analysis: Instantly analyzes video data from cameras, audio data from microphones, and data from various sensors. It uses image analysis algorithms for object recognition and audio analysis algorithms to classify ambient sounds and human voices.

[0977] Behavioral command generation: Based on the analysis results, behavioral commands are generated for the dog. This supports rescue dogs in acting efficiently.

[0978] User (handler) operation

[0979] The handler can monitor the rescue dog's perspective and situation in real time through a user interface (such as a smartphone or tablet) connected to a central system. They can also issue commands remotely as needed.

[0980] Program processing

[0981] Data collection

[0982] Device: A camera, microphone, and various sensors attached to the collar collect data. For example, if a dog is exploring through rubble, the camera will continuously film its surroundings, and the microphone will record ambient sounds.

[0983] Data transmission

[0984] Terminal: The collected data is sent to the server at regular intervals. This allows the server to receive the data in real time.

[0985] Data Analysis

[0986] Server: Analyzes received data and performs object recognition and audio analysis. For example, it can recognize human faces and limbs from camera images and detect voices such as "help" from audio data.

[0987] Action instruction generation

[0988] Server: Based on the analysis results, it generates appropriate instructions. These include specific instructions such as "move right" or "stay put."

[0989] Sending instructions

[0990] Server: The generated instructions are converted into audio format and sent to the terminal. These instructions are then transmitted to the dog through the terminal's speaker.

[0991] Feedback Collection

[0992] Terminal: Monitors the dog's behavior again after the command is given and sends the collected data to the server. The server continuously analyzes the data and generates new commands as needed.

[0993] Handler operation

[0994] User (Handler): The handler can monitor the rescue dog's status in real time through the user interface and issue additional instructions as needed.

[0995] Specific example

[0996] For example, after a major earthquake, when a rescue dog searches through rubble, video footage and audio recordings from the device are sent to a server. The server analyzes the received data and detects cries for help or people in the rubble. The server then generates an instruction, such as "move to the right," and transmits it to the rescue dog via the device. The handler can monitor the dog's perspective in real time and issue additional instructions as needed, such as "search carefully in that spot."

[0997] In this way, the present invention effectively supports the actions of disaster relief dogs and enables faster and more accurate rescue operations.

[0998] The following describes the processing flow.

[0999] Step 1:

[1000] The device is attached to the dog's collar, and the camera, microphone, and various sensors begin to operate. The camera captures images of the surroundings in real time at a frequency of several frames per second, and the microphone records ambient sounds. Temperature, humidity, and acceleration sensors measure environmental conditions and the dog's movements.

[1001] Step 2:

[1002] The device collects data, packets it sends to the server via wireless communication (e.g., Wi-Fi or Bluetooth), and transmits the data at regular intervals (e.g., every 10 seconds).

[1003] Step 3:

[1004] The server analyzes the data it receives. Specifically, it uses image analysis algorithms to analyze image data captured by cameras and perform object recognition. It also uses voice analysis algorithms to analyze audio data recorded by microphones and classify sounds such as cries for help and ambient noises.

[1005] Step 4:

[1006] The server generates behavioral instructions for the dog based on the analysis results. For example, the analysis results might generate an instruction such as "move right." These instructions are generated in text format.

[1007] Step 5:

[1008] The server converts the generated behavioral instructions into audio format and sends them to the terminal. The audio instructions are played back in a way that is easy for the dog to understand.

[1009] Step 6:

[1010] The device receives instructions and transmits them to the dog as voice commands through its speaker. For example, the command "Go right" is played through the speaker.

[1011] Step 7:

[1012] The device monitors the dog's behavior after the command is given and collects data again. This data includes whether the dog followed the command.

[1013] Step 8:

[1014] The device then packets the collected data and sends it to the server. This allows the server to always have up-to-date information on the dog's current behavior and location.

[1015] Step 9:

[1016] The user (handler) monitors information from the server and terminal in real time and issues additional instructions as needed. The handler uses a smartphone or tablet to give remote instructions via touch panel or voice commands.

[1017] Step 10:

[1018] The server receives additional instructions from the handler, converts them into voice commands, and sends them to the terminal. The terminal then relays the voice commands back to the dog.

[1019] This series of processes uses AI to complement the actions and judgment of disaster relief dogs, supporting effective and rapid rescue operations.

[1020] (Example 1)

[1021] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1022] In disaster relief operations, the swift and precise actions of rescue dogs are crucial. However, traditional methods involve time-consuming information gathering and analysis, making it difficult to accurately communicate instructions to the dogs. Furthermore, it is challenging for handlers in remote locations to grasp the situation in real time and give appropriate instructions to the dogs. This leads to a decrease in the efficiency of rescue operations and a reduction in the success rate of saving lives.

[1023] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1024] In this invention, the server includes means for a waterproof device with a built-in camera, microphone, and various sensors attached to a collar; means for transmitting information collected from the camera and microphone to the server in real time via wireless communication; means for analyzing the information received by the server and performing object recognition and voice analysis; means for converting instructions generated by the server into voice format and transmitting them to a speaker on the collar via wireless communication; and means for transmitting instructions to the dog from the speaker. This makes it possible to significantly improve the efficiency and success rate of rescue operations by enabling the rescue dog to collect accurate information in real time and the server to quickly analyze and issue instructions.

[1025] A "camera" is an optical device attached to a rescue dog's collar to collect visual information, capturing images of the direction and objects the dog is looking at.

[1026] A "microphone" is an acoustic device used to collect ambient sounds and human voices, and it is built into the collar.

[1027] "Various sensors" refers to devices including temperature sensors, humidity sensors, and acceleration sensors, which are used to monitor environmental conditions and the dog's movements in detail.

[1028] "Waterproof construction" means that the entire device is designed to be resistant to water and will function normally even in harsh environments.

[1029] A "device" is a device that incorporates a camera, microphone, and various sensors and is attached to a collar.

[1030] A "server" is a central computing device that receives information transmitted from devices, performs analysis, and generates instructions.

[1031] "Wireless communication" refers to technologies that transmit data without physical media such as cables, and includes Wi-Fi and LTE.

[1032] "Analysis" is the process of processing received data to extract meaningful information, and includes object recognition and speech analysis.

[1033] "Object recognition" is a technology that analyzes video data acquired from a camera to identify specific objects or people.

[1034] "Speech analysis" is a technology that processes audio data to identify meaningful sounds and voices.

[1035] An "instruction" is a command generated by the server that instructs the rescue dog to take a specific action.

[1036] "Voice format" refers to the format in which commands are converted into audio data for use in communicating instructions to dogs.

[1037] A "speaker" is an acoustic device that transmits commands converted into voice format to the dog, and it is built into the collar.

[1038] A "user interface" is a screen that allows handlers to check the status of rescue dogs and issue commands using a smartphone or tablet.

[1039] A "handler" is a person who trains and controls a rescue dog.

[1040] This invention relates to an AI support device for disaster relief dogs, and describes its specific usage method and processing flow. The system consists of a waterproof device with a built-in camera, microphone, and various sensors, which is attached to the dog's collar. It monitors and analyzes the dog's movements and surrounding environment in detail and generates appropriate commands.

[1041] Device Configuration

[1042] hardware

[1043] 1. Camera: A high-resolution camera is attached to the dog's collar to collect the dog's visual information in real time. It includes infrared capabilities for shooting at night or in dimly lit environments.

[1044] 2. Microphone: A microphone is built into the collar to collect ambient sounds and human voices. This allows for an accurate understanding of the surrounding sound environment.

[1045] 3. Various sensors: Includes temperature sensors, humidity sensors, acceleration sensors, etc., to monitor environmental conditions and the dog's movements in detail.

[1046] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[1047] Central system (server)

[1048] The server receives data transmitted from the device via wireless communication (e.g., Wi-Fi or LTE) and performs the following processing.

[1049] 1. Data Analysis: The server receives video data from cameras, audio data from microphones, and various sensor data, and analyzes them in real time. For example, it utilizes libraries such as TensorFlow and OpenCV to perform object recognition using image analysis algorithms. It also uses audio analysis algorithms to classify and recognize specific sounds and voices (e.g., "help").

[1050] 2. Behavioral Instruction Generation: Based on the analysis results, specific behavioral instructions are generated for the rescue dogs. A natural language generation (NLG) algorithm is used to ensure the instructions are specific and appropriate.

[1051] Communication of instructions and feedback

[1052] The behavioral commands generated by the server are converted into audio format and transmitted wirelessly to the device's speaker. For example, the Google Text-to-Speech API can be used to convert text commands into speech and transmit them to the dog. The dog's movements and surrounding environment after the command are collected again by the device and continuously transmitted to the server. This allows the server to continuously analyze the data and generate new commands.

[1053] User actions

[1054] The user (handler) can view the rescue dog's perspective and status in real time through a user interface such as a smartphone or tablet. Using a dedicated application, they can, for example, check the dog's location on a map or view past search history. The handler can also issue additional commands remotely.

[1055] Specific example

[1056] For example, when rescue dogs search through rubble after a major earthquake, the following specific measures are taken.

[1057] 1. Data Collection: As the dog moves around in the rubble, the device (attached camera, microphone, and sensors) collects video and audio data in real time.

[1058] 2. Data transmission: The collected data is transmitted to the server via wireless communication.

[1059] 3. Data Analysis: The server analyzes the received video data to identify human figures and voices. For example, the server recognizes a "person asking for help" from camera footage and detects the voice saying "help" through audio analysis.

[1060] 4. Generation of action instructions: Based on the analysis results, action instructions such as "move to the right" are generated.

[1061] 5. Sending Instructions: The generated instructions are sent to the device and communicated to the dog via the speaker.

[1062] 6. Feedback Collection: The dog's behavior after receiving instructions is closely monitored by a camera and accelerometer, and the data is retransmitted to a server to collect further information that helps in giving appropriate instructions for subsequent actions.

[1063] 7. Handler Operation: Handlers can use a dedicated app to check the dog's current location on a map and view its past search history. For example, if they want to command the dog to search an additional area, they can tap on the map to input the command.

[1064] Examples of prompts to input into a generative AI model

[1065] "Following a major earthquake, rescue dogs are searching through the rubble. The collected video and audio data is being analyzed immediately to detect human figures and cries for help. Please generate instructions for the next course of action."

[1066] Thus, this invention effectively supports the actions of disaster relief dogs, enabling faster and more accurate rescue operations.

[1067] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1068] Step 1: Data Collection

[1069] Input: The device (camera, microphone, and various sensors attached to the collar) collects information about the dog's movements and surrounding environment in real time.

[1070] Specific operation: The camera captures the dog's field of view and obtains high-resolution video. The microphone records ambient sounds, and temperature, humidity, and acceleration sensors collect environmental information.

[1071] Output: Various collected data (video, audio, temperature, humidity, acceleration) are temporarily stored on the device.

[1072] Step 2: Data transmission

[1073] Input: Various data acquired in the data collection step (video, audio, temperature, humidity, acceleration).

[1074] Specific operation: The device uses wireless communication (e.g., Wi-Fi or LTE) to send the collected data to the server at regular intervals.

[1075] Output: Video data, audio data, and various sensor data transmitted to the server.

[1076] Step 3: Data Analysis

[1077] Input: Received video data, audio data, and various sensor data.

[1078] Specific operation: The server processes video data using image analysis algorithms (e.g., TensorFlow or OpenCV) to analyze each received data, recognizing human faces, limbs, and the state of rubble. It also processes audio data using speech analysis algorithms to detect specific phrases such as "help."

[1079] Output: Recognized objects and audio information as analysis results.

[1080] Step 4: Generate action instructions

[1081] Input: Analysis results obtained in the data analysis step.

[1082] Specific operation: Based on the analysis results, the server uses an AI model to generate behavioral instructions for the dog. For example, specific instructions such as "move right" or "stay put" are generated.

[1083] Output: Generated action instructions.

[1084] Step 5: Send Instructions

[1085] Input: Generated action instructions.

[1086] Specific operation: The server converts the generated action instructions into speech format (for example, using the Google Text-to-Speech API) and sends them to the terminal via wireless communication.

[1087] Output: Action instructions converted into voice format are sent to the terminal.

[1088] Step 6: Instructions and communication

[1089] Input: Action instructions converted into voice format.

[1090] Specific operation: Voice commands are played from a speaker attached to the device, conveying instructions to the dog such as "move to the right" or "stay in place."

[1091] Output: The dog receives and executes the command.

[1092] Step 7: Gathering Feedback

[1093] Input: Information about the dog's movements after receiving a command and its surrounding environment.

[1094] Specific operation: The device monitors the dog's behavior again after the command is given and sends the collected data to the server.

[1095] Output: Feedback data sent to the server (information about the dog's movements and surrounding environment).

[1096] Step 8: Handler Operation

[1097] Input: Information that the user (handler) checks in real time.

[1098] Specific actions: The handler uses a dedicated application to check the rescue dog's current location, video, and audio data via a smartphone or tablet. They can also issue additional instructions remotely as needed.

[1099] Output: Additional instructions from the handler are sent to the terminal via the server and then transmitted to the rescue dog.

[1100] Through the steps described above, this system effectively supports the activities of disaster relief dogs and enables rapid and accurate rescue operations.

[1101] (Application Example 1)

[1102] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1103] In disaster areas, there is a constant need for means to conduct rescue operations quickly and efficiently. Traditional methods primarily rely on human rescuers and rescue dogs, but these have limitations, and efficiency is particularly low in large areas and dangerous environments. Furthermore, there is a lack of systematic systems for rapid information gathering, analysis, and command, so these challenges need to be addressed. The aim is to build a general-purpose disaster rescue system, including autonomous vehicles, to achieve more efficient information gathering and faster analysis during disasters.

[1104] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1105] In this invention, the server includes means for performing object recognition based on video data collected by a camera and using data from an acceleration sensor and a temperature sensor for environmental analysis; means for analyzing data collected by a generative AI model and generating appropriate action instructions for an autonomous vehicle; and means for operating based on prompt statements based on the results of analysis from sensor and voice data. This makes it possible to collect and analyze information quickly and efficiently at disaster sites and optimize rescue operations.

[1106] An "AI support device for disaster relief dogs" is a device equipped with a camera, microphone, and various sensors, and featuring a waterproof design, to support the activities of rescue dogs at disaster sites.

[1107] An "AI platform" is a central processing unit that analyzes information collected from cameras, microphones, and other sources in real time and generates action instructions. It is typically installed on a cloud server or similar location.

[1108] "Real-time analysis" is a data processing method that processes collected data immediately and produces results quickly.

[1109] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates appropriate action instructions based on collected data.

[1110] A "prompt message" is a sentence in the form of an instruction or question used when inputting specific data into a generative AI model.

[1111] A "camera" is a device that collects optical information as video data, and is used to acquire visual information from rescue dogs.

[1112] A "microphone" is a device that collects audio data and is used to record ambient sounds and human voices.

[1113] "Various sensors" refers to devices used to collect environmental data, such as temperature sensors, humidity sensors, and acceleration sensors.

[1114] "Waterproof construction" refers to a design that makes a device resistant to water and moisture, allowing it to function properly even in harsh environments.

[1115] A "speaker" is a device that outputs audio data and is used to transmit generated instructions as sound.

[1116] An "autonomous vehicle" is a vehicle that uses artificial intelligence to operate autonomously, and is intended for use in rescue operations at disaster sites.

[1117] "Object recognition" is a technology that identifies and detects specific objects from video data acquired by a camera.

[1118] "Environmental analysis" is the process of evaluating the surrounding physical and environmental conditions based on data acquired from various sensors.

[1119] "Behavioral instructions" are specific instructions on what actions the subject should take, generated based on the analysis results.

[1120] This invention relates to an AI support system applicable to disaster relief dogs and autonomous vehicles, and specifically describes its basic configuration and functions. It also touches upon actual operation methods and processing flows.

[1121] Basic System Configuration

[1122] This system consists of the following elements:

[1123] 1. Cameras: These are devices mounted on the front and sides of autonomous vehicles that collect images of the surroundings in real time.

[1124] 2. Microphone: A device used to collect ambient sounds, and multiple microphones are placed in the vehicle.

[1125] 3. Various sensors: These include temperature sensors, humidity sensors, acceleration sensors, LiDAR, etc., and are devices for detailed monitoring of the environment and vehicle movement.

[1126] 4. Waterproof structure: The entire vehicle and device are designed to be waterproof, ensuring normal operation even in harsh environments.

[1127] 5. Speaker: A device used to transmit voice instructions to vehicles or rescue dogs.

[1128] Data collection and transmission

[1129] The autonomous vehicle, acting as the terminal, collects data through cameras, microphones, and various sensors. This data is transmitted to a server at regular intervals via a communication module.

[1130] Data Analysis

[1131] The server uses the following hardware and software to analyze the received data:

[1132] Hardware: Cloud servers (e.g., AWS, Google Cloud)

[1133] Software: Image analysis (e.g., OpenCV, TensorFlow), audio analysis (e.g., Amazon Transcribe)

[1134] The server analyzes image data from the camera to detect specific objects. It analyzes audio data to identify ambient sounds and important phrases (such as "help"). Furthermore, it evaluates environmental conditions (temperature, humidity, motion) from sensor data.

[1135] Action instruction generation

[1136] Based on the analysis results, the server uses a generative AI model (for example, an algorithm implemented in Python) to generate appropriate action instructions for the autonomous vehicle. The generated instructions determine the specific action based on the prompt and are communicated to the vehicle.

[1137] Examples of prompt statements

[1138] For example, use the following prompt:

[1139] def generate_disaster_relief_instructions(camera_data, audio_data, sensor_data):

[1140] """

[1141] A function that analyzes camera footage, audio data, and sensor data to generate appropriate instructions for autonomous vehicles.

[1142] :param camera_data: np.ndarray, camera image data

[1143] :param audio_data: list, microphone audio data

[1144] :param sensor_data: dict, data from various sensors

[1145] :return: str, Instructions for the vehicle

[1146] """

[1147] Video data analysis

[1148] objects_detected = object_detection_model.predict(camera_data)

[1149] Audio data analysis

[1150] emergency_phrases = audio_analysis_model.detect_phrases(audio_data)

[1151] Comprehensive data analysis

[1152] if 'human' in objects_detected and 'help' in emergency_phrases:

[1153] Return "Continue moving forward. Human voice detected. Proceed to the right."

[1154] Return "Continue the search."

[1155] Feedback Collection

[1156] Based on the generated instructions, the autonomous vehicle operates, and subsequent data is collected again and sent to the server. The server continuously analyzes the data and generates new instructions as needed.

[1157] Handler operation

[1158] The user (handler) can monitor the perspective and status of the rescue dog or autonomous vehicle in real time through a user interface (such as a smartphone or tablet). They can also issue commands remotely as needed.

[1159] In this way, the present invention can effectively support the actions of disaster relief dogs and autonomous vehicles, enabling faster and more accurate rescue operations.

[1160] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1161] Step 1:

[1162] The terminal (autonomous vehicle) collects data using cameras, microphones, and various sensors. Specifically, it captures images of the surroundings with cameras and records ambient sounds and voices with microphones. It also acquires ambient environmental data using temperature sensors, acceleration sensors, and humidity sensors. This collected data is temporarily stored as raw data in the vehicle's internal storage.

[1163] Input: Environmental video, audio, temperature, acceleration, humidity

[1164] Output: Collected environmental data

[1165] Step 2:

[1166] The terminal transmits the collected data to the server via a communication module. The communication uses a 5G or 4G network and leverages a protocol (MQTT) to ensure data is transmitted in real time. The server temporarily stores the received data for analysis.

[1167] Input: Collected environmental data

[1168] Output: Data to send to the server

[1169] Step 3:

[1170] The server uses various analysis software to analyze the received data. Specifically, it performs object recognition on video data from the camera using OpenCV and TensorFlow to identify people and obstacles. Audio data is analyzed using Amazon Transcribe to detect important voice phrases (e.g., "Help"). Sensor data is used to detect anomalies in temperature and humidity, and acceleration data is used to evaluate the vehicle's operating state.

[1171] Input: Data sent to the server

[1172] Output: Analysis results (object recognition results, audio analysis results, sensor analysis results)

[1173] Step 4:

[1174] The server uses a generative AI model based on the analysis results to generate action instructions for the autonomous vehicle. These generated instructions are specifically constructed based on prompt statements. These prompt statements generate appropriate instructions (e.g., "Turn right," "Explore in place") based on the analysis results. The generative AI model's algorithm is implemented in Python.

[1175] Input: Analysis results

[1176] Output: Action Instructions

[1177] Step 5:

[1178] The server converts the generated action instructions into voice format and sends them back to the terminal. The terminal receives the instructions from the server via a communication module and outputs them through its speaker. These voice instructions prompt the rescue dog or autonomous vehicle to perform its specific action.

[1179] Input: Action Instructions

[1180] Output: Voice instructions

[1181] Step 6:

[1182] The terminal performs actions based on instructions. For example, if a vehicle receives the instruction "move right," it uses the vehicle's motion control algorithm (e.g., ROS - Robot Operating System) to perform the specific action. After the action is performed, the data is collected again and sent to the server.

[1183] Input: Voice Instructions

[1184] Output: Data after execution

[1185] Step 7:

[1186] The server receives the data sent from the terminal again and continues to analyze it. If necessary, it generates new action instructions and sends them to the terminal. By repeating this feedback loop, the rescue operation is optimized.

[1187] Input: Data after execution of the operation

[1188] Output: New analysis results and action instructions

[1189] As described above, this system enables efficient rescue operations by having the server, terminal, and user components work together and repeat a series of processes including data collection, analysis, instruction, and execution.

[1190] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1191] This invention relates to an AI support device for disaster relief dogs, and further describes a system that combines this with an emotion engine that recognizes the emotions of the user (handler). This invention consists of the following elements:

[1192] Basic device configuration

[1193] 1. Camera: This camera is used to collect visual information from rescue dogs. It is attached to the collar and captures images from a position close to the dog's line of sight.

[1194] 2. Microphone: This device is used to collect ambient sounds and the user's voice. It is built into the collar and collects voice data in accordance with the dog's movements.

[1195] 3. Various sensors: These include temperature sensors, humidity sensors, and acceleration sensors. This allows for detailed monitoring of environmental conditions and the dog's movements.

[1196] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[1197] 5. Speaker: This device transmits handler instructions and AI-generated instructions to the dog. It is built into the collar, allowing voice commands to reach the dog.

[1198] Central system (server)

[1199] Data collected from the device is transmitted wirelessly to a central system (server). The server analyzes the received data and performs the following processing:

[1200] Real-time analysis: Instantly analyzes video data from cameras, audio data from microphones, and data from various sensors. It uses image analysis algorithms for object recognition and audio analysis algorithms to classify ambient sounds and human voices.

[1201] Behavioral command generation: Based on the analysis results, behavioral commands are generated for the dog. This supports rescue dogs in acting efficiently.

[1202] Emotion Engine: Analyzes the user's (handler's) voice and video data to recognize their emotional state. For example, it can detect the user's tension or anxiety through voice analysis and detect changes in the user's facial expressions through facial expression analysis.

[1203] User (handler) operation

[1204] The handler can monitor the rescue dog's perspective and situation in real time through a user interface (such as a smartphone or tablet) connected to a central system. They can also issue commands remotely as needed. Furthermore, the system automatically generates more appropriate commands based on the emotional state analyzed by the emotion engine.

[1205] Program processing

[1206] Data collection

[1207] Device: A camera, microphone, and various sensors attached to the collar collect data. For example, if a dog is exploring through rubble, the camera will continuously film its surroundings, and the microphone will record ambient sounds.

[1208] Data transmission

[1209] Terminal: The collected data is sent to the server at regular intervals. This allows the server to receive the data in real time.

[1210] Data Analysis

[1211] Server: Analyzes received data and performs object recognition and audio analysis. For example, it can recognize human faces and limbs from camera images and detect voices such as "help" from audio data.

[1212] Action instruction generation

[1213] Server: Based on the analysis results, it generates appropriate instructions. These include specific instructions such as "move right" or "stay put."

[1214] Sending instructions

[1215] Server: The generated instructions are converted into audio format and sent to the terminal. The audio instructions are played back to the dog in a way that is easy for it to understand.

[1216] Executing instructions

[1217] Terminal: Receives commands and transmits them to the dog as voice commands through a speaker. For example, the command "Go right" is played through the speaker.

[1218] Feedback Collection

[1219] Terminal: Monitors the dog's behavior again after giving instructions and sends the collected data to the server. This data includes whether the dog followed the instructions.

[1220] Emotion analysis

[1221] Server: Analyzes the user's (handler's) voice and video data and recognizes their emotional state using an emotion engine. For example, it analyzes the user's level of tension from the voice data and recognizes changes in the user's facial expressions from the video data.

[1222] Emotion-based instruction generation

[1223] Server: Generates appropriate behavioral instructions based on the analyzed emotional state. The emotion engine understands the user's feelings and communicates corresponding instructions to the dog.

[1224] Handler operation

[1225] User (Handler): The handler can monitor the rescue dog's status in real time through the user interface and issue additional instructions as needed. Emotional analysis results are also displayed, allowing for appropriate instructions based on the situation.

[1226] Specific example

[1227] For example, after a major earthquake, when a rescue dog searches through rubble, the video and audio captured by the terminal are sent to a server. The server analyzes the received data and detects cries for help or people in the rubble. The server then generates an instruction such as "move to the right" and transmits it to the rescue dog via the terminal. On the other hand, if the emotion engine determines that the handler's voice is tense, it generates an instruction to alleviate that tension (for example, "search slowly"). The handler monitors the dog's perspective in real time and gives additional instructions as needed, such as "search carefully in place."

[1228] In this way, the present invention effectively supports the actions of disaster relief dogs and enables faster and more accurate rescue operations while taking into account the user's emotional state.

[1229] The following describes the processing flow.

[1230] Step 1:

[1231] The device is attached to the dog's collar, and the camera, microphone, and various sensors begin operating. The camera captures images of the surroundings in real time from a position close to the dog's line of sight, and the microphone collects ambient sounds. Temperature, humidity, and acceleration sensors measure environmental conditions and the dog's movements.

[1232] Step 2:

[1233] The device collects data (image data, audio data, sensor data), organizes it into packets, and sends them to a server via wireless communication (e.g., Wi-Fi or Bluetooth). This transmission occurs at regular intervals (e.g., every 10 seconds).

[1234] Step 3:

[1235] The server analyzes the received data. Specifically, it uses image analysis algorithms to perform object recognition from image data captured by the camera, and uses voice analysis algorithms to analyze audio data recorded by the microphone, classifying ambient sounds and human voices.

[1236] Step 4:

[1237] The server generates action commands for the rescue dogs based on the analysis results. For example, it generates specific commands such as "move to the right" or "stay put," and records them in text format.

[1238] Step 5:

[1239] The server generates behavioral commands, converts them into voice commands, and sends them to the terminal. The voice commands are then converted into a format that is easy for the dog to understand.

[1240] Step 6:

[1241] The device receives voice commands and transmits them to the rescue dog through a speaker. For example, the speaker might play the command "Go to the right."

[1242] Step 7:

[1243] The device monitors the rescue dog's actions after the command is given and collects data again. The newly collected data includes whether the rescue dog followed the command.

[1244] Step 8:

[1245] The device repackages the collected data into packets and sends them back to the server. This allows the server to always have up-to-date information on the rescue dog's current actions and location.

[1246] Step 9:

[1247] The server analyzes the user's (handler's) voice and video data. Using an emotion engine, it recognizes emotional states such as tension and anxiety through voice analysis and detects changes in the user's facial expressions through facial expression analysis.

[1248] Step 10:

[1249] The server generates additional action instructions based on the analyzed emotional state. For example, it might generate an instruction like "explore slowly" to reduce the handler's tension.

[1250] Step 11:

[1251] The server sends instructions, including those generated by the emotion engine, to the terminal, which then transmits them to the rescue dog via its speaker.

[1252] Step 12:

[1253] The user (handler) monitors information from the server and terminal in real time and issues additional instructions as needed. The handler uses a smartphone or tablet to remotely control the system via screen touch or voice commands.

[1254] Through these processing steps, the present invention effectively supports the actions of disaster relief dogs and enables accurate rescue operations that take into account the user's emotional state.

[1255] (Example 2)

[1256] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1257] While a wide range of technologies are needed to support the effective deployment of rescue dogs in disaster relief operations, challenges remain in efficient and rapid information gathering and analysis, as well as in generating commands that take into account the handler's emotional state. In particular, it is essential that devices function correctly in harsh environments, analyze collected information in real time, and make appropriate decisions quickly. Furthermore, there is a need for means for handlers to remotely and appropriately direct and manage the actions of rescue dogs.

[1258] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1259] In this invention, the server includes means equipped with an AI platform that analyzes information collected from a camera and microphone in real time, a speaker that transmits instructions generated by the server to the dog, and means including an emotion engine that analyzes the user's voice and video data and recognizes their emotional state. This not only effectively supports the actions of disaster relief dogs but also enables the generation of flexible instructions that take into account the handler's emotional state. Furthermore, the waterproof device ensures normal operation even in harsh environments and enables real-time instructions from a remote location.

[1260] A "camera" is a device used to collect video data, and it is attached to the collar of a rescue dog to record visual information of its surroundings.

[1261] A "microphone" is a device used to collect audio data, and it is built into the dog's collar to record ambient sounds and the user's voice.

[1262] "Various sensors" refers to multiple sensor devices such as temperature sensors, humidity sensors, and acceleration sensors, used to monitor environmental conditions and the dog's movements in detail.

[1263] "Waterproof construction" refers to a design that protects the entire device from water and moisture, ensuring it functions properly even in harsh environments.

[1264] A "device" refers to a set of equipment that includes a camera, microphone, and various sensors, which are attached to the collar of a rescue dog.

[1265] "AI infrastructure" refers to an artificial intelligence system for analyzing collected video and audio data in real time.

[1266] A "server" refers to a central processing unit that receives data transmitted from devices via wireless communication, performs analysis, and generates instructions.

[1267] A "speaker" refers to a device that plays back instructions generated from a server in audio format and transmits them to a dog.

[1268] An "emotion engine" refers to an analysis system that analyzes the voice and video data of a user (handler) to recognize their emotional state.

[1269] A "user interface" refers to the control screen that allows a handler to remotely monitor the device's status in real time and issue commands.

[1270] This invention relates to an AI support device and system for disaster relief dogs. Detailed embodiments thereof are described below.

[1271] Basic device configuration

[1272] The device in this system is attached to the collar of a rescue dog. The set of devices includes the following elements:

[1273] 1. Camera: This device is used to collect visual information from rescue dogs. The camera is attached to the collar and captures images from a position close to the dog's line of sight.

[1274] 2. Microphone: This device is used to collect ambient sounds and the user's voice. It is also built into the collar and collects voice data in accordance with the dog's movements.

[1275] 3. Various sensors: These include temperature sensors, humidity sensors, and acceleration sensors, allowing for detailed monitoring of environmental conditions and the dog's movements.

[1276] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[1277] 5. Speaker: This device transmits commands generated from the server to the dog. It is built into the collar, allowing voice commands to reach the dog.

[1278] Server configuration and functionality

[1279] Data collected from the device is transmitted to a central system (server) via wireless communication (e.g., Wi-Fi, Bluetooth). The server then performs the following processing:

[1280] Real-time analysis: Instantly analyzes video data from cameras, audio data from microphones, and data from various sensors. For example, it uses software libraries such as "TensorFlow" and "OpenCV" to perform image and audio analysis. It can recognize objects and people from camera images and classify ambient sounds and human voices from audio data.

[1281] Behavioral command generation: Based on the analysis results, behavioral commands are generated for the dog. These include specific commands such as "Go right" or "Stay where you are." The commands are then spoken using speech synthesis software (e.g., "Google Text-to-Speech").

[1282] Emotion Engine: Analyzes the user's (handler's) voice and video data to recognize their emotional state. Emotion analysis uses algorithms that analyze voice tone and facial expression changes from the video. This allows the user's emotional state to be recognized, and appropriate action instructions to be generated.

[1283] User (handler) operation

[1284] The user (handler) can view the rescue dog's perspective and situation in real time through a user interface connected to the server (e.g., a smartphone or tablet). The user can issue additional instructions as needed. Furthermore, the system automatically generates more appropriate instructions based on the emotional state analyzed by the emotion engine.

[1285] Specific example

[1286] For example, after a major earthquake, when a rescue dog searches through rubble, the video and audio captured by the device are sent to a server. The server analyzes the received data using TensorFlow and OpenCV to detect cries for help and people in the rubble. The server then generates an instruction such as "move to the right," converts it to speech using Google Text-to-Speech, and transmits it to the rescue dog via the device. Meanwhile, if the emotion engine detects tension in the handler's voice, it generates an instruction to alleviate that tension (for example, "search slowly"). The user (handler) can view the dog's perspective through their smartphone and issue additional instructions in real time, such as "search carefully in place."

[1287] Example of a Generated AI Model Prompt

[1288] Based on natural language processing, please describe the following device and system configuration:

[1289] A collar device for disaster relief dogs equipped with a camera, microphone, and sensors.

[1290] Central server for data analysis and action instruction generation.

[1291] An emotion engine that analyzes the emotions of users (handlers) during rescue operations.

[1292] A system that allows users to check the situation and give instructions through an interface.

[1293] Please describe the system's processes in order, using the server, terminal, and user as the subjects.

[1294] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1295] Step 1:

[1296] Data collection

[1297] Device: A camera, microphone, and various sensors attached to the collar collect data. When the dog is exploring through rubble, the camera continuously captures images of the surroundings, and the microphone records ambient sounds in real time. An accelerometer detects the dog's movements, and temperature and humidity sensors record environmental conditions. This collects data to provide the user with detailed information about the situation.

[1298] Input: Visual information of the surroundings, ambient sounds, dog movements, temperature, humidity, etc.

[1299] Output: Collected visual data, audio data, and sensor data.

[1300] Step 2:

[1301] Data transmission

[1302] Terminal: The collected data is transmitted to the server at regular intervals via wireless communication. For example, camera video, audio data, and sensor data are transmitted to the server every 10 seconds. Possible communication methods include Wi-Fi and Bluetooth. The server receives this data in real time and uses it for analysis.

[1303] Input: Collected visual data, audio data, and sensor data.

[1304] Output: Data sent to the server.

[1305] Step 3:

[1306] Data Analysis

[1307] Server: Analyzes received data and performs object recognition and audio analysis. It recognizes human faces and limbs from camera footage and detects cries for help from audio data. Software libraries such as "TensorFlow" and "OpenCV" are used for this analysis.

[1308] Input: Visual data, audio data, and sensor data sent to the server.

[1309] Output: Analysis results (e.g., human face detection, audio analysis results).

[1310] Step 4:

[1311] Action instruction generation

[1312] Server: Based on the analysis results, it generates appropriate instructions. For example, if a person is detected in the camera footage, it generates specific action instructions such as "move to the right." These instructions are converted into speech using speech synthesis software (e.g., "Google Text-to-Speech").

[1313] Input: Analysis results.

[1314] Output: Action instructions (audio data).

[1315] Step 5:

[1316] Sending instructions

[1317] Server: The generated instructions are converted into audio format and then sent to the terminal. The server sends the audio data to the terminal to quickly communicate the generated behavioral instructions to the dog.

[1318] Input: Action instructions (voice data).

[1319] Output: Audio data sent to the terminal.

[1320] Step 6:

[1321] Executing instructions

[1322] Terminal: Receives commands and transmits them to the dog as voice commands through a speaker. For example, the command "Go right" is played from the speaker, and the dog acts according to that command.

[1323] Input: Audio data sent to the terminal.

[1324] Output: Behavioral commands (voice) transmitted to the dog.

[1325] Step 7:

[1326] Feedback Collection

[1327] Terminal: Monitors the dog's behavior again after giving instructions and simultaneously sends the collected data to the server. Feedback data includes whether the dog followed instructions and changes in the surrounding environment. This feedback data is used for the next analysis.

[1328] Input: Dog behavior data, environmental data.

[1329] Output: Feedback data sent to the server.

[1330] Step 8:

[1331] Emotion analysis

[1332] Server: Analyzes the user's (handler's) voice and video data and recognizes their emotional state using an emotion engine. For example, it analyzes the user's level of tension from their voice tone and recognizes changes in their facial expressions from the video. Voice analysis algorithms and facial expression analysis algorithms are used for this purpose.

[1333] Input: User's voice data and video data.

[1334] Output: Analysis results of the user's emotional state.

[1335] Step 9:

[1336] Emotion-based instruction generation

[1337] Server: Based on the analyzed emotional state, it generates further appropriate action instructions. For example, if the emotion engine detects user tension, it generates instructions to alleviate that state (e.g., "Explore slowly").

[1338] Input: Analysis results of the user's emotional state.

[1339] Output: Action instructions (audio data).

[1340] Step 10:

[1341] Handler operation

[1342] User (Handler): The user can view the rescue dog's perspective and situation in real time through the user interface and issue additional instructions as needed. For example, the user can input the instruction "Carefully search the area" through the user interface.

[1343] Input: Rescue dog's point-of-view video, feedback data.

[1344] Output: Additional action instructions.

[1345] (Application Example 2)

[1346] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1347] Traditional disaster relief dogs and security devices primarily generated behavioral instructions by simply analyzing collected data, without considering the emotional state of the handler or personnel. This resulted in problems such as the inability to respond appropriately when the handler or personnel were experiencing tension or stress, making effective support and instruction difficult. Furthermore, the difficulty in real-time monitoring and issuing commands meant that responses in situations requiring quick and accurate action could be delayed.

[1348] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means equipped with an AI platform that analyzes information collected from the camera and microphone in real time, means that analyzes emotions using voice data and video data as an emotion engine that analyzes human emotional states and generates behavioral instructions based on the analysis results, and means equipped with a speaker that transmits the generated instructions to the dog or security device. This enables real-time monitoring and appropriate behavioral instructions while taking into account the emotional state of the handler or security personnel.

[1349] An "AI support device for disaster relief dogs" is a device that incorporates cameras, microphones, and various sensors to support rescue dogs in disaster relief operations. It utilizes AI to analyze data in real time and generate appropriate action instructions.

[1350] A "camera" is a device used to collect video data, and it is attached to the dog's collar to capture images of the surroundings from a position close to the dog's line of sight.

[1351] A "microphone" is a device used to collect audio data, and is built into the collar to record ambient sounds and the user's voice.

[1352] "Various sensors" are devices that collect data such as temperature, humidity, and acceleration to monitor environmental conditions and user behavior.

[1353] A "waterproof design" is a design feature that protects internal electronic components from water and moisture, enabling the device to function properly even in harsh environments.

[1354] A "server" is a central system that analyzes data collected from cameras, microphones, and various sensors in real time and sends the generated instructions to the devices.

[1355] An "AI platform" is a platform that uses artificial intelligence technology to analyze collected data and generates action instructions based on the analysis results.

[1356] A "speaker" is a device that transmits voice commands generated by a server to a dog, and has the function of conveying behavioral commands by voice.

[1357] An "emotion engine" is a technology for analyzing human emotional states. It uses audio and video data to analyze emotions and generates appropriate instructions based on the analysis results.

[1358] A "handler" is the person responsible for operating and managing the rescue dog, and their role is to monitor the dog's perspective in real time from a remote location and give necessary instructions.

[1359] This invention relates to a system that applies an AI support device for disaster relief dogs to security services. Specifically, it involves a waterproof device with a built-in camera, microphone, and various sensors that is attached to a collar, security robot, or surveillance drone. The system also includes a server equipped with an AI platform that analyzes the information collected by the device in real time, and transmits instructions generated by the server through a speaker.

[1360] Hardware to use

[1361] 1. Camera:

[1362] It will be attached to security robots and surveillance drones to collect video footage of the monitored area.

[1363] Examples: Typical webcams and surveillance cameras.

[1364] 2. Mike:

[1365] It is used to collect ambient sounds and detect suspicious noises or voices.

[1366] Example: High-sensitivity microphone.

[1367] 3. Various sensors:

[1368] Temperature sensors, acceleration sensors, and other sensors monitor environmental changes and anomalies.

[1369] Example: Multifunctional sensor.

[1370] 4. Speakers:

[1371] A device that transmits instructions from a server via voice.

[1372] Example: Portable speaker.

[1373] 5. Security robots:

[1374] It is used as a portable monitoring device.

[1375] Example: Autonomous robot.

[1376] Software to use

[1377] 1. AI foundation:

[1378] Image analysis algorithms (e.g., OpenCV, YOLO)

[1379] Speech analysis algorithms (e.g., Google Speech-to-Text API)

[1380] Emotion engine (e.g., IBM Watson Tone Analyzer)

[1381] Data collection and analysis process

[1382] The server collects data from cameras, microphones, and various sensors installed in the monitoring area and analyzes it in real time. Specifically, it analyzes video data captured by cameras using OpenCV and YOLO to detect specific objects (e.g., suspicious persons) through object recognition. It also analyzes collected audio data using the Google Speech-to-Text API to identify abnormal sounds and specific voices (e.g., voices calling for help).

[1383] Furthermore, IBM Watson Tone Analyzer is used as an emotion engine to understand the emotional state of security personnel. The system analyzes tension and stress levels from the personnel's voice data, and the server generates appropriate action instructions based on the results.

[1384] Generation and transmission of instructions

[1385] Based on the analysis results, the server generates specific action instructions. For example, if a suspicious person is detected, a warning such as "Please leave immediately" is played through the speaker. Also, if the person in charge is nervous, advice such as "Please relax" is conveyed via voice.

[1386] Specific example

[1387] For example, suppose a security robot is patrolling a surveillance area and its camera captures the movement of a suspicious person. This video data is immediately sent to a server and analyzed by OpenCV or YOLO. If the analysis reveals a suspicious person, the server generates an instruction such as "Please leave immediately" and issues a voice warning through the robot's speaker.

[1388] Example of a prompt

[1389] "Analyze the footage from this security camera to determine if there is a suspicious person present. If a suspicious person is detected, generate the voice message 'There is a suspicious person present, please leave immediately.'"

[1390] Thus, the present invention enables security services to grasp the situation in real time, take into account the emotional state of the person in charge, generate appropriate instructions, and respond quickly and accurately.

[1391] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1392] Step 1:

[1393] The terminal devices (security robots, surveillance drones) use mounted cameras, microphones, and various sensors to collect video, audio, and environmental data from the monitored area. Inputs include camera footage, microphone audio, and sensor readings, and this data is collected in real time.

[1394] Step 2:

[1395] The terminal transmits the collected data to the server via wireless communication. The input consists of video, audio, and sensor data acquired from the terminal, and by transmitting these to the server, the system is ready for real-time analysis.

[1396] Step 3:

[1397] The server analyzes the received video data using OpenCV and YOLO. Specifically, it performs image analysis to detect suspicious individuals and abnormal objects using object recognition algorithms. The input is video data, and the output is the detected object information as a result of the analysis.

[1398] Step 4:

[1399] The server converts the received audio data into text data using the Google Speech-to-Text API. The input is audio data, and the output is the converted text data. This text data is then analyzed using an algorithm that identifies abnormal sounds and specific voices.

[1400] Step 5:

[1401] The server analyzes the received and analyzed emotional data using IBM Watson Tone Analyzer. Input is audio and video data, and output is analyzed emotional state information. This is used to assess the tension and stress levels of security personnel.

[1402] Step 6:

[1403] Based on the analysis results from steps 3-5, the server generates instructions for the next course of action. Specifically, it generates warning instructions for suspicious individuals based on object recognition results, and advice for personnel based on emotion analysis results. The input is the analysis results, and the output is text data as action instructions.

[1404] Step 7:

[1405] The server converts the generated action instructions into audio data. This is done using text-to-speech (TTS) technology. The input is the text data of the action instructions, and the output is the audio data.

[1406] Step 8:

[1407] The server sends the generated audio data to the terminal. The input is audio data, and the output is the transmission of audio data to the terminal.

[1408] Step 9:

[1409] The device plays the received audio data through its speaker, transmitting instructions to the target (dog, suspicious person, security personnel). Input is audio data, and output is actual audio playback. For example, it can warn a suspicious person to "leave immediately."

[1410] Step 10:

[1411] The terminal monitors the situation again using cameras and sensors after the instruction is given, and sends the data to the server. The input is the re-collected video, audio, and sensor data, and the output is the transmission of data to the server. This step allows for the collection of feedback and further improvement of the accuracy of action instructions.

[1412] Through the steps outlined above, a cycle of information gathering, analysis, instruction generation and transmission, and feedback is realized, enabling the provision of effective security services.

[1413] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1414] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1415] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1416] [Fourth Embodiment]

[1417] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1418] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1419] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1420] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1421] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1422] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1423] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1424] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1425] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1426] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1427] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1428] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1429] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1430] This invention relates to an AI support device for disaster relief dogs, and its basic configuration and functions will be described in detail. The actual operation method and processing flow will also be discussed.

[1431] Basic device configuration

[1432] This device consists of the following elements:

[1433] 1. Camera: This camera is attached to the rescue dog's collar to collect visual information. Because the camera is mounted close to the dog's line of sight, it can accurately capture the direction and objects the dog is looking at.

[1434] 2. Microphone: This device is used to collect ambient sounds and human voices. It is built into the collar and collects sounds in accordance with the dog's movements.

[1435] 3. Various sensors: These include temperature sensors, humidity sensors, and acceleration sensors. This allows for detailed monitoring of environmental conditions and the dog's movements.

[1436] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[1437] 5. Speaker: This device transmits handler instructions and AI-generated instructions to the dog. It is built into the collar, allowing voice commands to reach the dog.

[1438] Central system (server)

[1439] Data collected from the device is transmitted wirelessly to a central system (server). The server analyzes the received data and performs the following processing:

[1440] Real-time analysis: Instantly analyzes video data from cameras, audio data from microphones, and data from various sensors. It uses image analysis algorithms for object recognition and audio analysis algorithms to classify ambient sounds and human voices.

[1441] Behavioral command generation: Based on the analysis results, behavioral commands are generated for the dog. This supports rescue dogs in acting efficiently.

[1442] User (handler) operation

[1443] The handler can monitor the rescue dog's perspective and situation in real time through a user interface (such as a smartphone or tablet) connected to a central system. They can also issue commands remotely as needed.

[1444] Program processing

[1445] Data collection

[1446] Device: A camera, microphone, and various sensors attached to the collar collect data. For example, if a dog is exploring through rubble, the camera will continuously film its surroundings, and the microphone will record ambient sounds.

[1447] Data transmission

[1448] Terminal: The collected data is sent to the server at regular intervals. This allows the server to receive the data in real time.

[1449] Data Analysis

[1450] Server: Analyzes received data and performs object recognition and audio analysis. For example, it can recognize human faces and limbs from camera images and detect voices such as "help" from audio data.

[1451] Action instruction generation

[1452] Server: Based on the analysis results, it generates appropriate instructions. These include specific instructions such as "move right" or "stay put."

[1453] Sending instructions

[1454] Server: The generated instructions are converted into audio format and sent to the terminal. These instructions are then transmitted to the dog through the terminal's speaker.

[1455] Feedback Collection

[1456] Terminal: Monitors the dog's behavior again after the command is given and sends the collected data to the server. The server continuously analyzes the data and generates new commands as needed.

[1457] Handler operation

[1458] User (Handler): The handler can monitor the rescue dog's status in real time through the user interface and issue additional instructions as needed.

[1459] Specific example

[1460] For example, after a major earthquake, when a rescue dog searches through rubble, video footage and audio recordings from the device are sent to a server. The server analyzes the received data and detects cries for help or people in the rubble. The server then generates an instruction, such as "move to the right," and transmits it to the rescue dog via the device. The handler can monitor the dog's perspective in real time and issue additional instructions as needed, such as "search carefully in that spot."

[1461] In this way, the present invention effectively supports the actions of disaster relief dogs and enables faster and more accurate rescue operations.

[1462] The following describes the processing flow.

[1463] Step 1:

[1464] The device is attached to the dog's collar, and the camera, microphone, and various sensors begin to operate. The camera captures images of the surroundings in real time at a frequency of several frames per second, and the microphone records ambient sounds. Temperature, humidity, and acceleration sensors measure environmental conditions and the dog's movements.

[1465] Step 2:

[1466] The device collects data, packets it sends to the server via wireless communication (e.g., Wi-Fi or Bluetooth), and transmits the data at regular intervals (e.g., every 10 seconds).

[1467] Step 3:

[1468] The server analyzes the data it receives. Specifically, it uses image analysis algorithms to analyze image data captured by cameras and perform object recognition. It also uses voice analysis algorithms to analyze audio data recorded by microphones and classify sounds such as cries for help and ambient noises.

[1469] Step 4:

[1470] The server generates behavioral instructions for the dog based on the analysis results. For example, the analysis results might generate an instruction such as "move right." These instructions are generated in text format.

[1471] Step 5:

[1472] The server converts the generated behavioral instructions into audio format and sends them to the terminal. The audio instructions are played back in a way that is easy for the dog to understand.

[1473] Step 6:

[1474] The device receives instructions and transmits them to the dog as voice commands through its speaker. For example, the command "Go right" is played through the speaker.

[1475] Step 7:

[1476] The device monitors the dog's behavior after the command is given and collects data again. This data includes whether the dog followed the command.

[1477] Step 8:

[1478] The device then packets the collected data and sends it to the server. This allows the server to always have up-to-date information on the dog's current behavior and location.

[1479] Step 9:

[1480] The user (handler) monitors information from the server and terminal in real time and issues additional instructions as needed. The handler uses a smartphone or tablet to give remote instructions via touch panel or voice commands.

[1481] Step 10:

[1482] The server receives additional instructions from the handler, converts them into voice commands, and sends them to the terminal. The terminal then relays the voice commands back to the dog.

[1483] This series of processes uses AI to complement the actions and judgment of disaster relief dogs, supporting effective and rapid rescue operations.

[1484] (Example 1)

[1485] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1486] In disaster relief operations, the swift and precise actions of rescue dogs are crucial. However, traditional methods involve time-consuming information gathering and analysis, making it difficult to accurately communicate instructions to the dogs. Furthermore, it is challenging for handlers in remote locations to grasp the situation in real time and give appropriate instructions to the dogs. This leads to a decrease in the efficiency of rescue operations and a reduction in the success rate of saving lives.

[1487] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1488] In this invention, the server includes means for a waterproof device with a built-in camera, microphone, and various sensors attached to a collar; means for transmitting information collected from the camera and microphone to the server in real time via wireless communication; means for analyzing the information received by the server and performing object recognition and voice analysis; means for converting instructions generated by the server into voice format and transmitting them to a speaker on the collar via wireless communication; and means for transmitting instructions to the dog from the speaker. This makes it possible to significantly improve the efficiency and success rate of rescue operations by enabling the rescue dog to collect accurate information in real time and the server to quickly analyze and issue instructions.

[1489] A "camera" is an optical device attached to a rescue dog's collar to collect visual information, capturing images of the direction and objects the dog is looking at.

[1490] A "microphone" is an acoustic device used to collect ambient sounds and human voices, and it is built into the collar.

[1491] "Various sensors" refers to devices including temperature sensors, humidity sensors, and acceleration sensors, which are used to monitor environmental conditions and the dog's movements in detail.

[1492] "Waterproof construction" means that the entire device is designed to be resistant to water and will function normally even in harsh environments.

[1493] A "device" is a device that incorporates a camera, microphone, and various sensors and is attached to a collar.

[1494] A "server" is a central computing device that receives information transmitted from devices, performs analysis, and generates instructions.

[1495] "Wireless communication" refers to technologies that transmit data without physical media such as cables, and includes Wi-Fi and LTE.

[1496] "Analysis" is the process of processing received data to extract meaningful information, and includes object recognition and speech analysis.

[1497] "Object recognition" is a technology that analyzes video data acquired from a camera to identify specific objects or people.

[1498] "Speech analysis" is a technology that processes audio data to identify meaningful sounds and voices.

[1499] An "instruction" is a command generated by the server that instructs the rescue dog to take a specific action.

[1500] "Voice format" refers to the format in which commands are converted into audio data for use in communicating instructions to dogs.

[1501] A "speaker" is an acoustic device that transmits commands converted into voice format to the dog, and it is built into the collar.

[1502] A "user interface" is a screen that allows handlers to check the status of rescue dogs and issue commands using a smartphone or tablet.

[1503] A "handler" is a person who trains and controls a rescue dog.

[1504] This invention relates to an AI support device for disaster relief dogs, and describes its specific usage method and processing flow. The system consists of a waterproof device with a built-in camera, microphone, and various sensors, which is attached to the dog's collar. It monitors and analyzes the dog's movements and surrounding environment in detail and generates appropriate commands.

[1505] Device Configuration

[1506] hardware

[1507] 1. Camera: A high-resolution camera is attached to the dog's collar to collect the dog's visual information in real time. It includes infrared capabilities for shooting at night or in dimly lit environments.

[1508] 2. Microphone: A microphone is built into the collar to collect ambient sounds and human voices. This allows for an accurate understanding of the surrounding sound environment.

[1509] 3. Various sensors: Includes temperature sensors, humidity sensors, acceleration sensors, etc., to monitor environmental conditions and the dog's movements in detail.

[1510] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[1511] Central system (server)

[1512] The server receives data transmitted from the device via wireless communication (e.g., Wi-Fi or LTE) and performs the following processing.

[1513] 1. Data Analysis: The server receives video data from cameras, audio data from microphones, and various sensor data, and analyzes them in real time. For example, it utilizes libraries such as TensorFlow and OpenCV to perform object recognition using image analysis algorithms. It also uses audio analysis algorithms to classify and recognize specific sounds and voices (e.g., "help").

[1514] 2. Behavioral Instruction Generation: Based on the analysis results, specific behavioral instructions are generated for the rescue dogs. A natural language generation (NLG) algorithm is used to ensure the instructions are specific and appropriate.

[1515] Communication of instructions and feedback

[1516] The behavioral commands generated by the server are converted into audio format and transmitted wirelessly to the device's speaker. For example, the Google Text-to-Speech API can be used to convert text commands into speech and transmit them to the dog. The dog's movements and surrounding environment after the command are collected again by the device and continuously transmitted to the server. This allows the server to continuously analyze the data and generate new commands.

[1517] User actions

[1518] The user (handler) can view the rescue dog's perspective and status in real time through a user interface such as a smartphone or tablet. Using a dedicated application, they can, for example, check the dog's location on a map or view past search history. The handler can also issue additional commands remotely.

[1519] Specific example

[1520] For example, when rescue dogs search through rubble after a major earthquake, the following specific measures are taken.

[1521] 1. Data Collection: As the dog moves around in the rubble, the device (attached camera, microphone, and sensors) collects video and audio data in real time.

[1522] 2. Data transmission: The collected data is transmitted to the server via wireless communication.

[1523] 3. Data Analysis: The server analyzes the received video data to identify human figures and voices. For example, the server recognizes a "person asking for help" from camera footage and detects the voice saying "help" through audio analysis.

[1524] 4. Generation of action instructions: Based on the analysis results, action instructions such as "move to the right" are generated.

[1525] 5. Sending Instructions: The generated instructions are sent to the device and communicated to the dog via the speaker.

[1526] 6. Feedback Collection: The dog's behavior after receiving instructions is closely monitored by a camera and accelerometer, and the data is retransmitted to a server to collect further information that helps in giving appropriate instructions for subsequent actions.

[1527] 7. Handler Operation: Handlers can use a dedicated app to check the dog's current location on a map and view its past search history. For example, if they want to command the dog to search an additional area, they can tap on the map to input the command.

[1528] Examples of prompts to input into a generative AI model

[1529] "Following a major earthquake, rescue dogs are searching through the rubble. The collected video and audio data is being analyzed immediately to detect human figures and cries for help. Please generate instructions for the next course of action."

[1530] Thus, this invention effectively supports the actions of disaster relief dogs, enabling faster and more accurate rescue operations.

[1531] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1532] Step 1: Data Collection

[1533] Input: The device (camera, microphone, and various sensors attached to the collar) collects information about the dog's movements and surrounding environment in real time.

[1534] Specific operation: The camera captures the dog's field of view and obtains high-resolution video. The microphone records ambient sounds, and temperature, humidity, and acceleration sensors collect environmental information.

[1535] Output: Various collected data (video, audio, temperature, humidity, acceleration) are temporarily stored on the device.

[1536] Step 2: Data transmission

[1537] Input: Various data acquired in the data collection step (video, audio, temperature, humidity, acceleration).

[1538] Specific operation: The device uses wireless communication (e.g., Wi-Fi or LTE) to send the collected data to the server at regular intervals.

[1539] Output: Video data, audio data, and various sensor data transmitted to the server.

[1540] Step 3: Data Analysis

[1541] Input: Received video data, audio data, and various sensor data.

[1542] Specific operation: The server processes video data using image analysis algorithms (e.g., TensorFlow or OpenCV) to analyze each received data, recognizing human faces, limbs, and the state of rubble. It also processes audio data using speech analysis algorithms to detect specific phrases such as "help."

[1543] Output: Recognized objects and audio information as analysis results.

[1544] Step 4: Generate action instructions

[1545] Input: Analysis results obtained in the data analysis step.

[1546] Specific operation: Based on the analysis results, the server uses an AI model to generate behavioral instructions for the dog. For example, specific instructions such as "move right" or "stay put" are generated.

[1547] Output: Generated action instructions.

[1548] Step 5: Send Instructions

[1549] Input: Generated action instructions.

[1550] Specific operation: The server converts the generated action instructions into speech format (for example, using the Google Text-to-Speech API) and sends them to the terminal via wireless communication.

[1551] Output: Action instructions converted into voice format are sent to the terminal.

[1552] Step 6: Instructions and communication

[1553] Input: Action instructions converted into voice format.

[1554] Specific operation: Voice commands are played from a speaker attached to the device, conveying instructions to the dog such as "move to the right" or "stay in place."

[1555] Output: The dog receives and executes the command.

[1556] Step 7: Gathering Feedback

[1557] Input: Information about the dog's movements after receiving a command and its surrounding environment.

[1558] Specific operation: The device monitors the dog's behavior again after the command is given and sends the collected data to the server.

[1559] Output: Feedback data sent to the server (information about the dog's movements and surrounding environment).

[1560] Step 8: Handler Operation

[1561] Input: Information that the user (handler) checks in real time.

[1562] Specific actions: The handler uses a dedicated application to check the rescue dog's current location, video, and audio data via a smartphone or tablet. They can also issue additional instructions remotely as needed.

[1563] Output: Additional instructions from the handler are sent to the terminal via the server and then transmitted to the rescue dog.

[1564] Through the steps described above, this system effectively supports the activities of disaster relief dogs and enables rapid and accurate rescue operations.

[1565] (Application Example 1)

[1566] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1567] In disaster areas, there is a constant need for means to conduct rescue operations quickly and efficiently. Traditional methods primarily rely on human rescuers and rescue dogs, but these have limitations, and efficiency is particularly low in large areas and dangerous environments. Furthermore, there is a lack of systematic systems for rapid information gathering, analysis, and command, so these challenges need to be addressed. The aim is to build a general-purpose disaster rescue system, including autonomous vehicles, to achieve more efficient information gathering and faster analysis during disasters.

[1568] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1569] In this invention, the server includes means for performing object recognition based on video data collected by a camera and using data from an acceleration sensor and a temperature sensor for environmental analysis; means for analyzing data collected by a generative AI model and generating appropriate action instructions for an autonomous vehicle; and means for operating based on prompt statements based on the results of analysis from sensor and voice data. This makes it possible to collect and analyze information quickly and efficiently at disaster sites and optimize rescue operations.

[1570] An "AI support device for disaster relief dogs" is a device equipped with a camera, microphone, and various sensors, and featuring a waterproof design, to support the activities of rescue dogs at disaster sites.

[1571] An "AI platform" is a central processing unit that analyzes information collected from cameras, microphones, and other sources in real time and generates action instructions. It is typically installed on a cloud server or similar location.

[1572] "Real-time analysis" is a data processing method that processes collected data immediately and produces results quickly.

[1573] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates appropriate action instructions based on collected data.

[1574] A "prompt message" is a sentence in the form of an instruction or question used when inputting specific data into a generative AI model.

[1575] A "camera" is a device that collects optical information as video data, and is used to acquire visual information from rescue dogs.

[1576] A "microphone" is a device that collects audio data and is used to record ambient sounds and human voices.

[1577] "Various sensors" refers to devices used to collect environmental data, such as temperature sensors, humidity sensors, and acceleration sensors.

[1578] "Waterproof construction" refers to a design that makes a device resistant to water and moisture, allowing it to function properly even in harsh environments.

[1579] A "speaker" is a device that outputs audio data and is used to transmit generated instructions as sound.

[1580] An "autonomous vehicle" is a vehicle that uses artificial intelligence to operate autonomously, and is intended for use in rescue operations at disaster sites.

[1581] "Object recognition" is a technology that identifies and detects specific objects from video data acquired by a camera.

[1582] "Environmental analysis" is the process of evaluating the surrounding physical and environmental conditions based on data acquired from various sensors.

[1583] "Behavioral instructions" are specific instructions on what actions the subject should take, generated based on the analysis results.

[1584] This invention relates to an AI support system applicable to disaster relief dogs and autonomous vehicles, and specifically describes its basic configuration and functions. It also touches upon actual operation methods and processing flows.

[1585] Basic System Configuration

[1586] This system consists of the following elements:

[1587] 1. Cameras: These are devices mounted on the front and sides of autonomous vehicles that collect images of the surroundings in real time.

[1588] 2. Microphone: A device used to collect ambient sounds, and multiple microphones are placed in the vehicle.

[1589] 3. Various sensors: These include temperature sensors, humidity sensors, acceleration sensors, LiDAR, etc., and are devices for detailed monitoring of the environment and vehicle movement.

[1590] 4. Waterproof structure: The entire vehicle and device are designed to be waterproof, ensuring normal operation even in harsh environments.

[1591] 5. Speaker: A device used to transmit voice instructions to vehicles or rescue dogs.

[1592] Data collection and transmission

[1593] The autonomous vehicle, acting as the terminal, collects data through cameras, microphones, and various sensors. This data is transmitted to a server at regular intervals via a communication module.

[1594] Data Analysis

[1595] The server uses the following hardware and software to analyze the received data:

[1596] Hardware: Cloud servers (e.g., AWS, Google Cloud)

[1597] Software: Image analysis (e.g., OpenCV, TensorFlow), audio analysis (e.g., Amazon Transcribe)

[1598] The server analyzes image data from the camera to detect specific objects. It analyzes audio data to identify ambient sounds and important phrases (such as "help"). Furthermore, it evaluates environmental conditions (temperature, humidity, motion) from sensor data.

[1599] Action instruction generation

[1600] Based on the analysis results, the server uses a generative AI model (for example, an algorithm implemented in Python) to generate appropriate action instructions for the autonomous vehicle. The generated instructions determine the specific action based on the prompt and are communicated to the vehicle.

[1601] Examples of prompt statements

[1602] For example, use the following prompt:

[1603] def generate_disaster_relief_instructions(camera_data, audio_data, sensor_data):

[1604] """

[1605] A function that analyzes camera footage, audio data, and sensor data to generate appropriate instructions for autonomous vehicles.

[1606] :param camera_data: np.ndarray, camera image data

[1607] :param audio_data: list, microphone audio data

[1608] :param sensor_data: dict, data from various sensors

[1609] :return: str, Instructions for the vehicle

[1610] """

[1611] Video data analysis

[1612] objects_detected = object_detection_model.predict(camera_data)

[1613] Audio data analysis

[1614] emergency_phrases = audio_analysis_model.detect_phrases(audio_data)

[1615] Comprehensive data analysis

[1616] if 'human' in objects_detected and 'help' in emergency_phrases:

[1617] Return "Continue moving forward. Human voice detected. Proceed to the right."

[1618] Return "Continue the search."

[1619] Feedback Collection

[1620] Based on the generated instructions, the autonomous vehicle operates, and subsequent data is collected again and sent to the server. The server continuously analyzes the data and generates new instructions as needed.

[1621] Handler operation

[1622] The user (handler) can monitor the perspective and status of the rescue dog or autonomous vehicle in real time through a user interface (such as a smartphone or tablet). They can also issue commands remotely as needed.

[1623] In this way, the present invention can effectively support the actions of disaster relief dogs and autonomous vehicles, enabling faster and more accurate rescue operations.

[1624] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1625] Step 1:

[1626] The terminal (autonomous vehicle) collects data using cameras, microphones, and various sensors. Specifically, it captures images of the surroundings with cameras and records ambient sounds and voices with microphones. It also acquires ambient environmental data using temperature sensors, acceleration sensors, and humidity sensors. This collected data is temporarily stored as raw data in the vehicle's internal storage.

[1627] Input: Environmental video, audio, temperature, acceleration, humidity

[1628] Output: Collected environmental data

[1629] Step 2:

[1630] The terminal transmits the collected data to the server via a communication module. The communication uses a 5G or 4G network and leverages a protocol (MQTT) to ensure data is transmitted in real time. The server temporarily stores the received data for analysis.

[1631] Input: Collected environmental data

[1632] Output: Data to send to the server

[1633] Step 3:

[1634] The server uses various analysis software to analyze the received data. Specifically, it performs object recognition on video data from the camera using OpenCV and TensorFlow to identify people and obstacles. Audio data is analyzed using Amazon Transcribe to detect important voice phrases (e.g., "Help"). Sensor data is used to detect anomalies in temperature and humidity, and acceleration data is used to evaluate the vehicle's operating state.

[1635] Input: Data sent to the server

[1636] Output: Analysis results (object recognition results, audio analysis results, sensor analysis results)

[1637] Step 4:

[1638] The server uses a generative AI model based on the analysis results to generate action instructions for the autonomous vehicle. These generated instructions are specifically constructed based on prompt statements. These prompt statements generate appropriate instructions (e.g., "Turn right," "Explore in place") based on the analysis results. The generative AI model's algorithm is implemented in Python.

[1639] Input: Analysis results

[1640] Output: Action Instructions

[1641] Step 5:

[1642] The server converts the generated action instructions into voice format and sends them back to the terminal. The terminal receives the instructions from the server via a communication module and outputs them through its speaker. These voice instructions prompt the rescue dog or autonomous vehicle to perform its specific action.

[1643] Input: Action Instructions

[1644] Output: Voice instructions

[1645] Step 6:

[1646] The terminal performs actions based on instructions. For example, if a vehicle receives the instruction "move right," it uses the vehicle's motion control algorithm (e.g., ROS - Robot Operating System) to perform the specific action. After the action is performed, the data is collected again and sent to the server.

[1647] Input: Voice Instructions

[1648] Output: Data after execution

[1649] Step 7:

[1650] The server receives the data sent from the terminal again and continues to analyze it. If necessary, it generates new action instructions and sends them to the terminal. By repeating this feedback loop, the rescue operation is optimized.

[1651] Input: Data after execution of the operation

[1652] Output: New analysis results and action instructions

[1653] As described above, this system enables efficient rescue operations by having the server, terminal, and user components work together and repeat a series of processes including data collection, analysis, instruction, and execution.

[1654] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1655] This invention relates to an AI support device for disaster relief dogs, and further describes a system that combines this with an emotion engine that recognizes the emotions of the user (handler). This invention consists of the following elements:

[1656] Basic device configuration

[1657] 1. Camera: This camera is used to collect visual information from rescue dogs. It is attached to the collar and captures images from a position close to the dog's line of sight.

[1658] 2. Microphone: This device is used to collect ambient sounds and the user's voice. It is built into the collar and collects voice data in accordance with the dog's movements.

[1659] 3. Various sensors: These include temperature sensors, humidity sensors, and acceleration sensors. This allows for detailed monitoring of environmental conditions and the dog's movements.

[1660] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[1661] 5. Speaker: This device transmits handler instructions and AI-generated instructions to the dog. It is built into the collar, allowing voice commands to reach the dog.

[1662] Central system (server)

[1663] Data collected from the device is transmitted wirelessly to a central system (server). The server analyzes the received data and performs the following processing:

[1664] Real-time analysis: Instantly analyzes video data from cameras, audio data from microphones, and data from various sensors. It uses image analysis algorithms for object recognition and audio analysis algorithms to classify ambient sounds and human voices.

[1665] Behavioral command generation: Based on the analysis results, behavioral commands are generated for the dog. This supports rescue dogs in acting efficiently.

[1666] Emotion Engine: Analyzes the user's (handler's) voice and video data to recognize their emotional state. For example, it can detect the user's tension or anxiety through voice analysis and detect changes in the user's facial expressions through facial expression analysis.

[1667] User (handler) operation

[1668] The handler can monitor the rescue dog's perspective and situation in real time through a user interface (such as a smartphone or tablet) connected to a central system. They can also issue commands remotely as needed. Furthermore, the system automatically generates more appropriate commands based on the emotional state analyzed by the emotion engine.

[1669] Program processing

[1670] Data collection

[1671] Device: A camera, microphone, and various sensors attached to the collar collect data. For example, if a dog is exploring through rubble, the camera will continuously film its surroundings, and the microphone will record ambient sounds.

[1672] Data transmission

[1673] Terminal: The collected data is sent to the server at regular intervals. This allows the server to receive the data in real time.

[1674] Data Analysis

[1675] Server: Analyzes received data and performs object recognition and audio analysis. For example, it can recognize human faces and limbs from camera images and detect voices such as "help" from audio data.

[1676] Action instruction generation

[1677] Server: Based on the analysis results, it generates appropriate instructions. These include specific instructions such as "move right" or "stay put."

[1678] Sending instructions

[1679] Server: The generated instructions are converted into audio format and sent to the terminal. The audio instructions are played back to the dog in a way that is easy for it to understand.

[1680] Executing instructions

[1681] Terminal: Receives commands and transmits them to the dog as voice commands through a speaker. For example, the command "Go right" is played through the speaker.

[1682] Feedback Collection

[1683] Terminal: Monitors the dog's behavior again after giving instructions and sends the collected data to the server. This data includes whether the dog followed the instructions.

[1684] Emotion analysis

[1685] Server: Analyzes the user's (handler's) voice and video data and recognizes their emotional state using an emotion engine. For example, it analyzes the user's level of tension from the voice data and recognizes changes in the user's facial expressions from the video data.

[1686] Emotion-based instruction generation

[1687] Server: Generates appropriate behavioral instructions based on the analyzed emotional state. The emotion engine understands the user's feelings and communicates corresponding instructions to the dog.

[1688] Handler operation

[1689] User (Handler): The handler can monitor the rescue dog's status in real time through the user interface and issue additional instructions as needed. Emotional analysis results are also displayed, allowing for appropriate instructions based on the situation.

[1690] Specific example

[1691] For example, after a major earthquake, when a rescue dog searches through rubble, the video and audio captured by the terminal are sent to a server. The server analyzes the received data and detects cries for help or people in the rubble. The server then generates an instruction such as "move to the right" and transmits it to the rescue dog via the terminal. On the other hand, if the emotion engine determines that the handler's voice is tense, it generates an instruction to alleviate that tension (for example, "search slowly"). The handler monitors the dog's perspective in real time and gives additional instructions as needed, such as "search carefully in place."

[1692] In this way, the present invention effectively supports the actions of disaster relief dogs and enables faster and more accurate rescue operations while taking into account the user's emotional state.

[1693] The following describes the processing flow.

[1694] Step 1:

[1695] The device is attached to the dog's collar, and the camera, microphone, and various sensors begin operating. The camera captures images of the surroundings in real time from a position close to the dog's line of sight, and the microphone collects ambient sounds. Temperature, humidity, and acceleration sensors measure environmental conditions and the dog's movements.

[1696] Step 2:

[1697] The device collects data (image data, audio data, sensor data), organizes it into packets, and sends them to a server via wireless communication (e.g., Wi-Fi or Bluetooth). This transmission occurs at regular intervals (e.g., every 10 seconds).

[1698] Step 3:

[1699] The server analyzes the received data. Specifically, it uses image analysis algorithms to perform object recognition from image data captured by the camera, and uses voice analysis algorithms to analyze audio data recorded by the microphone, classifying ambient sounds and human voices.

[1700] Step 4:

[1701] The server generates action commands for the rescue dogs based on the analysis results. For example, it generates specific commands such as "move to the right" or "stay put," and records them in text format.

[1702] Step 5:

[1703] The server generates behavioral commands, converts them into voice commands, and sends them to the terminal. The voice commands are then converted into a format that is easy for the dog to understand.

[1704] Step 6:

[1705] The device receives voice commands and transmits them to the rescue dog through a speaker. For example, the speaker might play the command "Go to the right."

[1706] Step 7:

[1707] The device monitors the rescue dog's actions after the command is given and collects data again. The newly collected data includes whether the rescue dog followed the command.

[1708] Step 8:

[1709] The device repackages the collected data into packets and sends them back to the server. This allows the server to always have up-to-date information on the rescue dog's current actions and location.

[1710] Step 9:

[1711] The server analyzes the user's (handler's) voice and video data. Using an emotion engine, it recognizes emotional states such as tension and anxiety through voice analysis and detects changes in the user's facial expressions through facial expression analysis.

[1712] Step 10:

[1713] The server generates additional action instructions based on the analyzed emotional state. For example, it might generate an instruction like "explore slowly" to reduce the handler's tension.

[1714] Step 11:

[1715] The server sends instructions, including those generated by the emotion engine, to the terminal, which then transmits them to the rescue dog via its speaker.

[1716] Step 12:

[1717] The user (handler) monitors information from the server and terminal in real time and issues additional instructions as needed. The handler uses a smartphone or tablet to remotely control the system via screen touch or voice commands.

[1718] Through these processing steps, the present invention effectively supports the actions of disaster relief dogs and enables accurate rescue operations that take into account the user's emotional state.

[1719] (Example 2)

[1720] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1721] While a wide range of technologies are needed to support the effective deployment of rescue dogs in disaster relief operations, challenges remain in efficient and rapid information gathering and analysis, as well as in generating commands that take into account the handler's emotional state. In particular, it is essential that devices function correctly in harsh environments, analyze collected information in real time, and make appropriate decisions quickly. Furthermore, there is a need for means for handlers to remotely and appropriately direct and manage the actions of rescue dogs.

[1722] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1723] In this invention, the server includes means equipped with an AI platform that analyzes information collected from a camera and microphone in real time, a speaker that transmits instructions generated by the server to the dog, and means including an emotion engine that analyzes the user's voice and video data and recognizes their emotional state. This not only effectively supports the actions of disaster relief dogs but also enables the generation of flexible instructions that take into account the handler's emotional state. Furthermore, the waterproof device ensures normal operation even in harsh environments and enables real-time instructions from a remote location.

[1724] A "camera" is a device used to collect video data, and it is attached to the collar of a rescue dog to record visual information of its surroundings.

[1725] A "microphone" is a device used to collect audio data, and it is built into the dog's collar to record ambient sounds and the user's voice.

[1726] "Various sensors" refers to multiple sensor devices such as temperature sensors, humidity sensors, and acceleration sensors, used to monitor environmental conditions and the dog's movements in detail.

[1727] "Waterproof construction" refers to a design that protects the entire device from water and moisture, ensuring it functions properly even in harsh environments.

[1728] A "device" refers to a set of equipment that includes a camera, microphone, and various sensors, which are attached to the collar of a rescue dog.

[1729] "AI infrastructure" refers to an artificial intelligence system for analyzing collected video and audio data in real time.

[1730] A "server" refers to a central processing unit that receives data transmitted from devices via wireless communication, performs analysis, and generates instructions.

[1731] A "speaker" refers to a device that plays back instructions generated from a server in audio format and transmits them to a dog.

[1732] An "emotion engine" refers to an analysis system that analyzes the voice and video data of a user (handler) to recognize their emotional state.

[1733] A "user interface" refers to the control screen that allows a handler to remotely monitor the device's status in real time and issue commands.

[1734] This invention relates to an AI support device and system for disaster relief dogs. Detailed embodiments thereof are described below.

[1735] Basic device configuration

[1736] The device in this system is attached to the collar of a rescue dog. The set of devices includes the following elements:

[1737] 1. Camera: This device is used to collect visual information from rescue dogs. The camera is attached to the collar and captures images from a position close to the dog's line of sight.

[1738] 2. Microphone: This device is used to collect ambient sounds and the user's voice. It is also built into the collar and collects voice data in accordance with the dog's movements.

[1739] 3. Various sensors: These include temperature sensors, humidity sensors, and acceleration sensors, allowing for detailed monitoring of environmental conditions and the dog's movements.

[1740] 4. Waterproof design: The entire device is waterproof and will function normally even in harsh environments.

[1741] 5. Speaker: This device transmits commands generated from the server to the dog. It is built into the collar, allowing voice commands to reach the dog.

[1742] Server configuration and functionality

[1743] Data collected from the device is transmitted to a central system (server) via wireless communication (e.g., Wi-Fi, Bluetooth). The server then performs the following processing:

[1744] Real-time analysis: Instantly analyzes video data from cameras, audio data from microphones, and data from various sensors. For example, it uses software libraries such as "TensorFlow" and "OpenCV" to perform image and audio analysis. It can recognize objects and people from camera images and classify ambient sounds and human voices from audio data.

[1745] Behavioral command generation: Based on the analysis results, behavioral commands are generated for the dog. These include specific commands such as "Go right" or "Stay where you are." The commands are then spoken using speech synthesis software (e.g., "Google Text-to-Speech").

[1746] Emotion Engine: Analyzes the user's (handler's) voice and video data to recognize their emotional state. Emotion analysis uses algorithms that analyze voice tone and facial expression changes from the video. This allows the user's emotional state to be recognized, and appropriate action instructions to be generated.

[1747] User (handler) operation

[1748] The user (handler) can view the rescue dog's perspective and situation in real time through a user interface connected to the server (e.g., a smartphone or tablet). The user can issue additional instructions as needed. Furthermore, the system automatically generates more appropriate instructions based on the emotional state analyzed by the emotion engine.

[1749] Specific example

[1750] For example, after a major earthquake, when a rescue dog searches through rubble, the video and audio captured by the device are sent to a server. The server analyzes the received data using TensorFlow and OpenCV to detect cries for help and people in the rubble. The server then generates an instruction such as "move to the right," converts it to speech using Google Text-to-Speech, and transmits it to the rescue dog via the device. Meanwhile, if the emotion engine detects tension in the handler's voice, it generates an instruction to alleviate that tension (for example, "search slowly"). The user (handler) can view the dog's perspective through their smartphone and issue additional instructions in real time, such as "search carefully in place."

[1751] Example of a Generated AI Model Prompt

[1752] Based on natural language processing, please describe the following device and system configuration:

[1753] A collar device for disaster relief dogs equipped with a camera, microphone, and sensors.

[1754] Central server for data analysis and action instruction generation.

[1755] An emotion engine that analyzes the emotions of users (handlers) during rescue operations.

[1756] A system that allows users to check the situation and give instructions through an interface.

[1757] Please describe the system's processes in order, using the server, terminal, and user as the subjects.

[1758] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1759] Step 1:

[1760] Data collection

[1761] Device: A camera, microphone, and various sensors attached to the collar collect data. When the dog is exploring through rubble, the camera continuously captures images of the surroundings, and the microphone records ambient sounds in real time. An accelerometer detects the dog's movements, and temperature and humidity sensors record environmental conditions. This collects data to provide the user with detailed information about the situation.

[1762] Input: Visual information of the surroundings, ambient sounds, dog movements, temperature, humidity, etc.

[1763] Output: Collected visual data, audio data, and sensor data.

[1764] Step 2:

[1765] Data transmission

[1766] Terminal: The collected data is transmitted to the server at regular intervals via wireless communication. For example, camera video, audio data, and sensor data are transmitted to the server every 10 seconds. Possible communication methods include Wi-Fi and Bluetooth. The server receives this data in real time and uses it for analysis.

[1767] Input: Collected visual data, audio data, and sensor data.

[1768] Output: Data sent to the server.

[1769] Step 3:

[1770] Data Analysis

[1771] Server: Analyzes received data and performs object recognition and audio analysis. It recognizes human faces and limbs from camera footage and detects cries for help from audio data. Software libraries such as "TensorFlow" and "OpenCV" are used for this analysis.

[1772] Input: Visual data, audio data, and sensor data sent to the server.

[1773] Output: Analysis results (e.g., human face detection, audio analysis results).

[1774] Step 4:

[1775] Action instruction generation

[1776] Server: Based on the analysis results, it generates appropriate instructions. For example, if a person is detected in the camera footage, it generates specific action instructions such as "move to the right." These instructions are converted into speech using speech synthesis software (e.g., "Google Text-to-Speech").

[1777] Input: Analysis results.

[1778] Output: Action instructions (audio data).

[1779] Step 5:

[1780] Sending instructions

[1781] Server: The generated instructions are converted into audio format and then sent to the terminal. The server sends the audio data to the terminal to quickly communicate the generated behavioral instructions to the dog.

[1782] Input: Action instructions (voice data).

[1783] Output: Audio data sent to the terminal.

[1784] Step 6:

[1785] Executing instructions

[1786] Terminal: Receives commands and transmits them to the dog as voice commands through a speaker. For example, the command "Go right" is played from the speaker, and the dog acts according to that command.

[1787] Input: Audio data sent to the terminal.

[1788] Output: Behavioral commands (voice) transmitted to the dog.

[1789] Step 7:

[1790] Feedback Collection

[1791] Terminal: Monitors the dog's behavior again after giving instructions and simultaneously sends the collected data to the server. Feedback data includes whether the dog followed instructions and changes in the surrounding environment. This feedback data is used for the next analysis.

[1792] Input: Dog behavior data, environmental data.

[1793] Output: Feedback data sent to the server.

[1794] Step 8:

[1795] Emotion analysis

[1796] Server: Analyzes the user's (handler's) voice and video data and recognizes their emotional state using an emotion engine. For example, it analyzes the user's level of tension from their voice tone and recognizes changes in their facial expressions from the video. Voice analysis algorithms and facial expression analysis algorithms are used for this purpose.

[1797] Input: User's voice data and video data.

[1798] Output: Analysis results of the user's emotional state.

[1799] Step 9:

[1800] Emotion-based instruction generation

[1801] Server: Based on the analyzed emotional state, it generates further appropriate action instructions. For example, if the emotion engine detects user tension, it generates instructions to alleviate that state (e.g., "Explore slowly").

[1802] Input: Analysis results of the user's emotional state.

[1803] Output: Action instructions (audio data).

[1804] Step 10:

[1805] Handler operation

[1806] User (Handler): The user can view the rescue dog's perspective and situation in real time through the user interface and issue additional instructions as needed. For example, the user can input the instruction "Carefully search the area" through the user interface.

[1807] Input: Rescue dog's point-of-view video, feedback data.

[1808] Output: Additional action instructions.

[1809] (Application Example 2)

[1810] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1811] Traditional disaster relief dogs and security devices primarily generated behavioral instructions by simply analyzing collected data, without considering the emotional state of the handler or personnel. This resulted in problems such as the inability to respond appropriately when the handler or personnel were experiencing tension or stress, making effective support and instruction difficult. Furthermore, the difficulty in real-time monitoring and issuing commands meant that responses in situations requiring quick and accurate action could be delayed.

[1812] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means equipped with an AI platform that analyzes information collected from the camera and microphone in real time, means that analyzes emotions using voice data and video data as an emotion engine that analyzes human emotional states and generates behavioral instructions based on the analysis results, and means equipped with a speaker that transmits the generated instructions to the dog or security device. This enables real-time monitoring and appropriate behavioral instructions while taking into account the emotional state of the handler or security personnel.

[1813] An "AI support device for disaster relief dogs" is a device that incorporates cameras, microphones, and various sensors to support rescue dogs in disaster relief operations. It utilizes AI to analyze data in real time and generate appropriate action instructions.

[1814] A "camera" is a device used to collect video data, and it is attached to the dog's collar to capture images of the surroundings from a position close to the dog's line of sight.

[1815] A "microphone" is a device used to collect audio data, and is built into the collar to record ambient sounds and the user's voice.

[1816] "Various sensors" are devices that collect data such as temperature, humidity, and acceleration to monitor environmental conditions and user behavior.

[1817] A "waterproof design" is a design feature that protects internal electronic components from water and moisture, enabling the device to function properly even in harsh environments.

[1818] A "server" is a central system that analyzes data collected from cameras, microphones, and various sensors in real time and sends the generated instructions to the devices.

[1819] An "AI platform" is a platform that uses artificial intelligence technology to analyze collected data and generates action instructions based on the analysis results.

[1820] A "speaker" is a device that transmits voice commands generated by a server to a dog, and has the function of conveying behavioral commands by voice.

[1821] An "emotion engine" is a technology for analyzing human emotional states. It uses audio and video data to analyze emotions and generates appropriate instructions based on the analysis results.

[1822] A "handler" is the person responsible for operating and managing the rescue dog, and their role is to monitor the dog's perspective in real time from a remote location and give necessary instructions.

[1823] This invention relates to a system that applies an AI support device for disaster relief dogs to security services. Specifically, it involves a waterproof device with a built-in camera, microphone, and various sensors that is attached to a collar, security robot, or surveillance drone. The system also includes a server equipped with an AI platform that analyzes the information collected by the device in real time, and transmits instructions generated by the server through a speaker.

[1824] Hardware to use

[1825] 1. Camera:

[1826] It will be attached to security robots and surveillance drones to collect video footage of the monitored area.

[1827] Examples: Typical webcams and surveillance cameras.

[1828] 2. Mike:

[1829] It is used to collect ambient sounds and detect suspicious noises or voices.

[1830] Example: High-sensitivity microphone.

[1831] 3. Various sensors:

[1832] Temperature sensors, acceleration sensors, and other sensors monitor environmental changes and anomalies.

[1833] Example: Multifunctional sensor.

[1834] 4. Speakers:

[1835] A device that transmits instructions from a server via voice.

[1836] Example: Portable speaker.

[1837] 5. Security robots:

[1838] It is used as a portable monitoring device.

[1839] Example: Autonomous robot.

[1840] Software to use

[1841] 1. AI foundation:

[1842] Image analysis algorithms (e.g., OpenCV, YOLO)

[1843] Speech analysis algorithms (e.g., Google Speech-to-Text API)

[1844] Emotion engine (e.g., IBM Watson Tone Analyzer)

[1845] Data collection and analysis process

[1846] The server collects data from cameras, microphones, and various sensors installed in the monitoring area and analyzes it in real time. Specifically, it analyzes video data captured by cameras using OpenCV and YOLO to detect specific objects (e.g., suspicious persons) through object recognition. It also analyzes collected audio data using the Google Speech-to-Text API to identify abnormal sounds and specific voices (e.g., voices calling for help).

[1847] Furthermore, IBM Watson Tone Analyzer is used as an emotion engine to understand the emotional state of security personnel. The system analyzes tension and stress levels from the personnel's voice data, and the server generates appropriate action instructions based on the results.

[1848] Generation and transmission of instructions

[1849] Based on the analysis results, the server generates specific action instructions. For example, if a suspicious person is detected, a warning such as "Please leave immediately" is played through the speaker. Also, if the person in charge is nervous, advice such as "Please relax" is conveyed via voice.

[1850] Specific example

[1851] For example, suppose a security robot is patrolling a surveillance area and its camera captures the movement of a suspicious person. This video data is immediately sent to a server and analyzed by OpenCV or YOLO. If the analysis reveals a suspicious person, the server generates an instruction such as "Please leave immediately" and issues a voice warning through the robot's speaker.

[1852] Example of a prompt

[1853] "Analyze the footage from this security camera to determine if there is a suspicious person present. If a suspicious person is detected, generate the voice message 'There is a suspicious person present, please leave immediately.'"

[1854] Thus, the present invention enables security services to grasp the situation in real time, take into account the emotional state of the person in charge, generate appropriate instructions, and respond quickly and accurately.

[1855] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1856] Step 1:

[1857] The terminal devices (security robots, surveillance drones) use mounted cameras, microphones, and various sensors to collect video, audio, and environmental data from the monitored area. Inputs include camera footage, microphone audio, and sensor readings, and this data is collected in real time.

[1858] Step 2:

[1859] The terminal transmits the collected data to the server via wireless communication. The input consists of video, audio, and sensor data acquired from the terminal, and by transmitting these to the server, the system is ready for real-time analysis.

[1860] Step 3:

[1861] The server analyzes the received video data using OpenCV and YOLO. Specifically, it performs image analysis to detect suspicious individuals and abnormal objects using object recognition algorithms. The input is video data, and the output is the detected object information as a result of the analysis.

[1862] Step 4:

[1863] The server converts the received audio data into text data using the Google Speech-to-Text API. The input is audio data, and the output is the converted text data. This text data is then analyzed using an algorithm that identifies abnormal sounds and specific voices.

[1864] Step 5:

[1865] The server analyzes the received and analyzed emotional data using IBM Watson Tone Analyzer. Input is audio and video data, and output is analyzed emotional state information. This is used to assess the tension and stress levels of security personnel.

[1866] Step 6:

[1867] Based on the analysis results from steps 3-5, the server generates instructions for the next course of action. Specifically, it generates warning instructions for suspicious individuals based on object recognition results, and advice for personnel based on emotion analysis results. The input is the analysis results, and the output is text data as action instructions.

[1868] Step 7:

[1869] The server converts the generated action instructions into audio data. This is done using text-to-speech (TTS) technology. The input is the text data of the action instructions, and the output is the audio data.

[1870] Step 8:

[1871] The server sends the generated audio data to the terminal. The input is audio data, and the output is the transmission of audio data to the terminal.

[1872] Step 9:

[1873] The device plays the received audio data through its speaker, transmitting instructions to the target (dog, suspicious person, security personnel). Input is audio data, and output is actual audio playback. For example, it can warn a suspicious person to "leave immediately."

[1874] Step 10:

[1875] The terminal monitors the situation again using cameras and sensors after the instruction is given, and sends the data to the server. The input is the re-collected video, audio, and sensor data, and the output is the transmission of data to the server. This step allows for the collection of feedback and further improvement of the accuracy of action instructions.

[1876] Through the steps outlined above, a cycle of information gathering, analysis, instruction generation and transmission, and feedback is realized, enabling the provision of effective security services.

[1877] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1878] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1879] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1880] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1881] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1882] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1883] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1884] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1885] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1886] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1887] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1888] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1889] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1890] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device...

Claims

1. This is an AI support device for disaster relief dogs. A waterproof device with a built-in camera, microphone, and various sensors is attached to the collar. A server equipped with an AI platform that analyzes information collected from the aforementioned camera and microphone in real time, A speaker that transmits instructions generated from the server to the dog, A system that includes this.

2. The system according to claim 1, comprising means for a handler to remotely check information in real time and issue instructions.

3. The system according to claim 1, wherein the AI ​​platform comprises means for analyzing images captured by a camera and detecting specific objects by object recognition.

Images