System

Abstract

A system is provided.SOLUTION: A system comprising: means for analyzing an audio datum using a AI model; means for obtaining and analyzing a remote video datum; means for determining an urgency from the audio datum and the video datum; and means for issuing an appropriate response command in response to the urgency.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, many of the calls coming into 119 are low-urgency or prank calls, which reduces the dispatch efficiency of ambulances and fire engines. Specifically, calls are made by drunk people, as a substitute for taxis, or to save on initial consultation fees. Of the 70% of calls made, 5% are prank calls. With the conventional method, multiple people in the control room manually determine the urgency of the call, which has limitations in terms of real-timeness and efficiency. Given this background, a new system is needed to improve the efficiency of 119 call centers and reduce the number of low-urgency calls. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for analyzing voice data using an AI model, a means for acquiring and analyzing remote video data, a means for determining the urgency of a call from the voice data and video data, and a means for issuing appropriate response commands according to the urgency. This system enables highly accurate determination of the urgency in real time by having the voice analysis means evaluate the tone, content, and urgency of the caller's voice, and the video analysis means detect movement and abnormal situations in the video. This reduces low-urgency incoming calls and enables fast and accurate emergency response.

[0006] An "AI model" is a computational algorithm designed to accomplish a specific task using artificial intelligence.

[0007] "Audio data" refers to the digital representation of acoustic signals obtained through a call or recording.

[0008] "Remote video data" refers to video information transmitted in real time from a remote location.

[0009] "Analysis" is the process of analyzing data and information in detail and extracting meaning and characteristics.

[0010] "Urgency" refers to the degree of urgency of the situation, the method of response, and the degree to which speed is required.

[0011] A "response order" refers to an order that instructs appropriate action to be taken. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] This invention is a system for determining whether a call made to 119 is urgent or not. A specific embodiment of this system is described below. The system of this invention analyzes audio data and remote video data and utilizes an AI model to determine the urgency of the call.

[0034] 1. Initialize the server

[0035] The server initializes the AI ​​model and video analysis module in advance to analyze the audio data and remote video data, so that the server is ready to process the data in real time.

[0036] 2. Receiving incoming calls

[0037] The server receives calls to 119 and acquires audio data and remote video data. The audio data is recorded in real time, and the video data is acquired from a remote device such as a camera. The audio data also contains information necessary for making emergency decisions, such as the content of the call and the caller's tone of voice.

[0038] 3. Analysis of audio and video data

[0039] The server uses an AI model to analyze the audio data, evaluating the caller's tone of voice, what was said, and the urgency score, while simultaneously analyzing the remote video data using a video analytics module to detect activity and unusual conditions in the video (e.g., fire, smoke, etc.).

[0040] As a concrete example, consider the case where a user witnesses a fire and calls 119. When the user shouts, "Fire! Help!", this voice data is sent to the server. At the same time, video of the scene is sent from the user's camera. The server analyzes the voice data to recognize the word "fire" and evaluate the level of tension in the caller's voice. It also uses video analysis to detect smoke in the video.

[0041] 4. Determining Urgency

[0042] The server combines the results of the analysis of the audio and video data to calculate an urgency score. If the urgency score is high, the server determines it as "high urgency," and if it is low, it determines it as "low urgency." This process allows the system to accurately determine the urgency based on audio and video.

[0043] 5. Issuance of response instructions

[0044] If the emergency is deemed high, the server immediately notifies the operator and issues a command to dispatch the appropriate rescue team or ambulance, enabling a prompt and appropriate response. If the emergency is not high, the server provides the operator with detailed information to help them make further confirmations and decisions.

[0045] In this way, the system of the present invention can accurately determine the urgency of calls made to 119, improving the efficiency of call centers. It also reduces prank calls and non-urgent calls, allowing for a quick response to truly urgent situations.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The server initializes the AI ​​model and video analysis module, which prepares the server for analyzing audio data and remote video data in real time. Specifically, the server loads the AI ​​model and initializes the video analysis module.

[0049] Step 2:

[0050] A user calls 119. The user reports an emergency to the operator. Once the call is connected, the server starts recording audio data and capturing remote video.

[0051] Step 3:

[0052] The server analyzes the voice data in real time, using AI models to assess the caller's tone of voice, content, and urgency score. This analysis uses voice recognition technology to specifically detect emergency words like "fire" and "accident."

[0053] Step 4:

[0054] The server receives the remote video data and analyzes it using a video analysis module. Video analysis detects movement and abnormal situations (e.g., fire, smoke, blood, etc.) in the video. The analysis results are converted into digital data and used to assess the emergency level.

[0055] Step 5:

[0056] The server combines the results of the audio and video analysis to calculate an urgency score. The overall urgency is determined based on the output data of the AI ​​model. In this process, the audio analysis urgency score and the video analysis urgency score are averaged.

[0057] Step 6:

[0058] The server creates a response plan based on the calculated urgency score. If the urgency score is high, it immediately notifies the operator and issues a command to dispatch the appropriate rescue team or ambulance. If the urgency score is low, it instructs the operator to check further.

[0059] Step 7:

[0060] The operator receives notifications from the server and responds according to specific instructions. In the case of a high-level emergency, they quickly dispatch a rescue team and take appropriate action. In the case of a low-level emergency, they obtain additional information from the user and take appropriate action.

[0061] As a concrete example, consider the case where a user witnesses a fire and calls 119. When the user shouts, "Fire! Help!", the server analyzes the voice and detects emergency words. At the same time, it analyzes the remote video to detect smoke. This allows the server to determine that the situation is a high urgency, immediately notify the operator, and issue a command to dispatch a fire engine.

[0062] Example 1

[0063] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0064] Conventional emergency call systems depended on the judgment of the operator, which meant it took time to determine the level of urgency and there was a risk of incorrect decisions being made. Furthermore, responding to prank calls and calls with low urgency could delay responses to actual emergency situations. Therefore, there is a need for a system that can analyze call audio and remote video data with high accuracy to quickly determine the level of urgency.

[0065] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0066] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for integrating the results of the analysis of the voice data and the video data to calculate an urgency score, and means for notifying an operator based on the urgency and issuing a command to dispatch an appropriate rescue team or ambulance. This enables the urgency to be determined with high accuracy using both voice and video, enabling a prompt and appropriate response.

[0067] An "AI model" is a mathematical model that uses artificial intelligence technology to analyze data and extract patterns and features for specific purposes.

[0068] "Voice Data" means sound waves converted into digital form that can be stored or analyzed and that contains the voice or sound content of a caller.

[0069] "Remote video data" means visual information captured from a remote location through a camera or other video capture device that can be stored in digital format or analyzed.

[0070] "Analysis means" refers to the technical means used to analyze data and extract necessary information using AI models, video analysis modules, etc.

[0071] The "urgency score" is an evaluation value that quantifies or quantifies the level of urgency based on the analysis results of the audio data and video data.

[0072] "Operator" means a person whose job is to receive and respond to emergency calls.

[0073] A "rescue team" is a public or private rescue team dispatched to respond to an emergency such as a fire or accident.

[0074] A "dispatch order" is an order issued to respond quickly in a highly urgent situation, and instructs the dispatch of rescue teams, ambulances, etc.

[0075] "Notification" means promptly conveying the results of a judgment on the level of urgency to an operator, and is the act of transmitting information to encourage appropriate response.

[0076] "Synthesis" is the process of combining multiple data and analysis results to make overall judgments and evaluations.

[0077] The invention is a system that determines the urgency of calls made to 911. The system utilizes an AI model to analyze audio data and remote video data.

[0078] The server first initializes the AI ​​model. The server uses an AI framework such as TENSORFLOW (registered trademark) or PyTorch. It also uses a video analysis library such as OpenCV for video analysis. Specifically, the server calls the "initialize_AI_model()" function to load the AI ​​model, and then initializes the video analysis module using the "initialize_video_analysis_module()" function.

[0079] Next, the server receives a call to 119. At this time, the server records the call's audio data in real time and obtains remote video data from the user's smartphone or camera. The server receives the call using the "receive_call()" function, records the audio data using the "record_audio()" function, and obtains the remote video data using the "get_video_feed()" function.

[0080] The server then analyzes the audio and video data. The server uses the "analyze_audio_data(audio_data)" function to analyze the audio data and evaluate the tone of the caller's voice and the content of what is being said. At the same time, the server uses the "analyze_video_data(video_data)" function to analyze the video data and detect movement or abnormal conditions in the video (e.g., fire, smoke, etc.).

[0081] Based on the analysis results, the server integrates the audio and video data and calculates an urgency score. This process is performed using the "integrate_analysis_results(audio_analysis, video_analysis)" function. This function integrates the audio and video analysis results. The "calculate_emergency_score(integrated_data)" function is also used to calculate the urgency score.

[0082] The server takes appropriate action based on the urgency score. If the urgency is deemed high, the server notifies the operator using the "notify_operator(emergency_level)" function and dispatches a rescue team or ambulance using the "dispatch_rescue_team()" function. On the other hand, if the urgency is low, the server provides detailed information to the operator using the "provide_additional_info()" function.

[0083] As a concrete example, consider the case where a user calls 119 at the scene of a fire. When the user shouts, "Fire! Help!", this audio data is sent to the server. At the same time, video of the scene is sent from the user's camera. The server analyzes the audio data using the "analyze_audio_data(audio_data)" function and recognizes the word "fire." It also uses video analysis to detect smoke and flames in the video. As the integrated analysis results in a high urgency score, the server notifies the operator using the "notify_operator(emergency_level)" function and dispatches an appropriate rescue team using the "dispatch_rescue_team()" function.

[0084] By inputting the following prompt sentence into the generative AI model, the user's voice data can be analyzed.

[0085] Example prompt sentence:

[0086] "Please analyze the urgent voice data. We have prepared the following sample voice data."

[0087] In this way, the system of the present invention can analyze audio and video data in real time and accurately determine emergency situations. This enables a prompt and appropriate response, improving the efficiency of call centers in emergency situations. It also minimizes the impact of prank calls and non-urgent calls, allowing for a prompt response to truly urgent situations.

[0088] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0089] Step 1: Initialize the server

[0090] The server initializes the AI ​​model and video analysis module.

[0091] Input: A library of pre-trained AI models and video analytics modules.

[0092] Processing: The server calls the "initialize_AI_model()" function to load the AI ​​model into memory, and initializes the video analysis module using the "initialize_video_analysis_module()" function.

[0093] Output: The AI ​​model and video analysis module have been initialized.

[0094] Step 2: Receiving the incoming call

[0095] The server receives calls to 119 and acquires audio data and remote video data.

[0096] Input: Phone call initiation signal, remote connection from user's smartphone or camera.

[0097] Processing: The server receives and initiates a call using the "receive_call()" function, records audio data in real time using the "record_audio()" function, and retrieves remote video data using the "get_video_feed()" function.

[0098] Output: Recorded audio data and captured remote video data.

[0099] Step 3: Analyzing the audio data

[0100] The server analyzes the acquired voice data using an AI model.

[0101] Input: Recorded audio data.

[0102] Processing: The server analyzes the audio data using the "analyze_audio_data(audio_data)" function, which evaluates the caller's tone of voice, the content of what is being said, and the urgency of the conversation.

[0103] Output: Analysis of the audio data (urgency score, identified keywords, tone rating).

[0104] Step 4: Analyzing the video data

[0105] The server analyzes the acquired video data using a video analysis module.

[0106] Input: Acquired remote video data.

[0107] Processing: The server analyzes the video data using the "analyze_video_data(video_data)" function, which detects abnormal conditions (e.g., fire or smoke) and movement in the video data.

[0108] Output: Analysis results of video data (detected abnormal situations and movements, urgency score).

[0109] Step 5: Integrating the analysis results

[0110] The server integrates the analysis results of the audio data and the video data.

[0111] Input: Audio data analysis results and video data analysis results.

[0112] Processing: The server integrates the audio and video analysis results using the "integrate_analysis_results(audio_analysis, video_analysis)" function.

[0113] Output: Consolidated analysis results (total urgency score).

[0114] Step 6: Determine the urgency

[0115] The server calculates an urgency score using the integrated analysis results.

[0116] Input: The integrated analysis results.

[0117] Processing: The server calculates the urgency score from the integrated data using the function "calculate_emergency_score(integrated_data)".

[0118] Output: Urgency score (high or low urgency).

[0119] Step 7: Sending response orders

[0120] The server issues appropriate response instructions based on the urgency score.

[0121] Input: Urgency score.

[0122] Processing: If the urgency score is high, the server notifies the operator using the "notify_operator(emergency_level)" function and dispatches a rescue team or ambulance using the "dispatch_rescue_team()" function. If the urgency score is low, the server provides more information to the operator using the "provide_additional_info()" function.

[0123] Output: Notify operator, dispatch rescue teams or ambulances, provide detailed information in low-risk situations.

[0124] In this way, the server can analyze the audio and video data in real time, determine the emergency situation with high accuracy, and take prompt action.

[0125] (Application example 1)

[0126] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0127] Conventional emergency notification systems analyze only audio data to determine the urgency of an emergency, which often limits information and makes it difficult to make accurate judgments. Furthermore, real-time processing of video data and linkage with audio are insufficient, making it difficult to respond appropriately to high-urgency situations. This reduces efficiency in situations where a rapid emergency response is required, and further improvements in accuracy and faster response are needed.

[0128] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0129] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for determining the level of urgency from the voice data and video data, means for issuing appropriate response commands depending on the level of urgency, and means for analyzing camera video data in real time in conjunction with the voice data in an emergency. This makes it possible to integrate information from both the voice and video data and determine the level of urgency quickly and with high accuracy.

[0130] An "AI model" is an artificial intelligence algorithm that analyzes audio and video data and performs pattern recognition and feature extraction.

[0131] "Voice data" refers to a voice signal that includes information necessary for determining the urgency of a call, such as the content of the call and the tone of the user's voice.

[0132] "Remote video data" is video information sent by a user from a smartphone or other camera device.

[0133] "Voice analysis means" refers to devices or software that use AI models to analyze the content, tone, and urgency of voice data.

[0134] "Video analytics" means devices or software that use AI models or other technologies to detect abnormal conditions or movements in remote video data.

[0135] The "means for determining the urgency" refers to a device or software that evaluates the urgency of a situation based on the results of analyzing audio data and video data.

[0136] The "means for issuing response commands" refers to a device or software that automatically issues commands to execute appropriate responses depending on the level of urgency.

[0137] "Means for analyzing camera video data in real time in conjunction with audio data in an emergency" refers to devices or software that instantly analyzes video data acquired in conjunction with a user's emergency call.

[0138] The present invention is a system for determining the urgency of an emergency using voice data and remote video data, and issuing prompt response instructions. A specific embodiment of this system will be described below.

[0139] System Overview

[0140] The system includes a voice data analyzer, a video data analyzer, an emergency assessment device, and a response command issuing device. All of these devices run on a server and are deployed in a cloud environment. The system acquires data via user devices such as smartphones and performs analysis in real time.

[0141] Voice data analysis

[0142] The server analyzes the voice data using an AI model for voice recognition. This AI model identifies the tone, content, and urgency of the user's voice. Once the voice data is acquired, the server converts and analyzes the content using a voice data analyzer.

[0143] Video Data Analysis

[0144] The server also analyzes remote video data acquired from users' smartphones and other camera devices, using AI models and video analysis modules to detect abnormal situations and movements within the video, enabling rapid recognition of emergencies such as fires and intruders.

[0145] Judgment of urgency

[0146] The server integrates the results of the analysis of the audio and video data and calculates an urgency score using an emergency assessment device. Based on this score, it determines with high accuracy whether an emergency has occurred. If the urgency is determined to be high, it immediately issues appropriate response instructions.

[0147] Output of response instructions

[0148] If the emergency is deemed high, the server will send a notification to the appropriate authorities, such as security companies or the police, via a response command system. This notification will include the user's voice data, video data, and location information, enabling a prompt and appropriate response.

[0149] Hardware and Software Use

[0150] The server uses the following hardware and software:

[0151] Voice recognition microphone (built-in microphone on smartphone)

[0152] Camera for capturing images (built-in camera on smartphone)

[0153] Python, speech_recognition library, transformers library

[0154] The voice data is converted into text by a speech recognition library and an AI model assesses the urgency of the call, while the video data is analyzed in real time via a video analytics module.

[0155] Specific examples

[0156] When an intruder suddenly breaks into the home, the user shouts "Help me!" into their smartphone. This voice data is sent to the server and analyzed by the AI ​​model. At the same time, the smartphone camera sends video data of the scene to the server, which is then analyzed by the video analysis module.

[0157] Prompt Sentence Examples

[0158] "Analyze the input voice data, and if it contains the keyword 'help,' assess the urgency of the data. Guide the user on what action to take in an emergency."

[0159] As a result, the present invention can integrate audio and video data to determine the urgency with high accuracy, thereby enabling a prompt and appropriate response.

[0160] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0161] Step 1:

[0162] The server performs initialization processing. Specifically, it loads and initializes the AI ​​model and video analysis module required to analyze audio and video data. This initialization prepares the server to execute real-time data processing. The AI ​​model and video analysis module are required as input, and the initialization-completed state is obtained as output.

[0163] Step 2:

[0164] When a user makes an emergency call, the smartphone sends voice data to a server. The smartphone acquires the voice data using a microphone for voice input. The input is the user's voice, and the acquired voice data is obtained as the output.

[0165] Step 3:

[0166] The server analyzes the received voice data in real time. Specifically, it uses a voice recognition library to convert the voice data into text, which is then analyzed using an AI model. The input is the voice data, and the output is the text data of the analysis results and an assessment of the urgency level.

[0167] Step 4:

[0168] It acquires remote video data from the user's smartphone. It uses a camera to record on-site video and transmits it to the server in real time. The input requires the camera on the user's device, and the output is the recorded video data.

[0169] Step 5:

[0170] The server analyzes the acquired video data. It uses a video analysis module to detect abnormal situations or movements (for example, fires or intruders). The input is video data, and the output is the analysis result, which indicates whether there is an abnormality or not and the urgency level.

[0171] Step 6:

[0172] The server combines the results of the analysis of the audio and video data to determine the urgency. It calculates an urgency score, and if it is high it is rated as high urgency, and if it is low it is rated as low urgency. The analysis results of the audio and video data are required as input, and the urgency score is obtained as output.

[0173] Step 7:

[0174] If the urgency is determined to be high, the server immediately issues appropriate response instructions. Specifically, it sends a notification to response agencies such as security companies and the police. The notification includes the user's audio data, video data, and location information. The input requires an urgency score, and the output is a notification of the response instructions.

[0175] This series of processing steps enables the system to integrate audio and video data, quickly and accurately determine the level of urgency, and prescribe appropriate responses.

[0176] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0177] This invention is a system that uses an AI model and an emotion engine to accurately determine the urgency of calls made to 119. The system includes means for analyzing audio data and remote video data and recognizing the user's emotional state from the audio and video.

[0178] 1. Initialize the server

[0179] The server initializes the AI ​​model, video analysis module, and emotion engine, which prepares the server to process audio and video data in real time. The emotion engine loads the libraries and models required to extract user emotions from audio and video.

[0180] 2. Receiving incoming calls

[0181] When a user calls 119, the server receives the call and captures the audio data and remote video data. The audio data is recorded in real time, and the remote video data is captured as a video stream from the camera.

[0182] 3. Analysis of audio and video data

[0183] The server uses AI models to analyze the audio data, assessing the caller's tone of voice, content, and urgency. At the same time, it uses a video analytics module to analyze the remote video data and detect activity or unusual conditions in the video (e.g., fire, smoke, blood, etc.).

[0184] Furthermore, the emotion engine recognizes the caller's emotional state based on the analysis of audio and video data. For example, the emotion engine analyzes the caller's level of tension from audio and identifies their emotional state by analyzing the caller's facial expressions and movements from video.

[0185] As a concrete example, consider the case where a user witnesses a fire. When the user shouts, "Fire! Help!", this voice data is sent to the server. At the same time, video footage of the fire scene captured by a remote camera is also sent to the server. The server recognizes the word "fire" through voice data analysis and assesses the user's level of urgency. It also detects smoke and flames at the scene through video analysis. Furthermore, the emotion engine recognizes that the caller is in a high-level emergency based on their tone of voice and facial expression.

[0186] 4. Determining Urgency

[0187] The server integrates the analysis results of the audio data, video data, and emotion engine to calculate an urgency score. In this process, the results of audio analysis, video analysis, and emotion recognition are combined to make a comprehensive urgency judgment. If the urgency score is high, the server judges it to be "high urgency," and if it is low, it judges it to be "low urgency."

[0188] 5. Issuance of response instructions

[0189] If the emergency is deemed high, the server immediately notifies the operator and issues instructions to dispatch the appropriate rescue team or ambulance. The server also provides the operator with information about the user's emotional state to help them respond appropriately. If the emergency is low, the server provides additional information and prompts the operator for further confirmation.

[0190] This system allows for highly accurate assessment of the urgency of 119 calls, enabling prompt and appropriate responses. Furthermore, by incorporating an emotion engine, it is possible to achieve even more accurate urgency assessments that take into account the caller's emotional state. In this way, the present invention significantly improves the efficiency of 119 call centers and reduces prank calls and non-urgent calls.

[0191] The processing flow will be explained below.

[0192] Step 1:

[0193] The server initializes the AI ​​model, video analysis module, and emotion engine. This enables real-time analysis of audio, video, and emotion data. Specifically, it loads the AI ​​model into memory, configures the video analysis module, and prepares the emotion engine library and model.

[0194] Step 2:

[0195] A user calls 119. To report an emergency, the user describes the situation over the phone. The user's words and tone of voice are sent to the server. Also, video data from the user's remote camera is sent to the server as needed.

[0196] Step 3:

[0197] The server starts recording audio data and simultaneously starts acquiring video streams from remote cameras, which are then processed in real time.

[0198] Step 4:

[0199] The server uses an AI model to analyze the recorded voice data, evaluating the caller's tone of voice, the content of what was said, and the urgency score. The AI ​​model detects specific keywords and quantifies the urgency level.

[0200] Step 5:

[0201] The server uses a video analytics module to analyze the remote video data, detecting movement and abnormal conditions (e.g., fire, smoke, blood, etc.) in the video and extracting this information as digital data.

[0202] Step 6:

[0203] The server uses an emotion engine to recognize the user's emotional state from audio and video data. From the audio data, the server analyzes the caller's tone and tempo to identify their emotional state. From the video data, the server analyzes the caller's facial expressions and gestures to evaluate emotions such as tension, fear, and excitement.

[0204] Step 7:

[0205] The server integrates the results of the analysis of the audio data, video data, and emotion data, and calculates an urgency score. The results of the audio analysis, video analysis, and emotion engine are combined to make a comprehensive urgency judgment.

[0206] Step 8:

[0207] The server issues appropriate response instructions based on the calculated urgency score. If the urgency is high, the server immediately notifies the operator and issues instructions to dispatch a rescue team or ambulance. The server also provides the operator with information about the user's emotional state, improving the quality of the response.

[0208] Step 9:

[0209] The operator receives a notification from the server and immediately takes appropriate action. If the emergency is high, they will instruct a quick response, and if the emergency is low, they will obtain additional information and take action.

[0210] As a concrete example, consider the case where a user witnesses a fire and calls 119. When the user yells, "Fire! Help!", the server uses voice analysis to recognize the word "fire" and assesses the level of urgency. At the same time, video analysis detects smoke, and the emotion engine determines that the user is extremely nervous. Based on this comprehensive assessment, the server determines that the level of urgency is high, and is able to immediately notify an operator and issue a command to dispatch a fire engine.

[0211] Example 2

[0212] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0213] Conventional 119 call centers determine the level of urgency based solely on voice information, making it difficult to effectively distinguish between high-urgency and low-urgency cases. Furthermore, they are unable to properly recognize the caller's emotional state, making it difficult to respond immediately and appropriately. This increases the risk of incorrect responses, posing challenges to the speed and accuracy of rescue responses.

[0214] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0215] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for recognizing the user's emotional state from the voice data and video data, means for determining the urgency from the voice data and video data, and means for issuing appropriate response instructions depending on the urgency. This enables quick and accurate identification of high-urgency cases and appropriate rescue responses that take into account the caller's emotional state.

[0216] An "AI model" is an artificial intelligence algorithm used to analyze a variety of data, including audio and video data.

[0217] "Voice data" refers to information in the form of voice signals transmitted by a user over a telephone.

[0218] "Remote video data" refers to information on video signals obtained by capturing the user's environment through a camera and transmitting it to a server.

[0219] "Emotional state" refers to the user's psychological and emotional state determined from the results of audio and video analysis.

[0220] "Voice analysis means" means means for using an AI model to evaluate the content, tone, and urgency of voice data.

[0221] "Video analysis means" refers to means for detecting movement or abnormal situations within remote video data.

[0222] The "urgency determination means" is a means for calculating an urgency score based on the analysis results of audio data and video data and the emotional state, and for assessing the urgency of the incident.

[0223] The "response command means" is a means for generating and outputting an appropriate command to dispatch a rescue team or an ambulance based on the result of the emergency assessment.

[0224] This invention is a system for accurately determining the urgency of an incoming call to 911 based on audio data and remote video data. The system includes a means for recognizing the emotional state of a user using an AI model, a video analysis module, and an emotion engine.

[0225] When the system first starts up, the server initializes the AI ​​model (e.g., TensorFlow), video analysis module (e.g., OpenCV), and emotion engine. This prepares the system for processing audio and video data in real time. The emotion engine is equipped with the necessary libraries and models for extracting user emotions from audio and video.

[0226] When a user calls 119, the device receives the call, acquires the audio data and remote video data, and sends them to the server in real time. For example, when a user yells "Help, there's a fire!", the audio is sent from the device to the server, and at the same time, the video of the scene captured by the remote camera is also sent to the server.

[0227] The server uses an AI model (e.g., Google® Speech-to-Text API or a proprietary voice analysis module) to analyze the received voice data. Here, the server analyzes the user's tone of voice and the content of what is being said to assess the level of urgency. For example, the keyword "fire" and the level of urgency in the voice are detected, and the call is judged to be "very urgent."

[0228] The server then uses OpenCV to analyze the video data. This video analysis module detects abnormal situations (e.g., fire, smoke, blood, etc.) and significant movement in the video sent from the remote camera. For example, it detects flame and smoke patterns in the video and concludes that a fire has been detected.

[0229] Furthermore, the emotion engine recognizes the user's emotional state based on the analysis of audio and video data. Specific emotions are extracted by analyzing tension and fear from audio and facial expressions and movements from video. For example, the server may analyze the user's trembling voice and teary eyes to determine that the user is "extremely tense."

[0230] Finally, the server calculates an urgency score by combining the results of the audio data, video data, and emotion engine analysis. If this score is above a certain level, it is judged to be a "high urgency," and if it is lower, it is judged to be a "low urgency." For example, if the audio analysis results in "fire," the video analysis shows "flames and smoke," and the emotion analysis shows "high tension," the overall urgency score will be high and the event will be judged to be a "high urgency."

[0231] If the emergency is deemed high, the server immediately notifies the operator and issues a command to dispatch the appropriate rescue team or ambulance. The operator is also provided with information about the user's emotional state, which encourages appropriate response. For example, the server may send a notification to the operator saying, "High emergency, fire. Dispatching rescue team to the scene. The caller is in a very tense situation."

[0232] As a concrete example of how this works, the prompt text is as follows:

[0233] "Audio data: 'Help, there's a fire!' Audio analysis: 'Fire' Video analysis: 'Flame and smoke detected' Emotional state: 'Very tense' Urgency score: 'High'"

[0234] This system will enable a highly accurate assessment of the urgency of calls to 119, enabling a swift and appropriate response. This will significantly improve the efficiency of 119 call centers, reducing prank calls and non-urgent calls. In addition, by combining it with an emotion engine, it will be possible to achieve a more accurate assessment of urgency by taking into account the caller's emotional state.

[0235] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0236] Step 1:

[0237] Initializing the Server

[0238] When the system starts up, the server loads the necessary libraries and models and initializes the AI ​​model (e.g., TensorFlow), video analysis module (e.g., OpenCV), and emotion engine. This prepares the system for analyzing audio and video data in real time. Specifically, each library is loaded and the model is set up. The input to this process is the "initialization command" and the output is the "initialization completion status."

[0239] Step 2:

[0240] Receiving incoming calls

[0241] When a user calls 119, the terminal receives the call and acquires the audio data and remote video data. This real-time data is sent from the terminal to the server. The input is the "user's telephone audio signal" and the "remote video signal," and the output is the "audio and video data sent to the server."

[0242] Step 3:

[0243] Analysis of audio data

[0244] The server analyzes the received voice data using an AI model. This analysis includes a process to evaluate the content and urgency of the voice. Specifically, the voice data is converted into text using the Google Speech-to-Text API or similar, and keywords and tone of voice are analyzed. The input is the "voice data sent to the server," and the output is the "analyzed voice text" and the "urgency assessment result."

[0245] Step 4:

[0246] Video data analysis

[0247] The server analyzes the remote video data using OpenCV. This analysis includes detecting abnormal situations in the video (for example, fire, smoke, blood, etc.). Specifically, it performs pattern recognition on each video frame to detect abnormal patterns. The input is the "video data sent to the server," and the output is "information about abnormal situations in the analyzed video."

[0248] Step 5:

[0249] Recognition of emotional states

[0250] The server uses an emotion engine to recognize the user's emotional state from the analysis of audio and video data. Specifically, emotions are determined based on the tone of voice, facial expressions read from the video, and movements. The inputs are the "analyzed audio text" and "analyzed abnormal situation information in the video," and the output is "recognized emotional state information."

[0251] Step 6:

[0252] Determining urgency

[0253] The server calculates an urgency score by integrating the analysis results of the audio data, video data, and emotion engine. Based on this score, if the urgency is judged to be high, it is judged as "high urgency," and if it is low, it is judged as "low urgency." The inputs are "analyzed audio text," "analyzed abnormal situation information in the video," and "recognized emotional state information," and the outputs are "urgency score" and "urgency judgment result."

[0254] Step 7:

[0255] Issuance of response instructions

[0256] Based on the urgency assessment result, the server generates appropriate response instructions and notifies the operator. In the case of a high urgency, an order to dispatch a rescue team or ambulance is issued immediately. The operator is also provided with information about the user's emotional state. The inputs are the "urgency assessment result" and "recognized emotional state information," and the outputs are "response instructions" and "operator notification."

[0257] (Application example 2)

[0258] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0259] In conventional emergency notification systems, callers rely solely on voice information to determine the level of urgency, which can result in a lack of immediate and appropriate response, even in highly urgent situations. It is particularly difficult to grasp the caller's situation and emotional state in detail using voice data alone. This can result in misjudgments and delayed responses in emergencies, posing a major challenge for security services, which require rapid and accurate responses.

[0260] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0261] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for determining the level of urgency from the voice data and video data, means for issuing appropriate response instructions depending on the level of urgency, and means for processing the voice data and video data collected using a smart device in real time. This allows for simultaneous analysis of the voice data and video data, enabling more accurate determination of the level of urgency and enabling faster and more appropriate responses.

[0262] An "AI model" is an artificial intelligence system that includes algorithms and statistical models that analyze data and make specific decisions.

[0263] "Audio data" is a digital representation of an audio signal captured using a microphone or the like.

[0264] "Remote video data" is a digital representation of a video signal captured from a remote location using a visual sensor such as a camera.

[0265] "Means of analysis" refers to the technical methods used to input data and extract and interpret the information contained in that data.

[0266] The "means for determining urgency" is a technical method for evaluating the urgency of a situation based on acquired data and outputting the results.

[0267] "Means for issuing appropriate response instructions" refers to a technical method for instructing a person or system to take specific actions based on the results of an emergency assessment.

[0268] A "smart device" is a portable or stationary electronic device that has Internet connectivity and is capable of sensing and data processing.

[0269] "Real-time processing means" refers to a technical method that analyzes data as it is generated and obtains results without delay.

[0270] This invention relates to a system that uses an AI model and an emotion engine to analyze audio data and remote video data collected from smart devices and accurately determine the urgency of a situation. This system is realized using the following hardware and software.

[0271] 1. Hardware and Software Used

[0272] Smart Devices: Electronic devices with internet connectivity, such as smartphones and security cameras.

[0273] Audio data collection microphone: A device for converting audio signals into digital data.

[0274] Remote video data collection camera: Equipment for converting video signals into digital data.

[0275] Server: A computer system with the computational resources to analyze audio and video data in real time.

[0276] software:

[0277] sounddevice library: A library for capturing and processing audio data.

[0278] cv2 (OpenCV): A library for acquiring and analyzing video data.

[0279] AI models: Algorithms and models for assessing urgency and issuing response instructions.

[0280] EmotionEngine: An engine for recognizing emotional states from audio and video data.

[0281] 2. Processing Flow

[0282] Once the server is initialized, it activates the AI ​​model, emotion engine, and video analysis module. Audio data and remote video data are sent to the server via smart devices, which process the data in real time. The AI ​​model analyzes the audio data to evaluate the caller's tone, content, and urgency. At the same time, the video analysis module detects abnormal situations in the video. The emotion engine analyzes the audio and video data to recognize the caller's emotional state.

[0283] 3. Urgency assessment and response instructions

[0284] The server calculates an urgency score by integrating the results of the audio and video analysis. If the urgency is determined to be high based on this score, the server immediately issues appropriate response instructions. For example, a notification may be sent to security guards or rescue teams. If the urgency is low, further confirmation is required and an operator is notified.

[0285] 4. Specific Examples

[0286] When a user reports "someone has broken into my house" in the middle of the night, the system collects audio data and remote video data. The server analyzes the tone of the voice and any abnormal activity in the video (e.g., broken windows or suspicious activity) in real time to determine the level of urgency. If the level of urgency is determined to be high, the server immediately issues a response command to call security guards or a rescue team.

[0287] 5. Examples of prompts

[0288] "Create an AI system that can recognize specific emotional states from audio and video data in real time and determine the level of urgency. The specific input data is as follows:

[0289] Audio data: A user shouts "Help!" in a tense voice at the scene of an accident

[0290] Video data: Includes footage of the accident scene, flames, smoke, and injured people.

[0291] As described above, the present invention makes it possible to judge the urgency of an emergency with significantly higher accuracy and to respond more quickly than conventional systems.

[0292] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0293] Step 1:

[0294] Initializing the Server

[0295] The server initializes the AI ​​model, emotion engine, and video analysis module. This prepares the server to process audio and video data in real time. Specifically, it loads various models and engines and imports related libraries. The input is the initialization command, and the output is the ready status.

[0296] Step 2:

[0297] Acquisition of audio and remote video data

[0298] When a user makes an emergency call, the microphone and camera on the terminal (smart device) collect audio and video data and send it to the server. The input is the user's audio and video, and the output is a digital signal of these data.

[0299] Step 3:

[0300] Analysis of audio data

[0301] The server receives the voice data and analyzes it using an AI model. The analysis evaluates the tone of voice, the content of the conversation, and the level of urgency. Specific operations include noise removal from the voice data, feature extraction, and a model-based urgency evaluation. The input is the voice data, and the output is the analysis results.

[0302] Step 4:

[0303] Video data analysis

[0304] The server receives the remote video data and analyzes it using a video analysis module. This analysis detects abnormal activity or situations in the video (e.g., fire, smoke, suspicious individuals, etc.). Specific operations include frame segmentation of the video data, feature extraction, and application of anomaly detection algorithms. The input is the video data, and the output is the analysis results.

[0305] Step 5:

[0306] Recognition of emotional states

[0307] The server uses an emotion engine to recognize the caller's emotional state from the analysis results of the audio and video data. Specifically, it integrates the audio and video analysis results and executes an algorithm to evaluate the emotional state. The input is the audio and video analysis results, and the output is the recognition result of the emotional state.

[0308] Step 6:

[0309] Determining the level of urgency

[0310] The server integrates the analysis results of the audio data, video data, and emotion engine to calculate an urgency score. The inputs are the audio analysis results, video analysis results, and emotion recognition results, and the output is an urgency score. Specifically, the server integrates the analysis results using a weighted average or other integration algorithm to calculate the urgency score.

[0311] Step 7:

[0312] Issuance of response instructions

[0313] Based on the urgency score, the server immediately issues an appropriate response command if the urgency is high, or notifies an operator if the urgency is low. Specifically, the server selects a command based on the urgency and sends the command to the appropriate response channel. For example, if the urgency is high, it notifies security guards or rescue teams, and if the urgency is low, it requests further confirmation from an operator. The input is the urgency score, and the output is a response command.

[0314] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0315] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0316] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0317] [Second embodiment]

[0318] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0319] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0320] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0321] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0322] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0323] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0324] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0325] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0326] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0327] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0328] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0329] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0330] This invention is a system for determining whether a call made to 119 is urgent or not. A specific embodiment of this system is described below. The system of this invention analyzes audio data and remote video data and utilizes an AI model to determine the urgency of the call.

[0331] 1. Initialize the server

[0332] The server initializes the AI ​​model and video analysis module in advance to analyze the audio data and remote video data, so that the server is ready to process the data in real time.

[0333] 2. Receiving incoming calls

[0334] The server receives calls to 119 and acquires audio data and remote video data. The audio data is recorded in real time, and the video data is acquired from a remote device such as a camera. The audio data also contains information necessary for making emergency decisions, such as the content of the call and the caller's tone of voice.

[0335] 3. Analysis of audio and video data

[0336] The server uses an AI model to analyze the audio data, evaluating the caller's tone of voice, what was said, and the urgency score, while simultaneously analyzing the remote video data using a video analytics module to detect activity and unusual conditions in the video (e.g., fire, smoke, etc.).

[0337] As a concrete example, consider the case where a user witnesses a fire and calls 119. When the user shouts, "Fire! Help!", this voice data is sent to the server. At the same time, video of the scene is sent from the user's camera. The server analyzes the voice data to recognize the word "fire" and evaluate the level of tension in the caller's voice. It also uses video analysis to detect smoke in the video.

[0338] 4. Determining Urgency

[0339] The server combines the results of the analysis of the audio and video data to calculate an urgency score. If the urgency score is high, the server determines it as "high urgency," and if it is low, it determines it as "low urgency." This process allows the system to accurately determine the urgency based on audio and video.

[0340] 5. Issuance of response instructions

[0341] If the emergency is deemed high, the server immediately notifies the operator and issues a command to dispatch the appropriate rescue team or ambulance, enabling a prompt and appropriate response. If the emergency is not high, the server provides the operator with detailed information to help them make further confirmations and decisions.

[0342] In this way, the system of the present invention can accurately determine the urgency of calls made to 119, improving the efficiency of call centers. It also reduces prank calls and non-urgent calls, allowing for a quick response to truly urgent situations.

[0343] The processing flow will be explained below.

[0344] Step 1:

[0345] The server initializes the AI ​​model and video analysis module, which prepares the server for analyzing audio data and remote video data in real time. Specifically, the server loads the AI ​​model and initializes the video analysis module.

[0346] Step 2:

[0347] A user calls 119. The user reports an emergency to the operator. Once the call is connected, the server starts recording audio data and capturing remote video.

[0348] Step 3:

[0349] The server analyzes the voice data in real time, using AI models to assess the caller's tone of voice, content, and urgency score. This analysis uses voice recognition technology to specifically detect emergency words like "fire" and "accident."

[0350] Step 4:

[0351] The server receives the remote video data and analyzes it using a video analysis module. Video analysis detects movement and abnormal situations (e.g., fire, smoke, blood, etc.) in the video. The analysis results are converted into digital data and used to assess the emergency level.

[0352] Step 5:

[0353] The server combines the results of the audio and video analysis to calculate an urgency score. The overall urgency is determined based on the output data of the AI ​​model. In this process, the audio analysis urgency score and the video analysis urgency score are averaged.

[0354] Step 6:

[0355] The server creates a response plan based on the calculated urgency score. If the urgency score is high, it immediately notifies the operator and issues a command to dispatch the appropriate rescue team or ambulance. If the urgency score is low, it instructs the operator to check further.

[0356] Step 7:

[0357] The operator receives notifications from the server and responds according to specific instructions. In the case of a high-level emergency, they quickly dispatch a rescue team and take appropriate action. In the case of a low-level emergency, they obtain additional information from the user and take appropriate action.

[0358] As a concrete example, consider the case where a user witnesses a fire and calls 119. When the user shouts, "Fire! Help!", the server analyzes the voice and detects emergency words. At the same time, it analyzes the remote video to detect smoke. This allows the server to determine that the situation is a high urgency, immediately notify the operator, and issue a command to dispatch a fire engine.

[0359] Example 1

[0360] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0361] Conventional emergency call systems depended on the judgment of the operator, which meant it took time to determine the level of urgency and there was a risk of incorrect decisions being made. Furthermore, responding to prank calls and calls with low urgency could delay responses to actual emergency situations. Therefore, there is a need for a system that can analyze call audio and remote video data with high accuracy to quickly determine the level of urgency.

[0362] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0363] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for integrating the results of the analysis of the voice data and the video data to calculate an urgency score, and means for notifying an operator based on the urgency and issuing a command to dispatch an appropriate rescue team or ambulance. This enables the urgency to be determined with high accuracy using both voice and video, enabling a prompt and appropriate response.

[0364] An "AI model" is a mathematical model that uses artificial intelligence technology to analyze data and extract patterns and features for specific purposes.

[0365] "Voice Data" means sound waves converted into digital form that can be stored or analyzed and that contains the voice or sound content of a caller.

[0366] "Remote video data" means visual information captured from a remote location through a camera or other video capture device that can be stored in digital format or analyzed.

[0367] "Analysis means" refers to the technical means used to analyze data and extract necessary information using AI models, video analysis modules, etc.

[0368] The "urgency score" is an evaluation value that quantifies or quantifies the level of urgency based on the analysis results of the audio data and video data.

[0369] "Operator" means a person whose job is to receive and respond to emergency calls.

[0370] A "rescue team" is a public or private rescue team dispatched to respond to an emergency such as a fire or accident.

[0371] A "dispatch order" is an order issued to respond quickly in a highly urgent situation, and instructs the dispatch of rescue teams, ambulances, etc.

[0372] "Notification" means promptly conveying the results of a judgment on the level of urgency to an operator, and is the act of transmitting information to encourage appropriate response.

[0373] "Synthesis" is the process of combining multiple data and analysis results to make overall judgments and evaluations.

[0374] The invention is a system that determines the urgency of calls made to 911. The system utilizes an AI model to analyze audio data and remote video data.

[0375] The server first initializes the AI ​​model. The server uses an AI framework such as TensorFlow or PyTorch. It also uses a video analysis library such as OpenCV for video analysis. Specifically, the server calls the "initialize_AI_model()" function to load the AI ​​model and initializes the video analysis module using the "initialize_video_analysis_module()" function.

[0376] Next, the server receives a call to 119. At this time, the server records the call's audio data in real time and obtains remote video data from the user's smartphone or camera. The server receives the call using the "receive_call()" function, records the audio data using the "record_audio()" function, and obtains the remote video data using the "get_video_feed()" function.

[0377] The server then analyzes the audio and video data. The server uses the "analyze_audio_data(audio_data)" function to analyze the audio data and evaluate the tone of the caller's voice and the content of what is being said. At the same time, the server uses the "analyze_video_data(video_data)" function to analyze the video data and detect movement or abnormal conditions in the video (e.g., fire, smoke, etc.).

[0378] Based on the analysis results, the server integrates the audio and video data and calculates an urgency score. This process is performed using the "integrate_analysis_results(audio_analysis, video_analysis)" function. This function integrates the audio and video analysis results. The "calculate_emergency_score(integrated_data)" function is also used to calculate the urgency score.

[0379] The server takes appropriate action based on the urgency score. If the urgency is deemed high, the server notifies the operator using the "notify_operator(emergency_level)" function and dispatches a rescue team or ambulance using the "dispatch_rescue_team()" function. On the other hand, if the urgency is low, the server provides detailed information to the operator using the "provide_additional_info()" function.

[0380] As a concrete example, consider the case where a user calls 119 at the scene of a fire. When the user shouts, "Fire! Help!", this audio data is sent to the server. At the same time, video of the scene is sent from the user's camera. The server analyzes the audio data using the "analyze_audio_data(audio_data)" function and recognizes the word "fire." It also uses video analysis to detect smoke and flames in the video. As the integrated analysis results in a high urgency score, the server notifies the operator using the "notify_operator(emergency_level)" function and dispatches an appropriate rescue team using the "dispatch_rescue_team()" function.

[0381] By inputting the following prompt sentence into the generative AI model, the user's voice data can be analyzed.

[0382] Example prompt sentence:

[0383] "Please analyze the urgent voice data. We have prepared the following sample voice data."

[0384] In this way, the system of the present invention can analyze audio and video data in real time and accurately determine emergency situations. This enables a prompt and appropriate response, improving the efficiency of call centers in emergency situations. It also minimizes the impact of prank calls and non-urgent calls, allowing for a prompt response to truly urgent situations.

[0385] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0386] Step 1: Initialize the server

[0387] The server initializes the AI ​​model and video analysis module.

[0388] Input: A library of pre-trained AI models and video analytics modules.

[0389] Processing: The server calls the "initialize_AI_model()" function to load the AI ​​model into memory, and initializes the video analysis module using the "initialize_video_analysis_module()" function.

[0390] Output: The AI ​​model and video analysis module have been initialized.

[0391] Step 2: Receiving the incoming call

[0392] The server receives calls to 119 and acquires audio data and remote video data.

[0393] Input: Phone call initiation signal, remote connection from user's smartphone or camera.

[0394] Processing: The server receives and initiates a call using the "receive_call()" function, records audio data in real time using the "record_audio()" function, and retrieves remote video data using the "get_video_feed()" function.

[0395] Output: Recorded audio data and captured remote video data.

[0396] Step 3: Analyzing the audio data

[0397] The server analyzes the acquired voice data using an AI model.

[0398] Input: Recorded audio data.

[0399] Processing: The server analyzes the audio data using the "analyze_audio_data(audio_data)" function, which evaluates the caller's tone of voice, the content of what is being said, and the urgency of the conversation.

[0400] Output: Analysis of the audio data (urgency score, identified keywords, tone rating).

[0401] Step 4: Analyzing the video data

[0402] The server analyzes the acquired video data using a video analysis module.

[0403] Input: Acquired remote video data.

[0404] Processing: The server analyzes the video data using the "analyze_video_data(video_data)" function, which detects abnormal conditions (e.g., fire or smoke) and movement in the video data.

[0405] Output: Analysis results of video data (detected abnormal situations and movements, urgency score).

[0406] Step 5: Integrating the analysis results

[0407] The server integrates the analysis results of the audio data and the video data.

[0408] Input: Audio data analysis results and video data analysis results.

[0409] Processing: The server integrates the audio and video analysis results using the "integrate_analysis_results(audio_analysis, video_analysis)" function.

[0410] Output: Consolidated analysis results (total urgency score).

[0411] Step 6: Determine the urgency

[0412] The server calculates an urgency score using the integrated analysis results.

[0413] Input: The integrated analysis results.

[0414] Processing: The server calculates the urgency score from the integrated data using the function "calculate_emergency_score(integrated_data)".

[0415] Output: Urgency score (high or low urgency).

[0416] Step 7: Sending response orders

[0417] The server issues appropriate response instructions based on the urgency score.

[0418] Input: Urgency score.

[0419] Processing: If the urgency score is high, the server notifies the operator using the "notify_operator(emergency_level)" function and dispatches a rescue team or ambulance using the "dispatch_rescue_team()" function. If the urgency score is low, the server provides more information to the operator using the "provide_additional_info()" function.

[0420] Output: Notify operator, dispatch rescue teams or ambulances, provide detailed information in low-risk situations.

[0421] In this way, the server can analyze the audio and video data in real time, determine the emergency situation with high accuracy, and take prompt action.

[0422] (Application example 1)

[0423] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0424] Conventional emergency notification systems analyze only audio data to determine the urgency of an emergency, which often limits information and makes it difficult to make accurate judgments. Furthermore, real-time processing of video data and linkage with audio are insufficient, making it difficult to respond appropriately to high-urgency situations. This reduces efficiency in situations where a rapid emergency response is required, and further improvements in accuracy and faster response are needed.

[0425] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0426] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for determining the level of urgency from the voice data and video data, means for issuing appropriate response commands depending on the level of urgency, and means for analyzing camera video data in real time in conjunction with the voice data in an emergency. This makes it possible to integrate information from both the voice and video data and determine the level of urgency quickly and with high accuracy.

[0427] An "AI model" is an artificial intelligence algorithm that analyzes audio and video data and performs pattern recognition and feature extraction.

[0428] "Voice data" refers to a voice signal that includes information necessary for determining the urgency of a call, such as the content of the call and the tone of the user's voice.

[0429] "Remote video data" is video information sent by a user from a smartphone or other camera device.

[0430] "Voice analysis means" refers to devices or software that use AI models to analyze the content, tone, and urgency of voice data.

[0431] "Video analytics" means devices or software that use AI models or other technologies to detect abnormal conditions or movements in remote video data.

[0432] The "means for determining the urgency" refers to a device or software that evaluates the urgency of a situation based on the results of analyzing audio data and video data.

[0433] The "means for issuing response commands" refers to a device or software that automatically issues commands to execute appropriate responses depending on the level of urgency.

[0434] "Means for analyzing camera video data in real time in conjunction with audio data in an emergency" refers to devices or software that instantly analyzes video data acquired in conjunction with a user's emergency call.

[0435] The present invention is a system for determining the urgency of an emergency using voice data and remote video data, and issuing prompt response instructions. A specific embodiment of this system will be described below.

[0436] System Overview

[0437] The system includes a voice data analyzer, a video data analyzer, an emergency assessment device, and a response command issuing device. All of these devices run on a server and are deployed in a cloud environment. The system acquires data via user devices such as smartphones and performs analysis in real time.

[0438] Voice data analysis

[0439] The server analyzes the voice data using an AI model for voice recognition. This AI model identifies the tone, content, and urgency of the user's voice. Once the voice data is acquired, the server converts and analyzes the content using a voice data analyzer.

[0440] Video Data Analysis

[0441] The server also analyzes remote video data acquired from users' smartphones and other camera devices, using AI models and video analysis modules to detect abnormal situations and movements within the video, enabling rapid recognition of emergencies such as fires and intruders.

[0442] Judgment of urgency

[0443] The server integrates the results of the analysis of the audio and video data and calculates an urgency score using an emergency assessment device. Based on this score, it determines with high accuracy whether an emergency has occurred. If the urgency is determined to be high, it immediately issues appropriate response instructions.

[0444] Output of response instructions

[0445] If the emergency is deemed high, the server will send a notification to the appropriate authorities, such as security companies or the police, via a response command system. This notification will include the user's voice data, video data, and location information, enabling a prompt and appropriate response.

[0446] Hardware and Software Use

[0447] The server uses the following hardware and software:

[0448] Voice recognition microphone (built-in microphone on smartphone)

[0449] Camera for capturing images (built-in camera on smartphone)

[0450] Python, speech_recognition library, transformers library

[0451] The voice data is converted into text by a speech recognition library and an AI model assesses the urgency of the call, while the video data is analyzed in real time via a video analytics module.

[0452] Specific examples

[0453] When an intruder suddenly breaks into the home, the user shouts "Help me!" into their smartphone. This voice data is sent to the server and analyzed by the AI ​​model. At the same time, the smartphone camera sends video data of the scene to the server, which is then analyzed by the video analysis module.

[0454] Prompt Sentence Examples

[0455] "Analyze the input voice data, and if it contains the keyword 'help,' assess the urgency of the data. Guide the user on what action to take in an emergency."

[0456] As a result, the present invention can integrate audio and video data to determine the urgency with high accuracy, thereby enabling a prompt and appropriate response.

[0457] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0458] Step 1:

[0459] The server performs initialization processing. Specifically, it loads and initializes the AI ​​model and video analysis module required to analyze audio and video data. This initialization prepares the server to execute real-time data processing. The AI ​​model and video analysis module are required as input, and the initialization-completed state is obtained as output.

[0460] Step 2:

[0461] When a user makes an emergency call, the smartphone sends voice data to a server. The smartphone acquires the voice data using a microphone for voice input. The input is the user's voice, and the acquired voice data is obtained as the output.

[0462] Step 3:

[0463] The server analyzes the received voice data in real time. Specifically, it uses a voice recognition library to convert the voice data into text, which is then analyzed using an AI model. The input is the voice data, and the output is the text data of the analysis results and an assessment of the urgency level.

[0464] Step 4:

[0465] It acquires remote video data from the user's smartphone. It uses a camera to record on-site video and transmits it to the server in real time. The input requires the camera on the user's device, and the output is the recorded video data.

[0466] Step 5:

[0467] The server analyzes the acquired video data. It uses a video analysis module to detect abnormal situations or movements (for example, fires or intruders). The input is video data, and the output is the analysis result, which indicates whether there is an abnormality or not and the urgency level.

[0468] Step 6:

[0469] The server combines the results of the analysis of the audio and video data to determine the urgency. It calculates an urgency score, and if it is high it is rated as high urgency, and if it is low it is rated as low urgency. The analysis results of the audio and video data are required as input, and the urgency score is obtained as output.

[0470] Step 7:

[0471] If the urgency is determined to be high, the server immediately issues appropriate response instructions. Specifically, it sends a notification to response agencies such as security companies and the police. The notification includes the user's audio data, video data, and location information. The input requires an urgency score, and the output is a notification of the response instructions.

[0472] This series of processing steps enables the system to integrate audio and video data, quickly and accurately determine the level of urgency, and prescribe appropriate responses.

[0473] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0474] This invention is a system that uses an AI model and an emotion engine to accurately determine the urgency of calls made to 119. The system includes means for analyzing audio data and remote video data and recognizing the user's emotional state from the audio and video.

[0475] 1. Initialize the server

[0476] The server initializes the AI ​​model, video analysis module, and emotion engine, which prepares the server to process audio and video data in real time. The emotion engine loads the libraries and models required to extract user emotions from audio and video.

[0477] 2. Receiving incoming calls

[0478] When a user calls 119, the server receives the call and captures the audio data and remote video data. The audio data is recorded in real time, and the remote video data is captured as a video stream from the camera.

[0479] 3. Analysis of audio and video data

[0480] The server uses AI models to analyze the audio data, assessing the caller's tone of voice, content, and urgency. At the same time, it uses a video analytics module to analyze the remote video data and detect activity or unusual conditions in the video (e.g., fire, smoke, blood, etc.).

[0481] Furthermore, the emotion engine recognizes the caller's emotional state based on the analysis of audio and video data. For example, the emotion engine analyzes the caller's level of tension from audio and identifies their emotional state by analyzing the caller's facial expressions and movements from video.

[0482] As a concrete example, consider the case where a user witnesses a fire. When the user shouts, "Fire! Help!", this voice data is sent to the server. At the same time, video footage of the fire scene captured by a remote camera is also sent to the server. The server recognizes the word "fire" through voice data analysis and assesses the user's level of urgency. It also detects smoke and flames at the scene through video analysis. Furthermore, the emotion engine recognizes that the caller is in a high-level emergency based on their tone of voice and facial expression.

[0483] 4. Determining Urgency

[0484] The server integrates the analysis results of the audio data, video data, and emotion engine to calculate an urgency score. In this process, the results of audio analysis, video analysis, and emotion recognition are combined to make a comprehensive urgency judgment. If the urgency score is high, the server judges it to be "high urgency," and if it is low, it judges it to be "low urgency."

[0485] 5. Issuance of response instructions

[0486] If the emergency is deemed high, the server immediately notifies the operator and issues instructions to dispatch the appropriate rescue team or ambulance. The server also provides the operator with information about the user's emotional state to help them respond appropriately. If the emergency is low, the server provides additional information and prompts the operator for further confirmation.

[0487] This system allows for highly accurate assessment of the urgency of 119 calls, enabling prompt and appropriate responses. Furthermore, by incorporating an emotion engine, it is possible to achieve even more accurate urgency assessments that take into account the caller's emotional state. In this way, the present invention significantly improves the efficiency of 119 call centers and reduces prank calls and non-urgent calls.

[0488] The processing flow will be explained below.

[0489] Step 1:

[0490] The server initializes the AI ​​model, video analysis module, and emotion engine. This enables real-time analysis of audio, video, and emotion data. Specifically, it loads the AI ​​model into memory, configures the video analysis module, and prepares the emotion engine library and model.

[0491] Step 2:

[0492] A user calls 119. To report an emergency, the user describes the situation over the phone. The user's words and tone of voice are sent to the server. Also, video data from the user's remote camera is sent to the server as needed.

[0493] Step 3:

[0494] The server starts recording audio data and simultaneously starts acquiring video streams from remote cameras, which are then processed in real time.

[0495] Step 4:

[0496] The server uses an AI model to analyze the recorded voice data, evaluating the caller's tone of voice, the content of what was said, and the urgency score. The AI ​​model detects specific keywords and quantifies the urgency level.

[0497] Step 5:

[0498] The server uses a video analytics module to analyze the remote video data, detecting movement and abnormal conditions (e.g., fire, smoke, blood, etc.) in the video and extracting this information as digital data.

[0499] Step 6:

[0500] The server uses an emotion engine to recognize the user's emotional state from audio and video data. From the audio data, the server analyzes the caller's tone and tempo to identify their emotional state. From the video data, the server analyzes the caller's facial expressions and gestures to evaluate emotions such as tension, fear, and excitement.

[0501] Step 7:

[0502] The server integrates the results of the analysis of the audio data, video data, and emotion data, and calculates an urgency score. The results of the audio analysis, video analysis, and emotion engine are combined to make a comprehensive urgency judgment.

[0503] Step 8:

[0504] The server issues appropriate response instructions based on the calculated urgency score. If the urgency is high, the server immediately notifies the operator and issues instructions to dispatch a rescue team or ambulance. The server also provides the operator with information about the user's emotional state, improving the quality of the response.

[0505] Step 9:

[0506] The operator receives a notification from the server and immediately takes appropriate action. If the emergency is high, they will instruct a quick response, and if the emergency is low, they will obtain additional information and take action.

[0507] As a concrete example, consider the case where a user witnesses a fire and calls 119. When the user yells, "Fire! Help!", the server uses voice analysis to recognize the word "fire" and assesses the level of urgency. At the same time, video analysis detects smoke, and the emotion engine determines that the user is extremely nervous. Based on this comprehensive assessment, the server determines that the level of urgency is high, and is able to immediately notify an operator and issue a command to dispatch a fire engine.

[0508] Example 2

[0509] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0510] Conventional 119 call centers determine the level of urgency based solely on voice information, making it difficult to effectively distinguish between high-urgency and low-urgency cases. Furthermore, they are unable to properly recognize the caller's emotional state, making it difficult to respond immediately and appropriately. This increases the risk of incorrect responses, posing challenges to the speed and accuracy of rescue responses.

[0511] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0512] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for recognizing the user's emotional state from the voice data and video data, means for determining the urgency from the voice data and video data, and means for issuing appropriate response instructions depending on the urgency. This enables quick and accurate identification of high-urgency cases and appropriate rescue responses that take into account the caller's emotional state.

[0513] An "AI model" is an artificial intelligence algorithm used to analyze a variety of data, including audio and video data.

[0514] "Voice data" refers to information in the form of voice signals transmitted by a user over a telephone.

[0515] "Remote video data" refers to information on video signals obtained by capturing the user's environment through a camera and transmitting it to a server.

[0516] "Emotional state" refers to the user's psychological and emotional state determined from the results of audio and video analysis.

[0517] "Voice analysis means" means means for using an AI model to evaluate the content, tone, and urgency of voice data.

[0518] "Video analysis means" refers to means for detecting movement or abnormal situations within remote video data.

[0519] The "urgency determination means" is a means for calculating an urgency score based on the analysis results of audio data and video data and the emotional state, and for assessing the urgency of the incident.

[0520] The "response command means" is a means for generating and outputting an appropriate command to dispatch a rescue team or an ambulance based on the result of the emergency assessment.

[0521] This invention is a system for accurately determining the urgency of an incoming call to 911 based on audio data and remote video data. The system includes a means for recognizing the emotional state of a user using an AI model, a video analysis module, and an emotion engine.

[0522] When the system first starts up, the server initializes the AI ​​model (e.g., TensorFlow), video analysis module (e.g., OpenCV), and emotion engine. This prepares the system for processing audio and video data in real time. The emotion engine is equipped with the necessary libraries and models for extracting user emotions from audio and video.

[0523] When a user calls 119, the device receives the call, acquires the audio data and remote video data, and sends them to the server in real time. For example, when a user yells "Help, there's a fire!", the audio is sent from the device to the server, and at the same time, the video of the scene captured by the remote camera is also sent to the server.

[0524] The server uses an AI model (e.g., Google Speech-to-Text API or a proprietary voice analysis module) to analyze the received voice data. Here, the server analyzes the user's tone of voice and the content of what is being said to assess the urgency. For example, the keyword "fire" and the level of urgency in the voice are detected, and the call is judged to be "very urgent."

[0525] The server then uses OpenCV to analyze the video data. This video analysis module detects abnormal situations (e.g., fire, smoke, blood, etc.) and significant movement in the video sent from the remote camera. For example, it detects flame and smoke patterns in the video and concludes that a fire has been detected.

[0526] Furthermore, the emotion engine recognizes the user's emotional state based on the analysis of audio and video data. Specific emotions are extracted by analyzing tension and fear from audio and facial expressions and movements from video. For example, the server may analyze the user's trembling voice and teary eyes to determine that the user is "extremely tense."

[0527] Finally, the server calculates an urgency score by combining the results of the audio data, video data, and emotion engine analysis. If this score is above a certain level, it is judged to be a "high urgency," and if it is lower, it is judged to be a "low urgency." For example, if the audio analysis results in "fire," the video analysis shows "flames and smoke," and the emotion analysis shows "high tension," the overall urgency score will be high and the event will be judged to be a "high urgency."

[0528] If the emergency is deemed high, the server immediately notifies the operator and issues a command to dispatch the appropriate rescue team or ambulance. The operator is also provided with information about the user's emotional state, which encourages appropriate response. For example, the server may send a notification to the operator saying, "High emergency, fire. Dispatching rescue team to the scene. The caller is in a very tense situation."

[0529] As a concrete example of how this works, the prompt text is as follows:

[0530] "Audio data: 'Help, there's a fire!' Audio analysis: 'Fire' Video analysis: 'Flame and smoke detected' Emotional state: 'Very tense' Urgency score: 'High'"

[0531] This system will enable a highly accurate assessment of the urgency of calls to 119, enabling a swift and appropriate response. This will significantly improve the efficiency of 119 call centers, reducing prank calls and non-urgent calls. In addition, by combining it with an emotion engine, it will be possible to achieve a more accurate assessment of urgency by taking into account the caller's emotional state.

[0532] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0533] Step 1:

[0534] Initializing the Server

[0535] When the system starts up, the server loads the necessary libraries and models and initializes the AI ​​model (e.g., TensorFlow), video analysis module (e.g., OpenCV), and emotion engine. This prepares the system for analyzing audio and video data in real time. Specifically, each library is loaded and the model is set up. The input to this process is the "initialization command" and the output is the "initialization completion status."

[0536] Step 2:

[0537] Receiving incoming calls

[0538] When a user calls 119, the terminal receives the call and acquires the audio data and remote video data. This real-time data is sent from the terminal to the server. The input is the "user's telephone audio signal" and the "remote video signal," and the output is the "audio and video data sent to the server."

[0539] Step 3:

[0540] Analysis of audio data

[0541] The server analyzes the received voice data using an AI model. This analysis includes a process to evaluate the content and urgency of the voice. Specifically, the voice data is converted into text using the Google Speech-to-Text API or similar, and keywords and tone of voice are analyzed. The input is the "voice data sent to the server," and the output is the "analyzed voice text" and the "urgency assessment result."

[0542] Step 4:

[0543] Video data analysis

[0544] The server analyzes the remote video data using OpenCV. This analysis includes detecting abnormal situations in the video (for example, fire, smoke, blood, etc.). Specifically, it performs pattern recognition on each video frame to detect abnormal patterns. The input is the "video data sent to the server," and the output is "information about abnormal situations in the analyzed video."

[0545] Step 5:

[0546] Recognition of emotional states

[0547] The server uses an emotion engine to recognize the user's emotional state from the analysis of audio and video data. Specifically, emotions are determined based on the tone of voice, facial expressions read from the video, and movements. The inputs are the "analyzed audio text" and "analyzed abnormal situation information in the video," and the output is "recognized emotional state information."

[0548] Step 6:

[0549] Determining urgency

[0550] The server calculates an urgency score by integrating the analysis results of the audio data, video data, and emotion engine. Based on this score, if the urgency is judged to be high, it is judged as "high urgency," and if it is low, it is judged as "low urgency." The inputs are "analyzed audio text," "analyzed abnormal situation information in the video," and "recognized emotional state information," and the outputs are "urgency score" and "urgency judgment result."

[0551] Step 7:

[0552] Issuance of response instructions

[0553] Based on the urgency assessment result, the server generates appropriate response instructions and notifies the operator. In the case of a high urgency, an order to dispatch a rescue team or ambulance is issued immediately. The operator is also provided with information about the user's emotional state. The inputs are the "urgency assessment result" and "recognized emotional state information," and the outputs are "response instructions" and "operator notification."

[0554] (Application example 2)

[0555] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0556] In conventional emergency notification systems, callers rely solely on voice information to determine the level of urgency, which can result in a lack of immediate and appropriate response, even in highly urgent situations. It is particularly difficult to grasp the caller's situation and emotional state in detail using voice data alone. This can result in misjudgments and delayed responses in emergencies, posing a major challenge for security services, which require rapid and accurate responses.

[0557] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0558] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for determining the level of urgency from the voice data and video data, means for issuing appropriate response instructions depending on the level of urgency, and means for processing the voice data and video data collected using a smart device in real time. This allows for simultaneous analysis of the voice data and video data, enabling more accurate determination of the level of urgency and enabling faster and more appropriate responses.

[0559] An "AI model" is an artificial intelligence system that includes algorithms and statistical models that analyze data and make specific decisions.

[0560] "Audio data" is a digital representation of an audio signal captured using a microphone or the like.

[0561] "Remote video data" is a digital representation of a video signal captured from a remote location using a visual sensor such as a camera.

[0562] "Means of analysis" refers to the technical methods used to input data and extract and interpret the information contained in that data.

[0563] The "means for determining urgency" is a technical method for evaluating the urgency of a situation based on acquired data and outputting the results.

[0564] "Means for issuing appropriate response instructions" refers to a technical method for instructing a person or system to take specific actions based on the results of an emergency assessment.

[0565] A "smart device" is a portable or stationary electronic device that has Internet connectivity and is capable of sensing and data processing.

[0566] "Real-time processing means" refers to a technical method that analyzes data as it is generated and obtains results without delay.

[0567] This invention relates to a system that uses an AI model and an emotion engine to analyze audio data and remote video data collected from smart devices and accurately determine the urgency of a situation. This system is realized using the following hardware and software.

[0568] 1. Hardware and Software Used

[0569] Smart Devices: Electronic devices with internet connectivity, such as smartphones and security cameras.

[0570] Audio data collection microphone: A device for converting audio signals into digital data.

[0571] Remote video data collection camera: Equipment for converting video signals into digital data.

[0572] Server: A computer system with the computational resources to analyze audio and video data in real time.

[0573] software:

[0574] sounddevice library: A library for capturing and processing audio data.

[0575] cv2 (OpenCV): A library for acquiring and analyzing video data.

[0576] AI models: Algorithms and models for assessing urgency and issuing response instructions.

[0577] EmotionEngine: An engine for recognizing emotional states from audio and video data.

[0578] 2. Processing Flow

[0579] Once the server is initialized, it activates the AI ​​model, emotion engine, and video analysis module. Audio data and remote video data are sent to the server via smart devices, which process the data in real time. The AI ​​model analyzes the audio data to evaluate the caller's tone, content, and urgency. At the same time, the video analysis module detects abnormal situations in the video. The emotion engine analyzes the audio and video data to recognize the caller's emotional state.

[0580] 3. Urgency assessment and response instructions

[0581] The server calculates an urgency score by integrating the results of the audio and video analysis. If the urgency is determined to be high based on this score, the server immediately issues appropriate response instructions. For example, a notification may be sent to security guards or rescue teams. If the urgency is low, further confirmation is required and an operator is notified.

[0582] 4. Specific Examples

[0583] When a user reports "someone has broken into my house" in the middle of the night, the system collects audio data and remote video data. The server analyzes the tone of the voice and any abnormal activity in the video (e.g., broken windows or suspicious activity) in real time to determine the level of urgency. If the level of urgency is determined to be high, the server immediately issues a response command to call security guards or a rescue team.

[0584] 5. Examples of prompts

[0585] "Create an AI system that can recognize specific emotional states from audio and video data in real time and determine the level of urgency. The specific input data is as follows:

[0586] Audio data: A user shouts "Help!" in a tense voice at the scene of an accident

[0587] Video data: Includes footage of the accident scene, flames, smoke, and injured people.

[0588] As described above, the present invention makes it possible to judge the urgency of an emergency with significantly higher accuracy and to respond more quickly than conventional systems.

[0589] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0590] Step 1:

[0591] Initializing the Server

[0592] The server initializes the AI ​​model, emotion engine, and video analysis module. This prepares the server to process audio and video data in real time. Specifically, it loads various models and engines and imports related libraries. The input is the initialization command, and the output is the ready status.

[0593] Step 2:

[0594] Acquisition of audio and remote video data

[0595] When a user makes an emergency call, the microphone and camera on the terminal (smart device) collect audio and video data and send it to the server. The input is the user's audio and video, and the output is a digital signal of these data.

[0596] Step 3:

[0597] Analysis of audio data

[0598] The server receives the voice data and analyzes it using an AI model. The analysis evaluates the tone of voice, the content of the conversation, and the level of urgency. Specific operations include noise removal from the voice data, feature extraction, and a model-based urgency evaluation. The input is the voice data, and the output is the analysis results.

[0599] Step 4:

[0600] Video data analysis

[0601] The server receives the remote video data and analyzes it using a video analysis module. This analysis detects abnormal activity or situations in the video (e.g., fire, smoke, suspicious individuals, etc.). Specific operations include frame segmentation of the video data, feature extraction, and application of anomaly detection algorithms. The input is the video data, and the output is the analysis results.

[0602] Step 5:

[0603] Recognition of emotional states

[0604] The server uses an emotion engine to recognize the caller's emotional state from the analysis results of the audio and video data. Specifically, it integrates the audio and video analysis results and executes an algorithm to evaluate the emotional state. The input is the audio and video analysis results, and the output is the recognition result of the emotional state.

[0605] Step 6:

[0606] Determining the level of urgency

[0607] The server integrates the analysis results of the audio data, video data, and emotion engine to calculate an urgency score. The inputs are the audio analysis results, video analysis results, and emotion recognition results, and the output is an urgency score. Specifically, the server integrates the analysis results using a weighted average or other integration algorithm to calculate the urgency score.

[0608] Step 7:

[0609] Issuance of response instructions

[0610] Based on the urgency score, the server immediately issues an appropriate response command if the urgency is high, or notifies an operator if the urgency is low. Specifically, the server selects a command based on the urgency and sends the command to the appropriate response channel. For example, if the urgency is high, it notifies security guards or rescue teams, and if the urgency is low, it requests further confirmation from an operator. The input is the urgency score, and the output is a response command.

[0611] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0612] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0613] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0614] [Third embodiment]

[0615] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0616] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0617] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0618] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0619] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0620] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0621] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0622] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0623] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0624] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0625] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0626] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0627] This invention is a system for determining whether a call made to 119 is urgent or not. A specific embodiment of this system is described below. The system of this invention analyzes audio data and remote video data and utilizes an AI model to determine the urgency of the call.

[0628] 1. Initialize the server

[0629] The server initializes the AI ​​model and video analysis module in advance to analyze the audio data and remote video data, so that the server is ready to process the data in real time.

[0630] 2. Receiving incoming calls

[0631] The server receives calls to 119 and acquires audio data and remote video data. The audio data is recorded in real time, and the video data is acquired from a remote device such as a camera. The audio data also contains information necessary for making emergency decisions, such as the content of the call and the caller's tone of voice.

[0632] 3. Analysis of audio and video data

[0633] The server uses an AI model to analyze the audio data, evaluating the caller's tone of voice, what was said, and the urgency score, while simultaneously analyzing the remote video data using a video analytics module to detect activity and unusual conditions in the video (e.g., fire, smoke, etc.).

[0634] As a concrete example, consider the case where a user witnesses a fire and calls 119. When the user shouts, "Fire! Help!", this voice data is sent to the server. At the same time, video of the scene is sent from the user's camera. The server analyzes the voice data to recognize the word "fire" and evaluate the level of tension in the caller's voice. It also uses video analysis to detect smoke in the video.

[0635] 4. Determining Urgency

[0636] The server combines the results of the analysis of the audio and video data to calculate an urgency score. If the urgency score is high, the server determines it as "high urgency," and if it is low, it determines it as "low urgency." This process allows the system to accurately determine the urgency based on audio and video.

[0637] 5. Issuance of response instructions

[0638] If the emergency is deemed high, the server immediately notifies the operator and issues a command to dispatch the appropriate rescue team or ambulance, enabling a prompt and appropriate response. If the emergency is not high, the server provides the operator with detailed information to help them make further confirmations and decisions.

[0639] In this way, the system of the present invention can accurately determine the urgency of calls made to 119, improving the efficiency of call centers. It also reduces prank calls and non-urgent calls, allowing for a quick response to truly urgent situations.

[0640] The processing flow will be explained below.

[0641] Step 1:

[0642] The server initializes the AI ​​model and video analysis module, which prepares the server for analyzing audio data and remote video data in real time. Specifically, the server loads the AI ​​model and initializes the video analysis module.

[0643] Step 2:

[0644] A user calls 119. The user reports an emergency to the operator. Once the call is connected, the server starts recording audio data and capturing remote video.

[0645] Step 3:

[0646] The server analyzes the voice data in real time, using AI models to assess the caller's tone of voice, content, and urgency score. This analysis uses voice recognition technology to specifically detect emergency words like "fire" and "accident."

[0647] Step 4:

[0648] The server receives the remote video data and analyzes it using a video analysis module. Video analysis detects movement and abnormal situations (e.g., fire, smoke, blood, etc.) in the video. The analysis results are converted into digital data and used to assess the emergency level.

[0649] Step 5:

[0650] The server combines the results of the audio and video analysis to calculate an urgency score. The overall urgency is determined based on the output data of the AI ​​model. In this process, the audio analysis urgency score and the video analysis urgency score are averaged.

[0651] Step 6:

[0652] The server creates a response plan based on the calculated urgency score. If the urgency score is high, it immediately notifies the operator and issues a command to dispatch the appropriate rescue team or ambulance. If the urgency score is low, it instructs the operator to check further.

[0653] Step 7:

[0654] The operator receives notifications from the server and responds according to specific instructions. In the case of a high-level emergency, they quickly dispatch a rescue team and take appropriate action. In the case of a low-level emergency, they obtain additional information from the user and take appropriate action.

[0655] As a concrete example, consider the case where a user witnesses a fire and calls 119. When the user shouts, "Fire! Help!", the server analyzes the voice and detects emergency words. At the same time, it analyzes the remote video to detect smoke. This allows the server to determine that the situation is a high urgency, immediately notify the operator, and issue a command to dispatch a fire engine.

[0656] Example 1

[0657] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0658] Conventional emergency call systems depended on the judgment of the operator, which meant it took time to determine the level of urgency and there was a risk of incorrect decisions being made. Furthermore, responding to prank calls and calls with low urgency could delay responses to actual emergency situations. Therefore, there is a need for a system that can analyze call audio and remote video data with high accuracy to quickly determine the level of urgency.

[0659] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0660] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for integrating the results of the analysis of the voice data and the video data to calculate an urgency score, and means for notifying an operator based on the urgency and issuing a command to dispatch an appropriate rescue team or ambulance. This enables the urgency to be determined with high accuracy using both voice and video, enabling a prompt and appropriate response.

[0661] An "AI model" is a mathematical model that uses artificial intelligence technology to analyze data and extract patterns and features for specific purposes.

[0662] "Voice Data" means sound waves converted into digital form that can be stored or analyzed and that contains the voice or sound content of a caller.

[0663] "Remote video data" means visual information captured from a remote location through a camera or other video capture device that can be stored in digital format or analyzed.

[0664] "Analysis means" refers to the technical means used to analyze data and extract necessary information using AI models, video analysis modules, etc.

[0665] The "urgency score" is an evaluation value that quantifies or quantifies the level of urgency based on the analysis results of the audio data and video data.

[0666] "Operator" means a person whose job is to receive and respond to emergency calls.

[0667] A "rescue team" is a public or private rescue team dispatched to respond to an emergency such as a fire or accident.

[0668] A "dispatch order" is an order issued to respond quickly in a highly urgent situation, and instructs the dispatch of rescue teams, ambulances, etc.

[0669] "Notification" means promptly conveying the results of a judgment on the level of urgency to an operator, and is the act of transmitting information to encourage appropriate response.

[0670] "Synthesis" is the process of combining multiple data and analysis results to make overall judgments and evaluations.

[0671] The invention is a system that determines the urgency of calls made to 911. The system utilizes an AI model to analyze audio data and remote video data.

[0672] The server first initializes the AI ​​model. The server uses an AI framework such as TensorFlow or PyTorch. It also uses a video analysis library such as OpenCV for video analysis. Specifically, the server calls the "initialize_AI_model()" function to load the AI ​​model and initializes the video analysis module using the "initialize_video_analysis_module()" function.

[0673] Next, the server receives a call to 119. At this time, the server records the call's audio data in real time and obtains remote video data from the user's smartphone or camera. The server receives the call using the "receive_call()" function, records the audio data using the "record_audio()" function, and obtains the remote video data using the "get_video_feed()" function.

[0674] The server then analyzes the audio and video data. The server uses the "analyze_audio_data(audio_data)" function to analyze the audio data and evaluate the tone of the caller's voice and the content of what is being said. At the same time, the server uses the "analyze_video_data(video_data)" function to analyze the video data and detect movement or abnormal conditions in the video (e.g., fire, smoke, etc.).

[0675] Based on the analysis results, the server integrates the audio and video data and calculates an urgency score. This process is performed using the "integrate_analysis_results(audio_analysis, video_analysis)" function. This function integrates the audio and video analysis results. The "calculate_emergency_score(integrated_data)" function is also used to calculate the urgency score.

[0676] The server takes appropriate action based on the urgency score. If the urgency is deemed high, the server notifies the operator using the "notify_operator(emergency_level)" function and dispatches a rescue team or ambulance using the "dispatch_rescue_team()" function. On the other hand, if the urgency is low, the server provides detailed information to the operator using the "provide_additional_info()" function.

[0677] As a concrete example, consider the case where a user calls 119 at the scene of a fire. When the user shouts, "Fire! Help!", this audio data is sent to the server. At the same time, video of the scene is sent from the user's camera. The server analyzes the audio data using the "analyze_audio_data(audio_data)" function and recognizes the word "fire." It also uses video analysis to detect smoke and flames in the video. As the integrated analysis results in a high urgency score, the server notifies the operator using the "notify_operator(emergency_level)" function and dispatches an appropriate rescue team using the "dispatch_rescue_team()" function.

[0678] By inputting the following prompt sentence into the generative AI model, the user's voice data can be analyzed.

[0679] Example prompt sentence:

[0680] "Please analyze the urgent voice data. We have prepared the following sample voice data."

[0681] In this way, the system of the present invention can analyze audio and video data in real time and accurately determine emergency situations. This enables a prompt and appropriate response, improving the efficiency of call centers in emergency situations. It also minimizes the impact of prank calls and non-urgent calls, allowing for a prompt response to truly urgent situations.

[0682] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0683] Step 1: Initialize the server

[0684] The server initializes the AI ​​model and video analysis module.

[0685] Input: A library of pre-trained AI models and video analytics modules.

[0686] Processing: The server calls the "initialize_AI_model()" function to load the AI ​​model into memory, and initializes the video analysis module using the "initialize_video_analysis_module()" function.

[0687] Output: The AI ​​model and video analysis module have been initialized.

[0688] Step 2: Receiving the incoming call

[0689] The server receives calls to 119 and acquires audio data and remote video data.

[0690] Input: Phone call initiation signal, remote connection from user's smartphone or camera.

[0691] Processing: The server receives and initiates a call using the "receive_call()" function, records audio data in real time using the "record_audio()" function, and retrieves remote video data using the "get_video_feed()" function.

[0692] Output: Recorded audio data and captured remote video data.

[0693] Step 3: Analyzing the audio data

[0694] The server analyzes the acquired voice data using an AI model.

[0695] Input: Recorded audio data.

[0696] Processing: The server analyzes the audio data using the "analyze_audio_data(audio_data)" function, which evaluates the caller's tone of voice, the content of what is being said, and the urgency of the conversation.

[0697] Output: Analysis of the audio data (urgency score, identified keywords, tone rating).

[0698] Step 4: Analyzing the video data

[0699] The server analyzes the acquired video data using a video analysis module.

[0700] Input: Acquired remote video data.

[0701] Processing: The server analyzes the video data using the "analyze_video_data(video_data)" function, which detects abnormal conditions (e.g., fire or smoke) and movement in the video data.

[0702] Output: Analysis results of video data (detected abnormal situations and movements, urgency score).

[0703] Step 5: Integrating the analysis results

[0704] The server integrates the analysis results of the audio data and the video data.

[0705] Input: Audio data analysis results and video data analysis results.

[0706] Processing: The server integrates the audio and video analysis results using the "integrate_analysis_results(audio_analysis, video_analysis)" function.

[0707] Output: Consolidated analysis results (total urgency score).

[0708] Step 6: Determine the urgency

[0709] The server calculates an urgency score using the integrated analysis results.

[0710] Input: The integrated analysis results.

[0711] Processing: The server calculates the urgency score from the integrated data using the function "calculate_emergency_score(integrated_data)".

[0712] Output: Urgency score (high or low urgency).

[0713] Step 7: Sending response orders

[0714] The server issues appropriate response instructions based on the urgency score.

[0715] Input: Urgency score.

[0716] Processing: If the urgency score is high, the server notifies the operator using the "notify_operator(emergency_level)" function and dispatches a rescue team or ambulance using the "dispatch_rescue_team()" function. If the urgency score is low, the server provides more information to the operator using the "provide_additional_info()" function.

[0717] Output: Notify operator, dispatch rescue teams or ambulances, provide detailed information in low-risk situations.

[0718] In this way, the server can analyze the audio and video data in real time, determine the emergency situation with high accuracy, and take prompt action.

[0719] (Application example 1)

[0720] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0721] Conventional emergency notification systems analyze only audio data to determine the urgency of an emergency, which often limits information and makes it difficult to make accurate judgments. Furthermore, real-time processing of video data and linkage with audio are insufficient, making it difficult to respond appropriately to high-urgency situations. This reduces efficiency in situations where a rapid emergency response is required, and further improvements in accuracy and faster response are needed.

[0722] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0723] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for determining the level of urgency from the voice data and video data, means for issuing appropriate response commands depending on the level of urgency, and means for analyzing camera video data in real time in conjunction with the voice data in an emergency. This makes it possible to integrate information from both the voice and video data and determine the level of urgency quickly and with high accuracy.

[0724] An "AI model" is an artificial intelligence algorithm that analyzes audio and video data and performs pattern recognition and feature extraction.

[0725] "Voice data" refers to a voice signal that includes information necessary for determining the urgency of a call, such as the content of the call and the tone of the user's voice.

[0726] "Remote video data" is video information sent by a user from a smartphone or other camera device.

[0727] "Voice analysis means" refers to devices or software that use AI models to analyze the content, tone, and urgency of voice data.

[0728] "Video analytics" means devices or software that use AI models or other technologies to detect abnormal conditions or movements in remote video data.

[0729] The "means for determining the urgency" refers to a device or software that evaluates the urgency of a situation based on the results of analyzing audio data and video data.

[0730] The "means for issuing response commands" refers to a device or software that automatically issues commands to execute appropriate responses depending on the level of urgency.

[0731] "Means for analyzing camera video data in real time in conjunction with audio data in an emergency" refers to devices or software that instantly analyzes video data acquired in conjunction with a user's emergency call.

[0732] The present invention is a system for determining the urgency of an emergency using voice data and remote video data, and issuing prompt response instructions. A specific embodiment of this system will be described below.

[0733] System Overview

[0734] The system includes a voice data analyzer, a video data analyzer, an emergency assessment device, and a response command issuing device. All of these devices run on a server and are deployed in a cloud environment. The system acquires data via user devices such as smartphones and performs analysis in real time.

[0735] Voice data analysis

[0736] The server analyzes the voice data using an AI model for voice recognition. This AI model identifies the tone, content, and urgency of the user's voice. Once the voice data is acquired, the server converts and analyzes the content using a voice data analyzer.

[0737] Video Data Analysis

[0738] The server also analyzes remote video data acquired from users' smartphones and other camera devices, using AI models and video analysis modules to detect abnormal situations and movements within the video, enabling rapid recognition of emergencies such as fires and intruders.

[0739] Judgment of urgency

[0740] The server integrates the results of the analysis of the audio and video data and calculates an urgency score using an emergency assessment device. Based on this score, it determines with high accuracy whether an emergency has occurred. If the urgency is determined to be high, it immediately issues appropriate response instructions.

[0741] Output of response instructions

[0742] If the emergency is deemed high, the server will send a notification to the appropriate authorities, such as security companies or the police, via a response command system. This notification will include the user's voice data, video data, and location information, enabling a prompt and appropriate response.

[0743] Hardware and Software Use

[0744] The server uses the following hardware and software:

[0745] Voice recognition microphone (built-in microphone on smartphone)

[0746] Camera for capturing images (built-in camera on smartphone)

[0747] Python, speech_recognition library, transformers library

[0748] The voice data is converted into text by a speech recognition library and an AI model assesses the urgency of the call, while the video data is analyzed in real time via a video analytics module.

[0749] Specific examples

[0750] When an intruder suddenly breaks into the house, the user shouts "Help me!" into their smartphone. This voice data is sent to the server and analyzed by the AI ​​model. At the same time, the smartphone camera sends video data of the scene to the server, which is then analyzed by the video analysis module.

[0751] Prompt Sentence Examples

[0752] "Analyze the input voice data, and if it contains the keyword 'help,' assess the urgency of the voice. Guide the user on what action to take in an emergency."

[0753] As a result, the present invention can integrate audio and video data to determine the urgency with high accuracy, thereby enabling a prompt and appropriate response.

[0754] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0755] Step 1:

[0756] The server performs initialization processing. Specifically, it loads and initializes the AI ​​model and video analysis module required to analyze audio and video data. This initialization prepares the server to execute real-time data processing. The AI ​​model and video analysis module are required as input, and the initialization-completed state is obtained as output.

[0757] Step 2:

[0758] When a user makes an emergency call, the smartphone sends voice data to a server. The smartphone acquires the voice data using a microphone for voice input. The input is the user's voice, and the acquired voice data is obtained as the output.

[0759] Step 3:

[0760] The server analyzes the received voice data in real time. Specifically, it uses a voice recognition library to convert the voice data into text, which is then analyzed using an AI model. The input is the voice data, and the output is the text data of the analysis results and an assessment of the urgency level.

[0761] Step 4:

[0762] It acquires remote video data from the user's smartphone. It uses a camera to record on-site video and transmits it to the server in real time. The input requires the camera on the user's device, and the output is the recorded video data.

[0763] Step 5:

[0764] The server analyzes the acquired video data. It uses a video analysis module to detect abnormal situations or movements (for example, fires or intruders). The input is video data, and the output is the analysis result, which indicates whether there is an abnormality or not and the urgency level.

[0765] Step 6:

[0766] The server combines the results of the analysis of the audio and video data to determine the urgency. It calculates an urgency score, and if it is high it is rated as high urgency, and if it is low it is rated as low urgency. The analysis results of the audio and video data are required as input, and the urgency score is obtained as output.

[0767] Step 7:

[0768] If the urgency is determined to be high, the server immediately issues appropriate response instructions. Specifically, it sends a notification to response agencies such as security companies and the police. The notification includes the user's audio data, video data, and location information. The input requires an urgency score, and the output is a notification of the response instructions.

[0769] This series of processing steps enables the system to integrate audio and video data, quickly and accurately determine the level of urgency, and prescribe appropriate responses.

[0770] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0771] This invention is a system that uses an AI model and an emotion engine to accurately determine the urgency of calls made to 119. The system includes means for analyzing audio data and remote video data and recognizing the user's emotional state from the audio and video.

[0772] 1. Initialize the server

[0773] The server initializes the AI ​​model, video analysis module, and emotion engine, which prepares the server to process audio and video data in real time. The emotion engine loads the libraries and models required to extract user emotions from audio and video.

[0774] 2. Receiving incoming calls

[0775] When a user calls 119, the server receives the call and captures the audio data and remote video data. The audio data is recorded in real time, and the remote video data is captured as a video stream from the camera.

[0776] 3. Analysis of audio and video data

[0777] The server uses AI models to analyze the audio data, assessing the caller's tone of voice, content, and urgency. At the same time, it uses a video analytics module to analyze the remote video data and detect activity or unusual conditions in the video (e.g., fire, smoke, blood, etc.).

[0778] Furthermore, the emotion engine recognizes the caller's emotional state based on the analysis of audio and video data. For example, the emotion engine analyzes the caller's level of tension from audio and identifies their emotional state by analyzing the caller's facial expressions and movements from video.

[0779] As a concrete example, consider the case where a user witnesses a fire. When the user shouts, "Fire! Help!", this voice data is sent to the server. At the same time, video footage of the fire scene captured by a remote camera is also sent to the server. The server recognizes the word "fire" through voice data analysis and assesses the user's level of urgency. It also detects smoke and flames at the scene through video analysis. Furthermore, the emotion engine recognizes that the caller is in a high-level emergency based on their tone of voice and facial expression.

[0780] 4. Determining Urgency

[0781] The server integrates the analysis results of the audio data, video data, and emotion engine to calculate an urgency score. In this process, the results of audio analysis, video analysis, and emotion recognition are combined to make a comprehensive urgency judgment. If the urgency score is high, the server judges it to be "high urgency," and if it is low, it judges it to be "low urgency."

[0782] 5. Issuance of response instructions

[0783] If the emergency is deemed high, the server immediately notifies the operator and issues instructions to dispatch the appropriate rescue team or ambulance. The server also provides the operator with information about the user's emotional state to help them respond appropriately. If the emergency is low, the server provides additional information and prompts the operator for further confirmation.

[0784] This system allows for highly accurate assessment of the urgency of 119 calls, enabling prompt and appropriate responses. Furthermore, by incorporating an emotion engine, it is possible to achieve even more accurate urgency assessments that take into account the caller's emotional state. In this way, the present invention significantly improves the efficiency of 119 call centers and reduces prank calls and non-urgent calls.

[0785] The processing flow will be explained below.

[0786] Step 1:

[0787] The server initializes the AI ​​model, video analysis module, and emotion engine. This enables real-time analysis of audio, video, and emotion data. Specifically, it loads the AI ​​model into memory, configures the video analysis module, and prepares the emotion engine library and model.

[0788] Step 2:

[0789] A user calls 119. To report an emergency, the user describes the situation over the phone. The user's words and tone of voice are sent to the server. Also, video data from the user's remote camera is sent to the server as needed.

[0790] Step 3:

[0791] The server starts recording audio data and simultaneously starts acquiring video streams from remote cameras, which are then processed in real time.

[0792] Step 4:

[0793] The server uses an AI model to analyze the recorded voice data, evaluating the caller's tone of voice, the content of what was said, and the urgency score. The AI ​​model detects specific keywords and quantifies the urgency level.

[0794] Step 5:

[0795] The server uses a video analytics module to analyze the remote video data, detecting movement and abnormal conditions (e.g., fire, smoke, blood, etc.) in the video and extracting this information as digital data.

[0796] Step 6:

[0797] The server uses an emotion engine to recognize the user's emotional state from audio and video data. From the audio data, the server analyzes the caller's tone and tempo to identify their emotional state. From the video data, the server analyzes the caller's facial expressions and gestures to evaluate emotions such as tension, fear, and excitement.

[0798] Step 7:

[0799] The server integrates the results of the analysis of the audio data, video data, and emotion data, and calculates an urgency score. The results of the audio analysis, video analysis, and emotion engine are combined to make a comprehensive urgency judgment.

[0800] Step 8:

[0801] The server issues appropriate response instructions based on the calculated urgency score. If the urgency is high, the server immediately notifies the operator and issues instructions to dispatch a rescue team or ambulance. The server also provides the operator with information about the user's emotional state, improving the quality of the response.

[0802] Step 9:

[0803] The operator receives a notification from the server and immediately takes appropriate action. If the emergency is high, they will instruct a quick response, and if the emergency is low, they will obtain additional information and take action.

[0804] As a concrete example, consider the case where a user witnesses a fire and calls 119. When the user yells, "Fire! Help!", the server uses voice analysis to recognize the word "fire" and assesses the level of urgency. At the same time, video analysis detects smoke, and the emotion engine determines that the user is extremely nervous. Based on this comprehensive assessment, the server determines that the level of urgency is high, and is able to immediately notify an operator and issue a command to dispatch a fire engine.

[0805] Example 2

[0806] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0807] Conventional 119 call centers determine the level of urgency based solely on voice information, making it difficult to effectively distinguish between high-urgency and low-urgency cases. Furthermore, they are unable to properly recognize the caller's emotional state, making it difficult to respond immediately and appropriately. This increases the risk of incorrect responses, posing challenges to the speed and accuracy of rescue responses.

[0808] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0809] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for recognizing the user's emotional state from the voice data and video data, means for determining the urgency from the voice data and video data, and means for issuing appropriate response instructions depending on the urgency. This enables quick and accurate identification of high-urgency cases and appropriate rescue responses that take into account the caller's emotional state.

[0810] An "AI model" is an artificial intelligence algorithm used to analyze a variety of data, including audio and video data.

[0811] "Voice data" refers to information in the form of voice signals transmitted by a user over a telephone.

[0812] "Remote video data" refers to information on video signals obtained by capturing the user's environment through a camera and transmitting it to a server.

[0813] "Emotional state" refers to the user's psychological and emotional state determined from the results of audio and video analysis.

[0814] "Voice analysis means" means means for using an AI model to evaluate the content, tone, and urgency of voice data.

[0815] "Video analysis means" refers to means for detecting movement or abnormal situations within remote video data.

[0816] The "urgency determination means" is a means for calculating an urgency score based on the analysis results of audio data and video data and the emotional state, and for assessing the urgency of the incident.

[0817] The "response command means" is a means for generating and outputting an appropriate command to dispatch a rescue team or an ambulance based on the result of the emergency assessment.

[0818] This invention is a system for accurately determining the urgency of an incoming call to 911 based on audio data and remote video data. The system includes a means for recognizing the emotional state of a user using an AI model, a video analysis module, and an emotion engine.

[0819] When the system first starts up, the server initializes the AI ​​model (e.g., TensorFlow), video analysis module (e.g., OpenCV), and emotion engine. This prepares the system for processing audio and video data in real time. The emotion engine is equipped with the necessary libraries and models for extracting user emotions from audio and video.

[0820] When a user calls 119, the device receives the call, acquires the audio data and remote video data, and sends them to the server in real time. For example, when a user yells "Help, there's a fire!", the audio is sent from the device to the server, and at the same time, the video of the scene captured by the remote camera is also sent to the server.

[0821] The server uses an AI model (e.g., Google Speech-to-Text API or a proprietary voice analysis module) to analyze the received voice data. Here, the server analyzes the user's tone of voice and the content of what is being said to assess the urgency. For example, the keyword "fire" and the level of urgency in the voice are detected, and the call is judged to be "very urgent."

[0822] The server then uses OpenCV to analyze the video data. This video analysis module detects abnormal situations (e.g., fire, smoke, blood, etc.) and significant movement in the video sent from the remote camera. For example, it detects flame and smoke patterns in the video and concludes that a fire has been detected.

[0823] Furthermore, the emotion engine recognizes the user's emotional state based on the analysis of audio and video data. Specific emotions are extracted by analyzing tension and fear from audio and facial expressions and movements from video. For example, the server may analyze the user's trembling voice and teary eyes to determine that the user is "extremely tense."

[0824] Finally, the server calculates an urgency score by combining the results of the audio data, video data, and emotion engine analysis. If this score is above a certain level, it is judged to be a "high urgency," and if it is lower, it is judged to be a "low urgency." For example, if the audio analysis results in "fire," the video analysis shows "flames and smoke," and the emotion analysis shows "high tension," the overall urgency score will be high and the event will be judged to be a "high urgency."

[0825] If the emergency is deemed high, the server immediately notifies the operator and issues a command to dispatch the appropriate rescue team or ambulance. The operator is also provided with information about the user's emotional state, which encourages appropriate response. For example, the server may send a notification to the operator saying, "High emergency, fire. Dispatching rescue team to the scene. The caller is in a very tense situation."

[0826] As a concrete example of how this works, the prompt text is as follows:

[0827] "Audio data: 'Help, there's a fire!' Audio analysis: 'Fire' Video analysis: 'Flame and smoke detected' Emotional state: 'Very tense' Urgency score: 'High'"

[0828] This system will enable a highly accurate assessment of the urgency of calls to 119, enabling a swift and appropriate response. This will significantly improve the efficiency of 119 call centers, reducing prank calls and non-urgent calls. In addition, by combining it with an emotion engine, it will be possible to achieve a more accurate assessment of urgency by taking into account the caller's emotional state.

[0829] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0830] Step 1:

[0831] Initializing the Server

[0832] When the system starts up, the server loads the necessary libraries and models and initializes the AI ​​model (e.g., TensorFlow), video analysis module (e.g., OpenCV), and emotion engine. This prepares the system for analyzing audio and video data in real time. Specifically, each library is loaded and the model is set up. The input to this process is the "initialization command" and the output is the "initialization completion status."

[0833] Step 2:

[0834] Receiving incoming calls

[0835] When a user calls 119, the terminal receives the call and acquires the audio data and remote video data. This real-time data is sent from the terminal to the server. The input is the "user's telephone audio signal" and the "remote video signal," and the output is the "audio and video data sent to the server."

[0836] Step 3:

[0837] Analysis of audio data

[0838] The server analyzes the received voice data using an AI model. This analysis includes a process to evaluate the content and urgency of the voice. Specifically, the voice data is converted into text using the Google Speech-to-Text API or similar, and keywords and tone of voice are analyzed. The input is the "voice data sent to the server," and the output is the "analyzed voice text" and the "urgency assessment result."

[0839] Step 4:

[0840] Video data analysis

[0841] The server analyzes the remote video data using OpenCV. This analysis includes detecting abnormal situations in the video (for example, fire, smoke, blood, etc.). Specifically, it performs pattern recognition on each video frame to detect abnormal patterns. The input is the "video data sent to the server," and the output is "information about abnormal situations in the analyzed video."

[0842] Step 5:

[0843] Recognition of emotional states

[0844] The server uses an emotion engine to recognize the user's emotional state from the analysis of audio and video data. Specifically, emotions are determined based on the tone of voice, facial expressions read from the video, and movements. The inputs are the "analyzed audio text" and "analyzed abnormal situation information in the video," and the output is "recognized emotional state information."

[0845] Step 6:

[0846] Determining urgency

[0847] The server calculates an urgency score by integrating the analysis results of the audio data, video data, and emotion engine. Based on this score, if the urgency is judged to be high, it is judged as "high urgency," and if it is low, it is judged as "low urgency." The inputs are "analyzed audio text," "analyzed abnormal situation information in the video," and "recognized emotional state information," and the outputs are "urgency score" and "urgency judgment result."

[0848] Step 7:

[0849] Issuance of response instructions

[0850] Based on the urgency assessment result, the server generates appropriate response instructions and notifies the operator. In the case of a high urgency, an order to dispatch a rescue team or ambulance is issued immediately. The operator is also provided with information about the user's emotional state. The inputs are the "urgency assessment result" and "recognized emotional state information," and the outputs are "response instructions" and "operator notification."

[0851] (Application example 2)

[0852] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0853] In conventional emergency notification systems, callers rely solely on voice information to determine the level of urgency, which can result in a lack of immediate and appropriate response, even in highly urgent situations. It is particularly difficult to grasp the caller's situation and emotional state in detail using voice data alone. This can result in misjudgments and delayed responses in emergencies, posing a major challenge for security services, which require rapid and accurate responses.

[0854] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0855] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for determining the level of urgency from the voice data and video data, means for issuing appropriate response instructions depending on the level of urgency, and means for processing the voice data and video data collected using a smart device in real time. This allows for simultaneous analysis of the voice data and video data, enabling more accurate determination of the level of urgency and enabling faster and more appropriate responses.

[0856] An "AI model" is an artificial intelligence system that includes algorithms and statistical models that analyze data and make specific decisions.

[0857] "Audio data" is a digital representation of an audio signal captured using a microphone or the like.

[0858] "Remote video data" is a digital representation of a video signal captured from a remote location using a visual sensor such as a camera.

[0859] "Means of analysis" refers to the technical methods used to input data and extract and interpret the information contained in that data.

[0860] The "means for determining urgency" is a technical method for evaluating the urgency of a situation based on acquired data and outputting the results.

[0861] "Means for issuing appropriate response instructions" refers to a technical method for instructing a person or system to take specific actions based on the results of an emergency assessment.

[0862] A "smart device" is a portable or stationary electronic device that has Internet connectivity and is capable of sensing and data processing.

[0863] "Real-time processing means" refers to a technical method that analyzes data as it is generated and obtains results without delay.

[0864] This invention relates to a system that uses an AI model and an emotion engine to analyze audio data and remote video data collected from smart devices and accurately determine the urgency of a situation. This system is realized using the following hardware and software.

[0865] 1. Hardware and Software Used

[0866] Smart Devices: Electronic devices with internet connectivity, such as smartphones and security cameras.

[0867] Audio data collection microphone: A device for converting audio signals into digital data.

[0868] Remote video data collection camera: Equipment for converting video signals into digital data.

[0869] Server: A computer system with the computational resources to analyze audio and video data in real time.

[0870] software:

[0871] sounddevice library: A library for capturing and processing audio data.

[0872] cv2 (OpenCV): A library for acquiring and analyzing video data.

[0873] AI models: Algorithms and models for assessing urgency and issuing response instructions.

[0874] EmotionEngine: An engine for recognizing emotional states from audio and video data.

[0875] 2. Processing Flow

[0876] Once the server is initialized, it activates the AI ​​model, emotion engine, and video analysis module. Audio data and remote video data are sent to the server via smart devices, which process the data in real time. The AI ​​model analyzes the audio data to evaluate the caller's tone, content, and urgency. At the same time, the video analysis module detects abnormal situations in the video. The emotion engine analyzes the audio and video data to recognize the caller's emotional state.

[0877] 3. Urgency assessment and response instructions

[0878] The server calculates an urgency score by integrating the results of the audio and video analysis. If the urgency is determined to be high based on this score, the server immediately issues appropriate response instructions. For example, a notification may be sent to security guards or rescue teams. If the urgency is low, further confirmation is required and an operator is notified.

[0879] 4. Specific Examples

[0880] When a user reports "someone has broken into my house" in the middle of the night, the system collects audio data and remote video data. The server analyzes the tone of the voice and any abnormal activity in the video (e.g., broken windows or suspicious activity) in real time to determine the level of urgency. If the level of urgency is determined to be high, the server immediately issues a response command to call security guards or a rescue team.

[0881] 5. Examples of prompts

[0882] "Create an AI system that can recognize specific emotional states from audio and video data in real time and determine the level of urgency. The specific input data is as follows:

[0883] Audio data: A user shouts "Help!" in a tense voice at the scene of an accident

[0884] Video data: Includes footage of the accident scene, flames, smoke, and injured people.

[0885] As described above, the present invention makes it possible to judge the urgency of an emergency with significantly higher accuracy and to respond more quickly than conventional systems.

[0886] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0887] Step 1:

[0888] Initializing the Server

[0889] The server initializes the AI ​​model, emotion engine, and video analysis module. This prepares the server to process audio and video data in real time. Specifically, it loads various models and engines and imports related libraries. The input is the initialization command, and the output is the ready status.

[0890] Step 2:

[0891] Acquisition of audio and remote video data

[0892] When a user makes an emergency call, the microphone and camera on the terminal (smart device) collect audio and video data and send it to the server. The input is the user's audio and video, and the output is a digital signal of these data.

[0893] Step 3:

[0894] Analysis of audio data

[0895] The server receives the voice data and analyzes it using an AI model. The analysis evaluates the tone of voice, the content of the conversation, and the level of urgency. Specific operations include noise removal from the voice data, feature extraction, and a model-based urgency evaluation. The input is the voice data, and the output is the analysis results.

[0896] Step 4:

[0897] Video data analysis

[0898] The server receives the remote video data and analyzes it using a video analysis module. This analysis detects abnormal activity or situations in the video (e.g., fire, smoke, suspicious individuals, etc.). Specific operations include frame segmentation of the video data, feature extraction, and application of anomaly detection algorithms. The input is the video data, and the output is the analysis results.

[0899] Step 5:

[0900] Recognition of emotional states

[0901] The server uses an emotion engine to recognize the caller's emotional state from the analysis results of the audio and video data. Specifically, it integrates the audio and video analysis results and executes an algorithm to evaluate the emotional state. The input is the audio and video analysis results, and the output is the recognition result of the emotional state.

[0902] Step 6:

[0903] Determining the level of urgency

[0904] The server integrates the analysis results of the audio data, video data, and emotion engine to calculate an urgency score. The inputs are the audio analysis results, video analysis results, and emotion recognition results, and the output is an urgency score. Specifically, the server integrates the analysis results using a weighted average or other integration algorithm to calculate the urgency score.

[0905] Step 7:

[0906] Issuance of response instructions

[0907] Based on the urgency score, the server immediately issues an appropriate response command if the urgency is high, or notifies an operator if the urgency is low. Specifically, the server selects a command based on the urgency and sends the command to the appropriate response channel. For example, if the urgency is high, it notifies security guards or rescue teams, and if the urgency is low, it requests further confirmation from an operator. The input is the urgency score, and the output is a response command.

[0908] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0909] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0910] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0911] [Fourth embodiment]

[0912] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0913] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0914] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0915] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0916] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0917] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0918] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0919] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0920] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0921] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0922] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0923] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0924] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0925] This invention is a system for determining whether a call made to 119 is urgent or not. A specific embodiment of this system is described below. The system of this invention analyzes audio data and remote video data and utilizes an AI model to determine the urgency of the call.

[0926] 1. Initialize the server

[0927] The server initializes the AI ​​model and video analysis module in advance to analyze the audio data and remote video data, so that the server is ready to process the data in real time.

[0928] 2. Receiving incoming calls

[0929] The server receives calls to 119 and acquires audio data and remote video data. The audio data is recorded in real time, and the video data is acquired from a remote device such as a camera. The audio data also contains information necessary for making emergency decisions, such as the content of the call and the caller's tone of voice.

[0930] 3. Analysis of audio and video data

[0931] The server uses an AI model to analyze the audio data, evaluating the caller's tone of voice, what was said, and the urgency score, while simultaneously analyzing the remote video data using a video analytics module to detect activity and unusual conditions in the video (e.g., fire, smoke, etc.).

[0932] As a concrete example, consider the case where a user witnesses a fire and calls 119. When the user shouts, "Fire! Help!", this voice data is sent to the server. At the same time, video of the scene is sent from the user's camera. The server analyzes the voice data to recognize the word "fire" and evaluate the level of tension in the caller's voice. It also uses video analysis to detect smoke in the video.

[0933] 4. Determining Urgency

[0934] The server combines the results of the analysis of the audio and video data to calculate an urgency score. If the urgency score is high, the server determines it as "high urgency," and if it is low, it determines it as "low urgency." This process allows the system to accurately determine the urgency based on audio and video.

[0935] 5. Issuance of response instructions

[0936] If the emergency is deemed high, the server immediately notifies the operator and issues a command to dispatch the appropriate rescue team or ambulance, enabling a prompt and appropriate response. If the emergency is not high, the server provides the operator with detailed information to help them make further confirmations and decisions.

[0937] In this way, the system of the present invention can accurately determine the urgency of calls made to 119, improving the efficiency of call centers. It also reduces prank calls and non-urgent calls, allowing for a quick response to truly urgent situations.

[0938] The processing flow will be explained below.

[0939] Step 1:

[0940] The server initializes the AI ​​model and video analysis module, which prepares the server for analyzing audio data and remote video data in real time. Specifically, the server loads the AI ​​model and initializes the video analysis module.

[0941] Step 2:

[0942] A user calls 119. The user reports an emergency to the operator. Once the call is connected, the server starts recording audio data and capturing remote video.

[0943] Step 3:

[0944] The server analyzes the voice data in real time, using AI models to assess the caller's tone of voice, content, and urgency score. This analysis uses voice recognition technology to specifically detect emergency words like "fire" and "accident."

[0945] Step 4:

[0946] The server receives the remote video data and analyzes it using a video analysis module. Video analysis detects movement and abnormal situations (e.g., fire, smoke, blood, etc.) in the video. The analysis results are converted into digital data and used to assess the emergency level.

[0947] Step 5:

[0948] The server combines the results of the audio and video analysis to calculate an urgency score. The overall urgency is determined based on the output data of the AI ​​model. In this process, the audio analysis urgency score and the video analysis urgency score are averaged.

[0949] Step 6:

[0950] The server creates a response plan based on the calculated urgency score. If the urgency score is high, it immediately notifies the operator and issues a command to dispatch the appropriate rescue team or ambulance. If the urgency score is low, it instructs the operator to check further.

[0951] Step 7:

[0952] The operator receives notifications from the server and responds according to specific instructions. In the case of a high-level emergency, they quickly dispatch a rescue team and take appropriate action. In the case of a low-level emergency, they obtain additional information from the user and take appropriate action.

[0953] As a concrete example, consider the case where a user witnesses a fire and calls 119. When the user shouts, "Fire! Help!", the server analyzes the voice and detects emergency words. At the same time, it analyzes the remote video to detect smoke. This allows the server to determine that the situation is a high urgency, immediately notify the operator, and issue a command to dispatch a fire engine.

[0954] Example 1

[0955] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0956] Conventional emergency call systems depended on the judgment of the operator, which meant it took time to determine the level of urgency and there was a risk of incorrect decisions being made. Furthermore, responding to prank calls and calls with low urgency could delay responses to actual emergency situations. Therefore, there is a need for a system that can analyze call audio and remote video data with high accuracy to quickly determine the level of urgency.

[0957] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0958] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for integrating the results of the analysis of the voice data and the video data to calculate an urgency score, and means for notifying an operator based on the urgency and issuing a command to dispatch an appropriate rescue team or ambulance. This enables the urgency to be determined with high accuracy using both voice and video, enabling a prompt and appropriate response.

[0959] An "AI model" is a mathematical model that uses artificial intelligence technology to analyze data and extract patterns and features for specific purposes.

[0960] "Voice Data" means sound waves converted into digital form that can be stored or analyzed and that contains the voice or sound content of a caller.

[0961] "Remote video data" means visual information captured from a remote location through a camera or other video capture device that can be stored in digital format or analyzed.

[0962] "Analysis means" refers to the technical means used to analyze data and extract necessary information using AI models, video analysis modules, etc.

[0963] The "urgency score" is an evaluation value that quantifies or quantifies the level of urgency based on the analysis results of the audio data and video data.

[0964] "Operator" means a person whose job is to receive and respond to emergency calls.

[0965] A "rescue team" is a public or private rescue team dispatched to respond to an emergency such as a fire or accident.

[0966] A "dispatch order" is an order issued to respond quickly in a highly urgent situation, and instructs the dispatch of rescue teams, ambulances, etc.

[0967] "Notification" means promptly conveying the results of a judgment on the level of urgency to an operator, and is the act of transmitting information to encourage appropriate response.

[0968] "Synthesis" is the process of combining multiple data and analysis results to make overall judgments and evaluations.

[0969] The invention is a system that determines the urgency of calls made to 911. The system utilizes an AI model to analyze audio data and remote video data.

[0970] The server first initializes the AI ​​model. The server uses an AI framework such as TensorFlow or PyTorch. It also uses a video analysis library such as OpenCV for video analysis. Specifically, the server calls the "initialize_AI_model()" function to load the AI ​​model and initializes the video analysis module using the "initialize_video_analysis_module()" function.

[0971] Next, the server receives a call to 119. At this time, the server records the call's audio data in real time and obtains remote video data from the user's smartphone or camera. The server receives the call using the "receive_call()" function, records the audio data using the "record_audio()" function, and obtains the remote video data using the "get_video_feed()" function.

[0972] The server then analyzes the audio and video data. The server uses the "analyze_audio_data(audio_data)" function to analyze the audio data and evaluate the tone of the caller's voice and the content of what is being said. At the same time, the server uses the "analyze_video_data(video_data)" function to analyze the video data and detect movement or abnormal conditions in the video (e.g., fire, smoke, etc.).

[0973] Based on the analysis results, the server integrates the audio and video data and calculates an urgency score. This process is performed using the "integrate_analysis_results(audio_analysis, video_analysis)" function. This function integrates the audio and video analysis results. The "calculate_emergency_score(integrated_data)" function is also used to calculate the urgency score.

[0974] The server takes appropriate action based on the urgency score. If the urgency is deemed high, the server notifies the operator using the "notify_operator(emergency_level)" function and dispatches a rescue team or ambulance using the "dispatch_rescue_team()" function. On the other hand, if the urgency is low, the server provides detailed information to the operator using the "provide_additional_info()" function.

[0975] As a concrete example, consider the case where a user calls 119 at the scene of a fire. When the user shouts, "Fire! Help!", this audio data is sent to the server. At the same time, video of the scene is sent from the user's camera. The server analyzes the audio data using the "analyze_audio_data(audio_data)" function and recognizes the word "fire." It also uses video analysis to detect smoke and flames in the video. As the integrated analysis results in a high urgency score, the server notifies the operator using the "notify_operator(emergency_level)" function and dispatches an appropriate rescue team using the "dispatch_rescue_team()" function.

[0976] By inputting the following prompt sentence into the generative AI model, the user's voice data can be analyzed.

[0977] Example prompt sentence:

[0978] "Please analyze the urgent voice data. We have prepared the following sample voice data."

[0979] In this way, the system of the present invention can analyze audio and video data in real time and accurately determine emergency situations. This enables a prompt and appropriate response, improving the efficiency of call centers in emergency situations. It also minimizes the impact of prank calls and non-urgent calls, allowing for a prompt response to truly urgent situations.

[0980] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0981] Step 1: Initialize the server

[0982] The server initializes the AI ​​model and video analysis module.

[0983] Input: A library of pre-trained AI models and video analytics modules.

[0984] Processing: The server calls the "initialize_AI_model()" function to load the AI ​​model into memory, and initializes the video analysis module using the "initialize_video_analysis_module()" function.

[0985] Output: The AI ​​model and video analysis module have been initialized.

[0986] Step 2: Receiving the incoming call

[0987] The server receives calls to 119 and acquires audio data and remote video data.

[0988] Input: Phone call initiation signal, remote connection from user's smartphone or camera.

[0989] Processing: The server receives and initiates a call using the "receive_call()" function, records audio data in real time using the "record_audio()" function, and retrieves remote video data using the "get_video_feed()" function.

[0990] Output: Recorded audio data and captured remote video data.

[0991] Step 3: Analyzing the audio data

[0992] The server analyzes the acquired voice data using an AI model.

[0993] Input: Recorded audio data.

[0994] Processing: The server analyzes the audio data using the "analyze_audio_data(audio_data)" function, which evaluates the caller's tone of voice, the content of what is being said, and the urgency of the conversation.

[0995] Output: Analysis of the audio data (urgency score, identified keywords, tone rating).

[0996] Step 4: Analyzing the video data

[0997] The server analyzes the acquired video data using a video analysis module.

[0998] Input: Acquired remote video data.

[0999] Processing: The server analyzes the video data using the "analyze_video_data(video_data)" function, which detects abnormal conditions (e.g., fire or smoke) and movement in the video data.

[1000] Output: Analysis results of video data (detected abnormal situations and movements, urgency score).

[1001] Step 5: Integrating the analysis results

[1002] The server integrates the analysis results of the audio data and the video data.

[1003] Input: Audio data analysis results and video data analysis results.

[1004] Processing: The server integrates the audio and video analysis results using the "integrate_analysis_results(audio_analysis, video_analysis)" function.

[1005] Output: Consolidated analysis results (total urgency score).

[1006] Step 6: Determine the urgency

[1007] The server calculates an urgency score using the integrated analysis results.

[1008] Input: The integrated analysis results.

[1009] Processing: The server calculates the urgency score from the integrated data using the function "calculate_emergency_score(integrated_data)".

[1010] Output: Urgency score (high or low urgency).

[1011] Step 7: Sending response orders

[1012] The server issues appropriate response instructions based on the urgency score.

[1013] Input: Urgency score.

[1014] Processing: If the urgency score is high, the server notifies the operator using the "notify_operator(emergency_level)" function and dispatches a rescue team or ambulance using the "dispatch_rescue_team()" function. If the urgency score is low, the server provides more information to the operator using the "provide_additional_info()" function.

[1015] Output: Notify operator, dispatch rescue teams or ambulances, provide detailed information in low-risk situations.

[1016] In this way, the server can analyze the audio and video data in real time, determine the emergency situation with high accuracy, and take prompt action.

[1017] (Application example 1)

[1018] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1019] Conventional emergency notification systems analyze only audio data to determine the urgency of an emergency, which often limits information and makes it difficult to make accurate judgments. Furthermore, real-time processing of video data and linkage with audio are insufficient, making it difficult to respond appropriately to high-urgency situations. This reduces efficiency in situations where a rapid emergency response is required, and further improvements in accuracy and faster response are needed.

[1020] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1021] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for determining the level of urgency from the voice data and video data, means for issuing appropriate response commands depending on the level of urgency, and means for analyzing camera video data in real time in conjunction with the voice data in an emergency. This makes it possible to integrate information from both the voice and video data and determine the level of urgency quickly and with high accuracy.

[1022] An "AI model" is an artificial intelligence algorithm that analyzes audio and video data and performs pattern recognition and feature extraction.

[1023] "Voice data" refers to a voice signal that includes information necessary for determining the urgency of a call, such as the content of the call and the tone of the user's voice.

[1024] "Remote video data" is video information sent by a user from a smartphone or other camera device.

[1025] "Voice analysis means" refers to devices or software that use AI models to analyze the content, tone, and urgency of voice data.

[1026] "Video analytics" means devices or software that use AI models or other technologies to detect abnormal conditions or movements in remote video data.

[1027] The "means for determining the urgency" refers to a device or software that evaluates the urgency of a situation based on the results of analyzing audio data and video data.

[1028] The "means for issuing response commands" refers to a device or software that automatically issues commands to execute appropriate responses depending on the level of urgency.

[1029] "Means for analyzing camera video data in real time in conjunction with audio data in an emergency" refers to devices or software that instantly analyzes video data acquired in conjunction with a user's emergency call.

[1030] The present invention is a system for determining the urgency of an emergency using voice data and remote video data, and issuing prompt response instructions. A specific embodiment of this system will be described below.

[1031] System Overview

[1032] The system includes a voice data analyzer, a video data analyzer, an emergency assessment device, and a response command issuing device. All of these devices run on a server and are deployed in a cloud environment. The system acquires data via user devices such as smartphones and performs analysis in real time.

[1033] Voice data analysis

[1034] The server analyzes the voice data using an AI model for voice recognition. This AI model identifies the tone, content, and urgency of the user's voice. Once the voice data is acquired, the server converts and analyzes the content using a voice data analyzer.

[1035] Video Data Analysis

[1036] The server also analyzes remote video data acquired from users' smartphones and other camera devices, using AI models and video analysis modules to detect abnormal situations and movements within the video, enabling rapid recognition of emergencies such as fires and intruders.

[1037] Judgment of urgency

[1038] The server integrates the results of the analysis of the audio and video data and calculates an urgency score using an emergency assessment device. Based on this score, it determines with high accuracy whether an emergency has occurred. If the urgency is determined to be high, it immediately issues appropriate response instructions.

[1039] Output of response instructions

[1040] If the emergency is deemed high, the server will send a notification to the appropriate authorities, such as security companies or the police, via a response command system. This notification will include the user's voice data, video data, and location information, enabling a prompt and appropriate response.

[1041] Hardware and Software Use

[1042] The server uses the following hardware and software:

[1043] Voice recognition microphone (built-in microphone on smartphone)

[1044] Camera for capturing images (built-in camera on smartphone)

[1045] Python, speech_recognition library, transformers library

[1046] The voice data is converted into text by a speech recognition library and an AI model assesses the urgency of the call, while the video data is analyzed in real time via a video analytics module.

[1047] Specific examples

[1048] When an intruder suddenly breaks into the home, the user shouts "Help me!" into their smartphone. This voice data is sent to the server and analyzed by the AI ​​model. At the same time, the smartphone camera sends video data of the scene to the server, which is then analyzed by the video analysis module.

[1049] Prompt Sentence Examples

[1050] "Analyze the input voice data, and if it contains the keyword 'help,' assess the urgency of the data. Guide the user on what action to take in an emergency."

[1051] As a result, the present invention can integrate audio and video data to determine the urgency with high accuracy, thereby enabling a prompt and appropriate response.

[1052] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1053] Step 1:

[1054] The server performs initialization processing. Specifically, it loads and initializes the AI ​​model and video analysis module required to analyze audio and video data. This initialization prepares the server to execute real-time data processing. The AI ​​model and video analysis module are required as input, and the initialization-completed state is obtained as output.

[1055] Step 2:

[1056] When a user makes an emergency call, the smartphone sends voice data to a server. The smartphone acquires the voice data using a microphone for voice input. The input is the user's voice, and the acquired voice data is obtained as the output.

[1057] Step 3:

[1058] The server analyzes the received voice data in real time. Specifically, it uses a voice recognition library to convert the voice data into text, which is then analyzed using an AI model. The input is the voice data, and the output is the text data of the analysis results and an assessment of the urgency level.

[1059] Step 4:

[1060] It acquires remote video data from the user's smartphone. It uses a camera to record on-site video and transmits it to the server in real time. The input requires the camera on the user's device, and the output is the recorded video data.

[1061] Step 5:

[1062] The server analyzes the acquired video data. It uses a video analysis module to detect abnormal situations or movements (for example, fires or intruders). The input is video data, and the output is the analysis result, which indicates whether there is an abnormality or not and the urgency level.

[1063] Step 6:

[1064] The server combines the results of the analysis of the audio and video data to determine the urgency. It calculates an urgency score, and if it is high it is rated as high urgency, and if it is low it is rated as low urgency. The analysis results of the audio and video data are required as input, and the urgency score is obtained as output.

[1065] Step 7:

[1066] If the urgency is determined to be high, the server immediately issues appropriate response instructions. Specifically, it sends a notification to response agencies such as security companies and the police. The notification includes the user's audio data, video data, and location information. The input requires an urgency score, and the output is a notification of the response instructions.

[1067] This series of processing steps enables the system to integrate audio and video data, quickly and accurately determine the level of urgency, and prescribe appropriate responses.

[1068] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1069] This invention is a system that uses an AI model and an emotion engine to accurately determine the urgency of calls made to 119. The system includes means for analyzing audio data and remote video data and recognizing the user's emotional state from the audio and video.

[1070] 1. Initialize the server

[1071] The server initializes the AI ​​model, video analysis module, and emotion engine, which prepares the server to process audio and video data in real time. The emotion engine loads the libraries and models required to extract user emotions from audio and video.

[1072] 2. Receiving incoming calls

[1073] When a user calls 119, the server receives the call and captures the audio data and remote video data. The audio data is recorded in real time, and the remote video data is captured as a video stream from the camera.

[1074] 3. Analysis of audio and video data

[1075] The server uses AI models to analyze the audio data, assessing the caller's tone of voice, content, and urgency. At the same time, it uses a video analytics module to analyze the remote video data and detect activity or unusual conditions in the video (e.g., fire, smoke, blood, etc.).

[1076] Furthermore, the emotion engine recognizes the caller's emotional state based on the analysis of audio and video data. For example, the emotion engine analyzes the caller's level of tension from audio and identifies their emotional state by analyzing the caller's facial expressions and movements from video.

[1077] As a concrete example, consider the case where a user witnesses a fire. When the user shouts, "Fire! Help!", this voice data is sent to the server. At the same time, video footage of the fire scene captured by a remote camera is also sent to the server. The server recognizes the word "fire" through voice data analysis and assesses the user's level of urgency. It also detects smoke and flames at the scene through video analysis. Furthermore, the emotion engine recognizes that the caller is in a high-level emergency based on their tone of voice and facial expression.

[1078] 4. Determining Urgency

[1079] The server integrates the analysis results of the audio data, video data, and emotion engine to calculate an urgency score. In this process, the results of audio analysis, video analysis, and emotion recognition are combined to make a comprehensive urgency judgment. If the urgency score is high, the server judges it to be "high urgency," and if it is low, it judges it to be "low urgency."

[1080] 5. Issuance of response instructions

[1081] If the emergency is deemed high, the server immediately notifies the operator and issues instructions to dispatch the appropriate rescue team or ambulance. The server also provides the operator with information about the user's emotional state to help them respond appropriately. If the emergency is low, the server provides additional information and prompts the operator for further confirmation.

[1082] This system allows for highly accurate assessment of the urgency of 119 calls, enabling prompt and appropriate responses. Furthermore, by incorporating an emotion engine, it is possible to achieve even more accurate urgency assessments that take into account the caller's emotional state. In this way, the present invention significantly improves the efficiency of 119 call centers and reduces prank calls and non-urgent calls.

[1083] The processing flow will be explained below.

[1084] Step 1:

[1085] The server initializes the AI ​​model, video analysis module, and emotion engine. This enables real-time analysis of audio, video, and emotion data. Specifically, it loads the AI ​​model into memory, configures the video analysis module, and prepares the emotion engine library and model.

[1086] Step 2:

[1087] A user calls 119. To report an emergency, the user describes the situation over the phone. The user's words and tone of voice are sent to the server. Also, video data from the user's remote camera is sent to the server as needed.

[1088] Step 3:

[1089] The server starts recording audio data and simultaneously starts acquiring video streams from remote cameras, which are then processed in real time.

[1090] Step 4:

[1091] The server uses an AI model to analyze the recorded voice data, evaluating the caller's tone of voice, the content of what was said, and the urgency score. The AI ​​model detects specific keywords and quantifies the urgency level.

[1092] Step 5:

[1093] The server uses a video analytics module to analyze the remote video data, detecting movement and abnormal conditions (e.g., fire, smoke, blood, etc.) in the video and extracting this information as digital data.

[1094] Step 6:

[1095] The server uses an emotion engine to recognize the user's emotional state from audio and video data. From the audio data, the server analyzes the caller's tone and tempo to identify their emotional state. From the video data, the server analyzes the caller's facial expressions and gestures to evaluate emotions such as tension, fear, and excitement.

[1096] Step 7:

[1097] The server integrates the results of the analysis of the audio data, video data, and emotion data, and calculates an urgency score. The results of the audio analysis, video analysis, and emotion engine are combined to make a comprehensive urgency judgment.

[1098] Step 8:

[1099] The server issues appropriate response instructions based on the calculated urgency score. If the urgency is high, the server immediately notifies the operator and issues instructions to dispatch a rescue team or ambulance. The server also provides the operator with information about the user's emotional state, improving the quality of the response.

[1100] Step 9:

[1101] The operator receives a notification from the server and immediately takes appropriate action. If the emergency is high, they will instruct a quick response, and if the emergency is low, they will obtain additional information and take action.

[1102] As a concrete example, consider the case where a user witnesses a fire and calls 119. When the user yells, "Fire! Help!", the server uses voice analysis to recognize the word "fire" and assesses the level of urgency. At the same time, video analysis detects smoke, and the emotion engine determines that the user is extremely nervous. Based on this comprehensive assessment, the server determines that the level of urgency is high, and is able to immediately notify an operator and issue a command to dispatch a fire engine.

[1103] Example 2

[1104] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1105] Conventional 119 call centers determine the level of urgency based solely on voice information, making it difficult to effectively distinguish between high-urgency and low-urgency cases. Furthermore, they are unable to properly recognize the caller's emotional state, making it difficult to respond immediately and appropriately. This increases the risk of incorrect responses, posing challenges to the speed and accuracy of rescue responses.

[1106] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1107] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for recognizing the user's emotional state from the voice data and video data, means for determining the urgency from the voice data and video data, and means for issuing appropriate response instructions depending on the urgency. This enables quick and accurate identification of high-urgency cases and appropriate rescue responses that take into account the caller's emotional state.

[1108] An "AI model" is an artificial intelligence algorithm used to analyze a variety of data, including audio and video data.

[1109] "Voice data" refers to information in the form of voice signals transmitted by a user over a telephone.

[1110] "Remote video data" refers to information on video signals obtained by capturing the user's environment through a camera and transmitting it to a server.

[1111] "Emotional state" refers to the user's psychological and emotional state determined from the results of audio and video analysis.

[1112] "Voice analysis means" means means for using an AI model to evaluate the content, tone, and urgency of voice data.

[1113] "Video analysis means" refers to means for detecting movement or abnormal situations within remote video data.

[1114] The "urgency determination means" is a means for calculating an urgency score based on the analysis results of audio data and video data and the emotional state, and for assessing the urgency of the incident.

[1115] The "response command means" is a means for generating and outputting an appropriate command to dispatch a rescue team or an ambulance based on the result of the emergency assessment.

[1116] This invention is a system for accurately determining the urgency of an incoming call to 911 based on audio data and remote video data. The system includes a means for recognizing the emotional state of a user using an AI model, a video analysis module, and an emotion engine.

[1117] When the system first starts up, the server initializes the AI ​​model (e.g., TensorFlow), video analysis module (e.g., OpenCV), and emotion engine. This prepares the system for processing audio and video data in real time. The emotion engine is equipped with the necessary libraries and models for extracting user emotions from audio and video.

[1118] When a user calls 119, the device receives the call, acquires the audio data and remote video data, and sends them to the server in real time. For example, when a user yells "Help, there's a fire!", the audio is sent from the device to the server, and at the same time, the video of the scene captured by the remote camera is also sent to the server.

[1119] The server uses an AI model (e.g., Google Speech-to-Text API or a proprietary voice analysis module) to analyze the received voice data. Here, the server analyzes the user's tone of voice and the content of what is being said to assess the urgency. For example, the keyword "fire" and the level of urgency in the voice are detected, and the call is judged to be "very urgent."

[1120] The server then uses OpenCV to analyze the video data. This video analysis module detects abnormal situations (e.g., fire, smoke, blood, etc.) and significant movement in the video sent from the remote camera. For example, it detects flame and smoke patterns in the video and concludes that a fire has been detected.

[1121] Furthermore, the emotion engine recognizes the user's emotional state based on the analysis of audio and video data. Specific emotions are extracted by analyzing tension and fear from audio and facial expressions and movements from video. For example, the server may analyze the user's trembling voice and teary eyes to determine that the user is "extremely tense."

[1122] Finally, the server calculates an urgency score by combining the results of the audio data, video data, and emotion engine analysis. If this score is above a certain level, it is judged to be a "high urgency," and if it is lower, it is judged to be a "low urgency." For example, if the audio analysis results in "fire," the video analysis shows "flames and smoke," and the emotion analysis shows "high tension," the overall urgency score will be high and the event will be judged to be a "high urgency."

[1123] If the emergency is deemed high, the server immediately notifies the operator and issues a command to dispatch the appropriate rescue team or ambulance. The operator is also provided with information about the user's emotional state, which encourages appropriate response. For example, the server may send a notification to the operator saying, "High emergency, fire. Dispatching rescue team to the scene. The caller is in a very tense situation."

[1124] As a concrete example of how this works, the prompt text is as follows:

[1125] "Audio data: 'Help, there's a fire!' Audio analysis: 'Fire' Video analysis: 'Flame and smoke detected' Emotional state: 'Very tense' Urgency score: 'High'"

[1126] This system will enable a highly accurate assessment of the urgency of calls to 119, enabling a swift and appropriate response. This will significantly improve the efficiency of 119 call centers, reducing prank calls and non-urgent calls. In addition, by combining it with an emotion engine, it will be possible to achieve a more accurate assessment of urgency by taking into account the caller's emotional state.

[1127] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1128] Step 1:

[1129] Initializing the Server

[1130] When the system starts up, the server loads the necessary libraries and models and initializes the AI ​​model (e.g., TensorFlow), video analysis module (e.g., OpenCV), and emotion engine. This prepares the system for analyzing audio and video data in real time. Specifically, each library is loaded and the model is set up. The input to this process is the "initialization command" and the output is the "initialization completion status."

[1131] Step 2:

[1132] Receiving incoming calls

[1133] When a user calls 119, the terminal receives the call and acquires the audio data and remote video data. This real-time data is sent from the terminal to the server. The input is the "user's telephone audio signal" and the "remote video signal," and the output is the "audio and video data sent to the server."

[1134] Step 3:

[1135] Analysis of audio data

[1136] The server analyzes the received voice data using an AI model. This analysis includes a process to evaluate the content and urgency of the voice. Specifically, the voice data is converted into text using the Google Speech-to-Text API or similar, and keywords and tone of voice are analyzed. The input is the "voice data sent to the server," and the output is the "analyzed voice text" and the "urgency assessment result."

[1137] Step 4:

[1138] Video data analysis

[1139] The server analyzes the remote video data using OpenCV. This analysis includes detecting abnormal situations in the video (for example, fire, smoke, blood, etc.). Specifically, it performs pattern recognition on each video frame to detect abnormal patterns. The input is the "video data sent to the server," and the output is "information about abnormal situations in the analyzed video."

[1140] Step 5:

[1141] Recognition of emotional states

[1142] The server uses an emotion engine to recognize the user's emotional state from the analysis of audio and video data. Specifically, emotions are determined based on the tone of voice, facial expressions read from the video, and movements. The inputs are the "analyzed audio text" and "analyzed abnormal situation information in the video," and the output is "recognized emotional state information."

[1143] Step 6:

[1144] Determining urgency

[1145] The server calculates an urgency score by integrating the analysis results of the audio data, video data, and emotion engine. Based on this score, if the urgency is judged to be high, it is judged as "high urgency," and if it is low, it is judged as "low urgency." The inputs are "analyzed audio text," "analyzed abnormal situation information in the video," and "recognized emotional state information," and the outputs are "urgency score" and "urgency judgment result."

[1146] Step 7:

[1147] Issuance of response instructions

[1148] Based on the urgency assessment result, the server generates appropriate response instructions and notifies the operator. In the case of a high urgency, an order to dispatch a rescue team or ambulance is issued immediately. The operator is also provided with information about the user's emotional state. The inputs are the "urgency assessment result" and "recognized emotional state information," and the outputs are "response instructions" and "operator notification."

[1149] (Application example 2)

[1150] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1151] In conventional emergency notification systems, callers rely solely on voice information to determine the level of urgency, which can result in a lack of immediate and appropriate response, even in highly urgent situations. It is particularly difficult to grasp the caller's situation and emotional state in detail using voice data alone. This can result in misjudgments and delayed responses in emergencies, posing a major challenge for security services, which require rapid and accurate responses.

[1152] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1153] In this invention, the server includes means for analyzing voice data using an AI model, means for acquiring and analyzing remote video data, means for determining the level of urgency from the voice data and video data, means for issuing appropriate response instructions depending on the level of urgency, and means for processing the voice data and video data collected using a smart device in real time. This allows for simultaneous analysis of the voice data and video data, enabling more accurate determination of the level of urgency and enabling faster and more appropriate responses.

[1154] An "AI model" is an artificial intelligence system that includes algorithms and statistical models that analyze data and make specific decisions.

[1155] "Audio data" is a digital representation of an audio signal captured using a microphone or the like.

[1156] "Remote video data" is a digital representation of a video signal captured from a remote location using a visual sensor such as a camera.

[1157] "Means of analysis" refers to the technical methods used to input data and extract and interpret the information contained in that data.

[1158] The "means for determining urgency" is a technical method for evaluating the urgency of a situation based on acquired data and outputting the results.

[1159] "Means for issuing appropriate response instructions" refers to a technical method for instructing a person or system to take specific actions based on the results of an emergency assessment.

[1160] A "smart device" is a portable or stationary electronic device that has Internet connectivity and is capable of sensing and data processing.

[1161] "Real-time processing means" refers to a technical method that analyzes data as it is generated and obtains results without delay.

[1162] This invention relates to a system that uses an AI model and an emotion engine to analyze audio data and remote video data collected from smart devices and accurately determine the urgency of a situation. This system is realized using the following hardware and software.

[1163] 1. Hardware and Software Used

[1164] Smart Devices: Electronic devices with internet connectivity, such as smartphones and security cameras.

[1165] Audio data collection microphone: A device for converting audio signals into digital data.

[1166] Remote video data collection camera: Equipment for converting video signals into digital data.

[1167] Server: A computer system with the computational resources to analyze audio and video data in real time.

[1168] software:

[1169] sounddevice library: A library for capturing and processing audio data.

[1170] cv2 (OpenCV): A library for acquiring and analyzing video data.

[1171] AI models: Algorithms and models for assessing urgency and issuing response instructions.

[1172] EmotionEngine: An engine for recognizing emotional states from audio and video data.

[1173] 2. Processing Flow

[1174] Once the server is initialized, it activates the AI ​​model, emotion engine, and video analysis module. Audio data and remote video data are sent to the server via smart devices, which process the data in real time. The AI ​​model analyzes the audio data to evaluate the caller's tone, content, and urgency. At the same time, the video analysis module detects abnormal situations in the video. The emotion engine analyzes the audio and video data to recognize the caller's emotional state.

[1175] 3. Urgency assessment and response instructions

[1176] The server calculates an urgency score by integrating the results of the audio and video analysis. If the urgency is determined to be high based on this score, the server immediately issues appropriate response instructions. For example, a notification may be sent to security guards or rescue teams. If the urgency is low, further confirmation is required and an operator is notified.

[1177] 4. Specific Examples

[1178] When a user reports "someone has broken into my house" in the middle of the night, the system collects audio data and remote video data. The server analyzes the tone of the voice and any abnormal activity in the video (e.g., broken windows or suspicious activity) in real time to determine the level of urgency. If the level of urgency is determined to be high, the server immediately issues a response command to call security guards or a rescue team.

[1179] 5. Examples of prompts

[1180] "Create an AI system that can recognize specific emotional states from audio and video data in real time and determine the level of urgency. The specific input data is as follows:

[1181] Audio data: A user shouts "Help!" in a tense voice at the scene of an accident

[1182] Video data: Includes footage of the accident scene, flames, smoke, and injured people.

[1183] As described above, the present invention makes it possible to judge the urgency of an emergency with significantly higher accuracy and to respond more quickly than conventional systems.

[1184] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1185] Step 1:

[1186] Initializing the Server

[1187] The server initializes the AI ​​model, emotion engine, and video analysis module. This prepares the server to process audio and video data in real time. Specifically, it loads various models and engines and imports related libraries. The input is the initialization command, and the output is the ready status.

[1188] Step 2:

[1189] Acquisition of audio and remote video data

[1190] When a user makes an emergency call, the microphone and camera on the terminal (smart device) collect audio and video data and send it to the server. The input is the user's audio and video, and the output is a digital signal of these data.

[1191] Step 3:

[1192] Analysis of audio data

[1193] The server receives the voice data and analyzes it using an AI model. The analysis evaluates the tone of voice, the content of the conversation, and the level of urgency. Specific operations include noise removal from the voice data, feature extraction, and a model-based urgency evaluation. The input is the voice data, and the output is the analysis results.

[1194] Step 4:

[1195] Video data analysis

[1196] The server receives the remote video data and analyzes it using a video analysis module. This analysis detects abnormal activity or situations in the video (e.g., fire, smoke, suspicious individuals, etc.). Specific operations include frame segmentation of the video data, feature extraction, and application of anomaly detection algorithms. The input is the video data, and the output is the analysis results.

[1197] Step 5:

[1198] Recognition of emotional states

[1199] The server uses an emotion engine to recognize the caller's emotional state from the analysis results of the audio and video data. Specifically, it integrates the audio and video analysis results and executes an algorithm to evaluate the emotional state. The input is the audio and video analysis results, and the output is the recognition result of the emotional state.

[1200] Step 6:

[1201] Determining the level of urgency

[1202] The server integrates the analysis results of the audio data, video data, and emotion engine to calculate an urgency score. The inputs are the audio analysis results, video analysis results, and emotion recognition results, and the output is an urgency score. Specifically, the server integrates the analysis results using a weighted average or other integration algorithm to calculate the urgency score.

[1203] Step 7:

[1204] Issuance of response instructions

[1205] Based on the urgency score, the server immediately issues an appropriate response command if the urgency is high, or notifies an operator if the urgency is low. Specifically, the server selects a command based on the urgency and sends the command to the appropriate response channel. For example, if the urgency is high, it notifies security guards or rescue teams, and if the urgency is low, it requests further confirmation from an operator. The input is the urgency score, and the output is a response command.

[1206] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1207] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1208] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1209] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1210] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1211] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1212] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1213] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1214] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1215] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1216] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1217] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1218] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1219] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1220] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1221] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1222] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1223] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1224] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1225] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1226] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1227] The following is further disclosed regarding the above embodiment.

[1228] (Claim 1)

[1229] A means for analyzing the voice data using an AI model; and

[1230] a means for acquiring and analyzing remote video data;

[1231] A means for determining urgency from the audio data and video data;

[1232] A means of issuing appropriate response instructions depending on the urgency;

[1233] A system including:

[1234] (Claim 2)

[1235] 10. The system of claim 1, wherein the voice analysis means includes means for evaluating the tone, content, and urgency of the caller's voice.

[1236] (Claim 3)

[1237] 10. The system of claim 1, wherein the video analysis means includes means for detecting motion or abnormal conditions within the video.

[1238] "Example 1"

[1239] (Claim 1)

[1240] A means for analyzing the voice data using an AI model; and

[1241] a means for acquiring and analyzing remote video data;

[1242] A means for integrating the analysis results of the audio data and the video data to calculate an urgency score;

[1243] A means of notifying an operator based on the level of urgency and issuing a command to dispatch the appropriate rescue team or ambulance;

[1244] A system including:

[1245] (Claim 2)

[1246] 10. The system of claim 1, wherein the voice analysis means includes means for evaluating the tone, content, and urgency of the caller's voice.

[1247] (Claim 3)

[1248] 10. The system of claim 1, wherein the video analysis means includes means for detecting motion or abnormal conditions within the video.

[1249] "Application Example 1"

[1250] (Claim 1)

[1251] A means for analyzing the voice data using an AI model; and

[1252] a means for acquiring and analyzing remote video data;

[1253] A means for determining urgency from the audio data and video data;

[1254] A means of issuing appropriate response instructions depending on the urgency;

[1255] A means of analyzing camera video data in real time in conjunction with audio data in an emergency,

[1256] A system including:

[1257] (Claim 2)

[1258] 10. The system of claim 1, wherein the voice analysis means includes means for evaluating the tone, content, and urgency of the user's voice in an emergency.

[1259] (Claim 3)

[1260] 10. The system of claim 1, wherein the video analysis means includes means for detecting abnormal situations or behaviors in the video.

[1261] "Example 2: Combining Emotion Engines"

[1262] (Claim 1)

[1263] A means for analyzing the voice data using an AI model; and

[1264] a means for acquiring and analyzing remote video data;

[1265] means for recognizing the emotional state of a user from audio data and video data;

[1266] A means for determining urgency from the audio data and video data;

[1267] A means of issuing appropriate response instructions depending on the urgency;

[1268] A system including:

[1269] (Claim 2)

[1270] 10. The system of claim 1, wherein the voice analysis means includes means for evaluating the tone, content, and urgency of the caller's voice and means for recognizing an emotional state.

[1271] (Claim 3)

[1272] 10. The system of claim 1, wherein the video analysis means includes means for detecting movement and abnormal situations in the video and means for recognizing the emotional state of the caller.

[1273] "Application example 2 when combining emotion engines"

[1274] (Claim 1)

[1275] A means for analyzing the voice data using an AI model; and

[1276] a means for acquiring and analyzing remote video data;

[1277] A means for determining urgency from the audio data and video data;

[1278] A means of issuing appropriate response instructions depending on the urgency;

[1279] A means for processing audio data and video data collected using a smart device in real time;

[1280] A system including:

[1281] (Claim 2)

[1282] 10. The system of claim 1, wherein the voice analysis means includes means for evaluating the tone, content, and urgency of the caller's voice.

[1283] (Claim 3)

[1284] 10. The system of claim 1, wherein the video analysis means includes means for detecting motion or abnormal conditions within the video. [Explanation of symbols]

[1285] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for analyzing the voice data using an AI model; and a means for acquiring and analyzing remote video data; A means for determining urgency from the audio data and video data; A means of issuing appropriate response instructions depending on the urgency; A system including:

2. 2. The system of claim 1, wherein the voice analysis means includes means for evaluating the tone, content, and urgency of the caller's voice.

3. 10. The system of claim 1, wherein the video analysis means includes means for detecting motion or abnormal conditions within the video.

Images