System

Abstract

A system is provided.SOLUTION: A system includes a means for recording a siren sound of an emergency vehicle, a means for transmitting the recorded voice data to an analysis server, a means for analyzing the voice data received by the server and generating position information of the emergency vehicle and a handling instruction, and a means for displaying the position information of the emergency vehicle and the handling instruction generated as an analysis result on a display of the vehicle.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When hearing-impaired people drive, they often find it difficult to hear the sirens of emergency vehicles, preventing them from responding appropriately. This makes it difficult for them to drive safely, potentially resulting in accidents or dangerous situations. Even for normal drivers, not receiving appropriate instructions when an emergency vehicle is approaching hinders safe driving. Therefore, a system that allows hearing-impaired people to visually recognize approaching emergency vehicles is needed to enable safe and smooth driving. [Means for solving the problem]

[0005] The present invention provides a means for recording the siren sound of an emergency vehicle and transmitting the recorded data to an analysis server. The analysis server is equipped with a means for analyzing the received audio data and generating emergency vehicle location information and response instructions. The system also includes a means for displaying the emergency vehicle location information and response instructions generated as a result of this analysis on a vehicle display. Furthermore, the system also includes a means for acquiring image data from a 360-degree camera and visually identifying a safe avoidance area from the captured video, as well as a means equipped with a network module for communicating audio data and analysis results in real time between the analysis server and the user's device. This allows hearing-impaired people to accurately and quickly detect approaching emergency vehicles and supports safe driving.

[0006] An "emergency vehicle" refers to a vehicle such as an ambulance, fire engine, or police vehicle that is equipped with special equipment to carry out emergency missions and uses sirens and lights to indicate an emergency to those around it.

[0007] A "siren sound" refers to the distinctive sound that an emergency vehicle makes to alert those around it to its approach.

[0008] "Recording means" refers to the audio input device and its control software used to capture ambient sounds.

[0009] "Analysis Server" refers to a computer system that processes and analyzes received data and generates necessary information.

[0010] "Transmitting means" refers to a combination of hardware and software for transmitting data over a communications network to other devices or systems.

[0011] "Location information" refers to data used to identify the current location and direction of movement of an emergency vehicle.

[0012] "Response instructions" refer to instructions that indicate the driving actions that a driver should take when an emergency vehicle is approaching.

[0013] "Display" means a display device for visually presenting textual and graphical information.

[0014] A "surrounding camera" refers to a camera system that captures the entire area around the vehicle and uses that footage to assist in safe driving.

[0015] "Network module" refers to a set of hardware and software required for data communication. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] As an embodiment of the present invention, the specific operation of the system will be described below.

[0038] overview

[0039] When driving a car, hearing-impaired people have difficulty visually recognizing approaching emergency vehicles. This invention provides a system that detects the siren sound of an emergency vehicle in real time and displays its location and response instructions on the vehicle's display. The system is mainly composed of a terminal, a server, and a user.

[0040] System configuration

[0041] 1. Device (smartphone)

[0042] Audio recording module: Uses the smartphone's microphone to record surrounding audio in real time.

[0043] Network communication module: A module that sends recorded audio data to the analysis server.

[0044] Display Audio Link Module: A module that receives analysis results and displays them on the vehicle's display.

[0045] All-around camera image acquisition module (optional): A module that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[0046] 2. Server

[0047] Voice data analysis module: Analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle.

[0048] Network communication module: A module that sends analysis results to the terminal.

[0049] 3. Users

[0050] Hearing-impaired driver: Check the approach of emergency vehicles through a smartphone and vehicle display and perform appropriate driving operations.

[0051] Program processing

[0052] Terminal

[0053] The device first records the surrounding sounds in real time, and the recorded data is sent to the analysis server via the network communication module.

[0054] server

[0055] The server's voice data analysis module analyzes the received voice data and identifies the siren sound of an emergency vehicle. From the identified siren sound, the type of emergency vehicle, its location, and direction of travel are estimated. The analysis results are sent to the device via the network communication module.

[0056] Terminal

[0057] The terminal receives the analysis results from the server and displays them on the vehicle's display via the display audio linkage module. For example, specific instructions such as "An ambulance is approaching 30 meters behind and to the right" can be displayed. In addition, by using the all-around camera image acquisition module, safe avoidance areas can also be visually displayed.

[0058] User

[0059] The user checks the vehicle's display and recognizes the approach of an emergency vehicle based on the analysis results. For example, if the display reads "An ambulance is approaching 30 meters behind the vehicle to the right," the user can follow the instructions and safely move the vehicle to the left.

[0060] Specific examples

[0061] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module records this audio and sends it to the server via the network communication module. The server's analysis module analyzes the audio data and identifies the location of the emergency vehicle. The analysis result is generated as "Ambulance approaching 30 meters behind and to the right," which is sent to the terminal. The terminal displays the received information on the vehicle's display. The user confirms this display and moves about 1 meter to the left, ensuring safe driving.

[0062] The above is an embodiment of the present invention, and this system enables even hearing-impaired people to properly recognize approaching emergency vehicles and drive safely.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] The device activates the audio recording module, which accesses the device's microphone and begins recording surrounding audio in real time.

[0066] Step 2:

[0067] The device sends the recorded audio data to the server via the network communication module, uploading the audio data approximately every second.

[0068] Step 3:

[0069] The server analyzes the received audio data using the audio data analysis module, and analyzes specific frequency patterns to detect the siren of an emergency vehicle.

[0070] Step 4:

[0071] The server identifies the type and location of the emergency vehicle based on the siren sound detection result, and estimates the direction and distance of the emergency vehicle based on the sound intensity and detection timing.

[0072] Step 5:

[0073] The server generates the analysis results and creates a text message containing the type of emergency vehicle, its location, direction of travel, and any other necessary information.

[0074] Step 6:

[0075] The analysis results generated by the server are sent to the terminal using a network communication module.

[0076] Step 7:

[0077] The analysis results received by the device from the server are displayed on the vehicle's display via the display audio link module. For example, the message "Ambulance approaching 30 meters behind the right" is displayed.

[0078] Step 8:

[0079] (Optional) The device uses the omnidirectional camera image acquisition module to analyze the video from the omnidirectional camera and visually inform the user of a safe area to avoid. For example, it displays an instruction such as "There is room to evacuate about 1 meter to the left."

[0080] Step 9:

[0081] The user checks the vehicle's display and sees an instruction such as "An ambulance is approaching 30 meters behind and to the right."

[0082] Step 10:

[0083] The system performs safe driving maneuvers based on instructions provided by the user. For example, the system follows the instructions on the display to move about 1 meter to the left.

[0084] The above series of processing steps enables hearing-impaired drivers to properly recognize the approach of an emergency vehicle and drive safely.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] With conventional technology, it has been difficult for hearing-impaired drivers to properly recognize approaching emergency vehicles. In particular, they often miss sirens, which can lead to traffic accidents. The present invention aims to provide a system that supports safe driving by detecting approaching emergency vehicles in real time and allowing even hearing-impaired drivers to visually recognize them.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes a means for recording surrounding sounds in real time, a means for transmitting the recorded sound data to an analysis server, a means for analyzing the sound data received by the server and estimating the type, location, and direction of travel of the emergency vehicle, and a means for displaying the analysis results on the vehicle display, thereby enabling even hearing-impaired persons to visually recognize the approach of an emergency vehicle and to operate the vehicle safely.

[0090] The "means for recording ambient sound in real time" refers to a device or module that has the function of continuously capturing ambient sound and recording it as digital data.

[0091] The "means for transmitting recorded voice data to an analysis server" refers to a device or module for transferring recorded voice data to another server via a network such as the Internet.

[0092] "Means for analyzing the voice data received by the server and estimating the type, location, and direction of travel of the emergency vehicle" refers to a program or algorithm that has the function of analyzing the voice data received by the server, identifying each emergency vehicle, and determining the location and direction of travel of the sound source.

[0093] The "means for displaying the analysis results on the vehicle display" refers to a device or system for visually displaying the analysis results received from the server on a display device within the vehicle.

[0094] "Means for acquiring image data from a 360-degree camera and visually displaying safe avoidance areas from the captured images" refers to a device or module that has the function of acquiring image data captured by a camera mounted on a vehicle, analyzing the surrounding situation, and displaying areas where it is safe to proceed on a display device inside the vehicle.

[0095] A "network module for communicating voice data and analysis results in real time between an analysis server and a terminal" is a device or software that provides the network communication functions necessary for the analysis server and a terminal to send and receive data in real time.

[0096] MODE FOR CARRYING OUT THE INVENTION

[0097] As an embodiment of the present invention, the specific operation of the system will be described below. The present invention is a system that assists hearing-impaired people in visually recognizing approaching emergency vehicles when driving a car. The system is composed of a terminal, a server, and a user.

[0098] Terminal

[0099] Ambient sound recording

[0100] The device uses a voice recording module to record surrounding sounds in real time through the smartphone's microphone, and the recording data is stored in digital format, for example, using a microphone device with an appropriate sampling rate.

[0101] Sending audio data

[0102] The device transmits the recorded audio data to the analysis server via a network communication module, primarily via an Internet connection, for example via HTTP or HTTPS protocols.

[0103] Receiving and displaying analysis results

[0104] The terminal receives the analysis results from the server and displays them on the vehicle's display via the display audio linkage module. The actual display is linked to the vehicle's information display system and is presented in a format that is intuitively easy for the user to understand.

[0105] Use of a surrounding camera (optional)

[0106] Additionally, the device can utilize a 360-degree camera image acquisition module, which captures and analyzes video data from cameras installed in the vehicle and displays safe areas to avoid.

[0107] server

[0108] Analysis of audio data

[0109] The server uses a voice data analysis module to analyze the received voice data. This analysis can utilize an AI voice recognition engine (e.g., Google Cloud Speech-to-Text). It identifies the sound of an emergency vehicle siren and estimates its type, location, and direction of travel.

[0110] Sending analysis results

[0111] The server then transmits the analysis results to the device in real time. This communication is fast and secure, providing information on approaching emergency vehicles without delay.

[0112] User

[0113] Checking information and driving operations

[0114] The user checks the analysis results displayed on the vehicle's display and recognizes the approach of an emergency vehicle. For example, if the display shows "An ambulance approaching 30 meters behind to the right," the user follows the instructions and safely moves the vehicle to the left.

[0115] Specific examples

[0116] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module records this audio and sends it to the server via the network communication module. The server's audio data analysis module analyzes the audio data and identifies the location information of the emergency vehicle. The analysis result, "Ambulance approaching 30 meters behind and to the right," is generated and sent to the terminal. The terminal displays the received information on the vehicle's display. The user can confirm this display and move about 1 meter to the left, ensuring safe driving.

[0117] Prompt Sentence Examples

[0118] "Detect the sound of an emergency vehicle siren. Display specific instructions to inform the user that an emergency vehicle is approaching 30 meters behind and to the right."

[0119] The above is an embodiment of the present invention, and this system enables even hearing-impaired people to properly recognize the approach of an emergency vehicle and drive safely.

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Step 1:

[0122] Ambient sound recording

[0123] The device uses the smartphone's audio recording module to record ambient audio in real time. The ambient sound is input from the smartphone's microphone. This audio is stored in a buffer as digitally recorded data. Specifically, the device sets a sampling rate and continuously captures audio and converts it into digital data, which is then used in subsequent processing steps.

[0124] Step 2:

[0125] Sending audio data

[0126] The device sends the recorded audio data to the analysis server via the network communication module. The recorded data buffer is used as input, and the data is sent to the server via an Internet connection. Specifically, the device divides the recorded data into packets at regular time intervals (e.g., every second) and sends them using the HTTP or HTTPS protocol. Once the transmission is complete, the buffer is cleared and prepared for the next data recording.

[0127] Step 3:

[0128] Analysis of audio data

[0129] The server analyzes the received voice data using the voice data analysis module. The voice data received from the device is used as input. Specifically, the server performs noise removal and normalization processing, and inputs this data into an AI voice recognition model. This model identifies the sound of an emergency vehicle siren and estimates the type, location, and direction of travel of the emergency vehicle. The analysis results are generated as JSON format data as output.

[0130] Step 4:

[0131] Sending analysis results

[0132] The server sends the generated analysis results to the terminal through the network communication module. The JSON data of the analysis results is used as input, and the data is sent to the terminal as output. Specifically, the server divides the data into packets and sends them to the terminal's receiving endpoint using the HTTPS protocol. This allows the analysis results to be updated in real time.

[0133] Step 5:

[0134] Receiving and displaying analysis results

[0135] The terminal displays the received analysis results on the vehicle's display via the display audio linkage module. The JSON data of the analysis results received from the server is used as input, and the information to be displayed on the vehicle's display is generated as output. Specifically, the terminal decodes the analysis results and converts them into an appropriate display format. For example, a message such as "Ambulance approaching 30 meters behind to the right" is displayed on the display.

[0136] Step 6:

[0137] Use of a surrounding camera (optional)

[0138] The device uses a 360-degree camera image acquisition module to acquire video data from the vehicle's onboard camera. It uses the camera's video data as input and generates information to display safe areas for avoidance as output. Specifically, the device analyzes the camera video in real time to identify the direction of travel and areas that can be avoided. This information is then visually displayed on the display.

[0139] Step 7:

[0140] Checking information and driving operations

[0141] The user checks the analysis results displayed on the vehicle's display and recognizes the approach of an emergency vehicle. The information displayed on the vehicle's display is used as input, and the user's driving operation is used as output. Specifically, the user checks the message "An ambulance is approaching 30 meters behind and to the right" and takes action to safely move the vehicle to the left. In this case, the user references the images from the all-around camera and the displayed information to safely avoid the approach.

[0142] (Application example 1)

[0143] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0144] It is difficult for the hearing impaired to visually recognize approaching emergency vehicles. This is particularly true when using autonomous vehicles, where a driver's failure to notice an approaching emergency vehicle can pose a serious risk. When such obstacles exist, effective evasive measures must be implemented quickly. However, current visual information systems lack sufficient functionality to provide real-time information on the location and direction of emergency vehicles.

[0145] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0146] In this invention, the server includes means for recording the siren sound of the emergency vehicle, means for transmitting the recorded audio data to an analysis server, means for the server to analyze the audio data received and generate emergency vehicle position information and response instructions, means for displaying the emergency vehicle position information and response instructions generated as a result of the analysis on a display of the vehicle, and means for displaying information about the approach of the emergency vehicle on a head-mounted display, thereby enabling hearing-impaired people to visually recognize the approach of an emergency vehicle in real time and take appropriate evasive action.

[0147] The "means for recording the siren sound of an emergency vehicle" is a device that records sounds in the surrounding environment of a vehicle and captures the siren sound of an emergency vehicle in real time.

[0148] The "means for transmitting recorded voice data to the analysis server" is a communication module for transmitting recorded voice data to the analysis server via a communication line such as the Internet.

[0149] "Means for analyzing the voice data received by the server and generating emergency vehicle location information and response instructions" refers to software or hardware that enables the server to analyze the voice data, identify the location and direction of the emergency vehicle, and provide instructions to the driver based on that information.

[0150] The "means for displaying the emergency vehicle location information and response instructions generated as the analysis result on the vehicle display" refers to an interface and display device for visually displaying the analysis result on the display inside the vehicle.

[0151] "Means for acquiring image data from a 360-degree camera and visually identifying safe avoidance areas from the captured image" refers to a device that analyzes images from a vehicle's 360-degree camera and visually identifies areas that can be safely avoided.

[0152] "Means having a network module for communicating voice data and analysis results in real time between the analysis server and the user's terminal" refers to a communication device for realizing data communication in real time between the analysis server and the vehicle's terminal.

[0153] The "means for displaying information about approaching emergency vehicles on a head-mounted display" is a device for displaying emergency vehicle location information and avoidance instructions on a head-mounted display.

[0154] As an embodiment of the present invention, the specific operation of the system will be described below.

[0155] overview

[0156] This system provides assistance to hearing-impaired people using self-driving vehicles, helping them visually recognize approaching emergency vehicles. The system consists of a terminal, a server, and a user. Terminals refer to smartphones, displays, head-mounted displays (HMDs), etc. installed in self-driving vehicles. The server analyzes the audio data and generates emergency vehicle location information and avoidance instructions. The user then performs safe driving operations based on the information provided visually.

[0157] System configuration

[0158] 1. Terminal (installed in autonomous vehicles)

[0159] Audio recording module: A device that uses the device's microphone to record surrounding audio in real time.

[0160] Network communication module: A device that sends recorded audio data to a server for analysis.

[0161] Display audio link module: A device that displays the analysis results received from the server on the display or HMD of the autonomous vehicle.

[0162] All-around camera image acquisition module (optional): A device that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[0163] 2. Server

[0164] Voice data analysis module: This module analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle. This analysis uses a deep learning model (for example, a model built with TensorFlow or PyTorch).

[0165] Network communication module: A device that transmits analysis results to the terminal.

[0166] 3. Users

[0167] Hearing-impaired person (driver): Check the approach of emergency vehicles through the self-driving vehicle's display or HMD and perform appropriate driving operations.

[0168] Program processing explanation

[0169] The device first records the surrounding sounds in real time, and the recorded data is sent to the analysis server via the network communication module.

[0170] The server analyzes the received audio data and identifies the siren sound of an emergency vehicle. From the identified siren sound, it estimates the type of emergency vehicle, its location, and its direction of travel. This analysis is performed using deep learning techniques (e.g., TensorFlow, PyTorch). The results are sent to the device using a network communication module.

[0171] The terminal receives the analysis results from the server and displays them on the autonomous vehicle's display or HMD via the display audio linkage module. For example, specific instructions such as "An ambulance is approaching 30 meters behind and to the right" can be displayed. In addition, by using the all-around camera image acquisition module, safe avoidance areas can also be visually displayed.

[0172] Users can check the autonomous vehicle's display or HMD, recognize the approach of an emergency vehicle based on the analysis results, and take appropriate evasive action. For example, if a user sees a message saying "An ambulance is approaching 30 meters behind and to the right," they will take action by moving about one meter to the left.

[0173] Specific examples

[0174] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module records this audio and sends it to the server via the network communication module. The server's analysis module analyzes the audio data and identifies the location information of the emergency vehicle. The analysis result is generated as "Ambulance approaching 30 meters behind and to the right," which is sent to the terminal. The terminal displays the received information on the display or HMD of the autonomous vehicle. The user confirms this display and moves about 1 meter to the left, ensuring safe driving.

[0175] Prompt Sentence Examples

[0176] For example, a prompt to a generative AI model might look like this:

[0177] "If an emergency vehicle siren is detected, generate a message with specific location and instructions on how to avoid it."

[0178] The above is an embodiment of the present invention, and this system enables even hearing-impaired people to properly recognize the approach of an emergency vehicle and drive safely.

[0179] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0180] Step 1:

[0181] The device records the surrounding audio. While the user is driving the autonomous vehicle, the audio recording module is activated when the engine starts and records the audio around the vehicle in real time. The input is the audio data of the surrounding environment, and the output is the recorded audio data.

[0182] Step 2:

[0183] The device sends the recorded voice data to the analysis server. The network communication module is used to upload the recorded voice data to the server via the Internet. The input is the recorded voice data, and the output is the voice data sent to the server.

[0184] Step 3:

[0185] The server analyzes the received audio data and identifies the siren sound of an emergency vehicle. The audio data analysis module uses deep learning technology to extract the characteristics of the siren sound and detect the approach of an emergency vehicle. The input is the audio data sent from the device, and the output is the siren sound detection result.

[0186] Step 4:

[0187] The server estimates the type, location, and direction of travel of the emergency vehicle based on the detection results. Based on the analysis results, the server identifies the emergency vehicle's detailed location and direction of travel, and generates corresponding driving instructions. The input is the detection result of the siren sound, and the output is data on the emergency vehicle's location and direction of travel.

[0188] Step 5:

[0189] The server generates analysis results and sends them to the terminal. The network communication module is used to return the analysis results to the terminal in real time. The input is data on the emergency vehicle's location and direction of travel, and the output is the analysis results sent to the terminal.

[0190] Step 6:

[0191] The terminal receives the analysis results and displays them on the autonomous vehicle's display or HMD. The display audio linkage module is used to visually display the emergency vehicle's location and response instructions. The input is the analysis results sent from the server, and the output is the emergency vehicle's location information and response instructions displayed on the display or HMD.

[0192] Step 7:

[0193] The user performs appropriate driving operations based on the information displayed on the display or HMD. Check the displayed information about approaching emergency vehicles and follow the instructions to safely avoid the vehicle. The input is the analysis results displayed on the display or HMD, and the output is the user's driving operations.

[0194] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0195] As an embodiment of the present invention, the specific operation of the system will be described below.

[0196] overview

[0197] When driving a car, hearing-impaired people have difficulty visually recognizing approaching emergency vehicles. This invention provides a system that detects emergency vehicle sirens in real time and displays their location information and response instructions on the vehicle's display. In addition, by combining it with an emotion engine that recognizes the user's emotions, driving assistance messages are optimized to the user's emotional state, supporting safe and comfortable driving.

[0198] System configuration

[0199] 1. Device (smartphone)

[0200] Audio recording module: Uses the smartphone's microphone to record surrounding audio in real time.

[0201] Network communication module: A module that sends recorded audio data to the analysis server.

[0202] Display Audio Link Module: A module that receives analysis results and displays them on the vehicle's display.

[0203] All-around camera image acquisition module (optional): A module that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[0204] Emotion recognition module: Recognizes user emotions in real time using cameras and sensors.

[0205] 2. Server

[0206] Voice data analysis module: Analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle.

[0207] Network communication module: A module that sends analysis results to the terminal.

[0208] Emotion analysis module: Analyzes received user emotion data and optimizes the content of driving assistance messages.

[0209] 3. Users

[0210] Hearing-impaired driver: Check the approach of emergency vehicles through a smartphone and vehicle display and perform appropriate driving operations.

[0211] Program processing

[0212] Terminal

[0213] The device first records the surrounding sounds in real time, and sends the recorded data to an analysis server via a network communication module. The device then uses cameras and sensors to recognize the user's emotional state, which is also sent to the server.

[0214] server

[0215] The server's voice data analysis module analyzes the received voice data and identifies the siren sound of an emergency vehicle. From the identified siren sound, the type of emergency vehicle, its location, and direction of travel are estimated. The analysis results are sent to the device via the network communication module. The emotion analysis module also analyzes the user's emotion data and optimizes the content of the driving assistance message to match the user's emotional state. For example, if the user is feeling stressed, more detailed and careful instructions are provided. If the user is relaxed, concise and smooth instructions are provided.

[0216] Terminal

[0217] The device receives analysis results from the server and displays driving assistance messages based on the driver's emotions on the vehicle's display via the display audio linkage module. Specifically, the device displays the message "An ambulance is approaching 30 meters behind the driver on the right" along with additional instructions tailored to the user's emotions. Safe avoidance areas can also be visually displayed by using the all-around camera image acquisition module.

[0218] User

[0219] The user checks the vehicle's display and reads driving assistance messages based on the analysis results and their emotions. For example, if the user sees a message saying "An ambulance is approaching 30 meters behind and to the right," they can follow the instructions and safely move the vehicle to the left. In addition, the response instructions are optimized to the user's emotional state, allowing the user to react more appropriately and quickly.

[0220] Specific examples

[0221] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module will record this audio and send it to the server via the network communication module. The server's analysis module will analyze the audio data and identify the location of the emergency vehicle. The analysis result will be "Ambulance approaching 30 meters behind and to the right," which will be sent to the device. At the same time, the emotion analysis module will analyze the user's emotional data, and if the user is feeling stressed, detailed instructions such as "Remain calm and move about 1 meter to the left" will be provided. Conversely, if the user is relaxed, simpler instructions will be given such as "Move to the left."

[0222] The above is a form for implementing the present invention, and this system allows even hearing-impaired people to properly recognize the approach of an emergency vehicle and receive driving assistance messages tailored to the user's emotional state, allowing them to drive safely and comfortably.

[0223] The processing flow will be explained below.

[0224] Step 1:

[0225] The device activates the audio recording module, which accesses the device's microphone and begins recording surrounding audio in real time.

[0226] Step 2:

[0227] The device sends the recorded audio data to the server via the network communication module, uploading the audio data approximately every second.

[0228] Step 3:

[0229] The server analyzes the received audio data using the audio data analysis module, and analyzes specific frequency patterns to detect the siren of an emergency vehicle.

[0230] Step 4:

[0231] The server identifies the type and location of the emergency vehicle based on the siren sound detection result, and estimates the direction and distance of the emergency vehicle based on the sound intensity and detection timing.

[0232] Step 5:

[0233] The server generates the analysis results and creates a text message containing the type of emergency vehicle, its location, direction of travel, and any other necessary information.

[0234] Step 6:

[0235] The device uses cameras and sensors to recognize the user's emotional state, and sends the recognized emotional data to a server.

[0236] Step 7:

[0237] The emotion data received by the server is analyzed by the emotion analysis module to determine whether the user is feeling stressed or anxious, or relaxed.

[0238] Step 8:

[0239] The server optimizes driving assistance messages based on the driver's emotional state, for example generating detailed and careful instructions when the driver is stressed and concise and smooth instructions when the driver is relaxed.

[0240] Step 9:

[0241] The analysis results and optimized driving assistance messages generated by the server are sent to the terminal using a network communication module.

[0242] Step 10:

[0243] The terminal receives the analysis results and driving assistance messages from the server and displays them on the vehicle's display via the display audio linkage module. For example, along with the message "An ambulance is approaching 30 meters behind and to the right," detailed instructions such as "Remain calm and move about 1 meter to the left" may be displayed.

[0244] Step 11:

[0245] (Optional) The device uses the omnidirectional camera image acquisition module to analyze the video from the omnidirectional camera and visually inform the user of a safe area to avoid. For example, it displays an instruction such as "There is room to evacuate about 1 meter to the left."

[0246] Step 12:

[0247] The user checks the vehicle's display and sees an instruction such as "An ambulance is approaching 30 meters behind and to the right."

[0248] Step 13:

[0249] The system performs safe driving maneuvers based on instructions provided by the user. For example, the system follows the instructions on the display to move about 1 meter to the left.

[0250] The above series of processing steps enables hearing-impaired drivers to properly recognize approaching emergency vehicles and drive safely. In addition, the emotion engine optimizes driving assistance messages, providing detailed assistance tailored to the user's mental state.

[0251] Example 2

[0252] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0253] When hearing-impaired people drive, it is difficult for them to visually recognize approaching emergency vehicles. This makes it difficult for them to quickly take appropriate evasive action, making safety a challenge. Therefore, there is a need for a system that can recognize approaching emergency vehicles in real time and provide appropriate support messages according to the driver's emotional state.

[0254] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for recording surrounding sounds; means for transmitting the recorded sound data and user emotion data to the analysis server; means having a network module for communicating the sound data, emotion data, and analysis results in real time between the analysis server and the user's terminal; means for analyzing the sound data received by the server and identifying the type, location information, and direction of travel of the emergency vehicle; means for analyzing the user emotion data received by the server and optimizing a driving assistance message; means for displaying the analysis result and the optimized driving assistance message on the vehicle display; and means for acquiring image data from a 360-degree camera and visually identifying a safe avoidance area from the captured image. This allows even hearing-impaired people to properly recognize the approach of an emergency vehicle and respond safely and quickly.

[0255] "Means for recording ambient sounds" refers to the device's ability to record ambient sounds in and around the vehicle in real time.

[0256] "Means for transmitting recorded voice data and user emotional data to an analysis server" refers to the function of transmitting the voice data collected by the terminal and the user's emotional state to a server via a network.

[0257] "Means having a network module for communicating voice data, emotion data, and analysis results in real time between the analysis server and the user's terminal" refers to a communications infrastructure for sending and receiving data in real time between the terminal and the server.

[0258] "Means for the server to analyze the received voice data and identify the type, location, and direction of travel of the emergency vehicle" refers to the function of the server to identify the sound of an emergency vehicle from the voice data and determine its type, location, and direction of travel.

[0259] "Means for analyzing the user's emotional data received by the server and optimizing driving assistance messages" refers to the function by which the server analyzes the user's emotional state and customizes safety instructions based on that information.

[0260] "Means for displaying the analysis results and optimized driving assistance messages on the vehicle's display" refers to the function of the terminal visually presenting the information obtained from the server on the vehicle's display.

[0261] "Means for acquiring image data from a 360-degree camera and visually identifying safe areas to avoid from the captured image" refers to a function that analyzes images acquired by a camera mounted on a vehicle and displays areas where it is possible to move safely.

[0262] As an embodiment for carrying out the present invention, the specific operation of the system will be described below.

[0263] overview

[0264] The purpose of this invention is to provide a system that allows hearing-impaired drivers to visually recognize approaching emergency vehicles in real time. In this system, a terminal records audio from inside and around the vehicle, and an analysis server analyzes the audio data. Furthermore, the system takes into account the user's emotional state and optimizes driving assistance messages to be displayed on the vehicle's display. This allows hearing-impaired drivers to respond safely and quickly.

[0265] System configuration

[0266] 1. Device (smartphone)

[0267] Audio recording module: Uses the smartphone's microphone to record surrounding audio in real time.

[0268] Network communication module: A module that transmits recorded voice data and user emotion data to the analysis server.

[0269] Display audio linkage module: A module that receives analysis results and driving assistance messages and displays them on the vehicle's display.

[0270] All-around camera image acquisition module (optional): A module that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[0271] Emotion recognition module: Recognizes user emotions in real time using cameras and sensors.

[0272] 2. Server

[0273] Voice data analysis module: Analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle.

[0274] Network communication module: A module that sends analysis results to the terminal.

[0275] Emotion analysis module: Analyzes received user emotion data and optimizes the content of driving assistance messages.

[0276] 3. Users

[0277] Hearing-impaired driver: Check the approach of emergency vehicles through a smartphone and vehicle display and perform appropriate driving operations.

[0278] Program processing

[0279] Terminal

[0280] The device first records the surrounding sounds in real time, and sends the recorded data to an analysis server via a network communication module. Furthermore, the device uses cameras and sensors to recognize the user's emotional state, and also sends this information to the server.

[0281] server

[0282] The server's voice data analysis module analyzes the received voice data and identifies the siren sound of an emergency vehicle. From the identified siren sound, the type of emergency vehicle, its location, and direction of travel are estimated. The analysis results are sent to the device via the network communication module. The emotion analysis module also analyzes the user's emotion data and optimizes the content of the driving assistance message to match the user's emotional state. For example, if the user is stressed, more detailed and careful instructions are provided. If the user is relaxed, concise and smooth instructions are provided.

[0283] Terminal

[0284] The device receives the analysis results from the server and displays the optimized driving assistance message on the vehicle's display via the display audio linkage module. Specifically, the message "Ambulance approaching 30 meters behind to the right" is displayed along with additional instructions tailored to the user's emotions. In addition, by using the all-around camera image acquisition module, safe avoidance areas can also be visually displayed.

[0285] User

[0286] The user checks the vehicle's display and reads driving assistance messages based on the analysis results and their emotions. For example, if the user sees a message saying "An ambulance is approaching 30 meters behind and to the right," they can follow the instructions and safely move the vehicle to the left. In addition, the response instructions are optimized to the user's emotional state, allowing the user to react more appropriately and quickly.

[0287] Specific examples

[0288] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module will record this audio and send it to the server via the network communication module. The server's analysis module will analyze the audio data and identify the location of the emergency vehicle. The analysis result will be "Ambulance approaching 30 meters behind and to the right," which will be sent to the device. At the same time, the emotion analysis module will analyze the user's emotional data, and if the user is feeling stressed, detailed instructions such as "Remain calm and move about 1 meter to the left" will be provided. Conversely, if the user is relaxed, simpler instructions will be given such as "Move to the left."

[0289] Prompt Sentence Examples

[0290] Examples of prompts for a generative AI model might include:

[0291] "What should I do if an ambulance is approaching 30 meters behind me to the right?"

[0292] "Please provide an example of how to instruct an emergency vehicle to evacuate if the user is experiencing stress."

[0293] "Please explain how to display a safe area to avoid using a 360-degree camera image."

[0294] This allows even hearing-impaired people to properly recognize approaching emergency vehicles and receive driving assistance messages tailored to the user's emotional state, enabling them to drive safely and comfortably.

[0295] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0296] Explain the program's processing flow in detail

[0297] Step 1:

[0298] The device uses the smartphone's audio recording module to record surrounding audio in real time.

[0299] Input: Surrounding environmental sounds

[0300] Data processing: Audio filtering (noise removal, volume adjustment)

[0301] Output: filtered audio data

[0302] Step 2:

[0303] The device sends the recorded voice data to the server via the network communication module, and at the same time, the device's camera and sensors capture the user's emotional data, which is also sent to the server.

[0304] Input: Filtered voice data and user emotion data

[0305] Data calculation: Data packing (combining voice data and emotion data)

[0306] Output: Sending consolidated data packets

[0307] Step 3:

[0308] The server's audio data analysis module analyzes the received audio data and identifies the siren sound of an emergency vehicle.

[0309] Input: Voice data in integrated data packets

[0310] Data Computing: Audio Analysis (Digital Signal Processing, Acoustic Pattern Recognition using Machine Learning Algorithms)

[0311] Output: Identification of emergency vehicle type, location, and direction of travel

[0312] Step 4:

[0313] The emotion analysis module of the server analyzes the received user emotion data and generates a driving assistance message.

[0314] Input: Emotion data in a consolidated data packet

[0315] Data calculation: Emotion analysis (facial recognition technology, estimation of emotional state through sensor data analysis)

[0316] Output: Driving assistance messages optimized for the user's emotional state

[0317] Step 5:

[0318] The server transmits the emergency vehicle information obtained as a result of the analysis and the optimized driving assistance message to the terminal via the network communication module.

[0319] Input: Emergency vehicle type, location, direction, optimized driver assistance message

[0320] Data calculation: Data packing (integration of emergency vehicle information and driver assistance messages)

[0321] Output: Sending consolidated data packets

[0322] Step 6:

[0323] The analysis results and driving assistance messages received by the terminal from the server are displayed on the vehicle's display via the display audio linkage module.

[0324] Input: Integrated data packet

[0325] Data processing: Data separation (extraction of emergency vehicle information and driving assistance messages)

[0326] Output: A message such as "Ambulance approaching 30m behind right" displayed on the vehicle's display

[0327] Step 7:

[0328] The user checks the vehicle's display and performs appropriate driving operations in accordance with the displayed analysis results and driving assistance messages.

[0329] Input: Information displayed on the vehicle's display

[0330] Specific operation: Safely evacuate the vehicle based on the analysis results and driver assistance messages

[0331] Output: Avoid danger through safe driving operations

[0332] In this way, by having each processing step work in cooperation with one another, a system is realized that allows the user to recognize the approach of an emergency vehicle in real time and respond safely and quickly.

[0333] (Application example 2)

[0334] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0335] There is a need for a system that enables hearing-impaired people, who have difficulty visually recognizing approaching emergency vehicles, to drive cars safely and comfortably. In particular, there is a technical challenge in providing support messages that adapt to the driver's emotional state when an emergency vehicle is approaching, enabling a quick and appropriate response.

[0336] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0337] In this invention, the server includes: [means for analyzing voice data and generating emergency vehicle location information and response instructions;] [means for generating a driving assistance message adapted based on the user's emotional state; and] [means for transmitting the analysis results and the driving assistance message to the user's terminal in real time.] This allows even hearing-impaired people to properly recognize the approach of an emergency vehicle and receive the optimal driving assistance message according to their emotional state, enabling safe and comfortable driving.

[0338] "Emergency vehicles" refers to ambulances, fire engines, police vehicles, etc. that emit sirens and travel in an emergency.

[0339] "Siren sound" refers to the special warning sound emitted by emergency vehicles.

[0340] "Means for recording" refers to a method of using a microphone to record surrounding sounds as digital audio data.

[0341] "Analysis Server" refers to a network server for processing and analyzing received voice and emotion data.

[0342] "Means for transmitting" refers to the method for transmitting data over a network to other devices or servers.

[0343] "Location Information" means information indicating the real-time geographic location of an emergency vehicle.

[0344] "Response instructions" refers to specific driving instructions to the driver regarding the approach of an emergency vehicle.

[0345] "Display" refers to a screen device installed inside a vehicle that visually displays analysis results and driving assistance messages.

[0346] An "emotion recognition module" refers to a device or system that uses sensors and cameras to detect a user's emotional state and digitize it.

[0347] "Driving assistance messages" refer to messages that provide the driver with instructions or warnings that are adapted to the current driving situation and the user's emotional state based on analyzed data.

[0348] A "surrounding camera" refers to a camera system that captures 360-degree images of the area around the vehicle and acquires video data.

[0349] A "safe avoidance area" refers to a space that allows drivers to safely move their vehicle when an emergency vehicle is approaching.

[0350] "Real-time" refers to a state in which data acquisition, processing, and communication occur instantly.

[0351] "Network module" refers to a hardware and software configuration for communicating data.

[0352] MODE FOR CARRYING OUT THE INVENTION

[0353] The following system will be described as an embodiment of the present invention. This system is equipped with voice recording and analysis functions and emotion recognition functions so that hearing-impaired people can sense the approach of an emergency vehicle with confidence and respond appropriately.

[0354] System Configuration and Operation

[0355] Terminal

[0356] The terminal (smartphone or in-car device) is equipped with the following modules:

[0357] 1. Audio recording module: Uses the smartphone's microphone to record surrounding sounds in real time, and the recorded audio data is saved in digital format.

[0358] 2. Network communication module: Sends recorded audio data to the analysis server via the network.

[0359] 3. Emotion Recognition Module: Using cameras and sensors, it collects the user's facial expressions and behavioral data to recognize their emotional state in real time.

[0360] 4. Display audio link module: Displays the analysis results and driving assistance messages received from the analysis server on the vehicle's display.

[0361] server

[0362] The server comprises several modules that perform the following functions:

[0363] 1. Voice data analysis module: Analyzes the received voice data and identifies the sound of an emergency vehicle siren. This analysis is performed using Google Speech-to-Text and Amazon Transcribe.

[0364] 2. Location information identification module: Identifies the location information of emergency vehicles and generates specific response instructions based on that information.

[0365] 3. Sentiment Analysis Module: Analyzes the emotional data sent by the user and generates driving assistance messages. This analysis is performed using Microsoft Azure Cognitive Services and IBM Watson Tone Analyzer.

[0366] 4. Network communication module: Transmits analysis results and driving assistance messages to the terminal in real time.

[0367] User

[0368] A user (hearing impaired person) uses the system as follows:

[0369] 1. Surrounding sounds are recorded, and when the sound of an emergency vehicle's siren is detected, the vehicle's display will show the vehicle's location and appropriate response instructions.

[0370] 2. Driving assistance messages are provided according to the user's emotional state, allowing the user to receive them intuitively.

[0371] 3. Information from the surrounding camera is also utilized to visually indicate areas where evasive action can be taken safely.

[0372] Specific examples

[0373] For example, if the sound of an emergency vehicle's siren is detected 30 meters behind and to the right, the audio recording module will record this sound. This recorded data is sent to the server via the network communication module, and the server's analysis module will identify the location of the emergency vehicle. The analysis result will be "Ambulance approaching 30 meters behind and to the right," which will be sent to the device. At the same time, the emotion analysis module will analyze the user's emotional data, and if the user is feeling stressed, detailed instructions such as "Remain calm and move about 1 meter to the left" will be provided. Conversely, if the user is relaxed, simpler instructions such as "Move to the left" will be provided.

[0374] Prompt Sentence Examples

[0375] Please provide an overview of a system that detects the approach of emergency vehicles and provides visual warnings to the driver. The system is designed for the hearing impaired, detects emergency vehicle sirens in real time, and displays their location on the vehicle's display. It also recognizes the driver's emotions and optimizes driver assistance messages accordingly. Specifically, please explain what hardware and software are used and what data processing is involved.

[0376] In this way, this invention is a system that provides drivers with optimal information in real time through voice analysis and emotion recognition, enabling hearing-impaired people to drive cars safely and comfortably.

[0377] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0378] Step 1:

[0379] The device uses a voice recording module to record surrounding sounds in real time, and the recorded voice data is stored in digital format.

[0380] Input: Ambient audio

[0381] Output: Digital audio data

[0382] Step 2:

[0383] The recorded voice data is sent to the analysis server via the network communication module.

[0384] Input: Digital audio data

[0385] Output: Audio data sent to the analysis server

[0386] Step 3:

[0387] The server's voice data analysis module analyzes the received voice data and identifies the sound of an emergency vehicle siren, using voice analysis software such as Google Speech-to-Text or Amazon Transcribe.

[0388] Input: Transmitted audio data

[0389] Data processing / calculation: Audio data analysis and siren sound identification

[0390] Output: Audio snippets containing emergency vehicle sirens and their metadata

[0391] Step 4:

[0392] The location information identification module identifies the location information of the emergency vehicle based on the metadata of the identified siren sound.

[0393] Input: Siren sound metadata

[0394] Data processing / calculation: Location information analysis

[0395] Output: Emergency vehicle location information

[0396] Step 5:

[0397] The emotion recognition module uses the device's camera and sensors to collect and analyze the user's facial expressions and behavioral data to recognize the user's emotional state. This data is also sent to the server.

[0398] Input: Camera images and sensor data from the device

[0399] Data processing / calculation: Analysis of user facial expression and behavior data

[0400] Output: User's emotional state data

[0401] Step 6:

[0402] The server's emotion analysis module analyzes the emotion data sent by the user and generates driving assistance messages using tools such as Microsoft Azure Cognitive Services and IBM Watson Tone Analyzer.

[0403] Input: User's emotional state data

[0404] Data processing / calculation: Emotional data analysis and generation of optimal driver assistance messages

[0405] Output: Emotion-based driving assistance message

[0406] Step 7:

[0407] The analysis server compiles the emergency vehicle's location information and the generated driving assistance message and transmits them to the terminal via the network communication module.

[0408] Input: Emergency vehicle location information, driving assistance message

[0409] Data processing / calculation: Integrating location information and messages

[0410] Output: Integrated location and driver assistance messages

[0411] Step 8:

[0412] The terminal displays the emergency vehicle location information and driving assistance message received from the analysis server on the vehicle's display via the display audio link module.

[0413] Input: Integrated location and driver assistance messages

[0414] Output: Emergency vehicle location information and driver assistance messages displayed on the vehicle display

[0415] Step 9:

[0416] The user checks the information displayed on the vehicle display and takes evasive action safely.

[0417] Input: Vehicle display information

[0418] Output: User's safe avoidance action

[0419] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0420] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0421] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0422] [Second embodiment]

[0423] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0424] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0425] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0426] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0427] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0428] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0429] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0430] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0431] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0432] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0433] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0434] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0435] As an embodiment of the present invention, the specific operation of the system will be described below.

[0436] overview

[0437] When driving a car, hearing-impaired people have difficulty visually recognizing approaching emergency vehicles. This invention provides a system that detects the siren sound of an emergency vehicle in real time and displays its location and response instructions on the vehicle's display. The system is mainly composed of a terminal, a server, and a user.

[0438] System configuration

[0439] 1. Device (smartphone)

[0440] Audio recording module: Uses the smartphone's microphone to record surrounding audio in real time.

[0441] Network communication module: A module that sends recorded audio data to the analysis server.

[0442] Display Audio Link Module: A module that receives analysis results and displays them on the vehicle's display.

[0443] All-around camera image acquisition module (optional): A module that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[0444] 2. Server

[0445] Voice data analysis module: Analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle.

[0446] Network communication module: A module that sends analysis results to the terminal.

[0447] 3. Users

[0448] Hearing-impaired driver: Check the approach of emergency vehicles through a smartphone and vehicle display and perform appropriate driving operations.

[0449] Program processing

[0450] Terminal

[0451] The device first records the surrounding sounds in real time, and the recorded data is sent to the analysis server via the network communication module.

[0452] server

[0453] The server's voice data analysis module analyzes the received voice data and identifies the siren sound of an emergency vehicle. From the identified siren sound, the type of emergency vehicle, its location, and direction of travel are estimated. The analysis results are sent to the device via the network communication module.

[0454] Terminal

[0455] The terminal receives the analysis results from the server and displays them on the vehicle's display via the display audio linkage module. For example, specific instructions such as "An ambulance is approaching 30 meters behind and to the right" can be displayed. In addition, by using the all-around camera image acquisition module, safe avoidance areas can also be visually displayed.

[0456] User

[0457] The user checks the vehicle's display and recognizes the approach of an emergency vehicle based on the analysis results. For example, if the display reads "An ambulance is approaching 30 meters behind the vehicle to the right," the user can follow the instructions and safely move the vehicle to the left.

[0458] Specific examples

[0459] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module records this audio and sends it to the server via the network communication module. The server's analysis module analyzes the audio data and identifies the location of the emergency vehicle. The analysis result is generated as "Ambulance approaching 30 meters behind and to the right," which is sent to the terminal. The terminal displays the received information on the vehicle's display. The user confirms this display and moves about 1 meter to the left, ensuring safe driving.

[0460] The above is an embodiment of the present invention, and this system enables even hearing-impaired people to properly recognize approaching emergency vehicles and drive safely.

[0461] The processing flow will be explained below.

[0462] Step 1:

[0463] The device activates the audio recording module, which accesses the device's microphone and begins recording surrounding audio in real time.

[0464] Step 2:

[0465] The device sends the recorded audio data to the server via the network communication module, uploading the audio data approximately every second.

[0466] Step 3:

[0467] The server analyzes the received audio data using the audio data analysis module, and analyzes specific frequency patterns to detect the siren of an emergency vehicle.

[0468] Step 4:

[0469] The server identifies the type and location of the emergency vehicle based on the siren sound detection result, and estimates the direction and distance of the emergency vehicle based on the sound intensity and detection timing.

[0470] Step 5:

[0471] The server generates the analysis results and creates a text message containing the type of emergency vehicle, its location, direction of travel, and any other necessary information.

[0472] Step 6:

[0473] The analysis results generated by the server are sent to the terminal using a network communication module.

[0474] Step 7:

[0475] The analysis results received by the device from the server are displayed on the vehicle's display via the display audio link module. For example, the message "Ambulance approaching 30 meters behind the right" is displayed.

[0476] Step 8:

[0477] (Optional) The device uses the omnidirectional camera image acquisition module to analyze the video from the omnidirectional camera and visually inform the user of a safe area to avoid. For example, it displays an instruction such as "There is room to evacuate about 1 meter to the left."

[0478] Step 9:

[0479] The user checks the vehicle's display and sees an instruction such as "An ambulance is approaching 30 meters behind and to the right."

[0480] Step 10:

[0481] The system performs safe driving maneuvers based on instructions provided by the user. For example, the system follows the instructions on the display to move about 1 meter to the left.

[0482] The above series of processing steps enables hearing-impaired drivers to properly recognize the approach of an emergency vehicle and drive safely.

[0483] Example 1

[0484] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0485] With conventional technology, it has been difficult for hearing-impaired drivers to properly recognize approaching emergency vehicles. In particular, they often miss sirens, which can lead to traffic accidents. The present invention aims to provide a system that supports safe driving by detecting approaching emergency vehicles in real time and allowing even hearing-impaired drivers to visually recognize them.

[0486] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0487] In this invention, the server includes a means for recording surrounding sounds in real time, a means for transmitting the recorded sound data to an analysis server, a means for analyzing the sound data received by the server and estimating the type, location, and direction of travel of the emergency vehicle, and a means for displaying the analysis results on the vehicle display, thereby enabling even hearing-impaired persons to visually recognize the approach of an emergency vehicle and to operate the vehicle safely.

[0488] The "means for recording ambient sound in real time" refers to a device or module that has the function of continuously capturing ambient sound and recording it as digital data.

[0489] The "means for transmitting recorded voice data to an analysis server" refers to a device or module for transferring recorded voice data to another server via a network such as the Internet.

[0490] "Means for analyzing the voice data received by the server and estimating the type, location, and direction of travel of the emergency vehicle" refers to a program or algorithm that has the function of analyzing the voice data received by the server, identifying each emergency vehicle, and determining the location and direction of travel of the sound source.

[0491] The "means for displaying the analysis results on the vehicle display" refers to a device or system for visually displaying the analysis results received from the server on a display device within the vehicle.

[0492] "Means for acquiring image data from a 360-degree camera and visually displaying safe avoidance areas from the captured images" refers to a device or module that has the function of acquiring image data captured by a camera mounted on a vehicle, analyzing the surrounding situation, and displaying areas where it is safe to proceed on a display device inside the vehicle.

[0493] A "network module for communicating voice data and analysis results in real time between an analysis server and a terminal" is a device or software that provides the network communication functions necessary for the analysis server and a terminal to send and receive data in real time.

[0494] MODE FOR CARRYING OUT THE INVENTION

[0495] As an embodiment of the present invention, the specific operation of the system will be described below. The present invention is a system that assists hearing-impaired people in visually recognizing approaching emergency vehicles when driving a car. The system is composed of a terminal, a server, and a user.

[0496] Terminal

[0497] Ambient sound recording

[0498] The device uses a voice recording module to record surrounding sounds in real time through the smartphone's microphone, and the recording data is stored in digital format, for example, using a microphone device with an appropriate sampling rate.

[0499] Sending audio data

[0500] The device transmits the recorded audio data to the analysis server via a network communication module, primarily via an Internet connection, for example via HTTP or HTTPS protocols.

[0501] Receiving and displaying analysis results

[0502] The terminal receives the analysis results from the server and displays them on the vehicle's display via the display audio linkage module. The actual display is linked to the vehicle's information display system and is presented in a format that is intuitively easy for the user to understand.

[0503] Use of a surrounding camera (optional)

[0504] Additionally, the device can utilize a 360-degree camera image acquisition module, which captures and analyzes video data from cameras installed in the vehicle and displays safe areas to avoid.

[0505] server

[0506] Analysis of audio data

[0507] The server uses a voice data analysis module to analyze the received voice data. This analysis can utilize an AI voice recognition engine (e.g., Google Cloud Speech-to-Text). It identifies the sound of an emergency vehicle siren and estimates its type, location, and direction of travel.

[0508] Sending analysis results

[0509] The server then transmits the analysis results to the device in real time. This communication is fast and secure, providing information on approaching emergency vehicles without delay.

[0510] User

[0511] Checking information and driving operations

[0512] The user checks the analysis results displayed on the vehicle's display and recognizes the approach of an emergency vehicle. For example, if the display shows "An ambulance approaching 30 meters behind to the right," the user follows the instructions and safely moves the vehicle to the left.

[0513] Specific examples

[0514] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module records this audio and sends it to the server via the network communication module. The server's audio data analysis module analyzes the audio data and identifies the location information of the emergency vehicle. The analysis result, "Ambulance approaching 30 meters behind and to the right," is generated and sent to the terminal. The terminal displays the received information on the vehicle's display. The user can confirm this display and move about 1 meter to the left, ensuring safe driving.

[0515] Prompt Sentence Examples

[0516] "Detect the sound of an emergency vehicle siren. Display specific instructions to inform the user that an emergency vehicle is approaching 30 meters behind and to the right."

[0517] The above is an embodiment of the present invention, and this system enables even hearing-impaired people to properly recognize the approach of an emergency vehicle and drive safely.

[0518] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0519] Step 1:

[0520] Ambient sound recording

[0521] The device uses the smartphone's audio recording module to record ambient audio in real time. The ambient sound is input from the smartphone's microphone. This audio is stored in a buffer as digitally recorded data. Specifically, the device sets a sampling rate and continuously captures audio and converts it into digital data, which is then used in subsequent processing steps.

[0522] Step 2:

[0523] Sending audio data

[0524] The device sends the recorded audio data to the analysis server via the network communication module. The recorded data buffer is used as input, and the data is sent to the server via an Internet connection. Specifically, the device divides the recorded data into packets at regular time intervals (e.g., every second) and sends them using the HTTP or HTTPS protocol. Once the transmission is complete, the buffer is cleared and prepared for the next data recording.

[0525] Step 3:

[0526] Analysis of audio data

[0527] The server analyzes the received voice data using the voice data analysis module. The voice data received from the device is used as input. Specifically, the server performs noise removal and normalization processing, and inputs this data into an AI voice recognition model. This model identifies the sound of an emergency vehicle siren and estimates the type, location, and direction of travel of the emergency vehicle. The analysis results are generated as JSON format data as output.

[0528] Step 4:

[0529] Sending analysis results

[0530] The server sends the generated analysis results to the terminal through the network communication module. The JSON data of the analysis results is used as input, and the data is sent to the terminal as output. Specifically, the server divides the data into packets and sends them to the terminal's receiving endpoint using the HTTPS protocol. This allows the analysis results to be updated in real time.

[0531] Step 5:

[0532] Receiving and displaying analysis results

[0533] The terminal displays the received analysis results on the vehicle's display via the display audio linkage module. The JSON data of the analysis results received from the server is used as input, and the information to be displayed on the vehicle's display is generated as output. Specifically, the terminal decodes the analysis results and converts them into an appropriate display format. For example, a message such as "Ambulance approaching 30 meters behind to the right" is displayed on the display.

[0534] Step 6:

[0535] Use of a surrounding camera (optional)

[0536] The device uses a 360-degree camera image acquisition module to acquire video data from the vehicle's onboard camera. It uses the camera's video data as input and generates information to display safe areas for avoidance as output. Specifically, the device analyzes the camera video in real time to identify the direction of travel and areas that can be avoided. This information is then visually displayed on the display.

[0537] Step 7:

[0538] Checking information and driving operations

[0539] The user checks the analysis results displayed on the vehicle's display and recognizes the approach of an emergency vehicle. The information displayed on the vehicle's display is used as input, and the user's driving operation is used as output. Specifically, the user checks the message "An ambulance is approaching 30 meters behind and to the right" and takes action to safely move the vehicle to the left. In this case, the user references the images from the all-around camera and the displayed information to safely avoid the approach.

[0540] (Application example 1)

[0541] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0542] It is difficult for the hearing impaired to visually recognize approaching emergency vehicles. This is particularly true when using autonomous vehicles, where a driver's failure to notice an approaching emergency vehicle can pose a serious risk. When such obstacles exist, effective evasive measures must be implemented quickly. However, current visual information systems lack sufficient functionality to provide real-time information on the location and direction of emergency vehicles.

[0543] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0544] In this invention, the server includes means for recording the siren sound of the emergency vehicle, means for transmitting the recorded audio data to an analysis server, means for the server to analyze the audio data received and generate emergency vehicle position information and response instructions, means for displaying the emergency vehicle position information and response instructions generated as a result of the analysis on a display of the vehicle, and means for displaying information about the approach of the emergency vehicle on a head-mounted display, thereby enabling hearing-impaired people to visually recognize the approach of an emergency vehicle in real time and take appropriate evasive action.

[0545] The "means for recording the siren sound of an emergency vehicle" is a device that records sounds in the surrounding environment of a vehicle and captures the siren sound of an emergency vehicle in real time.

[0546] The "means for transmitting recorded voice data to the analysis server" is a communication module for transmitting recorded voice data to the analysis server via a communication line such as the Internet.

[0547] "Means for analyzing the voice data received by the server and generating emergency vehicle location information and response instructions" refers to software or hardware that enables the server to analyze the voice data, identify the location and direction of the emergency vehicle, and provide instructions to the driver based on that information.

[0548] The "means for displaying the emergency vehicle location information and response instructions generated as the analysis result on the vehicle display" refers to an interface and display device for visually displaying the analysis result on the display inside the vehicle.

[0549] "Means for acquiring image data from a 360-degree camera and visually identifying safe avoidance areas from the captured image" refers to a device that analyzes images from a vehicle's 360-degree camera and visually identifies areas that can be safely avoided.

[0550] "Means having a network module for communicating voice data and analysis results in real time between the analysis server and the user's terminal" refers to a communication device for realizing data communication in real time between the analysis server and the vehicle's terminal.

[0551] The "means for displaying information about approaching emergency vehicles on a head-mounted display" is a device for displaying emergency vehicle location information and avoidance instructions on a head-mounted display.

[0552] As an embodiment of the present invention, the specific operation of the system will be described below.

[0553] overview

[0554] This system provides assistance to hearing-impaired people using self-driving vehicles, helping them visually recognize approaching emergency vehicles. The system consists of a terminal, a server, and a user. Terminals refer to smartphones, displays, head-mounted displays (HMDs), etc. installed in self-driving vehicles. The server analyzes the audio data and generates emergency vehicle location information and avoidance instructions. The user then performs safe driving operations based on the information provided visually.

[0555] System configuration

[0556] 1. Terminal (installed in autonomous vehicles)

[0557] Audio recording module: A device that uses the device's microphone to record surrounding audio in real time.

[0558] Network communication module: A device that sends recorded audio data to a server for analysis.

[0559] Display audio link module: A device that displays the analysis results received from the server on the display or HMD of the autonomous vehicle.

[0560] All-around camera image acquisition module (optional): A device that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[0561] 2. Server

[0562] Voice data analysis module: This module analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle. This analysis uses a deep learning model (for example, a model built with TensorFlow or PyTorch).

[0563] Network communication module: A device that transmits analysis results to the terminal.

[0564] 3. Users

[0565] Hearing-impaired person (driver): Check the approach of emergency vehicles through the self-driving vehicle's display or HMD and perform appropriate driving operations.

[0566] Program processing explanation

[0567] The device first records the surrounding sounds in real time, and the recorded data is sent to the analysis server via the network communication module.

[0568] The server analyzes the received audio data and identifies the siren sound of an emergency vehicle. From the identified siren sound, it estimates the type of emergency vehicle, its location, and its direction of travel. This analysis is performed using deep learning techniques (e.g., TensorFlow, PyTorch). The results are sent to the device using a network communication module.

[0569] The terminal receives the analysis results from the server and displays them on the autonomous vehicle's display or HMD via the display audio linkage module. For example, specific instructions such as "An ambulance is approaching 30 meters behind and to the right" can be displayed. In addition, by using the all-around camera image acquisition module, safe avoidance areas can also be visually displayed.

[0570] Users can check the autonomous vehicle's display or HMD, recognize the approach of an emergency vehicle based on the analysis results, and take appropriate evasive action. For example, if a user sees a message saying "An ambulance is approaching 30 meters behind and to the right," they will take action by moving about one meter to the left.

[0571] Specific examples

[0572] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module records this audio and sends it to the server via the network communication module. The server's analysis module analyzes the audio data and identifies the location information of the emergency vehicle. The analysis result is generated as "Ambulance approaching 30 meters behind and to the right," which is sent to the terminal. The terminal displays the received information on the display or HMD of the autonomous vehicle. The user confirms this display and moves about 1 meter to the left, ensuring safe driving.

[0573] Prompt Sentence Examples

[0574] For example, a prompt to a generative AI model might look like this:

[0575] "If an emergency vehicle siren is detected, generate a message with specific location and instructions on how to avoid it."

[0576] The above is an embodiment of the present invention, and this system enables even hearing-impaired people to properly recognize the approach of an emergency vehicle and drive safely.

[0577] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0578] Step 1:

[0579] The device records the surrounding audio. While the user is driving the autonomous vehicle, the audio recording module is activated when the engine starts and records the audio around the vehicle in real time. The input is the audio data of the surrounding environment, and the output is the recorded audio data.

[0580] Step 2:

[0581] The device sends the recorded voice data to the analysis server. The network communication module is used to upload the recorded voice data to the server via the Internet. The input is the recorded voice data, and the output is the voice data sent to the server.

[0582] Step 3:

[0583] The server analyzes the received audio data and identifies the siren sound of an emergency vehicle. The audio data analysis module uses deep learning technology to extract the characteristics of the siren sound and detect the approach of an emergency vehicle. The input is the audio data sent from the device, and the output is the siren sound detection result.

[0584] Step 4:

[0585] The server estimates the type, location, and direction of travel of the emergency vehicle based on the detection results. Based on the analysis results, the server identifies the emergency vehicle's detailed location and direction of travel, and generates corresponding driving instructions. The input is the detection result of the siren sound, and the output is data on the emergency vehicle's location and direction of travel.

[0586] Step 5:

[0587] The server generates analysis results and sends them to the terminal. The network communication module is used to return the analysis results to the terminal in real time. The input is data on the emergency vehicle's location and direction of travel, and the output is the analysis results sent to the terminal.

[0588] Step 6:

[0589] The terminal receives the analysis results and displays them on the autonomous vehicle's display or HMD. The display audio linkage module is used to visually display the emergency vehicle's location and response instructions. The input is the analysis results sent from the server, and the output is the emergency vehicle's location information and response instructions displayed on the display or HMD.

[0590] Step 7:

[0591] The user performs appropriate driving operations based on the information displayed on the display or HMD. Check the displayed information about approaching emergency vehicles and follow the instructions to safely avoid the vehicle. The input is the analysis results displayed on the display or HMD, and the output is the user's driving operations.

[0592] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0593] As an embodiment of the present invention, the specific operation of the system will be described below.

[0594] overview

[0595] When driving a car, hearing-impaired people have difficulty visually recognizing approaching emergency vehicles. This invention provides a system that detects emergency vehicle sirens in real time and displays their location information and response instructions on the vehicle's display. In addition, by combining it with an emotion engine that recognizes the user's emotions, driving assistance messages are optimized to the user's emotional state, supporting safe and comfortable driving.

[0596] System configuration

[0597] 1. Device (smartphone)

[0598] Audio recording module: Uses the smartphone's microphone to record surrounding audio in real time.

[0599] Network communication module: A module that sends recorded audio data to the analysis server.

[0600] Display Audio Link Module: A module that receives analysis results and displays them on the vehicle's display.

[0601] All-around camera image acquisition module (optional): A module that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[0602] Emotion recognition module: Recognizes user emotions in real time using cameras and sensors.

[0603] 2. Server

[0604] Voice data analysis module: Analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle.

[0605] Network communication module: A module that sends analysis results to the terminal.

[0606] Emotion analysis module: Analyzes received user emotion data and optimizes the content of driving assistance messages.

[0607] 3. Users

[0608] Hearing-impaired driver: Check the approach of emergency vehicles through a smartphone and vehicle display and perform appropriate driving operations.

[0609] Program processing

[0610] Terminal

[0611] The device first records the surrounding sounds in real time, and sends the recorded data to an analysis server via a network communication module. The device then uses cameras and sensors to recognize the user's emotional state, which is also sent to the server.

[0612] server

[0613] The server's voice data analysis module analyzes the received voice data and identifies the siren sound of an emergency vehicle. From the identified siren sound, the type of emergency vehicle, its location, and direction of travel are estimated. The analysis results are sent to the device via the network communication module. The emotion analysis module also analyzes the user's emotion data and optimizes the content of the driving assistance message to match the user's emotional state. For example, if the user is feeling stressed, more detailed and careful instructions are provided. If the user is relaxed, concise and smooth instructions are provided.

[0614] Terminal

[0615] The device receives analysis results from the server and displays driving assistance messages based on the driver's emotions on the vehicle's display via the display audio linkage module. Specifically, the device displays the message "An ambulance is approaching 30 meters behind the driver on the right" along with additional instructions tailored to the user's emotions. Safe avoidance areas can also be visually displayed by using the all-around camera image acquisition module.

[0616] User

[0617] The user checks the vehicle's display and reads driving assistance messages based on the analysis results and their emotions. For example, if the user sees a message saying "An ambulance is approaching 30 meters behind and to the right," they can follow the instructions and safely move the vehicle to the left. In addition, the response instructions are optimized to the user's emotional state, allowing the user to react more appropriately and quickly.

[0618] Specific examples

[0619] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module will record this audio and send it to the server via the network communication module. The server's analysis module will analyze the audio data and identify the location of the emergency vehicle. The analysis result will be "Ambulance approaching 30 meters behind and to the right," which will be sent to the device. At the same time, the emotion analysis module will analyze the user's emotional data, and if the user is feeling stressed, detailed instructions such as "Remain calm and move about 1 meter to the left" will be provided. Conversely, if the user is relaxed, simpler instructions will be given such as "Move to the left."

[0620] The above is a form for implementing the present invention, and this system allows even hearing-impaired people to properly recognize the approach of an emergency vehicle and receive driving assistance messages tailored to the user's emotional state, allowing them to drive safely and comfortably.

[0621] The processing flow will be explained below.

[0622] Step 1:

[0623] The device activates the audio recording module, which accesses the device's microphone and begins recording surrounding audio in real time.

[0624] Step 2:

[0625] The device sends the recorded audio data to the server via the network communication module, uploading the audio data approximately every second.

[0626] Step 3:

[0627] The server analyzes the received audio data using the audio data analysis module, and analyzes specific frequency patterns to detect the siren of an emergency vehicle.

[0628] Step 4:

[0629] The server identifies the type and location of the emergency vehicle based on the siren sound detection result, and estimates the direction and distance of the emergency vehicle based on the sound intensity and detection timing.

[0630] Step 5:

[0631] The server generates the analysis results and creates a text message containing the type of emergency vehicle, its location, direction of travel, and any other necessary information.

[0632] Step 6:

[0633] The device uses cameras and sensors to recognize the user's emotional state, and sends the recognized emotional data to a server.

[0634] Step 7:

[0635] The emotion data received by the server is analyzed by the emotion analysis module to determine whether the user is feeling stressed or anxious, or relaxed.

[0636] Step 8:

[0637] The server optimizes driving assistance messages based on the driver's emotional state, for example generating detailed and careful instructions when the driver is stressed and concise and smooth instructions when the driver is relaxed.

[0638] Step 9:

[0639] The analysis results and optimized driving assistance messages generated by the server are sent to the terminal using a network communication module.

[0640] Step 10:

[0641] The terminal receives the analysis results and driving assistance messages from the server and displays them on the vehicle's display via the display audio linkage module. For example, along with the message "An ambulance is approaching 30 meters behind and to the right," detailed instructions such as "Remain calm and move about 1 meter to the left" may be displayed.

[0642] Step 11:

[0643] (Optional) The device uses the omnidirectional camera image acquisition module to analyze the video from the omnidirectional camera and visually inform the user of a safe area to avoid. For example, it displays an instruction such as "There is room to evacuate about 1 meter to the left."

[0644] Step 12:

[0645] The user checks the vehicle's display and sees an instruction such as "An ambulance is approaching 30 meters behind and to the right."

[0646] Step 13:

[0647] The system performs safe driving maneuvers based on instructions provided by the user. For example, the system follows the instructions on the display to move about 1 meter to the left.

[0648] The above series of processing steps enables hearing-impaired drivers to properly recognize approaching emergency vehicles and drive safely. In addition, the emotion engine optimizes driving assistance messages, providing detailed assistance tailored to the user's mental state.

[0649] Example 2

[0650] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0651] When hearing-impaired people drive, it is difficult for them to visually recognize approaching emergency vehicles. This makes it difficult for them to quickly take appropriate evasive action, making safety a challenge. Therefore, there is a need for a system that can recognize approaching emergency vehicles in real time and provide appropriate support messages according to the driver's emotional state.

[0652] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for recording surrounding sounds; means for transmitting the recorded sound data and user emotion data to the analysis server; means having a network module for communicating the sound data, emotion data, and analysis results in real time between the analysis server and the user's terminal; means for analyzing the sound data received by the server and identifying the type, location information, and direction of travel of the emergency vehicle; means for analyzing the user emotion data received by the server and optimizing a driving assistance message; means for displaying the analysis result and the optimized driving assistance message on the vehicle display; and means for acquiring image data from a 360-degree camera and visually identifying a safe avoidance area from the captured image. This allows even hearing-impaired people to properly recognize the approach of an emergency vehicle and respond safely and quickly.

[0653] "Means for recording ambient sounds" refers to the device's ability to record ambient sounds in and around the vehicle in real time.

[0654] "Means for transmitting recorded voice data and user emotional data to an analysis server" refers to the function of transmitting the voice data collected by the terminal and the user's emotional state to a server via a network.

[0655] "Means having a network module for communicating voice data, emotion data, and analysis results in real time between the analysis server and the user's terminal" refers to a communications infrastructure for sending and receiving data in real time between the terminal and the server.

[0656] "Means for the server to analyze the received voice data and identify the type, location, and direction of travel of the emergency vehicle" refers to the function of the server to identify the sound of an emergency vehicle from the voice data and determine its type, location, and direction of travel.

[0657] "Means for analyzing the user's emotional data received by the server and optimizing driving assistance messages" refers to the function by which the server analyzes the user's emotional state and customizes safety instructions based on that information.

[0658] "Means for displaying the analysis results and optimized driving assistance messages on the vehicle's display" refers to the function of the terminal visually presenting the information obtained from the server on the vehicle's display.

[0659] "Means for acquiring image data from a 360-degree camera and visually identifying safe areas to avoid from the captured image" refers to a function that analyzes images acquired by a camera mounted on a vehicle and displays areas where it is possible to move safely.

[0660] As an embodiment for carrying out the present invention, the specific operation of the system will be described below.

[0661] overview

[0662] The purpose of this invention is to provide a system that allows hearing-impaired drivers to visually recognize approaching emergency vehicles in real time. In this system, a terminal records audio from inside and around the vehicle, and an analysis server analyzes the audio data. Furthermore, the system takes into account the user's emotional state and optimizes driving assistance messages to be displayed on the vehicle's display. This allows hearing-impaired drivers to respond safely and quickly.

[0663] System configuration

[0664] 1. Device (smartphone)

[0665] Audio recording module: Uses the smartphone's microphone to record surrounding audio in real time.

[0666] Network communication module: A module that transmits recorded voice data and user emotion data to the analysis server.

[0667] Display audio linkage module: A module that receives analysis results and driving assistance messages and displays them on the vehicle's display.

[0668] All-around camera image acquisition module (optional): A module that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[0669] Emotion recognition module: Recognizes user emotions in real time using cameras and sensors.

[0670] 2. Server

[0671] Voice data analysis module: Analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle.

[0672] Network communication module: A module that sends analysis results to the terminal.

[0673] Emotion analysis module: Analyzes received user emotion data and optimizes the content of driving assistance messages.

[0674] 3. Users

[0675] Hearing-impaired driver: Check the approach of emergency vehicles through a smartphone and vehicle display and perform appropriate driving operations.

[0676] Program processing

[0677] Terminal

[0678] The device first records the surrounding sounds in real time, and sends the recorded data to an analysis server via a network communication module. Furthermore, the device uses cameras and sensors to recognize the user's emotional state, and also sends this information to the server.

[0679] server

[0680] The server's voice data analysis module analyzes the received voice data and identifies the siren sound of an emergency vehicle. From the identified siren sound, the type of emergency vehicle, its location, and direction of travel are estimated. The analysis results are sent to the device via the network communication module. The emotion analysis module also analyzes the user's emotion data and optimizes the content of the driving assistance message to match the user's emotional state. For example, if the user is stressed, more detailed and careful instructions are provided. If the user is relaxed, concise and smooth instructions are provided.

[0681] Terminal

[0682] The device receives the analysis results from the server and displays the optimized driving assistance message on the vehicle's display via the display audio linkage module. Specifically, the message "Ambulance approaching 30 meters behind to the right" is displayed along with additional instructions tailored to the user's emotions. In addition, by using the all-around camera image acquisition module, safe avoidance areas can also be visually displayed.

[0683] User

[0684] The user checks the vehicle's display and reads driving assistance messages based on the analysis results and their emotions. For example, if the user sees a message saying "An ambulance is approaching 30 meters behind and to the right," they can follow the instructions and safely move the vehicle to the left. In addition, the response instructions are optimized to the user's emotional state, allowing the user to react more appropriately and quickly.

[0685] Specific examples

[0686] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module will record this audio and send it to the server via the network communication module. The server's analysis module will analyze the audio data and identify the location of the emergency vehicle. The analysis result will be "Ambulance approaching 30 meters behind and to the right," which will be sent to the device. At the same time, the emotion analysis module will analyze the user's emotional data, and if the user is feeling stressed, detailed instructions such as "Remain calm and move about 1 meter to the left" will be provided. Conversely, if the user is relaxed, simpler instructions will be given such as "Move to the left."

[0687] Prompt Sentence Examples

[0688] Examples of prompts for a generative AI model might include:

[0689] "What should I do if an ambulance is approaching 30 meters behind me to the right?"

[0690] "Please provide an example of how to instruct an emergency vehicle to evacuate if the user is experiencing stress."

[0691] "Please explain how to display a safe area to avoid using a 360-degree camera image."

[0692] This allows even hearing-impaired people to properly recognize approaching emergency vehicles and receive driving assistance messages tailored to the user's emotional state, enabling them to drive safely and comfortably.

[0693] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0694] Explain the program's processing flow in detail

[0695] Step 1:

[0696] The device uses the smartphone's audio recording module to record surrounding audio in real time.

[0697] Input: Surrounding environmental sounds

[0698] Data processing: Audio filtering (noise removal, volume adjustment)

[0699] Output: filtered audio data

[0700] Step 2:

[0701] The device sends the recorded voice data to the server via the network communication module, and at the same time, the device's camera and sensors capture the user's emotional data, which is also sent to the server.

[0702] Input: Filtered voice data and user emotion data

[0703] Data calculation: Data packing (combining voice data and emotion data)

[0704] Output: Sending consolidated data packets

[0705] Step 3:

[0706] The server's audio data analysis module analyzes the received audio data and identifies the siren sound of an emergency vehicle.

[0707] Input: Voice data in integrated data packets

[0708] Data Computing: Audio Analysis (Digital Signal Processing, Acoustic Pattern Recognition using Machine Learning Algorithms)

[0709] Output: Identification of emergency vehicle type, location, and direction of travel

[0710] Step 4:

[0711] The emotion analysis module of the server analyzes the received user emotion data and generates a driving assistance message.

[0712] Input: Emotion data in a consolidated data packet

[0713] Data calculation: Emotion analysis (facial recognition technology, estimation of emotional state through sensor data analysis)

[0714] Output: Driving assistance messages optimized for the user's emotional state

[0715] Step 5:

[0716] The server transmits the emergency vehicle information obtained as a result of the analysis and the optimized driving assistance message to the terminal via the network communication module.

[0717] Input: Emergency vehicle type, location, direction, optimized driver assistance message

[0718] Data calculation: Data packing (integration of emergency vehicle information and driver assistance messages)

[0719] Output: Sending consolidated data packets

[0720] Step 6:

[0721] The analysis results and driving assistance messages received by the terminal from the server are displayed on the vehicle's display via the display audio linkage module.

[0722] Input: Integrated data packet

[0723] Data processing: Data separation (extraction of emergency vehicle information and driving assistance messages)

[0724] Output: A message such as "Ambulance approaching 30m behind right" displayed on the vehicle's display

[0725] Step 7:

[0726] The user checks the vehicle's display and performs appropriate driving operations in accordance with the displayed analysis results and driving assistance messages.

[0727] Input: Information displayed on the vehicle's display

[0728] Specific operation: Safely evacuate the vehicle based on the analysis results and driver assistance messages

[0729] Output: Avoid danger through safe driving operations

[0730] In this way, by having each processing step work in cooperation with one another, a system is realized that allows the user to recognize the approach of an emergency vehicle in real time and respond safely and quickly.

[0731] (Application example 2)

[0732] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0733] There is a need for a system that enables hearing-impaired people, who have difficulty visually recognizing approaching emergency vehicles, to drive cars safely and comfortably. In particular, there is a technical challenge in providing support messages that adapt to the driver's emotional state when an emergency vehicle is approaching, enabling a quick and appropriate response.

[0734] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0735] In this invention, the server includes: [means for analyzing voice data and generating emergency vehicle location information and response instructions;] [means for generating a driving assistance message adapted based on the user's emotional state; and] [means for transmitting the analysis results and the driving assistance message to the user's terminal in real time.] This allows even hearing-impaired people to properly recognize the approach of an emergency vehicle and receive the optimal driving assistance message according to their emotional state, enabling safe and comfortable driving.

[0736] "Emergency vehicles" refers to ambulances, fire engines, police vehicles, etc. that emit sirens and travel in an emergency.

[0737] "Siren sound" refers to the special warning sound emitted by emergency vehicles.

[0738] "Means for recording" refers to a method of using a microphone to record surrounding sounds as digital audio data.

[0739] "Analysis Server" refers to a network server for processing and analyzing received voice and emotion data.

[0740] "Means for transmitting" refers to the method for transmitting data over a network to other devices or servers.

[0741] "Location Information" means information indicating the real-time geographic location of an emergency vehicle.

[0742] "Response instructions" refers to specific driving instructions to the driver regarding the approach of an emergency vehicle.

[0743] "Display" refers to a screen device installed inside a vehicle that visually displays analysis results and driving assistance messages.

[0744] An "emotion recognition module" refers to a device or system that uses sensors and cameras to detect a user's emotional state and digitize it.

[0745] "Driving assistance messages" refer to messages that provide the driver with instructions or warnings that are adapted to the current driving situation and the user's emotional state based on analyzed data.

[0746] A "surrounding camera" refers to a camera system that captures 360-degree images of the area around the vehicle and acquires video data.

[0747] A "safe avoidance area" refers to a space that allows drivers to safely move their vehicle when an emergency vehicle is approaching.

[0748] "Real-time" refers to a state in which data acquisition, processing, and communication occur instantly.

[0749] "Network module" refers to a hardware and software configuration for communicating data.

[0750] MODE FOR CARRYING OUT THE INVENTION

[0751] The following system will be described as an embodiment of the present invention. This system is equipped with voice recording and analysis functions and emotion recognition functions so that hearing-impaired people can sense the approach of an emergency vehicle with confidence and respond appropriately.

[0752] System Configuration and Operation

[0753] Terminal

[0754] The terminal (smartphone or in-car device) is equipped with the following modules:

[0755] 1. Audio recording module: Uses the smartphone's microphone to record surrounding sounds in real time, and the recorded audio data is saved in digital format.

[0756] 2. Network communication module: Sends recorded audio data to the analysis server via the network.

[0757] 3. Emotion Recognition Module: Using cameras and sensors, it collects the user's facial expressions and behavioral data to recognize their emotional state in real time.

[0758] 4. Display audio link module: Displays the analysis results and driving assistance messages received from the analysis server on the vehicle's display.

[0759] server

[0760] The server comprises several modules that perform the following functions:

[0761] 1. Voice data analysis module: Analyzes the received voice data and identifies the sound of an emergency vehicle siren. This analysis is performed using Google Speech-to-Text and Amazon Transcribe.

[0762] 2. Location information identification module: Identifies the location information of emergency vehicles and generates specific response instructions based on that information.

[0763] 3. Sentiment Analysis Module: Analyzes the emotional data sent by the user and generates driving assistance messages. This analysis is performed using Microsoft Azure Cognitive Services and IBM Watson Tone Analyzer.

[0764] 4. Network communication module: Transmits analysis results and driving assistance messages to the terminal in real time.

[0765] User

[0766] A user (hearing impaired person) uses the system as follows:

[0767] 1. Surrounding sounds are recorded, and when the sound of an emergency vehicle's siren is detected, the vehicle's display will show the vehicle's location and appropriate response instructions.

[0768] 2. Driving assistance messages are provided according to the user's emotional state, allowing the user to receive them intuitively.

[0769] 3. Information from the surrounding camera is also utilized to visually indicate areas where evasive action can be taken safely.

[0770] Specific examples

[0771] For example, if the sound of an emergency vehicle's siren is detected 30 meters behind and to the right, the audio recording module will record this sound. This recorded data is sent to the server via the network communication module, and the server's analysis module will identify the location of the emergency vehicle. The analysis result will be "Ambulance approaching 30 meters behind and to the right," which will be sent to the device. At the same time, the emotion analysis module will analyze the user's emotional data, and if the user is feeling stressed, detailed instructions such as "Remain calm and move about 1 meter to the left" will be provided. Conversely, if the user is relaxed, simpler instructions such as "Move to the left" will be provided.

[0772] Prompt Sentence Examples

[0773] Please provide an overview of a system that detects the approach of emergency vehicles and provides visual warnings to the driver. The system is designed for the hearing impaired, detects emergency vehicle sirens in real time, and displays their location on the vehicle's display. It also recognizes the driver's emotions and optimizes driver assistance messages accordingly. Specifically, please explain what hardware and software are used and what data processing is involved.

[0774] In this way, this invention is a system that provides drivers with optimal information in real time through voice analysis and emotion recognition, enabling hearing-impaired people to drive cars safely and comfortably.

[0775] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0776] Step 1:

[0777] The device uses a voice recording module to record surrounding sounds in real time, and the recorded voice data is stored in digital format.

[0778] Input: Ambient audio

[0779] Output: Digital audio data

[0780] Step 2:

[0781] The recorded voice data is sent to the analysis server via the network communication module.

[0782] Input: Digital audio data

[0783] Output: Audio data sent to the analysis server

[0784] Step 3:

[0785] The server's voice data analysis module analyzes the received voice data and identifies the sound of an emergency vehicle siren, using voice analysis software such as Google Speech-to-Text or Amazon Transcribe.

[0786] Input: Transmitted audio data

[0787] Data processing / calculation: Audio data analysis and siren sound identification

[0788] Output: Audio snippets containing emergency vehicle sirens and their metadata

[0789] Step 4:

[0790] The location information identification module identifies the location information of the emergency vehicle based on the metadata of the identified siren sound.

[0791] Input: Siren sound metadata

[0792] Data processing / calculation: Location information analysis

[0793] Output: Emergency vehicle location information

[0794] Step 5:

[0795] The emotion recognition module uses the device's camera and sensors to collect and analyze the user's facial expressions and behavioral data to recognize the user's emotional state. This data is also sent to the server.

[0796] Input: Camera images and sensor data from the device

[0797] Data processing / calculation: Analysis of user facial expression and behavior data

[0798] Output: User's emotional state data

[0799] Step 6:

[0800] The server's emotion analysis module analyzes the emotion data sent by the user and generates driving assistance messages using tools such as Microsoft Azure Cognitive Services and IBM Watson Tone Analyzer.

[0801] Input: User's emotional state data

[0802] Data processing / calculation: Emotional data analysis and generation of optimal driver assistance messages

[0803] Output: Emotion-based driving assistance message

[0804] Step 7:

[0805] The analysis server compiles the emergency vehicle's location information and the generated driving assistance message and transmits them to the terminal via the network communication module.

[0806] Input: Emergency vehicle location information, driving assistance message

[0807] Data processing / calculation: Integrating location information and messages

[0808] Output: Integrated location and driver assistance messages

[0809] Step 8:

[0810] The terminal displays the emergency vehicle location information and driving assistance message received from the analysis server on the vehicle's display via the display audio link module.

[0811] Input: Integrated location and driver assistance messages

[0812] Output: Emergency vehicle location information and driver assistance messages displayed on the vehicle display

[0813] Step 9:

[0814] The user checks the information displayed on the vehicle display and takes evasive action safely.

[0815] Input: Vehicle display information

[0816] Output: User's safe avoidance action

[0817] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0818] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0819] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0820] [Third embodiment]

[0821] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0822] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0823] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0824] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0825] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0826] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0827] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0828] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0829] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0830] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0831] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0832] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0833] As an embodiment of the present invention, the specific operation of the system will be described below.

[0834] overview

[0835] When driving a car, hearing-impaired people have difficulty visually recognizing approaching emergency vehicles. This invention provides a system that detects the siren sound of an emergency vehicle in real time and displays its location and response instructions on the vehicle's display. The system is mainly composed of a terminal, a server, and a user.

[0836] System configuration

[0837] 1. Device (smartphone)

[0838] Audio recording module: Uses the smartphone's microphone to record surrounding audio in real time.

[0839] Network communication module: A module that sends recorded audio data to the analysis server.

[0840] Display Audio Link Module: A module that receives analysis results and displays them on the vehicle's display.

[0841] All-around camera image acquisition module (optional): A module that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[0842] 2. Server

[0843] Voice data analysis module: Analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle.

[0844] Network communication module: A module that sends analysis results to the terminal.

[0845] 3. Users

[0846] Hearing-impaired driver: Check the approach of emergency vehicles through a smartphone and vehicle display and perform appropriate driving operations.

[0847] Program processing

[0848] Terminal

[0849] The device first records the surrounding sounds in real time, and the recorded data is sent to the analysis server via the network communication module.

[0850] server

[0851] The server's voice data analysis module analyzes the received voice data and identifies the siren sound of an emergency vehicle. From the identified siren sound, the type of emergency vehicle, its location, and direction of travel are estimated. The analysis results are sent to the device via the network communication module.

[0852] Terminal

[0853] The terminal receives the analysis results from the server and displays them on the vehicle's display via the display audio linkage module. For example, specific instructions such as "An ambulance is approaching 30 meters behind and to the right" can be displayed. In addition, by using the all-around camera image acquisition module, safe avoidance areas can also be visually displayed.

[0854] User

[0855] The user checks the vehicle's display and recognizes the approach of an emergency vehicle based on the analysis results. For example, if the display reads "An ambulance is approaching 30 meters behind the vehicle to the right," the user can follow the instructions and safely move the vehicle to the left.

[0856] Specific examples

[0857] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module records this audio and sends it to the server via the network communication module. The server's analysis module analyzes the audio data and identifies the location of the emergency vehicle. The analysis result is generated as "Ambulance approaching 30 meters behind and to the right," which is sent to the terminal. The terminal displays the received information on the vehicle's display. The user confirms this display and moves about 1 meter to the left, ensuring safe driving.

[0858] The above is an embodiment of the present invention, and this system enables even hearing-impaired people to properly recognize approaching emergency vehicles and drive safely.

[0859] The processing flow will be explained below.

[0860] Step 1:

[0861] The device activates the audio recording module, which accesses the device's microphone and begins recording surrounding audio in real time.

[0862] Step 2:

[0863] The device sends the recorded audio data to the server via the network communication module, uploading the audio data approximately every second.

[0864] Step 3:

[0865] The server analyzes the received audio data using the audio data analysis module, and analyzes specific frequency patterns to detect the siren of an emergency vehicle.

[0866] Step 4:

[0867] The server identifies the type and location of the emergency vehicle based on the siren sound detection result, and estimates the direction and distance of the emergency vehicle based on the sound intensity and detection timing.

[0868] Step 5:

[0869] The server generates the analysis results and creates a text message containing the type of emergency vehicle, its location, direction of travel, and any other necessary information.

[0870] Step 6:

[0871] The analysis results generated by the server are sent to the terminal using a network communication module.

[0872] Step 7:

[0873] The analysis results received by the device from the server are displayed on the vehicle's display via the display audio link module. For example, the message "Ambulance approaching 30 meters behind the right" is displayed.

[0874] Step 8:

[0875] (Optional) The device uses the omnidirectional camera image acquisition module to analyze the video from the omnidirectional camera and visually inform the user of a safe area to avoid. For example, it displays an instruction such as "There is room to evacuate about 1 meter to the left."

[0876] Step 9:

[0877] The user checks the vehicle's display and sees an instruction such as "An ambulance is approaching 30 meters behind and to the right."

[0878] Step 10:

[0879] The system performs safe driving maneuvers based on instructions provided by the user. For example, the system follows the instructions on the display to move about 1 meter to the left.

[0880] The above series of processing steps enables hearing-impaired drivers to properly recognize the approach of an emergency vehicle and drive safely.

[0881] Example 1

[0882] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0883] With conventional technology, it has been difficult for hearing-impaired drivers to properly recognize approaching emergency vehicles. In particular, they often miss sirens, which can lead to traffic accidents. The present invention aims to provide a system that supports safe driving by detecting approaching emergency vehicles in real time and allowing even hearing-impaired drivers to visually recognize them.

[0884] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0885] In this invention, the server includes a means for recording surrounding sounds in real time, a means for transmitting the recorded sound data to an analysis server, a means for analyzing the sound data received by the server and estimating the type, location, and direction of travel of the emergency vehicle, and a means for displaying the analysis results on the vehicle display, thereby enabling even hearing-impaired persons to visually recognize the approach of an emergency vehicle and to operate the vehicle safely.

[0886] The "means for recording ambient sound in real time" refers to a device or module that has the function of continuously capturing ambient sound and recording it as digital data.

[0887] The "means for transmitting recorded voice data to an analysis server" refers to a device or module for transferring recorded voice data to another server via a network such as the Internet.

[0888] "Means for analyzing the voice data received by the server and estimating the type, location, and direction of travel of the emergency vehicle" refers to a program or algorithm that has the function of analyzing the voice data received by the server, identifying each emergency vehicle, and determining the location and direction of travel of the sound source.

[0889] The "means for displaying the analysis results on the vehicle display" refers to a device or system for visually displaying the analysis results received from the server on a display device within the vehicle.

[0890] "Means for acquiring image data from a 360-degree camera and visually displaying safe avoidance areas from the captured images" refers to a device or module that has the function of acquiring image data captured by a camera mounted on a vehicle, analyzing the surrounding situation, and displaying areas where it is safe to proceed on a display device inside the vehicle.

[0891] A "network module for communicating voice data and analysis results in real time between an analysis server and a terminal" is a device or software that provides the network communication functions necessary for the analysis server and a terminal to send and receive data in real time.

[0892] MODE FOR CARRYING OUT THE INVENTION

[0893] As an embodiment of the present invention, the specific operation of the system will be described below. The present invention is a system that assists hearing-impaired people in visually recognizing approaching emergency vehicles when driving a car. The system is composed of a terminal, a server, and a user.

[0894] Terminal

[0895] Ambient sound recording

[0896] The device uses a voice recording module to record surrounding sounds in real time through the smartphone's microphone, and the recording data is stored in digital format, for example, using a microphone device with an appropriate sampling rate.

[0897] Sending audio data

[0898] The device transmits the recorded audio data to the analysis server via a network communication module, primarily via an Internet connection, for example via HTTP or HTTPS protocols.

[0899] Receiving and displaying analysis results

[0900] The terminal receives the analysis results from the server and displays them on the vehicle's display via the display audio linkage module. The actual display is linked to the vehicle's information display system and is presented in a format that is intuitively easy for the user to understand.

[0901] Use of a surrounding camera (optional)

[0902] Additionally, the device can utilize a 360-degree camera image acquisition module, which captures and analyzes video data from cameras installed in the vehicle and displays safe areas to avoid.

[0903] server

[0904] Analysis of audio data

[0905] The server uses a voice data analysis module to analyze the received voice data. This analysis can utilize an AI voice recognition engine (e.g., Google Cloud Speech-to-Text). It identifies the sound of an emergency vehicle siren and estimates its type, location, and direction of travel.

[0906] Sending analysis results

[0907] The server then transmits the analysis results to the device in real time. This communication is fast and secure, providing information on approaching emergency vehicles without delay.

[0908] User

[0909] Checking information and driving operations

[0910] The user checks the analysis results displayed on the vehicle's display and recognizes the approach of an emergency vehicle. For example, if the display shows "An ambulance approaching 30 meters behind to the right," the user follows the instructions and safely moves the vehicle to the left.

[0911] Specific examples

[0912] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module records this audio and sends it to the server via the network communication module. The server's audio data analysis module analyzes the audio data and identifies the location information of the emergency vehicle. The analysis result, "Ambulance approaching 30 meters behind and to the right," is generated and sent to the terminal. The terminal displays the received information on the vehicle's display. The user can confirm this display and move about 1 meter to the left, ensuring safe driving.

[0913] Prompt Sentence Examples

[0914] "Detect the sound of an emergency vehicle siren. Display specific instructions to inform the user that an emergency vehicle is approaching 30 meters behind and to the right."

[0915] The above is an embodiment of the present invention, and this system enables even hearing-impaired people to properly recognize the approach of an emergency vehicle and drive safely.

[0916] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0917] Step 1:

[0918] Ambient sound recording

[0919] The device uses the smartphone's audio recording module to record ambient audio in real time. The ambient sound is input from the smartphone's microphone. This audio is stored in a buffer as digitally recorded data. Specifically, the device sets a sampling rate and continuously captures audio and converts it into digital data, which is then used in subsequent processing steps.

[0920] Step 2:

[0921] Sending audio data

[0922] The device sends the recorded audio data to the analysis server via the network communication module. The recorded data buffer is used as input, and the data is sent to the server via an Internet connection. Specifically, the device divides the recorded data into packets at regular time intervals (e.g., every second) and sends them using the HTTP or HTTPS protocol. Once the transmission is complete, the buffer is cleared and prepared for the next data recording.

[0923] Step 3:

[0924] Analysis of audio data

[0925] The server analyzes the received voice data using the voice data analysis module. The voice data received from the device is used as input. Specifically, the server performs noise removal and normalization processing, and inputs this data into an AI voice recognition model. This model identifies the sound of an emergency vehicle siren and estimates the type, location, and direction of travel of the emergency vehicle. The analysis results are generated as JSON format data as output.

[0926] Step 4:

[0927] Sending analysis results

[0928] The server sends the generated analysis results to the terminal through the network communication module. The JSON data of the analysis results is used as input, and the data is sent to the terminal as output. Specifically, the server divides the data into packets and sends them to the terminal's receiving endpoint using the HTTPS protocol. This allows the analysis results to be updated in real time.

[0929] Step 5:

[0930] Receiving and displaying analysis results

[0931] The terminal displays the received analysis results on the vehicle's display via the display audio linkage module. The JSON data of the analysis results received from the server is used as input, and the information to be displayed on the vehicle's display is generated as output. Specifically, the terminal decodes the analysis results and converts them into an appropriate display format. For example, a message such as "Ambulance approaching 30 meters behind to the right" is displayed on the display.

[0932] Step 6:

[0933] Use of a surrounding camera (optional)

[0934] The device uses a 360-degree camera image acquisition module to acquire video data from the vehicle's onboard camera. It uses the camera's video data as input and generates information to display safe areas for avoidance as output. Specifically, the device analyzes the camera video in real time to identify the direction of travel and areas that can be avoided. This information is then visually displayed on the display.

[0935] Step 7:

[0936] Checking information and driving operations

[0937] The user checks the analysis results displayed on the vehicle's display and recognizes the approach of an emergency vehicle. The information displayed on the vehicle's display is used as input, and the user's driving operation is used as output. Specifically, the user checks the message "An ambulance is approaching 30 meters behind and to the right" and takes action to safely move the vehicle to the left. In this case, the user references the images from the all-around camera and the displayed information to safely avoid the approach.

[0938] (Application example 1)

[0939] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0940] It is difficult for the hearing impaired to visually recognize approaching emergency vehicles. This is particularly true when using autonomous vehicles, where a driver's failure to notice an approaching emergency vehicle can pose a serious risk. When such obstacles exist, effective evasive measures must be implemented quickly. However, current visual information systems lack sufficient functionality to provide real-time information on the location and direction of emergency vehicles.

[0941] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0942] In this invention, the server includes means for recording the siren sound of the emergency vehicle, means for transmitting the recorded audio data to an analysis server, means for the server to analyze the audio data received and generate emergency vehicle position information and response instructions, means for displaying the emergency vehicle position information and response instructions generated as a result of the analysis on a display of the vehicle, and means for displaying information about the approach of the emergency vehicle on a head-mounted display, thereby enabling hearing-impaired people to visually recognize the approach of an emergency vehicle in real time and take appropriate evasive action.

[0943] The "means for recording the siren sound of an emergency vehicle" is a device that records sounds in the surrounding environment of a vehicle and captures the siren sound of an emergency vehicle in real time.

[0944] The "means for transmitting recorded voice data to the analysis server" is a communication module for transmitting recorded voice data to the analysis server via a communication line such as the Internet.

[0945] "Means for analyzing the voice data received by the server and generating emergency vehicle location information and response instructions" refers to software or hardware that enables the server to analyze the voice data, identify the location and direction of the emergency vehicle, and provide instructions to the driver based on that information.

[0946] The "means for displaying the emergency vehicle location information and response instructions generated as the analysis result on the vehicle display" refers to an interface and display device for visually displaying the analysis result on the display inside the vehicle.

[0947] "Means for acquiring image data from a 360-degree camera and visually identifying safe avoidance areas from the captured image" refers to a device that analyzes images from a vehicle's 360-degree camera and visually identifies areas that can be safely avoided.

[0948] "Means having a network module for communicating voice data and analysis results in real time between the analysis server and the user's terminal" refers to a communication device for realizing data communication in real time between the analysis server and the vehicle's terminal.

[0949] The "means for displaying information about approaching emergency vehicles on a head-mounted display" is a device for displaying emergency vehicle location information and avoidance instructions on a head-mounted display.

[0950] As an embodiment of the present invention, the specific operation of the system will be described below.

[0951] overview

[0952] This system provides assistance to hearing-impaired people using self-driving vehicles, helping them visually recognize approaching emergency vehicles. The system consists of a terminal, a server, and a user. Terminals refer to smartphones, displays, head-mounted displays (HMDs), etc. installed in self-driving vehicles. The server analyzes the audio data and generates emergency vehicle location information and avoidance instructions. The user then performs safe driving operations based on the information provided visually.

[0953] System configuration

[0954] 1. Terminal (installed in autonomous vehicles)

[0955] Audio recording module: A device that uses the device's microphone to record surrounding audio in real time.

[0956] Network communication module: A device that sends recorded audio data to a server for analysis.

[0957] Display audio link module: A device that displays the analysis results received from the server on the display or HMD of the autonomous vehicle.

[0958] All-around camera image acquisition module (optional): A device that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[0959] 2. Server

[0960] Voice data analysis module: This module analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle. This analysis uses a deep learning model (for example, a model built with TensorFlow or PyTorch).

[0961] Network communication module: A device that transmits analysis results to the terminal.

[0962] 3. Users

[0963] Hearing-impaired person (driver): Check the approach of emergency vehicles through the self-driving vehicle's display or HMD and perform appropriate driving operations.

[0964] Program processing explanation

[0965] The device first records the surrounding sounds in real time, and the recorded data is sent to the analysis server via the network communication module.

[0966] The server analyzes the received audio data and identifies the siren sound of an emergency vehicle. From the identified siren sound, it estimates the type of emergency vehicle, its location, and its direction of travel. This analysis is performed using deep learning techniques (e.g., TensorFlow, PyTorch). The results are sent to the device using a network communication module.

[0967] The terminal receives the analysis results from the server and displays them on the autonomous vehicle's display or HMD via the display audio linkage module. For example, specific instructions such as "An ambulance is approaching 30 meters behind and to the right" can be displayed. In addition, by using the all-around camera image acquisition module, safe avoidance areas can also be visually displayed.

[0968] Users can check the autonomous vehicle's display or HMD, recognize the approach of an emergency vehicle based on the analysis results, and take appropriate evasive action. For example, if a user sees a message saying "An ambulance is approaching 30 meters behind and to the right," they will take action by moving about one meter to the left.

[0969] Specific examples

[0970] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module records this audio and sends it to the server via the network communication module. The server's analysis module analyzes the audio data and identifies the location information of the emergency vehicle. The analysis result is generated as "Ambulance approaching 30 meters behind and to the right," which is sent to the terminal. The terminal displays the received information on the display or HMD of the autonomous vehicle. The user confirms this display and moves about 1 meter to the left, ensuring safe driving.

[0971] Prompt Sentence Examples

[0972] For example, a prompt to a generative AI model might look like this:

[0973] "If an emergency vehicle siren is detected, generate a message with specific location and instructions on how to avoid it."

[0974] The above is an embodiment of the present invention, and this system enables even hearing-impaired people to properly recognize the approach of an emergency vehicle and drive safely.

[0975] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0976] Step 1:

[0977] The device records the surrounding audio. While the user is driving the autonomous vehicle, the audio recording module is activated when the engine starts and records the audio around the vehicle in real time. The input is the audio data of the surrounding environment, and the output is the recorded audio data.

[0978] Step 2:

[0979] The device sends the recorded voice data to the analysis server. The network communication module is used to upload the recorded voice data to the server via the Internet. The input is the recorded voice data, and the output is the voice data sent to the server.

[0980] Step 3:

[0981] The server analyzes the received audio data and identifies the siren sound of an emergency vehicle. The audio data analysis module uses deep learning technology to extract the characteristics of the siren sound and detect the approach of an emergency vehicle. The input is the audio data sent from the device, and the output is the siren sound detection result.

[0982] Step 4:

[0983] The server estimates the type, location, and direction of travel of the emergency vehicle based on the detection results. Based on the analysis results, the server identifies the emergency vehicle's detailed location and direction of travel, and generates corresponding driving instructions. The input is the detection result of the siren sound, and the output is data on the emergency vehicle's location and direction of travel.

[0984] Step 5:

[0985] The server generates analysis results and sends them to the terminal. The network communication module is used to return the analysis results to the terminal in real time. The input is data on the emergency vehicle's location and direction of travel, and the output is the analysis results sent to the terminal.

[0986] Step 6:

[0987] The terminal receives the analysis results and displays them on the autonomous vehicle's display or HMD. The display audio linkage module is used to visually display the emergency vehicle's location and response instructions. The input is the analysis results sent from the server, and the output is the emergency vehicle's location information and response instructions displayed on the display or HMD.

[0988] Step 7:

[0989] The user performs appropriate driving operations based on the information displayed on the display or HMD. Check the displayed information about approaching emergency vehicles and follow the instructions to safely avoid the vehicle. The input is the analysis results displayed on the display or HMD, and the output is the user's driving operations.

[0990] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0991] As an embodiment of the present invention, the specific operation of the system will be described below.

[0992] overview

[0993] When driving a car, hearing-impaired people have difficulty visually recognizing approaching emergency vehicles. This invention provides a system that detects emergency vehicle sirens in real time and displays their location information and response instructions on the vehicle's display. In addition, by combining it with an emotion engine that recognizes the user's emotions, driving assistance messages are optimized to the user's emotional state, supporting safe and comfortable driving.

[0994] System configuration

[0995] 1. Device (smartphone)

[0996] Audio recording module: Uses the smartphone's microphone to record surrounding audio in real time.

[0997] Network communication module: A module that sends recorded audio data to the analysis server.

[0998] Display Audio Link Module: A module that receives analysis results and displays them on the vehicle's display.

[0999] All-around camera image acquisition module (optional): A module that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[1000] Emotion recognition module: Recognizes user emotions in real time using cameras and sensors.

[1001] 2. Server

[1002] Voice data analysis module: Analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle.

[1003] Network communication module: A module that sends analysis results to the terminal.

[1004] Emotion analysis module: Analyzes received user emotion data and optimizes the content of driving assistance messages.

[1005] 3. Users

[1006] Hearing-impaired driver: Check the approach of emergency vehicles through a smartphone and vehicle display and perform appropriate driving operations.

[1007] Program processing

[1008] Terminal

[1009] The device first records the surrounding sounds in real time, and sends the recorded data to an analysis server via a network communication module. The device then uses cameras and sensors to recognize the user's emotional state, which is also sent to the server.

[1010] server

[1011] The server's voice data analysis module analyzes the received voice data and identifies the siren sound of an emergency vehicle. From the identified siren sound, the type of emergency vehicle, its location, and direction of travel are estimated. The analysis results are sent to the device via the network communication module. The emotion analysis module also analyzes the user's emotion data and optimizes the content of the driving assistance message to match the user's emotional state. For example, if the user is feeling stressed, more detailed and careful instructions are provided. If the user is relaxed, concise and smooth instructions are provided.

[1012] Terminal

[1013] The device receives analysis results from the server and displays driving assistance messages based on the driver's emotions on the vehicle's display via the display audio linkage module. Specifically, the device displays the message "An ambulance is approaching 30 meters behind the driver on the right" along with additional instructions tailored to the user's emotions. Safe avoidance areas can also be visually displayed by using the all-around camera image acquisition module.

[1014] User

[1015] The user checks the vehicle's display and reads driving assistance messages based on the analysis results and their emotions. For example, if the user sees a message saying "An ambulance is approaching 30 meters behind and to the right," they can follow the instructions and safely move the vehicle to the left. In addition, the response instructions are optimized to the user's emotional state, allowing the user to react more appropriately and quickly.

[1016] Specific examples

[1017] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module will record this audio and send it to the server via the network communication module. The server's analysis module will analyze the audio data and identify the location of the emergency vehicle. The analysis result will be "Ambulance approaching 30 meters behind and to the right," which will be sent to the device. At the same time, the emotion analysis module will analyze the user's emotional data, and if the user is feeling stressed, detailed instructions such as "Remain calm and move about 1 meter to the left" will be provided. Conversely, if the user is relaxed, simpler instructions will be given such as "Move to the left."

[1018] The above is a form for implementing the present invention, and this system allows even hearing-impaired people to properly recognize the approach of an emergency vehicle and receive driving assistance messages tailored to the user's emotional state, allowing them to drive safely and comfortably.

[1019] The processing flow will be explained below.

[1020] Step 1:

[1021] The device activates the audio recording module, which accesses the device's microphone and begins recording surrounding audio in real time.

[1022] Step 2:

[1023] The device sends the recorded audio data to the server via the network communication module, uploading the audio data approximately every second.

[1024] Step 3:

[1025] The server analyzes the received audio data using the audio data analysis module, and analyzes specific frequency patterns to detect the siren of an emergency vehicle.

[1026] Step 4:

[1027] The server identifies the type and location of the emergency vehicle based on the siren sound detection result, and estimates the direction and distance of the emergency vehicle based on the sound intensity and detection timing.

[1028] Step 5:

[1029] The server generates the analysis results and creates a text message containing the type of emergency vehicle, its location, direction of travel, and any other necessary information.

[1030] Step 6:

[1031] The device uses cameras and sensors to recognize the user's emotional state, and sends the recognized emotional data to a server.

[1032] Step 7:

[1033] The emotion data received by the server is analyzed by the emotion analysis module to determine whether the user is feeling stressed or anxious, or relaxed.

[1034] Step 8:

[1035] The server optimizes driving assistance messages based on the driver's emotional state, for example generating detailed and careful instructions when the driver is stressed and concise and smooth instructions when the driver is relaxed.

[1036] Step 9:

[1037] The analysis results and optimized driving assistance messages generated by the server are sent to the terminal using a network communication module.

[1038] Step 10:

[1039] The terminal receives the analysis results and driving assistance messages from the server and displays them on the vehicle's display via the display audio linkage module. For example, along with the message "An ambulance is approaching 30 meters behind and to the right," detailed instructions such as "Remain calm and move about 1 meter to the left" may be displayed.

[1040] Step 11:

[1041] (Optional) The device uses the omnidirectional camera image acquisition module to analyze the video from the omnidirectional camera and visually inform the user of a safe area to avoid. For example, it displays an instruction such as "There is room to evacuate about 1 meter to the left."

[1042] Step 12:

[1043] The user checks the vehicle's display and sees an instruction such as "An ambulance is approaching 30 meters behind and to the right."

[1044] Step 13:

[1045] The system performs safe driving maneuvers based on instructions provided by the user. For example, the system follows the instructions on the display to move about 1 meter to the left.

[1046] The above series of processing steps enables hearing-impaired drivers to properly recognize approaching emergency vehicles and drive safely. In addition, the emotion engine optimizes driving assistance messages, providing detailed assistance tailored to the user's mental state.

[1047] Example 2

[1048] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1049] When hearing-impaired people drive, it is difficult for them to visually recognize approaching emergency vehicles. This makes it difficult for them to quickly take appropriate evasive action, making safety a challenge. Therefore, there is a need for a system that can recognize approaching emergency vehicles in real time and provide appropriate support messages according to the driver's emotional state.

[1050] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for recording surrounding sounds; means for transmitting the recorded sound data and user emotion data to the analysis server; means having a network module for communicating the sound data, emotion data, and analysis results in real time between the analysis server and the user's terminal; means for analyzing the sound data received by the server and identifying the type, location information, and direction of travel of the emergency vehicle; means for analyzing the user emotion data received by the server and optimizing a driving assistance message; means for displaying the analysis result and the optimized driving assistance message on the vehicle display; and means for acquiring image data from a 360-degree camera and visually identifying a safe avoidance area from the captured image. This allows even hearing-impaired people to properly recognize the approach of an emergency vehicle and respond safely and quickly.

[1051] "Means for recording ambient sounds" refers to the device's ability to record ambient sounds in and around the vehicle in real time.

[1052] "Means for transmitting recorded voice data and user emotional data to an analysis server" refers to the function of transmitting the voice data collected by the terminal and the user's emotional state to a server via a network.

[1053] "Means having a network module for communicating voice data, emotion data, and analysis results in real time between the analysis server and the user's terminal" refers to a communications infrastructure for sending and receiving data in real time between the terminal and the server.

[1054] "Means for the server to analyze the received voice data and identify the type, location, and direction of travel of the emergency vehicle" refers to the function of the server to identify the sound of an emergency vehicle from the voice data and determine its type, location, and direction of travel.

[1055] "Means for analyzing the user's emotional data received by the server and optimizing driving assistance messages" refers to the function by which the server analyzes the user's emotional state and customizes safety instructions based on that information.

[1056] "Means for displaying the analysis results and optimized driving assistance messages on the vehicle's display" refers to the function of the terminal visually presenting the information obtained from the server on the vehicle's display.

[1057] "Means for acquiring image data from a 360-degree camera and visually identifying safe areas to avoid from the captured image" refers to a function that analyzes images acquired by a camera mounted on a vehicle and displays areas where it is possible to move safely.

[1058] As an embodiment for carrying out the present invention, the specific operation of the system will be described below.

[1059] overview

[1060] The purpose of this invention is to provide a system that allows hearing-impaired drivers to visually recognize approaching emergency vehicles in real time. In this system, a terminal records audio from inside and around the vehicle, and an analysis server analyzes the audio data. Furthermore, the system takes into account the user's emotional state and optimizes driving assistance messages to be displayed on the vehicle's display. This allows hearing-impaired drivers to respond safely and quickly.

[1061] System configuration

[1062] 1. Device (smartphone)

[1063] Audio recording module: Uses the smartphone's microphone to record surrounding audio in real time.

[1064] Network communication module: A module that transmits recorded voice data and user emotion data to the analysis server.

[1065] Display audio linkage module: A module that receives analysis results and driving assistance messages and displays them on the vehicle's display.

[1066] All-around camera image acquisition module (optional): A module that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[1067] Emotion recognition module: Recognizes user emotions in real time using cameras and sensors.

[1068] 2. Server

[1069] Voice data analysis module: Analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle.

[1070] Network communication module: A module that sends analysis results to the terminal.

[1071] Emotion analysis module: Analyzes received user emotion data and optimizes the content of driving assistance messages.

[1072] 3. Users

[1073] Hearing-impaired driver: Check the approach of emergency vehicles through a smartphone and vehicle display and perform appropriate driving operations.

[1074] Program processing

[1075] Terminal

[1076] The device first records the surrounding sounds in real time, and sends the recorded data to an analysis server via a network communication module. Furthermore, the device uses cameras and sensors to recognize the user's emotional state, and also sends this information to the server.

[1077] server

[1078] The server's voice data analysis module analyzes the received voice data and identifies the siren sound of an emergency vehicle. From the identified siren sound, the type of emergency vehicle, its location, and direction of travel are estimated. The analysis results are sent to the device via the network communication module. The emotion analysis module also analyzes the user's emotion data and optimizes the content of the driving assistance message to match the user's emotional state. For example, if the user is stressed, more detailed and careful instructions are provided. If the user is relaxed, concise and smooth instructions are provided.

[1079] Terminal

[1080] The device receives the analysis results from the server and displays the optimized driving assistance message on the vehicle's display via the display audio linkage module. Specifically, the message "Ambulance approaching 30 meters behind to the right" is displayed along with additional instructions tailored to the user's emotions. In addition, by using the all-around camera image acquisition module, safe avoidance areas can also be visually displayed.

[1081] User

[1082] The user checks the vehicle's display and reads driving assistance messages based on the analysis results and their emotions. For example, if the user sees a message saying "An ambulance is approaching 30 meters behind and to the right," they can follow the instructions and safely move the vehicle to the left. In addition, the response instructions are optimized to the user's emotional state, allowing the user to react more appropriately and quickly.

[1083] Specific examples

[1084] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module will record this audio and send it to the server via the network communication module. The server's analysis module will analyze the audio data and identify the location of the emergency vehicle. The analysis result will be "Ambulance approaching 30 meters behind and to the right," which will be sent to the device. At the same time, the emotion analysis module will analyze the user's emotional data, and if the user is feeling stressed, detailed instructions such as "Remain calm and move about 1 meter to the left" will be provided. Conversely, if the user is relaxed, simpler instructions will be given such as "Move to the left."

[1085] Prompt Sentence Examples

[1086] Examples of prompts for a generative AI model might include:

[1087] "What should I do if an ambulance is approaching 30 meters behind me to the right?"

[1088] "Please provide an example of how to instruct an emergency vehicle to evacuate if the user is experiencing stress."

[1089] "Please explain how to display a safe area to avoid using a 360-degree camera image."

[1090] This allows even hearing-impaired people to properly recognize approaching emergency vehicles and receive driving assistance messages tailored to the user's emotional state, enabling them to drive safely and comfortably.

[1091] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1092] Explain the program's processing flow in detail

[1093] Step 1:

[1094] The device uses the smartphone's audio recording module to record surrounding audio in real time.

[1095] Input: Surrounding environmental sounds

[1096] Data processing: Audio filtering (noise removal, volume adjustment)

[1097] Output: filtered audio data

[1098] Step 2:

[1099] The device sends the recorded voice data to the server via the network communication module, and at the same time, the device's camera and sensors capture the user's emotional data, which is also sent to the server.

[1100] Input: Filtered voice data and user emotion data

[1101] Data calculation: Data packing (combining voice data and emotion data)

[1102] Output: Sending consolidated data packets

[1103] Step 3:

[1104] The server's audio data analysis module analyzes the received audio data and identifies the siren sound of an emergency vehicle.

[1105] Input: Voice data in integrated data packets

[1106] Data Computing: Audio Analysis (Digital Signal Processing, Acoustic Pattern Recognition using Machine Learning Algorithms)

[1107] Output: Identification of emergency vehicle type, location, and direction of travel

[1108] Step 4:

[1109] The emotion analysis module of the server analyzes the received user emotion data and generates a driving assistance message.

[1110] Input: Emotion data in a consolidated data packet

[1111] Data calculation: Emotion analysis (facial recognition technology, estimation of emotional state through sensor data analysis)

[1112] Output: Driving assistance messages optimized for the user's emotional state

[1113] Step 5:

[1114] The server transmits the emergency vehicle information obtained as a result of the analysis and the optimized driving assistance message to the terminal via the network communication module.

[1115] Input: Emergency vehicle type, location, direction, optimized driver assistance message

[1116] Data calculation: Data packing (integration of emergency vehicle information and driver assistance messages)

[1117] Output: Sending consolidated data packets

[1118] Step 6:

[1119] The analysis results and driving assistance messages received by the terminal from the server are displayed on the vehicle's display via the display audio linkage module.

[1120] Input: Integrated data packet

[1121] Data processing: Data separation (extraction of emergency vehicle information and driving assistance messages)

[1122] Output: A message such as "Ambulance approaching 30m behind right" displayed on the vehicle's display

[1123] Step 7:

[1124] The user checks the vehicle's display and performs appropriate driving operations in accordance with the displayed analysis results and driving assistance messages.

[1125] Input: Information displayed on the vehicle's display

[1126] Specific operation: Safely evacuate the vehicle based on the analysis results and driver assistance messages

[1127] Output: Avoid danger through safe driving operations

[1128] In this way, by having each processing step work in cooperation with one another, a system is realized that allows the user to recognize the approach of an emergency vehicle in real time and respond safely and quickly.

[1129] (Application example 2)

[1130] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1131] There is a need for a system that enables hearing-impaired people, who have difficulty visually recognizing approaching emergency vehicles, to drive cars safely and comfortably. In particular, there is a technical challenge in providing support messages that adapt to the driver's emotional state when an emergency vehicle is approaching, enabling a quick and appropriate response.

[1132] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1133] In this invention, the server includes: [means for analyzing voice data and generating emergency vehicle location information and response instructions;] [means for generating a driving assistance message adapted based on the user's emotional state; and] [means for transmitting the analysis results and the driving assistance message to the user's terminal in real time.] This allows even hearing-impaired people to properly recognize the approach of an emergency vehicle and receive the optimal driving assistance message according to their emotional state, enabling safe and comfortable driving.

[1134] "Emergency vehicles" refers to ambulances, fire engines, police vehicles, etc. that emit sirens and travel in an emergency.

[1135] "Siren sound" refers to the special warning sound emitted by emergency vehicles.

[1136] "Means for recording" refers to a method of using a microphone to record surrounding sounds as digital audio data.

[1137] "Analysis Server" refers to a network server for processing and analyzing received voice and emotion data.

[1138] "Means for transmitting" refers to the method for transmitting data over a network to other devices or servers.

[1139] "Location Information" means information indicating the real-time geographic location of an emergency vehicle.

[1140] "Response instructions" refers to specific driving instructions to the driver regarding the approach of an emergency vehicle.

[1141] "Display" refers to a screen device installed inside a vehicle that visually displays analysis results and driving assistance messages.

[1142] An "emotion recognition module" refers to a device or system that uses sensors and cameras to detect a user's emotional state and digitize it.

[1143] "Driving assistance messages" refer to messages that provide the driver with instructions or warnings that are adapted to the current driving situation and the user's emotional state based on analyzed data.

[1144] A "surrounding camera" refers to a camera system that captures 360-degree images of the area around the vehicle and acquires video data.

[1145] A "safe avoidance area" refers to a space that allows drivers to safely move their vehicle when an emergency vehicle is approaching.

[1146] "Real-time" refers to a state in which data acquisition, processing, and communication occur instantly.

[1147] "Network module" refers to a hardware and software configuration for communicating data.

[1148] MODE FOR CARRYING OUT THE INVENTION

[1149] The following system will be described as an embodiment of the present invention. This system is equipped with voice recording and analysis functions and emotion recognition functions so that hearing-impaired people can sense the approach of an emergency vehicle with confidence and respond appropriately.

[1150] System Configuration and Operation

[1151] Terminal

[1152] The terminal (smartphone or in-car device) is equipped with the following modules:

[1153] 1. Audio recording module: Uses the smartphone's microphone to record surrounding sounds in real time, and the recorded audio data is saved in digital format.

[1154] 2. Network communication module: Sends recorded audio data to the analysis server via the network.

[1155] 3. Emotion Recognition Module: Using cameras and sensors, it collects the user's facial expressions and behavioral data to recognize their emotional state in real time.

[1156] 4. Display audio link module: Displays the analysis results and driving assistance messages received from the analysis server on the vehicle's display.

[1157] server

[1158] The server comprises several modules that perform the following functions:

[1159] 1. Voice data analysis module: Analyzes the received voice data and identifies the sound of an emergency vehicle siren. This analysis is performed using Google Speech-to-Text and Amazon Transcribe.

[1160] 2. Location information identification module: Identifies the location information of emergency vehicles and generates specific response instructions based on that information.

[1161] 3. Sentiment Analysis Module: Analyzes the emotional data sent by the user and generates driving assistance messages. This analysis is performed using Microsoft Azure Cognitive Services and IBM Watson Tone Analyzer.

[1162] 4. Network communication module: Transmits analysis results and driving assistance messages to the terminal in real time.

[1163] User

[1164] A user (hearing impaired person) uses the system as follows:

[1165] 1. Surrounding sounds are recorded, and when the sound of an emergency vehicle's siren is detected, the vehicle's display will show the vehicle's location and appropriate response instructions.

[1166] 2. Driving assistance messages are provided according to the user's emotional state, allowing the user to receive them intuitively.

[1167] 3. Information from the surrounding camera is also utilized to visually indicate areas where evasive action can be taken safely.

[1168] Specific examples

[1169] For example, if the sound of an emergency vehicle's siren is detected 30 meters behind and to the right, the audio recording module will record this sound. This recorded data is sent to the server via the network communication module, and the server's analysis module will identify the location of the emergency vehicle. The analysis result will be "Ambulance approaching 30 meters behind and to the right," which will be sent to the device. At the same time, the emotion analysis module will analyze the user's emotional data, and if the user is feeling stressed, detailed instructions such as "Remain calm and move about 1 meter to the left" will be provided. Conversely, if the user is relaxed, simpler instructions such as "Move to the left" will be provided.

[1170] Prompt Sentence Examples

[1171] Please provide an overview of a system that detects the approach of emergency vehicles and provides visual warnings to the driver. The system is designed for the hearing impaired, detects emergency vehicle sirens in real time, and displays their location on the vehicle's display. It also recognizes the driver's emotions and optimizes driver assistance messages accordingly. Specifically, please explain what hardware and software are used and what data processing is involved.

[1172] In this way, this invention is a system that provides drivers with optimal information in real time through voice analysis and emotion recognition, enabling hearing-impaired people to drive cars safely and comfortably.

[1173] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1174] Step 1:

[1175] The device uses a voice recording module to record surrounding sounds in real time, and the recorded voice data is stored in digital format.

[1176] Input: Ambient audio

[1177] Output: Digital audio data

[1178] Step 2:

[1179] The recorded voice data is sent to the analysis server via the network communication module.

[1180] Input: Digital audio data

[1181] Output: Audio data sent to the analysis server

[1182] Step 3:

[1183] The server's voice data analysis module analyzes the received voice data and identifies the sound of an emergency vehicle siren, using voice analysis software such as Google Speech-to-Text or Amazon Transcribe.

[1184] Input: Transmitted audio data

[1185] Data processing / calculation: Audio data analysis and siren sound identification

[1186] Output: Audio snippets containing emergency vehicle sirens and their metadata

[1187] Step 4:

[1188] The location information identification module identifies the location information of the emergency vehicle based on the metadata of the identified siren sound.

[1189] Input: Siren sound metadata

[1190] Data processing / calculation: Location information analysis

[1191] Output: Emergency vehicle location information

[1192] Step 5:

[1193] The emotion recognition module uses the device's camera and sensors to collect and analyze the user's facial expressions and behavioral data to recognize the user's emotional state. This data is also sent to the server.

[1194] Input: Camera images and sensor data from the device

[1195] Data processing / calculation: Analysis of user facial expression and behavior data

[1196] Output: User's emotional state data

[1197] Step 6:

[1198] The server's emotion analysis module analyzes the emotion data sent by the user and generates driving assistance messages using tools such as Microsoft Azure Cognitive Services and IBM Watson Tone Analyzer.

[1199] Input: User's emotional state data

[1200] Data processing / calculation: Emotional data analysis and generation of optimal driver assistance messages

[1201] Output: Emotion-based driving assistance message

[1202] Step 7:

[1203] The analysis server compiles the emergency vehicle's location information and the generated driving assistance message and transmits them to the terminal via the network communication module.

[1204] Input: Emergency vehicle location information, driving assistance message

[1205] Data processing / calculation: Integrating location information and messages

[1206] Output: Integrated location and driver assistance messages

[1207] Step 8:

[1208] The terminal displays the emergency vehicle location information and driving assistance message received from the analysis server on the vehicle's display via the display audio link module.

[1209] Input: Integrated location and driver assistance messages

[1210] Output: Emergency vehicle location information and driver assistance messages displayed on the vehicle display

[1211] Step 9:

[1212] The user checks the information displayed on the vehicle display and takes evasive action safely.

[1213] Input: Vehicle display information

[1214] Output: User's safe avoidance action

[1215] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1216] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1217] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1218] [Fourth embodiment]

[1219] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1220] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1221] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1222] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1223] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1224] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1225] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1226] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1227] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1228] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1229] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1230] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1231] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1232] As an embodiment of the present invention, the specific operation of the system will be described below.

[1233] overview

[1234] When driving a car, hearing-impaired people have difficulty visually recognizing approaching emergency vehicles. This invention provides a system that detects the siren sound of an emergency vehicle in real time and displays its location and response instructions on the vehicle's display. The system is mainly composed of a terminal, a server, and a user.

[1235] System configuration

[1236] 1. Device (smartphone)

[1237] Audio recording module: Uses the smartphone's microphone to record surrounding audio in real time.

[1238] Network communication module: A module that sends recorded audio data to the analysis server.

[1239] Display Audio Link Module: A module that receives analysis results and displays them on the vehicle's display.

[1240] All-around camera image acquisition module (optional): A module that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[1241] 2. Server

[1242] Voice data analysis module: Analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle.

[1243] Network communication module: A module that sends analysis results to the terminal.

[1244] 3. Users

[1245] Hearing-impaired driver: Check the approach of emergency vehicles through a smartphone and vehicle display and perform appropriate driving operations.

[1246] Program processing

[1247] Terminal

[1248] The device first records the surrounding sounds in real time, and the recorded data is sent to the analysis server via the network communication module.

[1249] server

[1250] The server's voice data analysis module analyzes the received voice data and identifies the siren sound of an emergency vehicle. From the identified siren sound, the type of emergency vehicle, its location, and direction of travel are estimated. The analysis results are sent to the device via the network communication module.

[1251] Terminal

[1252] The terminal receives the analysis results from the server and displays them on the vehicle's display via the display audio linkage module. For example, specific instructions such as "An ambulance is approaching 30 meters behind and to the right" can be displayed. In addition, by using the all-around camera image acquisition module, safe avoidance areas can also be visually displayed.

[1253] User

[1254] The user checks the vehicle's display and recognizes the approach of an emergency vehicle based on the analysis results. For example, if the display reads "An ambulance is approaching 30 meters behind the vehicle to the right," the user can follow the instructions and safely move the vehicle to the left.

[1255] Specific examples

[1256] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module records this audio and sends it to the server via the network communication module. The server's analysis module analyzes the audio data and identifies the location of the emergency vehicle. The analysis result is generated as "Ambulance approaching 30 meters behind and to the right," which is sent to the terminal. The terminal displays the received information on the vehicle's display. The user confirms this display and moves about 1 meter to the left, ensuring safe driving.

[1257] The above is an embodiment of the present invention, and this system enables even hearing-impaired people to properly recognize approaching emergency vehicles and drive safely.

[1258] The processing flow will be explained below.

[1259] Step 1:

[1260] The device activates the audio recording module, which accesses the device's microphone and begins recording surrounding audio in real time.

[1261] Step 2:

[1262] The device sends the recorded audio data to the server via the network communication module, uploading the audio data approximately every second.

[1263] Step 3:

[1264] The server analyzes the received audio data using the audio data analysis module, and analyzes specific frequency patterns to detect the siren of an emergency vehicle.

[1265] Step 4:

[1266] The server identifies the type and location of the emergency vehicle based on the siren sound detection result, and estimates the direction and distance of the emergency vehicle based on the sound intensity and detection timing.

[1267] Step 5:

[1268] The server generates the analysis results and creates a text message containing the type of emergency vehicle, its location, direction of travel, and any other necessary information.

[1269] Step 6:

[1270] The analysis results generated by the server are sent to the terminal using a network communication module.

[1271] Step 7:

[1272] The analysis results received by the device from the server are displayed on the vehicle's display via the display audio link module. For example, the message "Ambulance approaching 30 meters behind the right" is displayed.

[1273] Step 8:

[1274] (Optional) The device uses the omnidirectional camera image acquisition module to analyze the video from the omnidirectional camera and visually inform the user of a safe area to avoid. For example, it displays an instruction such as "There is room to evacuate about 1 meter to the left."

[1275] Step 9:

[1276] The user checks the vehicle's display and sees an instruction such as "An ambulance is approaching 30 meters behind and to the right."

[1277] Step 10:

[1278] The system performs safe driving maneuvers based on instructions provided by the user. For example, the system follows the instructions on the display to move about 1 meter to the left.

[1279] The above series of processing steps enables hearing-impaired drivers to properly recognize the approach of an emergency vehicle and drive safely.

[1280] Example 1

[1281] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1282] With conventional technology, it has been difficult for hearing-impaired drivers to properly recognize approaching emergency vehicles. In particular, they often miss sirens, which can lead to traffic accidents. The present invention aims to provide a system that supports safe driving by detecting approaching emergency vehicles in real time and allowing even hearing-impaired drivers to visually recognize them.

[1283] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1284] In this invention, the server includes a means for recording surrounding sounds in real time, a means for transmitting the recorded sound data to an analysis server, a means for analyzing the sound data received by the server and estimating the type, location, and direction of travel of the emergency vehicle, and a means for displaying the analysis results on the vehicle display, thereby enabling even hearing-impaired persons to visually recognize the approach of an emergency vehicle and to operate the vehicle safely.

[1285] The "means for recording ambient sound in real time" refers to a device or module that has the function of continuously capturing ambient sound and recording it as digital data.

[1286] The "means for transmitting recorded voice data to an analysis server" refers to a device or module for transferring recorded voice data to another server via a network such as the Internet.

[1287] "Means for analyzing the voice data received by the server and estimating the type, location, and direction of travel of the emergency vehicle" refers to a program or algorithm that has the function of analyzing the voice data received by the server, identifying each emergency vehicle, and determining the location and direction of travel of the sound source.

[1288] The "means for displaying the analysis results on the vehicle display" refers to a device or system for visually displaying the analysis results received from the server on a display device within the vehicle.

[1289] "Means for acquiring image data from a 360-degree camera and visually displaying safe avoidance areas from the captured images" refers to a device or module that has the function of acquiring image data captured by a camera mounted on a vehicle, analyzing the surrounding situation, and displaying areas where it is safe to proceed on a display device inside the vehicle.

[1290] A "network module for communicating voice data and analysis results in real time between an analysis server and a terminal" is a device or software that provides the network communication functions necessary for the analysis server and a terminal to send and receive data in real time.

[1291] MODE FOR CARRYING OUT THE INVENTION

[1292] As an embodiment of the present invention, the specific operation of the system will be described below. The present invention is a system that assists hearing-impaired people in visually recognizing approaching emergency vehicles when driving a car. The system is composed of a terminal, a server, and a user.

[1293] Terminal

[1294] Ambient sound recording

[1295] The device uses a voice recording module to record surrounding sounds in real time through the smartphone's microphone, and the recording data is stored in digital format, for example, using a microphone device with an appropriate sampling rate.

[1296] Sending audio data

[1297] The device transmits the recorded audio data to the analysis server via a network communication module, primarily via an Internet connection, for example via HTTP or HTTPS protocols.

[1298] Receiving and displaying analysis results

[1299] The terminal receives the analysis results from the server and displays them on the vehicle's display via the display audio linkage module. The actual display is linked to the vehicle's information display system and is presented in a format that is intuitively easy for the user to understand.

[1300] Use of a surrounding camera (optional)

[1301] Additionally, the device can utilize a 360-degree camera image acquisition module, which captures and analyzes video data from cameras installed in the vehicle and displays safe areas to avoid.

[1302] server

[1303] Analysis of audio data

[1304] The server uses a voice data analysis module to analyze the received voice data. This analysis can utilize an AI voice recognition engine (e.g., Google Cloud Speech-to-Text). It identifies the sound of an emergency vehicle siren and estimates its type, location, and direction of travel.

[1305] Sending analysis results

[1306] The server then transmits the analysis results to the device in real time. This communication is fast and secure, providing information on approaching emergency vehicles without delay.

[1307] User

[1308] Checking information and driving operations

[1309] The user checks the analysis results displayed on the vehicle's display and recognizes the approach of an emergency vehicle. For example, if the display shows "An ambulance approaching 30 meters behind to the right," the user follows the instructions and safely moves the vehicle to the left.

[1310] Specific examples

[1311] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module records this audio and sends it to the server via the network communication module. The server's audio data analysis module analyzes the audio data and identifies the location information of the emergency vehicle. The analysis result, "Ambulance approaching 30 meters behind and to the right," is generated and sent to the terminal. The terminal displays the received information on the vehicle's display. The user can confirm this display and move about 1 meter to the left, ensuring safe driving.

[1312] Prompt Sentence Examples

[1313] "Detect the sound of an emergency vehicle siren. Display specific instructions to inform the user that an emergency vehicle is approaching 30 meters behind and to the right."

[1314] The above is an embodiment of the present invention, and this system enables even hearing-impaired people to properly recognize the approach of an emergency vehicle and drive safely.

[1315] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1316] Step 1:

[1317] Ambient sound recording

[1318] The device uses the smartphone's audio recording module to record ambient audio in real time. The ambient sound is input from the smartphone's microphone. This audio is stored in a buffer as digitally recorded data. Specifically, the device sets a sampling rate and continuously captures audio and converts it into digital data, which is then used in subsequent processing steps.

[1319] Step 2:

[1320] Sending audio data

[1321] The device sends the recorded audio data to the analysis server via the network communication module. The recorded data buffer is used as input, and the data is sent to the server via an Internet connection. Specifically, the device divides the recorded data into packets at regular time intervals (e.g., every second) and sends them using the HTTP or HTTPS protocol. Once the transmission is complete, the buffer is cleared and prepared for the next data recording.

[1322] Step 3:

[1323] Analysis of audio data

[1324] The server analyzes the received voice data using the voice data analysis module. The voice data received from the device is used as input. Specifically, the server performs noise removal and normalization processing, and inputs this data into an AI voice recognition model. This model identifies the sound of an emergency vehicle siren and estimates the type, location, and direction of travel of the emergency vehicle. The analysis results are generated as JSON format data as output.

[1325] Step 4:

[1326] Sending analysis results

[1327] The server sends the generated analysis results to the terminal through the network communication module. The JSON data of the analysis results is used as input, and the data is sent to the terminal as output. Specifically, the server divides the data into packets and sends them to the terminal's receiving endpoint using the HTTPS protocol. This allows the analysis results to be updated in real time.

[1328] Step 5:

[1329] Receiving and displaying analysis results

[1330] The terminal displays the received analysis results on the vehicle's display via the display audio linkage module. The JSON data of the analysis results received from the server is used as input, and the information to be displayed on the vehicle's display is generated as output. Specifically, the terminal decodes the analysis results and converts them into an appropriate display format. For example, a message such as "Ambulance approaching 30 meters behind to the right" is displayed on the display.

[1331] Step 6:

[1332] Use of a surrounding camera (optional)

[1333] The device uses a 360-degree camera image acquisition module to acquire video data from the vehicle's onboard camera. It uses the camera's video data as input and generates information to display safe areas for avoidance as output. Specifically, the device analyzes the camera video in real time to identify the direction of travel and areas that can be avoided. This information is then visually displayed on the display.

[1334] Step 7:

[1335] Checking information and driving operations

[1336] The user checks the analysis results displayed on the vehicle's display and recognizes the approach of an emergency vehicle. The information displayed on the vehicle's display is used as input, and the user's driving operation is used as output. Specifically, the user checks the message "An ambulance is approaching 30 meters behind and to the right" and takes action to safely move the vehicle to the left. In this case, the user references the images from the all-around camera and the displayed information to safely avoid the approach.

[1337] (Application example 1)

[1338] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1339] It is difficult for the hearing impaired to visually recognize approaching emergency vehicles. This is particularly true when using autonomous vehicles, where a driver's failure to notice an approaching emergency vehicle can pose a serious risk. When such obstacles exist, effective evasive measures must be implemented quickly. However, current visual information systems lack sufficient functionality to provide real-time information on the location and direction of emergency vehicles.

[1340] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1341] In this invention, the server includes means for recording the siren sound of the emergency vehicle, means for transmitting the recorded audio data to an analysis server, means for the server to analyze the audio data received and generate emergency vehicle position information and response instructions, means for displaying the emergency vehicle position information and response instructions generated as a result of the analysis on a display of the vehicle, and means for displaying information about the approach of the emergency vehicle on a head-mounted display, thereby enabling hearing-impaired people to visually recognize the approach of an emergency vehicle in real time and take appropriate evasive action.

[1342] The "means for recording the siren sound of an emergency vehicle" is a device that records sounds in the surrounding environment of a vehicle and captures the siren sound of an emergency vehicle in real time.

[1343] The "means for transmitting recorded voice data to the analysis server" is a communication module for transmitting recorded voice data to the analysis server via a communication line such as the Internet.

[1344] "Means for analyzing the voice data received by the server and generating emergency vehicle location information and response instructions" refers to software or hardware that enables the server to analyze the voice data, identify the location and direction of the emergency vehicle, and provide instructions to the driver based on that information.

[1345] The "means for displaying the emergency vehicle location information and response instructions generated as the analysis result on the vehicle display" refers to an interface and display device for visually displaying the analysis result on the display inside the vehicle.

[1346] "Means for acquiring image data from a 360-degree camera and visually identifying safe avoidance areas from the captured image" refers to a device that analyzes images from a vehicle's 360-degree camera and visually identifies areas that can be safely avoided.

[1347] "Means having a network module for communicating voice data and analysis results in real time between the analysis server and the user's terminal" refers to a communication device for realizing data communication in real time between the analysis server and the vehicle's terminal.

[1348] The "means for displaying information about approaching emergency vehicles on a head-mounted display" is a device for displaying emergency vehicle location information and avoidance instructions on a head-mounted display.

[1349] As an embodiment of the present invention, the specific operation of the system will be described below.

[1350] overview

[1351] This system provides assistance to hearing-impaired people using self-driving vehicles, helping them visually recognize approaching emergency vehicles. The system consists of a terminal, a server, and a user. Terminals refer to smartphones, displays, head-mounted displays (HMDs), etc. installed in self-driving vehicles. The server analyzes the audio data and generates emergency vehicle location information and avoidance instructions. The user then performs safe driving operations based on the information provided visually.

[1352] System configuration

[1353] 1. Terminal (installed in autonomous vehicles)

[1354] Audio recording module: A device that uses the device's microphone to record surrounding audio in real time.

[1355] Network communication module: A device that sends recorded audio data to a server for analysis.

[1356] Display audio link module: A device that displays the analysis results received from the server on the display or HMD of the autonomous vehicle.

[1357] All-around camera image acquisition module (optional): A device that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[1358] 2. Server

[1359] Voice data analysis module: This module analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle. This analysis uses a deep learning model (for example, a model built with TensorFlow or PyTorch).

[1360] Network communication module: A device that transmits analysis results to the terminal.

[1361] 3. Users

[1362] Hearing-impaired person (driver): Check the approach of emergency vehicles through the self-driving vehicle's display or HMD and perform appropriate driving operations.

[1363] Program processing explanation

[1364] The device first records the surrounding sounds in real time, and the recorded data is sent to the analysis server via the network communication module.

[1365] The server analyzes the received audio data and identifies the siren sound of an emergency vehicle. From the identified siren sound, it estimates the type of emergency vehicle, its location, and its direction of travel. This analysis is performed using deep learning techniques (e.g., TensorFlow, PyTorch). The results are sent to the device using a network communication module.

[1366] The terminal receives the analysis results from the server and displays them on the autonomous vehicle's display or HMD via the display audio linkage module. For example, specific instructions such as "An ambulance is approaching 30 meters behind and to the right" can be displayed. In addition, by using the all-around camera image acquisition module, safe avoidance areas can also be visually displayed.

[1367] Users can check the autonomous vehicle's display or HMD, recognize the approach of an emergency vehicle based on the analysis results, and take appropriate evasive action. For example, if a user sees a message saying "An ambulance is approaching 30 meters behind and to the right," they will take action by moving about one meter to the left.

[1368] Specific examples

[1369] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module records this audio and sends it to the server via the network communication module. The server's analysis module analyzes the audio data and identifies the location information of the emergency vehicle. The analysis result is generated as "Ambulance approaching 30 meters behind and to the right," which is sent to the terminal. The terminal displays the received information on the display or HMD of the autonomous vehicle. The user confirms this display and moves about 1 meter to the left, ensuring safe driving.

[1370] Prompt Sentence Examples

[1371] For example, a prompt to a generative AI model might look like this:

[1372] "If an emergency vehicle siren is detected, generate a message with specific location and instructions on how to avoid it."

[1373] The above is an embodiment of the present invention, and this system enables even hearing-impaired people to properly recognize the approach of an emergency vehicle and drive safely.

[1374] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1375] Step 1:

[1376] The device records the surrounding audio. While the user is driving the autonomous vehicle, the audio recording module is activated when the engine starts and records the audio around the vehicle in real time. The input is the audio data of the surrounding environment, and the output is the recorded audio data.

[1377] Step 2:

[1378] The device sends the recorded voice data to the analysis server. The network communication module is used to upload the recorded voice data to the server via the Internet. The input is the recorded voice data, and the output is the voice data sent to the server.

[1379] Step 3:

[1380] The server analyzes the received audio data and identifies the siren sound of an emergency vehicle. The audio data analysis module uses deep learning technology to extract the characteristics of the siren sound and detect the approach of an emergency vehicle. The input is the audio data sent from the device, and the output is the siren sound detection result.

[1381] Step 4:

[1382] The server estimates the type, location, and direction of travel of the emergency vehicle based on the detection results. Based on the analysis results, the server identifies the emergency vehicle's detailed location and direction of travel, and generates corresponding driving instructions. The input is the detection result of the siren sound, and the output is data on the emergency vehicle's location and direction of travel.

[1383] Step 5:

[1384] The server generates analysis results and sends them to the terminal. The network communication module is used to return the analysis results to the terminal in real time. The input is data on the emergency vehicle's location and direction of travel, and the output is the analysis results sent to the terminal.

[1385] Step 6:

[1386] The terminal receives the analysis results and displays them on the autonomous vehicle's display or HMD. The display audio linkage module is used to visually display the emergency vehicle's location and response instructions. The input is the analysis results sent from the server, and the output is the emergency vehicle's location information and response instructions displayed on the display or HMD.

[1387] Step 7:

[1388] The user performs appropriate driving operations based on the information displayed on the display or HMD. Check the displayed information about approaching emergency vehicles and follow the instructions to safely avoid the vehicle. The input is the analysis results displayed on the display or HMD, and the output is the user's driving operations.

[1389] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1390] As an embodiment of the present invention, the specific operation of the system will be described below.

[1391] overview

[1392] When driving a car, hearing-impaired people have difficulty visually recognizing approaching emergency vehicles. This invention provides a system that detects emergency vehicle sirens in real time and displays their location information and response instructions on the vehicle's display. In addition, by combining it with an emotion engine that recognizes the user's emotions, driving assistance messages are optimized to the user's emotional state, supporting safe and comfortable driving.

[1393] System configuration

[1394] 1. Device (smartphone)

[1395] Audio recording module: Uses the smartphone's microphone to record surrounding audio in real time.

[1396] Network communication module: A module that sends recorded audio data to the analysis server.

[1397] Display Audio Link Module: A module that receives analysis results and displays them on the vehicle's display.

[1398] All-around camera image acquisition module (optional): A module that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[1399] Emotion recognition module: Recognizes user emotions in real time using cameras and sensors.

[1400] 2. Server

[1401] Voice data analysis module: Analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle.

[1402] Network communication module: A module that sends analysis results to the terminal.

[1403] Emotion analysis module: Analyzes received user emotion data and optimizes the content of driving assistance messages.

[1404] 3. Users

[1405] Hearing-impaired driver: Check the approach of emergency vehicles through a smartphone and vehicle display and perform appropriate driving operations.

[1406] Program processing

[1407] Terminal

[1408] The device first records the surrounding sounds in real time, and sends the recorded data to an analysis server via a network communication module. The device then uses cameras and sensors to recognize the user's emotional state, which is also sent to the server.

[1409] server

[1410] The server's voice data analysis module analyzes the received voice data and identifies the siren sound of an emergency vehicle. From the identified siren sound, the type of emergency vehicle, its location, and direction of travel are estimated. The analysis results are sent to the device via the network communication module. The emotion analysis module also analyzes the user's emotion data and optimizes the content of the driving assistance message to match the user's emotional state. For example, if the user is feeling stressed, more detailed and careful instructions are provided. If the user is relaxed, concise and smooth instructions are provided.

[1411] Terminal

[1412] The device receives analysis results from the server and displays driving assistance messages based on the driver's emotions on the vehicle's display via the display audio linkage module. Specifically, the device displays the message "An ambulance is approaching 30 meters behind the driver on the right" along with additional instructions tailored to the user's emotions. Safe avoidance areas can also be visually displayed by using the all-around camera image acquisition module.

[1413] User

[1414] The user checks the vehicle's display and reads driving assistance messages based on the analysis results and their emotions. For example, if the user sees a message saying "An ambulance is approaching 30 meters behind and to the right," they can follow the instructions and safely move the vehicle to the left. In addition, the response instructions are optimized to the user's emotional state, allowing the user to react more appropriately and quickly.

[1415] Specific examples

[1416] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module will record this audio and send it to the server via the network communication module. The server's analysis module will analyze the audio data and identify the location of the emergency vehicle. The analysis result will be "Ambulance approaching 30 meters behind and to the right," which will be sent to the device. At the same time, the emotion analysis module will analyze the user's emotional data, and if the user is feeling stressed, detailed instructions such as "Remain calm and move about 1 meter to the left" will be provided. Conversely, if the user is relaxed, simpler instructions will be given such as "Move to the left."

[1417] The above is a form for implementing the present invention, and this system allows even hearing-impaired people to properly recognize the approach of an emergency vehicle and receive driving assistance messages tailored to the user's emotional state, allowing them to drive safely and comfortably.

[1418] The processing flow will be explained below.

[1419] Step 1:

[1420] The device activates the audio recording module, which accesses the device's microphone and begins recording surrounding audio in real time.

[1421] Step 2:

[1422] The device sends the recorded audio data to the server via the network communication module, uploading the audio data approximately every second.

[1423] Step 3:

[1424] The server analyzes the received audio data using the audio data analysis module, and analyzes specific frequency patterns to detect the siren of an emergency vehicle.

[1425] Step 4:

[1426] The server identifies the type and location of the emergency vehicle based on the siren sound detection result, and estimates the direction and distance of the emergency vehicle based on the sound intensity and detection timing.

[1427] Step 5:

[1428] The server generates the analysis results and creates a text message containing the type of emergency vehicle, its location, direction of travel, and any other necessary information.

[1429] Step 6:

[1430] The device uses cameras and sensors to recognize the user's emotional state, and sends the recognized emotional data to a server.

[1431] Step 7:

[1432] The emotion data received by the server is analyzed by the emotion analysis module to determine whether the user is feeling stressed or anxious, or relaxed.

[1433] Step 8:

[1434] The server optimizes driving assistance messages based on the driver's emotional state, for example generating detailed and careful instructions when the driver is stressed and concise and smooth instructions when the driver is relaxed.

[1435] Step 9:

[1436] The analysis results and optimized driving assistance messages generated by the server are sent to the terminal using a network communication module.

[1437] Step 10:

[1438] The terminal receives the analysis results and driving assistance messages from the server and displays them on the vehicle's display via the display audio linkage module. For example, along with the message "An ambulance is approaching 30 meters behind and to the right," detailed instructions such as "Remain calm and move about 1 meter to the left" may be displayed.

[1439] Step 11:

[1440] (Optional) The device uses the omnidirectional camera image acquisition module to analyze the video from the omnidirectional camera and visually inform the user of a safe area to avoid. For example, it displays an instruction such as "There is room to evacuate about 1 meter to the left."

[1441] Step 12:

[1442] The user checks the vehicle's display and sees an instruction such as "An ambulance is approaching 30 meters behind and to the right."

[1443] Step 13:

[1444] The system performs safe driving maneuvers based on instructions provided by the user. For example, the system follows the instructions on the display to move about 1 meter to the left.

[1445] The above series of processing steps enables hearing-impaired drivers to properly recognize approaching emergency vehicles and drive safely. In addition, the emotion engine optimizes driving assistance messages, providing detailed assistance tailored to the user's mental state.

[1446] Example 2

[1447] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1448] When hearing-impaired people drive, it is difficult for them to visually recognize approaching emergency vehicles. This makes it difficult for them to quickly take appropriate evasive action, making safety a challenge. Therefore, there is a need for a system that can recognize approaching emergency vehicles in real time and provide appropriate support messages according to the driver's emotional state.

[1449] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for recording surrounding sounds; means for transmitting the recorded sound data and user emotion data to the analysis server; means having a network module for communicating the sound data, emotion data, and analysis results in real time between the analysis server and the user's terminal; means for analyzing the sound data received by the server and identifying the type, location information, and direction of travel of the emergency vehicle; means for analyzing the user emotion data received by the server and optimizing a driving assistance message; means for displaying the analysis result and the optimized driving assistance message on the vehicle display; and means for acquiring image data from a 360-degree camera and visually identifying a safe avoidance area from the captured image. This allows even hearing-impaired people to properly recognize the approach of an emergency vehicle and respond safely and quickly.

[1450] "Means for recording ambient sounds" refers to the device's ability to record ambient sounds in and around the vehicle in real time.

[1451] "Means for transmitting recorded voice data and user emotional data to an analysis server" refers to the function of transmitting the voice data collected by the terminal and the user's emotional state to a server via a network.

[1452] "Means having a network module for communicating voice data, emotion data, and analysis results in real time between the analysis server and the user's terminal" refers to a communications infrastructure for sending and receiving data in real time between the terminal and the server.

[1453] "Means for the server to analyze the received voice data and identify the type, location, and direction of travel of the emergency vehicle" refers to the function of the server to identify the sound of an emergency vehicle from the voice data and determine its type, location, and direction of travel.

[1454] "Means for analyzing the user's emotional data received by the server and optimizing driving assistance messages" refers to the function by which the server analyzes the user's emotional state and customizes safety instructions based on that information.

[1455] "Means for displaying the analysis results and optimized driving assistance messages on the vehicle's display" refers to the function of the terminal visually presenting the information obtained from the server on the vehicle's display.

[1456] "Means for acquiring image data from a 360-degree camera and visually identifying safe areas to avoid from the captured image" refers to a function that analyzes images acquired by a camera mounted on a vehicle and displays areas where it is possible to move safely.

[1457] As an embodiment for carrying out the present invention, the specific operation of the system will be described below.

[1458] overview

[1459] The purpose of this invention is to provide a system that allows hearing-impaired drivers to visually recognize approaching emergency vehicles in real time. In this system, a terminal records audio from inside and around the vehicle, and an analysis server analyzes the audio data. Furthermore, the system takes into account the user's emotional state and optimizes driving assistance messages to be displayed on the vehicle's display. This allows hearing-impaired drivers to respond safely and quickly.

[1460] System configuration

[1461] 1. Device (smartphone)

[1462] Audio recording module: Uses the smartphone's microphone to record surrounding audio in real time.

[1463] Network communication module: A module that transmits recorded voice data and user emotion data to the analysis server.

[1464] Display audio linkage module: A module that receives analysis results and driving assistance messages and displays them on the vehicle's display.

[1465] All-around camera image acquisition module (optional): A module that acquires images from all-around cameras and analyzes the direction of travel and avoidable areas.

[1466] Emotion recognition module: Recognizes user emotions in real time using cameras and sensors.

[1467] 2. Server

[1468] Voice data analysis module: Analyzes the voice data sent from the device and identifies the type and location of the emergency vehicle.

[1469] Network communication module: A module that sends analysis results to the terminal.

[1470] Emotion analysis module: Analyzes received user emotion data and optimizes the content of driving assistance messages.

[1471] 3. Users

[1472] Hearing-impaired driver: Check the approach of emergency vehicles through a smartphone and vehicle display and perform appropriate driving operations.

[1473] Program processing

[1474] Terminal

[1475] The device first records the surrounding sounds in real time, and sends the recorded data to an analysis server via a network communication module. Furthermore, the device uses cameras and sensors to recognize the user's emotional state, and also sends this information to the server.

[1476] server

[1477] The server's voice data analysis module analyzes the received voice data and identifies the siren sound of an emergency vehicle. From the identified siren sound, the type of emergency vehicle, its location, and direction of travel are estimated. The analysis results are sent to the device via the network communication module. The emotion analysis module also analyzes the user's emotion data and optimizes the content of the driving assistance message to match the user's emotional state. For example, if the user is stressed, more detailed and careful instructions are provided. If the user is relaxed, concise and smooth instructions are provided.

[1478] Terminal

[1479] The device receives the analysis results from the server and displays the optimized driving assistance message on the vehicle's display via the display audio linkage module. Specifically, the message "Ambulance approaching 30 meters behind to the right" is displayed along with additional instructions tailored to the user's emotions. In addition, by using the all-around camera image acquisition module, safe avoidance areas can also be visually displayed.

[1480] User

[1481] The user checks the vehicle's display and reads driving assistance messages based on the analysis results and their emotions. For example, if the user sees a message saying "An ambulance is approaching 30 meters behind and to the right," they can follow the instructions and safely move the vehicle to the left. In addition, the response instructions are optimized to the user's emotional state, allowing the user to react more appropriately and quickly.

[1482] Specific examples

[1483] For example, if the siren of an emergency vehicle is detected 30 meters behind and to the right, the audio recording module will record this audio and send it to the server via the network communication module. The server's analysis module will analyze the audio data and identify the location of the emergency vehicle. The analysis result will be "Ambulance approaching 30 meters behind and to the right," which will be sent to the device. At the same time, the emotion analysis module will analyze the user's emotional data, and if the user is feeling stressed, detailed instructions such as "Remain calm and move about 1 meter to the left" will be provided. Conversely, if the user is relaxed, simpler instructions will be given such as "Move to the left."

[1484] Prompt Sentence Examples

[1485] Examples of prompts for a generative AI model might include:

[1486] "What should I do if an ambulance is approaching 30 meters behind me to the right?"

[1487] "Please provide an example of how to instruct an emergency vehicle to evacuate if the user is experiencing stress."

[1488] "Please explain how to display a safe area to avoid using a 360-degree camera image."

[1489] This allows even hearing-impaired people to properly recognize approaching emergency vehicles and receive driving assistance messages tailored to the user's emotional state, enabling them to drive safely and comfortably.

[1490] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1491] Explain the program's processing flow in detail

[1492] Step 1:

[1493] The device uses the smartphone's audio recording module to record surrounding audio in real time.

[1494] Input: Surrounding environmental sounds

[1495] Data processing: Audio filtering (noise removal, volume adjustment)

[1496] Output: filtered audio data

[1497] Step 2:

[1498] The device sends the recorded voice data to the server via the network communication module, and at the same time, the device's camera and sensors capture the user's emotional data, which is also sent to the server.

[1499] Input: Filtered voice data and user emotion data

[1500] Data calculation: Data packing (combining voice data and emotion data)

[1501] Output: Sending consolidated data packets

[1502] Step 3:

[1503] The server's audio data analysis module analyzes the received audio data and identifies the siren sound of an emergency vehicle.

[1504] Input: Voice data in integrated data packets

[1505] Data Computing: Audio Analysis (Digital Signal Processing, Acoustic Pattern Recognition using Machine Learning Algorithms)

[1506] Output: Identification of emergency vehicle type, location, and direction of travel

[1507] Step 4:

[1508] The emotion analysis module of the server analyzes the received user emotion data and generates a driving assistance message.

[1509] Input: Emotion data in a consolidated data packet

[1510] Data calculation: Emotion analysis (facial recognition technology, estimation of emotional state through sensor data analysis)

[1511] Output: Driving assistance messages optimized for the user's emotional state

[1512] Step 5:

[1513] The server transmits the emergency vehicle information obtained as a result of the analysis and the optimized driving assistance message to the terminal via the network communication module.

[1514] Input: Emergency vehicle type, location, direction, optimized driver assistance message

[1515] Data calculation: Data packing (integration of emergency vehicle information and driver assistance messages)

[1516] Output: Sending consolidated data packets

[1517] Step 6:

[1518] The analysis results and driving assistance messages received by the terminal from the server are displayed on the vehicle's display via the display audio linkage module.

[1519] Input: Integrated data packet

[1520] Data processing: Data separation (extraction of emergency vehicle information and driving assistance messages)

[1521] Output: A message such as "Ambulance approaching 30m behind right" displayed on the vehicle's display

[1522] Step 7:

[1523] The user checks the vehicle's display and performs appropriate driving operations in accordance with the displayed analysis results and driving assistance messages.

[1524] Input: Information displayed on the vehicle's display

[1525] Specific operation: Safely evacuate the vehicle based on the analysis results and driver assistance messages

[1526] Output: Avoid danger through safe driving operations

[1527] In this way, by having each processing step work in cooperation with one another, a system is realized that allows the user to recognize the approach of an emergency vehicle in real time and respond safely and quickly.

[1528] (Application example 2)

[1529] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1530] There is a need for a system that enables hearing-impaired people, who have difficulty visually recognizing approaching emergency vehicles, to drive cars safely and comfortably. In particular, there is a technical challenge in providing support messages that adapt to the driver's emotional state when an emergency vehicle is approaching, enabling a quick and appropriate response.

[1531] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1532] In this invention, the server includes: [means for analyzing voice data and generating emergency vehicle location information and response instructions;] [means for generating a driving assistance message adapted based on the user's emotional state; and] [means for transmitting the analysis results and the driving assistance message to the user's terminal in real time.] This allows even hearing-impaired people to properly recognize the approach of an emergency vehicle and receive the optimal driving assistance message according to their emotional state, enabling safe and comfortable driving.

[1533] "Emergency vehicles" refers to ambulances, fire engines, police vehicles, etc. that emit sirens and travel in an emergency.

[1534] "Siren sound" refers to the special warning sound emitted by emergency vehicles.

[1535] "Means for recording" refers to a method of using a microphone to record surrounding sounds as digital audio data.

[1536] "Analysis Server" refers to a network server for processing and analyzing received voice and emotion data.

[1537] "Means for transmitting" refers to the method for transmitting data over a network to other devices or servers.

[1538] "Location Information" means information indicating the real-time geographic location of an emergency vehicle.

[1539] "Response instructions" refers to specific driving instructions to the driver regarding the approach of an emergency vehicle.

[1540] "Display" refers to a screen device installed inside a vehicle that visually displays analysis results and driving assistance messages.

[1541] An "emotion recognition module" refers to a device or system that uses sensors and cameras to detect a user's emotional state and digitize it.

[1542] "Driving assistance messages" refer to messages that provide the driver with instructions or warnings that are adapted to the current driving situation and the user's emotional state based on analyzed data.

[1543] A "surrounding camera" refers to a camera system that captures 360-degree images of the area around the vehicle and acquires video data.

[1544] A "safe avoidance area" refers to a space that allows drivers to safely move their vehicle when an emergency vehicle is approaching.

[1545] "Real-time" refers to a state in which data acquisition, processing, and communication occur instantly.

[1546] "Network module" refers to a hardware and software configuration for communicating data.

[1547] MODE FOR CARRYING OUT THE INVENTION

[1548] The following system will be described as an embodiment of the present invention. This system is equipped with voice recording and analysis functions and emotion recognition functions so that hearing-impaired people can sense the approach of an emergency vehicle with confidence and respond appropriately.

[1549] System Configuration and Operation

[1550] Terminal

[1551] The terminal (smartphone or in-car device) is equipped with the following modules:

[1552] 1. Audio recording module: Uses the smartphone's microphone to record surrounding sounds in real time, and the recorded audio data is saved in digital format.

[1553] 2. Network communication module: Sends recorded audio data to the analysis server via the network.

[1554] 3. Emotion Recognition Module: Using cameras and sensors, it collects the user's facial expressions and behavioral data to recognize their emotional state in real time.

[1555] 4. Display audio link module: Displays the analysis results and driving assistance messages received from the analysis server on the vehicle's display.

[1556] server

[1557] The server comprises several modules that perform the following functions:

[1558] 1. Voice data analysis module: Analyzes the received voice data and identifies the sound of an emergency vehicle siren. This analysis is performed using Google Speech-to-Text and Amazon Transcribe.

[1559] 2. Location information identification module: Identifies the location information of emergency vehicles and generates specific response instructions based on that information.

[1560] 3. Sentiment Analysis Module: Analyzes the emotional data sent by the user and generates driving assistance messages. This analysis is performed using Microsoft Azure Cognitive Services and IBM Watson Tone Analyzer.

[1561] 4. Network communication module: Transmits analysis results and driving assistance messages to the terminal in real time.

[1562] User

[1563] A user (hearing impaired person) uses the system as follows:

[1564] 1. Surrounding sounds are recorded, and when the sound of an emergency vehicle's siren is detected, the vehicle's display will show the vehicle's location and appropriate response instructions.

[1565] 2. Driving assistance messages are provided according to the user's emotional state, allowing the user to receive them intuitively.

[1566] 3. Information from the surrounding camera is also utilized to visually indicate areas where evasive action can be taken safely.

[1567] Specific examples

[1568] For example, if the sound of an emergency vehicle's siren is detected 30 meters behind and to the right, the audio recording module will record this sound. This recorded data is sent to the server via the network communication module, and the server's analysis module will identify the location of the emergency vehicle. The analysis result will be "Ambulance approaching 30 meters behind and to the right," which will be sent to the device. At the same time, the emotion analysis module will analyze the user's emotional data, and if the user is feeling stressed, detailed instructions such as "Remain calm and move about 1 meter to the left" will be provided. Conversely, if the user is relaxed, simpler instructions such as "Move to the left" will be provided.

[1569] Prompt Sentence Examples

[1570] Please provide an overview of a system that detects the approach of emergency vehicles and provides visual warnings to the driver. The system is designed for the hearing impaired, detects emergency vehicle sirens in real time, and displays their location on the vehicle's display. It also recognizes the driver's emotions and optimizes driver assistance messages accordingly. Specifically, please explain what hardware and software are used and what data processing is involved.

[1571] In this way, this invention is a system that provides drivers with optimal information in real time through voice analysis and emotion recognition, enabling hearing-impaired people to drive cars safely and comfortably.

[1572] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1573] Step 1:

[1574] The device uses a voice recording module to record surrounding sounds in real time, and the recorded voice data is stored in digital format.

[1575] Input: Ambient audio

[1576] Output: Digital audio data

[1577] Step 2:

[1578] The recorded voice data is sent to the analysis server via the network communication module.

[1579] Input: Digital audio data

[1580] Output: Audio data sent to the analysis server

[1581] Step 3:

[1582] The server's voice data analysis module analyzes the received voice data and identifies the sound of an emergency vehicle siren, using voice analysis software such as Google Speech-to-Text or Amazon Transcribe.

[1583] Input: Transmitted audio data

[1584] Data processing / calculation: Audio data analysis and siren sound identification

[1585] Output: Audio snippets containing emergency vehicle sirens and their metadata

[1586] Step 4:

[1587] The location information identification module identifies the location information of the emergency vehicle based on the metadata of the identified siren sound.

[1588] Input: Siren sound metadata

[1589] Data processing / calculation: Location information analysis

[1590] Output: Emergency vehicle location information

[1591] Step 5:

[1592] The emotion recognition module uses the device's camera and sensors to collect and analyze the user's facial expressions and behavioral data to recognize the user's emotional state. This data is also sent to the server.

[1593] Input: Camera images and sensor data from the device

[1594] Data processing / calculation: Analysis of user facial expression and behavior data

[1595] Output: User's emotional state data

[1596] Step 6:

[1597] The server's emotion analysis module analyzes the emotion data sent by the user and generates driving assistance messages using tools such as Microsoft Azure Cognitive Services and IBM Watson Tone Analyzer.

[1598] Input: User's emotional state data

[1599] Data processing / calculation: Emotional data analysis and generation of optimal driver assistance messages

[1600] Output: Emotion-based driving assistance message

[1601] Step 7:

[1602] The analysis server compiles the emergency vehicle's location information and the generated driving assistance message and transmits them to the terminal via the network communication module.

[1603] Input: Emergency vehicle location information, driving assistance message

[1604] Data processing / calculation: Integrating location information and messages

[1605] Output: Integrated location and driver assistance messages

[1606] Step 8:

[1607] The terminal displays the emergency vehicle location information and driving assistance message received from the analysis server on the vehicle's display via the display audio link module.

[1608] Input: Integrated location and driver assistance messages

[1609] Output: Emergency vehicle location information and driver assistance messages displayed on the vehicle display

[1610] Step 9:

[1611] The user checks the information displayed on the vehicle display and takes evasive action safely.

[1612] Input: Vehicle display information

[1613] Output: User's safe avoidance action

[1614] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1615] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1616] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1617] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1618] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1619] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1620] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1621] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1622] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1623] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1624] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1625] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1626] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1627] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1628] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1629] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1630] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1631] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1632] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1633] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1634] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1635] The following is further disclosed regarding the above embodiment.

[1636] (Claim 1)

[1637] [Means for recording emergency vehicle sirens,

[1638] [Means for transmitting the recorded voice data to an analysis server;

[1639] [Means for analyzing the voice data received by the server and generating location information and response instructions for the emergency vehicle;

[1640] [Means for displaying the emergency vehicle's location information and response instructions generated as a result of the analysis on a vehicle display;

[1641] A system including:

[1642] (Claim 2)

[1643] [Means for acquiring image data from a 360-degree camera and visually identifying a safe avoidance area from the captured image;

[1644] The system of claim 1 further comprising:

[1645] (Claim 3)

[1646] [Means for providing a network module for communicating voice data and analysis results in real time between the analysis server and the user's terminal;

[1647] 10. The system of claim 1, comprising:

[1648] "Example 1"

[1649] (Claim 1)

[1650] [Means for recording surrounding sounds in real time,

[1651] [Means for transmitting the recorded voice data to an analysis server;

[1652] [Means for analyzing the voice data received by the server and estimating the type, location information, and direction of travel of the emergency vehicle;

[1653] [Means for displaying the analysis results on a vehicle display;

[1654] A system including:

[1655] (Claim 2)

[1656] [The system according to claim 1 acquires image data from a 360-degree camera and visually displays a safe avoidance area from the captured image.

[1657] (Claim 3)

[1658] [The system according to claim 1, further comprising a network module for communicating voice data and analysis results in real time between the analysis server and the terminal.

[1659] "Application Example 1"

[1660] (Claim 1)

[1661] [Means for recording emergency vehicle sirens,

[1662] [Means for transmitting the recorded voice data to an analysis server;

[1663] [Means for analyzing the voice data received by the server and generating location information and response instructions for the emergency vehicle;

[1664] [Means for displaying the emergency vehicle's location information and response instructions generated as a result of the analysis on a vehicle display;

[1665] [Means for displaying approaching information of an emergency vehicle on a head-mounted display;

[1666] A system including:

[1667] (Claim 2)

[1668] [The system of claim 1 acquires image data from a 360-degree camera and visually identifies a safe avoidance area from the captured image.

[1669] (Claim 3)

[1670] [The system according to claim 1, further comprising a network module for communicating voice data and analysis results in real time between the analysis server and the user's terminal.

[1671] "Example 2: Combining Emotion Engines"

[1672] (Claim 1)

[1673] [Means for recording surrounding sounds and

[1674] [Means for transmitting the recorded voice data and the user's emotion data to an analysis server;

[1675] [Means for analyzing the received voice data by the server and identifying the type, location information, and direction of travel of the emergency vehicle;

[1676] [Means for analyzing the user's emotion data received by the server and optimizing the driving assistance message;

[1677] [Means for displaying the analysis results and optimized driving assistance messages on a display of the vehicle;

[1678] A system including:

[1679] (Claim 2)

[1680] [Means for acquiring image data from a 360-degree camera and visually identifying a safe avoidance area from the captured image] The system according to claim 1.

[1681] (Claim 3)

[1682] [The system according to claim 1, comprising a network module for communicating voice data, emotion data, and analysis results in real time between the analysis server and the user's terminal.

[1683] "Application example 2 when combining emotion engines"

[1684] (Claim 1)

[1685] [Means for recording emergency vehicle sirens,

[1686] [Means for transmitting the recorded voice data to an analysis server;

[1687] [Means for analyzing the voice data received by the server and generating location information and response instructions for the emergency vehicle;

[1688] [Means for displaying the emergency vehicle's location information and response instructions generated as a result of the analysis on a vehicle display;

[1689] [Means for detecting the user's emotional state with a sensor and transmitting the information to an analysis server;

[1690] [means for generating an adapted driving assistance message based on an emotional state;

[1691] [means for displaying the generated driving assistance message on a display of the vehicle;

[1692] A system including:

[1693] (Claim 2)

[1694] [Means for acquiring image data from a 360-degree camera and visually identifying a safe avoidance area from the captured image;

[1695] The system of claim 1 further comprising:

[1696] (Claim 3)

[1697] [Means for providing a network module for communicating voice data, emotion data, and analysis results in real time between the analysis server and the user's terminal;

[1698] 10. The system of claim 1, comprising: [Explanation of symbols]

[1699] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for recording the siren sound of an emergency vehicle; A means for transmitting the recorded voice data to an analysis server; A means for analyzing the voice data received by the server and generating location information and response instructions for the emergency vehicle; a means for displaying the emergency vehicle's location information and response instructions generated as a result of the analysis on a display of the vehicle; A system including:

2. A means for acquiring image data from a 360-degree camera and visually identifying a safe avoidance area from the captured image; The system of claim 1 further comprising:

3. a network module for communicating voice data and analysis results in real time between the analysis server and the user's terminal; The system of claim 1 , comprising:

Images