System

Abstract

To provide a system which explains, through voice, abnormality content indicated by alarm sound.SOLUTION: A system comprises: means for identifying an alarm sound produced by a vehicle; means for explaining an abnormality content to an identified alarm sound to a user, using a prompt text that instructs explanation of the abnormality content to the identified alarm sound to the user and a generative AI model; and means for explaining the countermeasures for the abnormality to the user, using a prompt text that instructs explanation of the countermeasures for the abnormality to the user and the generative AI model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] It is difficult for users to intuitively understand what the warning sounds emitted from the vehicle mean, and especially in an emergency, there may not be time to consult the manual. [Means for solving the problem]

[0005] To provide a system that automatically identifies warning sounds emitted from a vehicle and provides audible explanations to a user about the abnormality indicated by the warning sound, and further provides a method of dealing with the abnormality by using an internet search tool so that the user can deal with the abnormality. [Brief explanation of the drawings]

[0006] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, a voice recognition system is used to identify warning sounds emitted from a vehicle. This voice recognition system can learn various warning sounds emitted from the vehicle and identify the meaning of each warning sound. For example, it can identify warning sounds from an intelligent key or a collision warning function.

[0029] "Example 2"

[0030] A speech synthesis system is used as a means for explaining to the user the details of the abnormality corresponding to the identified alarm sound. This speech synthesis system acquires the details of the abnormality corresponding to the identified alarm sound as text data, converts it into voice, and conveys it to the user. For example, when an alarm sound indicating an engine abnormality sounds, the speech synthesis system will convey to the user, "There is an abnormality in the engine. Please stop the vehicle immediately and contact a professional repair shop."

[0031] "Example 3"

[0032] An internet search tool is used as a means for allowing the user to deal with the abnormality. This internet search tool searches the internet for information on the abnormality corresponding to the identified alarm sound and provides the results to the user. For example, if an alarm sound is heard indicating low tire pressure, the internet search tool provides the user with a solution such as "Your tire pressure is low. Please refill the tire with air at the nearest gas station or vehicle repair shop."

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The warning sound emitted from the vehicle is collected by a microphone.

[0036] Step 2: The collected alarm sounds are input into a speech recognition system, which uses a database of previously trained alarm sounds to identify the meaning of the input alarm sound.

[0037] Step 3: The user is notified of the identification result by, for example, displaying the identification result on a display in the vehicle.

[0038] "Example 2"

[0039] Step 1: Receive the identification result from the speech recognition system.

[0040] Step 2: Obtain the abnormality details corresponding to the identification result as text data. This text data is obtained, for example, from a database.

[0041] Step 3: The acquired text data is input into a speech synthesis system and converted into speech.

[0042] Step 4: The converted audio is output from the speaker to inform the user of the abnormality.

[0043] "Example 3"

[0044] Step 1: Receive the identification result from the speech recognition system.

[0045] Step 2: Use an internet search tool to search for information about the anomaly that corresponds to the identification result.

[0046] Step 3: The search results are notified to the user. This notification is performed, for example, by displaying the search results on the vehicle display.

[0047] Example 1

[0048] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0049] Conventional vehicle alarm sound identification systems have had the problem of making it difficult to quickly and accurately convey the meaning of an alarm sound to the user when it occurs. Furthermore, the information provided to users is insufficient to enable them to understand the meaning of the alarm sound and take appropriate action. This can lead to users being unable to take appropriate action, potentially compromising the safety and convenience of the vehicle.

[0050] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0051] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to a user the details of an abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the details of the abnormality, voice recognition means for analyzing the voice data, and means for notifying the user of the analysis results, thereby making it possible to quickly and accurately convey the meaning of the alarm sound to the user and provide information for the user to take appropriate action.

[0052] A "vehicle" is a mechanical device designed to travel on roads and powered by an engine or motor.

[0053] An "alarm sound" is an audio signal that is generated to notify the user of an abnormality in the vehicle or a situation that requires attention.

[0054] An "identification means" is a technique or device that analyzes a particular audio signal and determines what the audio means.

[0055] "Abnormality details" is information that indicates a state in which the vehicle is deviating from normal operation or a situation that requires attention.

[0056] A "user" is a person who drives or uses a vehicle.

[0057] "Explanatory means" refers to techniques and devices for communicating the details of the identified abnormality to the user in an easily understandable manner.

[0058] The "means for enabling a user to take action" refers to a technology or device that provides information or instructions to the user so that the user can take appropriate action against the identified abnormality.

[0059] "Speech recognition means" refers to technology or devices for analyzing voice data and converting it into text data.

[0060] "Means of notification" refers to the technology or device used to notify the user of the analysis results and abnormality details.

[0061] MODE FOR CARRYING OUT THE INVENTION

[0062] The present invention relates to a system for identifying an alarm sound emitted from a vehicle and notifying a user of the content of the abnormality. Specific embodiments of this system will be described below.

[0063] System Overview

[0064] This system uses voice recognition technology to identify warning sounds emitted by the vehicle. The voice recognition system can learn the various warning sounds emitted by the vehicle and identify the meaning of each warning sound. For example, it can identify the warning sound from the intelligent key or the warning sound from the collision warning function.

[0065] Hardware and software used

[0066] The server hosts the voice recognition system and receives voice data sent from the vehicle. The server analyzes the voice data using voice recognition software (e.g., a general voice recognition API). The analyzed voice data is compared with a database of pre-trained warning sounds.

[0067] When an alarm sound is generated, the terminal (e.g., a vehicle's on-board computer) transmits the sound data to a server. The terminal collects data from various sensors in the vehicle and records the timing and circumstances under which the alarm sound was generated.

[0068] To find out what the vehicle's alarm means, users will receive a notification via a smartphone app or on the vehicle's display, where they can view a detailed explanation of the alarm and what to do.

[0069] Specific examples

[0070] When an intelligent key warning sound is emitted while the vehicle is in motion, the device sends the audio data to the server. The server uses a common voice recognition API to analyze the audio data and compares the analysis results with an alarm sound database. The server identifies the sound as an intelligent key warning sound and notifies the user's smartphone app. The user receives a warning message through the app stating, "The intelligent key is not in the vehicle."

[0071] Prompt Sentence Examples

[0072] "Design a system to identify warning sounds emitted by a vehicle. The system uses speech recognition technology to analyze the warning sounds and notify the user of what each sound means. As a specific use case, consider a system that can identify the warning sounds from an intelligent key and the warning sounds from a collision warning function."

[0073] In this way, by clarifying the roles of servers, terminals, and users and naming specific hardware and software, it is possible to understand the system's processing in detail.

[0074] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0075] Program processing flow

[0076] Step 1:

[0077] The terminal monitors various sensors inside the vehicle and detects when an alarm is generated. For example, if the intelligent key is not inside the vehicle, an alarm will sound.

[0078] Input: Data from sensors inside the vehicle

[0079] Output: Signal to detect the occurrence of an alarm sound

[0080] Specific operation: The terminal collects data in real time from sensors inside the vehicle and determines whether an alarm has been sounded.

[0081] Step 2:

[0082] The device will record the alarm sound using the built-in microphone, and the recorded audio data will be temporarily stored on the device.

[0083] Input: Alarm sound

[0084] Output: Recorded audio data

[0085] Specific operation: When the device detects an alarm sound, it activates the built-in microphone to record the sound and saves the data in memory.

[0086] Step 3:

[0087] The terminal transmits the recorded voice data to the server using the vehicle's communication module (e.g., LTE module).

[0088] Input: Recorded audio data

[0089] Output: Audio data sent to the server

[0090] Specific operation: The terminal compresses the recorded data and sends it to the server via the vehicle's communication module.

[0091] Step 4:

[0092] The server analyzes the received voice data using a voice recognition API and converts it into text data.

[0093] Input: Audio data sent to the server

[0094] Output: Parsed text data

[0095] Specific operation: The server calls the speech recognition API and converts the speech data into text data.

[0096] Step 5:

[0097] The server compares the converted text data with a pre-trained alarm sound database.

[0098] Input: Parsed text data

[0099] Output: Identified alarm type

[0100] Specific operation: The server compares the text data with the alarm sound database to identify matching alarm sound types.

[0101] Step 6:

[0102] The server notifies the user's smartphone app of the identification result using a push notification service (e.g., a general push notification service).

[0103] Input: identified alarm type

[0104] Output: Notification sent to the user's smartphone

[0105] Specific operation: The server formats the identification results and sends them to the user's smartphone via a push notification service.

[0106] Step 7:

[0107] The user receives a notification via a smartphone app, for example, a message saying "Intelligent key not in vehicle."

[0108] Input: Notification sent to the user's smartphone

[0109] Output: Notification message received by the user

[0110] Specific operation: The user checks the notification on their smartphone and takes action, such as returning the intelligent key to the vehicle, if necessary.

[0111] In this way, by explaining each processing step in detail and clearly indicating the inputs and outputs, the processing flow of the system can be understood in detail.

[0112] (Application example 1)

[0113] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0114] Conventional vehicle alarm sound identification systems are required not only to identify alarm sounds but also to explain the nature of the abnormality to the user and provide countermeasures. However, these systems lack the functionality to notify the user of the type and meaning of the alarm sound in real time and to suggest necessary countermeasures or actions. This makes it difficult for the user to respond quickly and appropriately.

[0115] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0116] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to a user the details of an abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the details of the abnormality, means for notifying the user of the type and meaning of the identified alarm sound, and means for suggesting necessary countermeasures or actions, thereby enabling the user to understand the type and meaning of the alarm sound in real time and to respond promptly and appropriately.

[0117] A "vehicle" is a mechanical device designed to travel on roads and powered by an engine or motor.

[0118] An "alarm sound" is an audio signal emitted by a vehicle and is used to notify of abnormalities or dangers.

[0119] An "identification means" is a device or system that analyzes a specific audio signal and determines its type and meaning.

[0120] "Abnormality details" refers to the specific circumstances or conditions of a problem or malfunction that prevents the vehicle from operating normally.

[0121] The "means for explaining to the user" refers to a device or system for communicating the identified abnormality to the user in an easy-to-understand manner.

[0122] The "means for enabling a user to take action" refers to a device or system that provides information and instructions for the user to take appropriate action against the abnormality.

[0123] The "notification means" is a device or system for informing the user of the type of identified alarm sound and its meaning.

[0124] "Proposed measures" are devices or systems that present specific measures or actions that users should take.

[0125] The system for implementing this invention identifies warning sounds emitted by a vehicle, explains the nature of the abnormality to the user, and provides countermeasures. The system uses a microphone installed in the vehicle, a smartphone, the voice recognition library "SpeechRecognition," and the machine learning library "TENSORFLOW (registered trademark)."

[0126] System configuration

[0127] 1. Hardware

[0128] Vehicle-mounted microphones: collect warning sounds emitted by the vehicle.

[0129] Smartphone: A device for processing voice data and notifying the user.

[0130] 2. Software

[0131] Speech Recognition library: Analyzes collected voice data and identifies alarm sounds.

[0132] TensorFlow machine learning library: Uses an alarm sound identification model to determine the type of alarm sound.

[0133] Processing flow

[0134] 1. Acquisition of audio data: A microphone installed in the vehicle collects the alarm sound and transmits it to a smartphone.

[0135] 2. Preprocessing of audio data: Using the speech recognition library "SpeechRecognition" on the smartphone, the audio data is analyzed and the features of the alarm sound are extracted.

[0136] 3. Alarm sound identification: Using the machine learning library "TensorFlow," audio data is input into a pre-trained alarm sound identification model to identify the type of alarm sound.

[0137] 4. Notification and Suggestion: Inform the user of the identified alarm type and its meaning, and suggest any necessary countermeasures or actions.

[0138] Specific examples

[0139] For example, if the vehicle emits a collision warning sound, the system will identify the sound and notify the smartphone, "There is a risk of a collision. Please apply the brakes."

[0140] Prompt Sentence Examples

[0141] "Develop an application that identifies warning sounds emitted by a vehicle and notifies the user of their type and meaning. The libraries used are SpeechRecognition and TensorFlow. Types of warning sounds include intelligent key warning sounds, collision warning sounds, and engine abnormality warning sounds."

[0142] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0143] Step 1:

[0144] A microphone installed in the vehicle collects the alarm sound and transmits it to a smartphone.

[0145] Input: Vehicle alarm sound

[0146] Output: Audio data sent to your smartphone

[0147] How it works: The vehicle's microphone detects an alarm sound and transmits the audio data in real time to a smartphone, which receives the audio data and proceeds to the next processing step.

[0148] Step 2:

[0149] The voice recognition library "SpeechRecognition" is used on the smartphone to analyze the voice data and extract the characteristics of the alarm sound.

[0150] Input: Voice data sent to smartphone

[0151] Output: Extracted alarm sound feature data

[0152] Specific operation: The smartphone inputs the received voice data into the "SpeechRecognition" library and analyzes the voice data. As a result of the analysis, feature data of the alarm sound is extracted. This feature data is used in the next step.

[0153] Step 3:

[0154] Using the machine learning library "TensorFlow," audio data is input into a pre-trained alarm sound identification model to identify the type of alarm sound.

[0155] Input: Extracted alarm sound feature data

[0156] Output: Identified alarm type

[0157] How it works: The smartphone inputs the extracted feature data into the TensorFlow library and uses the alarm sound classification model to identify the type of alarm sound.

[0158] Step 4:

[0159] The system notifies the user of the type of alarm sound identified and its meaning, and suggests necessary countermeasures or actions.

[0160] Input: Identified alarm type

[0161] Output: The type of alarm that will be notified to the user, its meaning, and any suggested countermeasures or actions.

[0162] Specific behavior: The smartphone notifies the user of the meaning of the identified warning sound based on the type of sound. For example, if a "collision warning sound" is identified, the smartphone will notify the user, "There is a risk of a collision. Please apply the brakes." It will also suggest necessary countermeasures and actions.

[0163] Example 2

[0164] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0165] Conventional vehicle alarm systems have the problem that it is difficult for users to quickly and accurately understand the nature of the abnormality when an alarm sounds. In particular, when an alarm sounds while driving, users must refer to the manual to understand the meaning of the alarm, which can hinder safe driving. In addition, because users cannot immediately know the appropriate way to deal with the abnormality, there is a possibility that they will be slow to respond in situations that require a prompt response.

[0166] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0167] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for acquiring details of an abnormality corresponding to the identified alarm sound as text data, voice synthesis means for converting the acquired text data into voice data, means for notifying the user of the generated voice data, and means for enabling the user to deal with the details of the abnormality. Thus, when an alarm sound is emitted, the user can immediately understand the details of the abnormality through voice and can quickly learn an appropriate way to deal with it.

[0168] An "alarm sound" is an audio signal emitted by a vehicle to notify of an abnormality or warning.

[0169] The "means for identifying" refers to a device or software that analyzes the received alarm sound and identifies its type and content.

[0170] "Anomaly details" is information that indicates the specific problem or situation corresponding to the identified alarm sound.

[0171] "Text data" is data that expresses the content of an abnormality as text information.

[0172] "Voice data" refers to a voice signal generated from text data using voice synthesis technology.

[0173] "Speech synthesis means" refers to a device or software for converting text data into voice data.

[0174] The "notification means" refers to a device or software for notifying the user of the generated voice data.

[0175] "Means for enabling a user to take action" refers to devices or software that provide information and methods for the user to take appropriate action against the abnormality.

[0176] MODE FOR CARRYING OUT THE INVENTION

[0177] The present invention relates to a system for identifying an alarm sound emitted from a vehicle and notifying a user of the content of the abnormality by voice. Specific embodiments of this system will be described below.

[0178] Hardware and software used

[0179] Hardware: Servers, devices (smartphones, tablets, etc.)

[0180] Software: Speech synthesis system (e.g., Google® Text-to-Speech API), alarm sound identification system (e.g., speech recognition model using TensorFlow)

[0181] System configuration

[0182] 1. Receiving an alarm sound

[0183] The device records the alarm sounds generated inside the vehicle in real time via a microphone.

[0184] The terminal transmits the recorded voice data to the server.

[0185] 2. Alarm sound identification

[0186] The server inputs the received audio data into the warning sound identification system.

[0187] The server uses an alarm sound identification system to analyze the audio data and identify the type of alarm sound.

[0188] 3. Obtaining the details of the abnormality

[0189] The server retrieves the abnormality content corresponding to the identified alarm sound from the database.

[0190] For example, as the abnormality content corresponding to "engine abnormality," text data such as "There is an abnormality in the engine. Stop the vehicle immediately and contact a specialized repair company" is acquired.

[0191] 4. Generating Audio Data

[0192] The server inputs the acquired text data into a speech synthesis system to generate speech data.

[0193] For example, using the Google Text-to-Speech API, we can generate voice data that says, "There is a problem with the engine. Stop the vehicle immediately and contact a professional repair shop."

[0194] 5. Notice to Users

[0195] The server transmits the generated voice data to the terminal.

[0196] The terminal reproduces the received audio data and notifies the user of the details of the abnormality.

[0197] Specific examples

[0198] For example, if an alarm sounds indicating an engine malfunction while the user is driving a car, the system operates as follows.

[0199] 1. The device records the alarm sound and sends the audio data to the server.

[0200] 2. The server uses the alarm sound identification system to identify the sound as "engine abnormality."

[0201] 3. The server retrieves text data from the database stating, "There is an engine malfunction. Stop the vehicle immediately and contact a professional repair company."

[0202] 4. The server converts the text data into voice data using a voice synthesis system.

[0203] 5. The device plays the generated voice data and informs the user of the abnormality.

[0204] Prompt Sentence Examples

[0205] Input: "Please explain what to do if an engine malfunction alarm sounds."

[0206] Output: "There is an engine malfunction. Stop the vehicle immediately and contact a professional repair service."

[0207] The system aims to ensure user safety by quickly identifying alarm sounds and providing users with audible instructions on how to respond appropriately.

[0208] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0209] Step 1:

[0210] Receiving alarm sounds

[0211] The device records the alarm sounds generated inside the vehicle in real time via a microphone.

[0212] Input: Alarm sound generated inside a car

[0213] Output: Recorded audio data

[0214] What it does: The device's microphone captures the alarm sound and uploads the audio data to a server via the Internet.

[0215] Step 2:

[0216] Alarm sound identification

[0217] The server inputs the received audio data into the warning sound identification system.

[0218] Input: Audio data sent from the device

[0219] Output: Type of alarm identified (e.g. engine abnormality, brake abnormality, etc.)

[0220] Specific operation: The server analyzes the audio data and identifies the type of alarm sound using an alarm sound identification system (e.g., a speech recognition model using TensorFlow).

[0221] Step 3:

[0222] Acquisition of abnormality details

[0223] The server retrieves the abnormality content corresponding to the identified alarm sound from the database.

[0224] Input: Identified alarm type

[0225] Output: Text data about the problem (e.g., "There is a problem with the engine. Stop the vehicle immediately and contact a professional repair shop.")

[0226] Specific operation: The server executes a database query to obtain the corresponding text data of the anomaly.

[0227] Step 4:

[0228] Generating audio data

[0229] The server inputs the acquired text data into a speech synthesis system to generate speech data.

[0230] Input: Text data of the anomaly

[0231] Output: Generated audio data

[0232] Specific operation: The server sends text data to a speech synthesis API (e.g., Google Text-to-Speech API) and receives the generated speech data.

[0233] Step 5:

[0234] User Notification

[0235] The server transmits the generated voice data to the terminal.

[0236] The terminal reproduces the received audio data and notifies the user of the details of the abnormality.

[0237] Input: Generated audio data

[0238] Output: A voice message to be announced to the user

[0239] Specific operation: The device plays audio data over the speaker, telling the user, "There is an engine malfunction. Please stop the vehicle immediately and contact a professional repair company."

[0240] (Application example 2)

[0241] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0242] In conventional vehicles, when an alarm sound is emitted, it is difficult for the user to immediately understand the meaning and take appropriate action. In particular, in autonomous vehicles, users are required to quickly understand the meaning of the alarm sound and take appropriate action, but current systems do not provide this support sufficiently. This may result in a decrease in user safety and vehicle operating efficiency.

[0243] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0244] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to the user the details of the abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the details of the abnormality, and means for audibly communicating the details of the abnormality to the user using a voice synthesis system, thereby enabling the user to immediately understand the meaning of the alarm sound and take appropriate action.

[0245] A "vehicle" is a mechanical device for moving on land, powered by an engine or motor.

[0246] An "alarm sound" is an audio signal that is generated to warn of an abnormality or danger in the vehicle.

[0247] "Discrimination" means distinguishing a particular sound or signal from others and recognizing its content.

[0248] "Abnormality details" is information that indicates a state or problem that deviates from the normal operation of the vehicle.

[0249] "User" refers to a person who operates or uses a vehicle.

[0250] "Explanation" is the act of conveying specific information or content to others so that they can understand it.

[0251] "Response" means taking appropriate action in response to problems or abnormalities that have occurred.

[0252] A "speech synthesis system" is a technology or device that converts text data into speech.

[0253] An "Internet search tool" is software or a service that allows you to search for information on the Internet and obtain the information you need.

[0254] The following system is constructed as an embodiment of this invention. First, a vehicle is equipped with a voice recognition system for identifying warning sounds. This voice recognition system identifies warning sounds generated inside the vehicle in real time and acquires their contents as text data. The speech_recognition library is used for voice recognition.

[0255] Next, a database is prepared to acquire the details of the abnormality corresponding to the identified alarm sound. This database stores the details of the abnormality corresponding to each alarm sound in text format. For example, for an alarm sound that says "engine abnormality," the following abnormality details are stored: "There is an abnormality in the engine. Stop the vehicle immediately and contact a professional repair shop."

[0256] The acquired abnormality information is converted into voice using a voice synthesis system. The gTTS library is used for voice synthesis. This voice synthesis system converts text data into voice data and conveys it to the user. This allows the user to immediately understand the meaning of the alarm sound and take appropriate action.

[0257] As a specific example, consider the case where an alarm sounds indicating an engine malfunction. In this case, the speech recognition system receives the text "engine malfunction" and retrieves the corresponding malfunction details from the database. The speech synthesis system then generates a voice message to the user saying, "There is an engine malfunction. Stop the vehicle immediately and contact a professional repair shop."

[0258] Examples of prompts to input to a generative AI model might include:

[0259] I would like to develop a system that can identify alarm sounds and inform the user of the nature of the abnormality by voice. Specifically, when an alarm indicating an engine abnormality sounds, a voice message should be sent saying, "There is an abnormality in the engine. Please stop the vehicle immediately and contact a professional repair shop." Please provide the Python code to achieve this.

[0260] This system allows users to quickly understand the meaning of the warning sound and take appropriate action, thereby improving vehicle safety and operational efficiency.

[0261] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0262] Step 1:

[0263] The server uses a voice recognition system to identify warning sounds generated inside the vehicle in real time. Specifically, it uses a voice recognition system (speech_recognition library) to convert the warning sounds into text data. The input is the voice data of the warning sounds, and the output is the text data of the identified warning sounds.

[0264] Step 2:

[0265] The server retrieves the corresponding abnormality details from the database based on the text data of the identified alarm sound. Specifically, it searches the database that stores the abnormality details corresponding to the alarm sound and retrieves the text data of the corresponding abnormality details. The input is the text data of the identified alarm sound, and the output is the text data of the corresponding abnormality details.

[0266] Step 3:

[0267] The server converts the acquired text data of the anomaly into voice data using a voice synthesis system. Specifically, it uses a voice synthesis system (gTTS library) to convert the text data into voice data. The input is the text data of the anomaly, and the output is voice data.

[0268] Step 4:

[0269] The terminal transmits the generated voice data to the user. Specifically, the voice data is played through a speaker to inform the user of the abnormality by voice. The input is the voice data, and the output is the voice information that the user hears.

[0270] Step 5:

[0271] The user takes appropriate action based on the details of the abnormality communicated by voice. Specifically, the user follows the instructions communicated by voice and takes appropriate action, such as stopping the vehicle or contacting a repair shop. The input is voice information, and the output is the user's response action.

[0272] Example 3

[0273] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0274] Conventional vehicle systems have a problem in that when an alarm sound is emitted, it is difficult for the user to quickly and accurately understand the nature of the abnormality and find an appropriate way to deal with it. In particular, when the abnormality is complex or the user does not have technical knowledge, there is a high possibility that appropriate action will be delayed. To solve this problem, a system is needed that can analyze the alarm sound, identify the nature of the abnormality, and quickly provide a way to deal with it.

[0275] The identification process by the identification processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes means for receiving an alarm sound emitted from a vehicle, means for analyzing the received alarm sound to identify the nature of the abnormality, means for searching for a solution from the Internet based on the identified nature of the abnormality, and means for providing the searched solution to the user. This allows the user, when an alarm sound is emitted, to quickly and accurately understand the nature of the abnormality and immediately obtain an appropriate solution.

[0276] A "vehicle" is a mechanical device that uses an engine or motor as a power source for movement, and primarily refers to automobiles that run on roads.

[0277] The "alarm sound" is an audio signal that is generated to alert the user to an abnormality or danger in the vehicle, and is intended to draw the user's attention.

[0278] "Means for receiving" refers to devices or systems for receiving data transmitted from a vehicle, specifically using communication technologies such as Bluetooth or Wi-Fi.

[0279] "Means of analysis" refers to software or algorithms used to process and understand the received data.

[0280] "Abnormality" refers to the specific condition of a problem or malfunction that prevents the vehicle from operating normally.

[0281] "Means of identification" refers to methods or techniques for identifying specific anomalies based on the analysis results.

[0282] "Internet solution search" refers to technology or software that uses Internet resources to find solutions to specific anomalies.

[0283] "Means of providing" refers to the method or device for informing the user of the searched solution, specifically using a display device such as a smartphone or an in-car display.

[0284] MODE FOR CARRYING OUT THE INVENTION

[0285] The present invention is a system for analyzing an alarm sound emitted from a vehicle, identifying the nature of the abnormality, and providing a user with a method for dealing with the abnormality. Specific embodiments of this system will be described below.

[0286] System configuration

[0287] This system mainly uses the following hardware and software:

[0288] Server: The central component that receives, analyzes, searches, and serves data.

[0289] Terminal: A device used by a user, including a smartphone or an in-car display.

[0290] Speech recognition software: Analyzes alarm sounds using the Google Cloud Speech-to-Text API, among others.

[0291] Search engine API: Search for solutions on the Internet using Google Search API, etc.

[0292] Program processing

[0293] The server receives the alarm sound data sent from the user's vehicle. The received data is analyzed using voice recognition software. Based on the analysis results, the server identifies the nature of the abnormality. Based on the identified abnormality, the server uses a search engine API to search for related information from the Internet. The server extracts the most appropriate solution from the search results and provides it to the user.

[0294] Specific examples

[0295] For example, if a user's vehicle emits an alarm sound indicating low tire pressure, the server analyzes the alarm sound and identifies the abnormality as "Your tire pressure is low." The server then uses the Google Search API to search for "What to do if your tire pressure is low" and provides the user with the solution of "Please refill your tires with air at the nearest gas station or vehicle repair shop."

[0296] Prompt Sentence Examples

[0297] Analyze the alarm sound sent from the user's vehicle and identify the abnormality. Based on the identified abnormality, search the Internet for a solution and provide it to the user.

[0298] This system allows the user to quickly deal with any abnormalities in the vehicle. The server performs all processes from analyzing the alarm sound to providing a solution, so the user does not need to perform any complicated operations. This improves the safety of the vehicle and the convenience of the user. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[0299] Step 1: Receiving the alarm sound

[0300] Input: Alarm sound data sent from the user's vehicle

[0301] Specific operation: When the user's vehicle emits an alarm, the audio data is captured by the vehicle's sensors. The device then transmits the captured audio data to the server.

[0302] Output: Alarm sound data sent to the server

[0303] Step 2: Analyze the alarm sound

[0304] Input: Alarm sound data sent to the server

[0305] Specific operation: The server sends the received alarm sound data to speech recognition software (e.g., Google Cloud Speech-to-Text API) and converts it into text data. The server then analyzes the converted text data to identify the content of the alarm sound.

[0306] Output: Parsed text data and alarm sound content

[0307] Step 3: Identifying the anomaly

[0308] Input: Parsed text data and alarm sound content

[0309] Specific operation: The server identifies the abnormality based on the analysis results. For example, it identifies the abnormality as "tire pressure is low."

[0310] Output: Identified anomalies

[0311] Step 4: Internet Search

[0312] Input: Identified anomaly

[0313] What it does: The server uses the Google Search API to search for "What to do when tire pressure is low." The server then extracts the most relevant information from the search results.

[0314] Output: Search results and extracted solutions

[0315] Step 5: Provide solutions

[0316] Input: Search results and extracted solutions

[0317] Specific operation: The server sends the extracted solutions to the user's device. The user checks the solutions displayed on the device and takes appropriate action.

[0318] Output: Actions displayed on the user's device

[0319] In this way, the user can quickly respond to vehicle abnormalities. The server handles everything from analyzing the alarm sound to providing countermeasures, so the user does not need to perform complicated operations. This improves vehicle safety and user convenience.

[0320] (Application example 3)

[0321] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0322] Conventional vehicle systems have a problem in that when an alarm sound is issued, it is difficult for the user to understand the nature of the abnormality and quickly find an appropriate solution. In particular, when the abnormality is complex or the user does not have technical knowledge, appropriate action is often delayed. This leads to issues such as reduced vehicle safety and user convenience.

[0323] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0324] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to a user the nature of an abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the abnormality, means for searching information relating to the abnormality from the Internet, and means for providing the search results to the user. This allows the user to quickly understand the nature of the abnormality when an alarm sound is emitted and to search for and obtain an appropriate solution from the Internet. Furthermore, by notifying the user of the search results by voice, the user can safely receive information while driving.

[0325] A "vehicle" is a mechanical device for moving on roads, powered by an engine or motor.

[0326] An "alarm sound" is an audio signal that is generated to warn of an abnormality or danger in the vehicle.

[0327] An "identification means" is a device or software that detects a particular sound or signal and determines what it means.

[0328] "Abnormality details" refer to the specific condition or cause of a problem or malfunction that prevents the vehicle from operating normally.

[0329] A "user" is a person who operates a vehicle or receives information about the vehicle.

[0330] "Explanatory means" refers to a method or device for helping the user understand the nature of the abnormality.

[0331] The "means for enabling a user to take action" refers to a method or device that supports the user in taking appropriate action in response to the abnormality.

[0332] The "Internet" is an information and communications network that interconnects computer networks around the world.

[0333] A "search means" is a method or device for locating specific information on the Internet.

[0334] "Means for providing" refers to the method or device for conveying search results and information to users.

[0335] "Audio notification" is a method of conveying information using audio.

[0336] The system for carrying out the present invention includes a sensor mounted on a vehicle, a server with an Internet connection function, and a terminal for providing information to a user. The specific configuration and operation of the system will be described below.

[0337] System configuration

[0338] 1. Vehicle sensors

[0339] Vehicles are equipped with various sensors to detect tire pressure, engine temperature, collision warnings, etc. These sensors emit an alarm when they detect an abnormality.

[0340] 2. Server

[0341] The server includes software to identify the warning sounds sent from the vehicle and identify the nature of the abnormality. Specifically, it uses programming languages ​​such as Python to identify the warning sounds and the nature of the abnormality. The server also has the function of searching for information on the abnormality using internet search tools.

[0342] 3. User Device

[0343] The user terminal is a device such as a smartphone or tablet that receives information sent from the server. The user terminal is equipped with a voice notification function that notifies the user of the nature of the abnormality and how to deal with it.

[0344] System Operation

[0345] 1. Abnormality detection and alarm generation

[0346] When the vehicle's sensors detect an abnormality, an alarm sounds, which is transmitted to the server via the vehicle's microphone.

[0347] 2. Identifying alarm sounds and identifying abnormalities

[0348] The server identifies the received alarm sound and determines the abnormality. For example, if the tire pressure is low, the server determines the abnormality as "Tire pressure is low."

[0349] 3. Internet search and information provision

[0350] The server searches the Internet based on the identified anomaly to obtain information on how to deal with the problem, and then transmits the obtained information to the user terminal.

[0351] 4. Audio Notifications

[0352] The user terminal notifies the user of the received information by voice, allowing the user to receive information safely while driving.

[0353] Specific examples

[0354] For example, if a vehicle sensor detects an abnormality such as low tire pressure, the system will operate as follows:

[0355] 1. The sensor detects an abnormal sound and sends it to the server.

[0356] 2. The server identifies the abnormality "Tire pressure is low."

[0357] 3. The server performs an internet search and obtains the solution, "Please refill the air at the nearest gas station or vehicle repair shop."

[0358] 4. The user device will give an audio notification and tell the user how to deal with the problem.

[0359] Prompt Sentence Examples

[0360] "Your tire pressure is low. Please refill it at the nearest gas station or repair shop. Search the internet for solutions."

[0361] In this way, the user can deal with the abnormality promptly and appropriately.

[0362] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0363] Step 1:

[0364] The vehicle's sensors detect an abnormality and sound an alarm.

[0365] Input: Abnormality detection signal from vehicle sensor

[0366] Output: Alarm sound

[0367] Specific operation: If the tire pressure drops, the tire pressure sensor will detect the abnormality and sound an alarm.

[0368] Step 2:

[0369] The warning sound is transmitted to the server through a microphone in the vehicle.

[0370] Input: Alarm

[0371] Output: Audio data to the server

[0372] Specific operation: A microphone inside the vehicle captures the warning sound and sends the audio data to a server.

[0373] Step 3:

[0374] The server identifies the received alarm sound and identifies the nature of the abnormality.

[0375] Input: Audio data

[0376] Output: Abnormality details

[0377] Specific operation: The server uses voice recognition software to analyze the alarm sound and identify the abnormality: "Tire pressure is low."

[0378] Step 4:

[0379] The server performs an internet search based on the identified abnormality and obtains information on how to deal with the problem.

[0380] Input: Abnormality details

[0381] Output: Information on what to do

[0382] Specific operation: The server uses an internet search engine to search for and retrieve the solution, "Your tire pressure is low. Please refill the tire with air at the nearest gas station or vehicle repair shop."

[0383] Step 5:

[0384] The server transmits the acquired information to the user terminal.

[0385] Input: Information on how to proceed

[0386] Output: Sending information to the user's device

[0387] Specific operation: The server sends the acquired information on how to deal with the problem to the user terminal.

[0388] Step 6:

[0389] The user terminal notifies the user of the received information by voice.

[0390] Input: Information on how to proceed

[0391] Output: Audio notification

[0392] Specific operation: The user device uses speech synthesis software to notify the user by voice the message, "Your tire pressure is low. Please refill the tire with air at the nearest gas station or vehicle repair shop."

[0393] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0394] "Example 1"

[0395] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system identifies an alarm sound emitted from a vehicle and explains to the user the nature of the abnormality corresponding to the identified alarm sound. Furthermore, the system allows the user to deal with the abnormality. The system further includes an emotion engine that recognizes the user's emotion. This emotion engine recognizes the user's emotion from, for example, the user's tone of voice, facial expression, choice of words, etc.

[0396] "Example 2"

[0397] The emotion engine adjusts the way it explains the abnormality depending on the user's emotions. For example, if the user is panicking, the emotion engine will explain the abnormality in a concise and clear manner. On the other hand, if the user is calm, the emotion engine will explain the abnormality in more detail. This provides an optimal explanation according to the user's emotional state, allowing the user to deal with the abnormality appropriately.

[0398] "Example 3"

[0399] Furthermore, the emotion engine uses an internet search tool to provide a method for dealing with the abnormality according to the user's emotions. For example, if the user is feeling anxious, the emotion engine searches for detailed methods for dealing with the abnormality and provides them to the user. This provides the optimal method for dealing with the abnormality according to the user's emotional state, allowing the user to deal with the abnormality appropriately.

[0400] The processing flow of each embodiment will be described below.

[0401] "Example 1"

[0402] Step 1: The system identifies the warning sound coming from the vehicle.

[0403] Step 2: The content of the abnormality corresponding to the identified alarm sound is explained to the user.

[0404] Step 3: Activate the emotion engine that recognizes the user's emotions. This emotion engine recognizes the user's emotions from the user's tone of voice, facial expressions, choice of words, etc.

[0405] "Example 2"

[0406] Step 1: The emotion engine recognizes the user's emotion.

[0407] Step 2: The emotion engine adjusts the way it explains the abnormality according to the user's emotions. Step 3: For example, if the user is panicking, the emotion engine will make the explanation of the abnormality concise and clear. If the user is calm, the emotion engine will make the explanation of the abnormality more detailed.

[0408] "Example 3"

[0409] Step 1: The emotion engine recognizes the user's emotion.

[0410] Step 2: The emotion engine uses an internet search tool to provide a solution to the abnormal content according to the user's emotion.

[0411] Step 3: For example, if the user is feeling anxious, the emotion engine will search for detailed coping methods and provide them to the user.

[0412] Example 1

[0413] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0414] In conventional vehicle systems, it was difficult to properly convey the meaning of an alarm sound to the user, making it difficult for the user to take appropriate action. Furthermore, a uniform response was given without considering the user's feelings, which could cause anxiety and confusion for the user. This led to problems such as a loss of vehicle safety and a sense of security for the user.

[0415] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0416] In this invention, the server includes means for identifying an alarm sound emitted from the vehicle, means for explaining to the user the details of the abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the abnormality, means for recognizing the user's emotion, and means for adjusting the explanation of the abnormality based on the recognized emotion. This makes it possible to accurately convey the meaning of the alarm sound to the user and enable the user to deal with the abnormality appropriately. Furthermore, by taking the user's emotion into consideration, it is possible to increase the user's sense of security and improve the safety of the vehicle.

[0417] An "alarm sound" is an audio signal emitted by a vehicle to warn of abnormalities or to alert the driver.

[0418] "Anomaly Description" is a description of the specific problem or situation that corresponds to the identified alarm sound.

[0419] "User" refers to a person who drives or uses a vehicle.

[0420] "Emotion recognition" is a technology that analyzes a user's emotions from their tone of voice, facial expressions, choice of words, etc.

[0421] A "voice recognition system" is a system that analyzes voice data and identifies specific sounds and words.

[0422] An "emotion engine" is software or hardware for analyzing and recognizing a user's emotions.

[0423] An "on-board computer" is a computer system installed in a vehicle that processes and communicates various data.

[0424] An "Internet search tool" is software for searching for information using the Internet.

[0425] A "generative AI model" is a model that uses artificial intelligence to generate new information and content from data.

[0426] This invention is a system that identifies warning sounds emitted by a vehicle, explains the appropriate nature of the abnormality to the user, and adjusts the response by recognizing the user's emotions. This system is realized using hardware such as a microphone, camera, and on-board computer in the vehicle, and software such as a voice recognition system, emotion engine, and generative AI model.

[0427] Hardware and software used

[0428] Hardware: In-vehicle microphones, cameras, on-board computers

[0429] Software: Speech recognition system, emotion engine, generative AI model

[0430] System Operation

[0431] 1. Acquiring audio data

[0432] The terminal (a microphone inside the vehicle) captures voice data generated inside the vehicle in real time.

[0433] The terminal transmits the acquired voice data to the vehicle-mounted computer.

[0434] The server receives the voice data from the on-board computer.

[0435] 2. Alarm sound identification

[0436] The server inputs the received voice data into a voice recognition system.

[0437] The server uses a voice recognition system to analyze the voice data and identify the alarm sound.

[0438] The server identifies the meaning of the identified warning sound, for example, the warning sound from the intelligent key or the warning sound from the collision warning function.

[0439] 3. Generating Anomalous Content

[0440] The server generates anomaly content corresponding to the identified alarm sound.

[0441] The server inputs the abnormality details into the generative AI model and generates a specific message to explain to the user.

[0442] 4. Notification of abnormalities

[0443] The server sends the generated abnormality information to the vehicle-mounted computer.

[0444] The terminal (on-board computer) notifies the user of the abnormality by voice or text.

[0445] The user checks the details of the notified abnormality.

[0446] 5. User Emotion Recognition

[0447] The device (cameras and microphones inside the vehicle) captures the user's tone of voice, facial expressions, and word choice in real time.

[0448] The terminal transmits the acquired data to the vehicle-mounted computer.

[0449] The server receives data obtained from the vehicle's computer and inputs it into the emotion engine.

[0450] The server uses an emotion engine to recognize the user's emotions.

[0451] 6. Emotional responses

[0452] The server adjusts the description of the anomaly based on the perceived user emotion.

[0453] The server generates additional messages according to the emotion and sends them to the vehicle's computer.

[0454] The terminal (on-board computer) notifies the user of additional messages.

[0455] The user sees the additional message and takes appropriate action.

[0456] Specific examples

[0457] Example 1: When the vehicle emits an intelligent key warning sound

[0458] The terminal receives the warning sound and transmits it to the vehicle's computer.

[0459] The server uses a voice recognition system to distinguish the warning sound from the warning sound of the intelligent key.

[0460] The server generates an error message stating "The intelligent key is not inside the vehicle" and sends it to the on-board computer.

[0461] The terminal notifies the user that "the intelligent key is not inside the vehicle."

[0462] The device captures the user's tone of voice and facial expressions and transmits them to the vehicle's computer.

[0463] The server uses an emotion engine to recognize that the user is in a hurry.

[0464] The server generates an additional message saying "Please stay calm. Find your keys" and sends it to the vehicle's computer.

[0465] The terminal will tell the user to "Please stay calm. Find your key."

[0466] Example 2: When the vehicle emits an alarm from the collision warning function

[0467] The terminal captures the alarm sound and transmits it to the vehicle's computer.

[0468] The server uses a voice recognition system to identify the warning sound as a collision warning sound.

[0469] The server generates an abnormality message such as "There is an obstacle ahead. Please apply the brakes," and sends it to the onboard computer.

[0470] The device notifies the user, "There is an obstacle ahead. Please apply the brakes."

[0471] The device captures the user's tone of voice and facial expressions and transmits them to the vehicle's computer.

[0472] The server uses an emotion engine to recognize that the user is surprised.

[0473] The server generates an additional message saying "Please stay calm" and sends it to the vehicle computer.

[0474] The device notifies the user to "remain calm."

[0475] Prompt Sentence Examples

[0476] Prompt 1: "The vehicle's intelligent key is beeping. How would you explain this to the user?"

[0477] Prompt 2: "A vehicle collision warning sound is heard. If the user is startled, how would they respond?"

[0478] The above is an embodiment of the present invention.

[0479] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0480] Step 1:

[0481] Acquiring audio data

[0482] The terminal (a microphone inside the vehicle) captures voice data generated inside the vehicle in real time.

[0483] Input: In-vehicle audio data

[0484] Output: Acquired audio data

[0485] What it does: A microphone captures audio and sends the data to the car's computer.

[0486] Step 2:

[0487] Sending audio data

[0488] The terminal transmits the acquired voice data to the vehicle-mounted computer.

[0489] Input: Acquired audio data

[0490] Output: Audio data sent to the on-board computer

[0491] Specific operation: The onboard computer receives audio data from the microphone and prepares it to be sent to the server.

[0492] Step 3:

[0493] Receiving audio data

[0494] The server receives the voice data from the on-board computer.

[0495] Input: Audio data sent from the on-board computer

[0496] Output: Received audio data

[0497] Specific operation: The server receives the voice data from the vehicle's computer and prepares it for analysis.

[0498] Step 4:

[0499] Alarm sound identification

[0500] The server inputs the received voice data into a voice recognition system.

[0501] Input: Received audio data

[0502] Output: Identified alarm sound

[0503] How it works: The voice recognition system analyzes the voice data and identifies specific warning sounds, such as the intelligent key warning sound or collision warning sound.

[0504] Step 5:

[0505] Anomaly generation

[0506] The server generates anomaly content corresponding to the identified alarm sound.

[0507] Input: Identified alarm sound

[0508] Output: Anomaly details

[0509] Specific operation: Based on the identified alarm sound, the server inputs the abnormal content into the generative AI model and generates a specific message to explain to the user.

[0510] Step 6:

[0511] Notification of abnormality details

[0512] The server sends the generated abnormality information to the vehicle-mounted computer.

[0513] Input: Anomaly

[0514] Output: Anomaly details sent to the onboard computer

[0515] Specific operation: The server sends the generated abnormality information to the on-board computer, which then prepares to notify the user.

[0516] Step 7:

[0517] Display of abnormality details

[0518] The terminal (on-board computer) notifies the user of the abnormality by voice or text.

[0519] Input: Anomaly sent to on-board computer

[0520] Output: The anomaly reported to the user

[0521] Specific operation: The on-board computer notifies the user of the abnormality via an audio output device or display.

[0522] Step 8:

[0523] User Emotion Recognition

[0524] The device (cameras and microphones inside the vehicle) captures the user's tone of voice, facial expressions, and word choice in real time.

[0525] Input: Your tone of voice, facial expressions, and word choice

[0526] Output: Retrieved emotion data

[0527] What it does: Cameras and microphones capture data about the user's emotions and send it to the car's computer.

[0528] Step 9:

[0529] Sending emotional data

[0530] The terminal transmits the acquired emotion data to the vehicle-mounted computer.

[0531] Input: Acquired emotion data

[0532] Output: Emotion data sent to the on-board computer

[0533] Specific operation: The onboard computer receives emotion data from the camera and microphone and prepares it to be sent to the server.

[0534] Step 10:

[0535] Receiving emotion data

[0536] The server receives emotion data from the vehicle computer and inputs it into the emotion engine.

[0537] Input: Emotion data sent from the vehicle's computer

[0538] Output: Emotion data input to the emotion engine

[0539] Specific operation: The server receives emotion data from the vehicle computer and inputs it into the emotion engine.

[0540] Step 11:

[0541] Emotion recognition

[0542] The server uses an emotion engine to recognize the user's emotions.

[0543] Input: Emotion data input to the emotion engine

[0544] Output: Perceived user emotion

[0545] Specific operation: The emotion engine analyzes the emotion data and identifies the user's emotion. For example, it recognizes whether the user is anxious or surprised.

[0546] Step 12:

[0547] Emotion-based responses

[0548] The server adjusts the description of the anomaly based on the perceived user emotion.

[0549] Input: Perceived user emotion

[0550] Output: Adjusted anomaly description

[0551] Specific operation: The server generates an additional message based on the user's emotion and sends it to the vehicle's computer.

[0552] Step 13:

[0553] Additional message notifications

[0554] The terminal (on-board computer) notifies the user of additional messages.

[0555] Input: Additional message sent to the onboard computer

[0556] Output: Additional message notified to the user

[0557] Specific operation: The on-board computer notifies the user of an additional message via an audio output device or display.

[0558] The above is the specific flow of the program processing of this system.

[0559] (Application example 1)

[0560] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0561] In conventional vehicle systems, when an alarm sound is generated, there are limited means to appropriately inform the user of the content of the alarm, which can make it difficult for the user to take prompt action. Furthermore, since information is provided without taking into account the user's emotional state, the user may feel stressed or be unable to take appropriate action. This may result in a loss of vehicle safety and the user's sense of security.

[0562] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0563] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to the user the details of the abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the details of the abnormality, means for recognizing the user's emotions, and means for providing appropriate advice and support according to the user's emotions, thereby making it possible to quickly and accurately convey the details of the alarm sound to the user and provide appropriate support according to the user's emotional state.

[0564] A "vehicle" is a mechanical device designed to travel on roads and powered by an engine or motor.

[0565] An "alarm sound" is an audio signal emitted from the vehicle to notify the user of an abnormality or danger.

[0566] "Discrimination" means recognizing a particular sound or signal and determining what it means.

[0567] "Abnormality" refers to a state or problem that deviates from the normal operation of the vehicle.

[0568] "User" refers to a person operating or riding in a vehicle.

[0569] "Emotion" refers to the user's psychological state or mood, and is determined from the tone of voice, facial expression, etc.

[0570] "Advice" refers to advice or instructions provided to help a user take appropriate action.

[0571] "Support" refers to assistance and help in dealing with problems or anomalies that a user encounters.

[0572] An "Internet search tool" is software or a service that searches for information on the Internet and provides it to users.

[0573] The following system configuration will be described as an embodiment of the present invention.

[0574] System Configuration

[0575] Hardware

[0576] Smartphone: Microphone, camera, processor, memory, storage

[0577] Vehicle: Alarm generator

[0578] software

[0579] Speech recognition library: Google Cloud Speech-to-Text API

[0580] Machine learning model: TensorFlow

[0581] Emotion Recognition Library: Microsoft(R) Azure(R) Emotion API

[0582] Notification system: Firebase Cloud Messaging

[0583] Program processing explanation

[0584] 1. Speech Recognition System

[0585] The smartphone's microphone is used to capture the warning sound emitted from the vehicle.

[0586] The captured audio data is converted into text data using the Google Cloud Speech-to-Text API.

[0587] 2. Alarm sound identification

[0588] The text data obtained from the voice recognition system is input into a machine learning model using TensorFlow to identify the type of alarm sound and the nature of the abnormality.

[0589] 3. Emotion Engine

[0590] It uses the smartphone's camera and microphone to capture the user's tone of voice and facial expressions.

[0591] The captured data is used to analyze user emotions using the Microsoft Azure Emotion API.

[0592] 4. Notice and Support

[0593] Based on the identified alarm sound and user emotion, appropriate notifications and advice are provided to the user using Firebase Cloud Messaging.

[0594] Specific examples

[0595] For example, if a collision warning occurs while the vehicle is moving and the user is startled, the system operates as follows.

[0596] 1. A voice recognition system captures collision warning sounds and converts them into text data.

[0597] 2. The alarm sound identification identifies the collision alarm sound and identifies the abnormality as "A collision alarm has occurred."

[0598] 3. The emotion engine analyzes the user's tone of voice and facial expressions and recognizes that the user is surprised.

[0599] 4. Notification and support provide appropriate advice to the user, such as "A collision warning has occurred. Please apply the brakes. Don't be alarmed, take a deep breath and stay calm."

[0600] Prompt Sentence Examples

[0601] "Your application must be able to identify the warning sounds coming from a vehicle and notify the user of any abnormalities. You must also add a function that recognizes the user's emotions and provides appropriate advice."

[0602] In this way, the embodiments for carrying out the invention have been specifically described.

[0603] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0604] Step 1:

[0605] Audio Capture

[0606] The microphone on the device (smartphone) captures the warning sound emitted from the vehicle. The input is the warning sound from the vehicle, and the output is voice data. This voice data is used for subsequent voice recognition processing.

[0607] Step 2:

[0608] Voice Recognition

[0609] The device sends the captured audio data to the Google Cloud Speech-to-Text API and converts it into text data. The input is audio data, and the output is text data. This text data is used to identify the type of alarm sound.

[0610] Step 3:

[0611] Alarm sound identification

[0612] The server inputs the text data into a machine learning model using TensorFlow to identify the type of alarm sound and the details of the abnormality. The input is text data, and the output is the type of alarm sound and the details of the abnormality. This identification result is used to notify the user.

[0613] Step 4:

[0614] Emotion Capture

[0615] The device's camera and microphone capture the user's tone of voice and facial expressions. The input is the user's voice and facial expressions, and the output is emotional data. This emotional data is used to analyze the user's emotional state.

[0616] Step 5:

[0617] emotion recognition

[0618] The server sends the captured emotion data to the Microsoft Azure Emotion API to analyze the user's emotions. The input is emotion data, and the output is the user's emotional state. This emotional state is used to provide appropriate advice.

[0619] Step 6:

[0620] Notifications and Support

[0621] The server generates appropriate notifications and advice based on the identified alarm type, abnormality details, and the user's emotional state, and sends them to the device using Firebase Cloud Messaging. The input is the alarm type, abnormality details, and the user's emotional state, and the output is a notification message. This notification message provides the user with appropriate countermeasures.

[0622] In this way, the specific operations performed at each processing step and their inputs and outputs have been explained.

[0623] Example 2

[0624] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0625] Conventional vehicle systems have the problem that when an alarm sound is issued, it is difficult for the user to quickly and accurately understand the nature of the abnormality and take appropriate action. In addition, because an appropriate explanation based on the user's emotional state is not provided, users in a panic state may not be able to take appropriate action. This can compromise the safety of the vehicle and the user's sense of security.

[0626] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0627] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for acquiring details of an abnormality corresponding to the identified alarm sound as text data, voice synthesis means for converting the acquired text data into voice and conveying the voice to the user, means for detecting the emotional state of the user, and emotion engine means for adjusting the method of explaining the abnormality according to the emotional state of the user, thereby enabling the user to quickly and accurately understand the details of the abnormality corresponding to the alarm sound and receive an appropriate explanation according to their emotional state.

[0628] An "alarm sound" is a sound emitted by a vehicle to notify of abnormalities or warnings.

[0629] "Means for identifying" refers to devices or software that analyze the alarm sound and identify what the sound indicates.

[0630] The "content of abnormality" is the specific content of the abnormality or problem in the vehicle that corresponds to the warning sound.

[0631] "Text data" is data that expresses the content of an abnormality as text information.

[0632] "Speech synthesis means" refers to a device or software for converting text data into speech.

[0633] A "user" is a person who drives or uses a vehicle.

[0634] "Emotional state" refers to the user's current psychological state or mood.

[0635] The "emotion engine means" is a device or software for detecting the user's emotional state and adjusting the method of explaining the abnormality accordingly.

[0636] MODE FOR CARRYING OUT THE INVENTION

[0637] This invention is a system that identifies warning sounds emitted from a vehicle and provides audible explanations of the abnormality to the user. Furthermore, by adjusting the explanation method according to the user's emotional state, the system allows the user to deal with the abnormality appropriately.

[0638] Hardware and software used

[0639] Hardware: Servers, terminals (user devices)

[0640] Software: speech recognition software, speech synthesis system, emotion engine, database

[0641] System configuration

[0642] 1. Server

[0643] The server receives the warning sounds generated by the vehicle and analyzes them using voice recognition software.

[0644] The abnormality content corresponding to the identified alarm sound is retrieved from the database.

[0645] The acquired abnormality information is temporarily saved as text data.

[0646] An emotion engine is used to adjust the way the anomaly is explained depending on the user's emotional state.

[0647] The adjusted text data is converted into speech using a speech synthesis system.

[0648] 2. Terminal

[0649] The device uses a camera and microphone to detect the user's emotional state.

[0650] The detected emotional state data is transmitted to a server.

[0651] The voice data sent from the server is transmitted to the user through a speaker.

[0652] Specific examples

[0653] For example, if an alarm sounds indicating an engine malfunction, the server identifies the sound as an engine malfunction. Next, the server retrieves text data from the database that reads, "There is an engine malfunction. Stop the vehicle immediately and contact a professional repair shop." The device analyzes the user's tone of voice and facial expression and determines that the user is in a panic. The server uses an emotion engine to adjust the explanation to be concise and clear, and converts it into speech using a speech synthesis system. Finally, the device plays the generated speech over the speaker and conveys it to the user.

[0654] Prompt Sentence Examples

[0655] "When an engine alarm sounds, generate a description for when the user is panicking."

[0656] "When an alarm sounds indicating a brake malfunction, generate an explanation for the user to remain calm."

[0657] In this way, the system can identify alarm sounds and provide an optimal audio explanation based on the user's emotional state, allowing the user to quickly and accurately understand the nature of the anomaly and take appropriate action.

[0658] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0659] Program processing flow

[0660] Step 1: Receiving and identifying the alarm sound

[0661] Input: Vehicle alarm sound

[0662] Process: The server receives the alarm sound and analyzes it using voice recognition software.

[0663] Output: Type of alarm identified (e.g. engine abnormality, brake abnormality)

[0664] What it does: The server receives the alarm sound through a microphone and analyzes it with voice recognition software, for example by detecting specific frequencies or patterns and identifying it as an "engine malfunction."

[0665] Step 2: Obtaining the details of the anomaly

[0666] Input: Identified alarm type

[0667] Processing: The server retrieves the corresponding anomaly details from the database as text data.

[0668] Output: Text data of the abnormality (e.g. "There is an abnormality in the engine. Stop the vehicle immediately and contact a professional repair shop.")

[0669] Specific operation: The server accesses the database, searches for and retrieves the anomaly content corresponding to the identified alarm sound, and temporarily stores the retrieved text data.

[0670] Step 3: Detecting the user's emotional state

[0671] Input: User voice and facial expression data

[0672] Processing: The device detects the user's emotional state using the camera and microphone.

[0673] Output: User's emotional state (e.g., panicked, calm)

[0674] How it works: The device captures the user's facial expressions with a camera and records the user's voice with a microphone. This data is then analyzed to determine whether the user is panicking or calm.

[0675] Step 4: Adjust your explanation

[0676] Input: Textual data of the anomaly, user's emotional state

[0677] Processing: The server uses an emotion engine to adjust the way the anomaly is explained based on the user's emotional state.

[0678] Output: Adjusted text data of the anomaly (e.g., brief description, detailed description)

[0679] Specific behavior: The server uses an emotion engine to adjust the text data based on the user's emotional state. For example, if the user is panicking, the explanation will be concise and clear.

[0680] Step 5: Text-to-Speech

[0681] Input: Adjusted text of the anomaly

[0682] Processing: The server uses a speech synthesis system to convert the text data into speech.

[0683] Output: Audio data describing the problem (e.g., "There is a problem with the engine. Stop the vehicle immediately and contact a professional repair shop.")

[0684] What happens: The server uses speech synthesis software (e.g., a speech synthesis API) to convert the adjusted text data into speech data.

[0685] Step 6: Notify users

[0686] Input: Audio recording of the anomaly

[0687] Processing: The device plays the generated audio to the user.

[0688] Output: The user receives an audio description of the anomaly.

[0689] Specific operation: The device plays audio data through the speaker to inform the user of the abnormality. For example, a voice message may be played from the smartphone saying, "There is an abnormality in the engine. Please stop the vehicle immediately and contact a professional repair shop."

[0690] In this way, the entire system functions by processing input data at each step and generating the output data required for the next step.

[0691] (Application example 2)

[0692] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0693] In conventional vehicle systems, when an alarm sounds, it can be difficult for users to understand the nature of the abnormality and take appropriate action. Furthermore, because an appropriate explanation is not provided that reflects the user's emotional state, the user may panic or miss important information. This poses a challenge, hindering user safety and proper vehicle maintenance.

[0694] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0695] In this invention, the server includes means for identifying an alarm sound emitted from the vehicle, means for explaining to the user the details of the abnormality corresponding to the identified alarm sound, means for analyzing the emotional state of the user, means for adjusting the method of explaining the details of the abnormality in accordance with the analyzed emotional state, and means for enabling the user to deal with the details of the abnormality. This enables the user to deal appropriately with the alarm sound and provides an optimal explanation according to the user's emotional state, thereby ensuring the user's safety and enabling appropriate maintenance of the vehicle.

[0696] "Vehicle" means a mechanical device for moving on land, including automobiles and trucks powered by engines or motors.

[0697] An "alarm sound" is an audio signal that is generated to alert the driver to abnormalities or dangers in the vehicle, and different sounds are generated depending on the situation, such as engine abnormalities or collision warnings.

[0698] "Means for identifying" refers to technology or equipment that recognizes a particular audio signal or data and determines what it means.

[0699] "Abnormality" refers to the specific problem or malfunction that prevents the vehicle from operating normally, including engine failure and flat tires.

[0700] "User" refers to a person who operates or uses a vehicle, including the driver and passengers.

[0701] "Means for explaining" refers to a method or device for conveying specific information to a user, including a voice synthesis system and a display.

[0702] "Emotional state" refers to the user's psychological state and includes emotions such as panic or calm.

[0703] "Analytical tools" refers to techniques or devices used to analyze data and derive specific conclusions or information.

[0704] "Adjusting means" refers to techniques or devices for changing operations or settings in response to specific conditions.

[0705] "Means for enabling a user to cope" refers to a method or device that assists a user in appropriately dealing with a particular problem or situation.

[0706] The system for carrying out the present invention identifies warning sounds emitted from a vehicle, explains to a user the nature of the abnormality corresponding to the identified warning sound, and has the function of analyzing the emotional state of the user and adjusting the method of explaining the abnormality based on the analysis result.

[0707] System configuration

[0708] The system consists of the following main components:

[0709] 1. Warning sound identification module: This module identifies warning sounds emitted by vehicles and uses speech recognition technology. Specifically, it uses the speech_recognition library.

[0710] 2. Anomaly Description Module: This module obtains the anomaly description corresponding to the identified alarm sound as text data and conveys it to the user using a speech synthesis system. Specifically, it uses the gTTS library.

[0711] 3. Emotional State Analysis Module: This module analyzes the user's emotional state and uses image recognition technology. Specifically, it uses the emotion_recognition library.

[0712] 4. Explanation method adjustment module: This module adjusts the explanation method for abnormal content according to the analyzed emotional state.

[0713] System Operation

[0714] The server uses voice recognition technology to identify warning sounds emitted by the vehicle. It obtains the details of the abnormality corresponding to the identified warning sound as text data and communicates this to the user using a voice synthesis system. It analyzes the user's facial expressions using image recognition technology to analyze the user's emotional state. It adjusts the way it explains the abnormality based on the analysis results. For example, if the user is in a panic, it provides a brief explanation, and if the user is calm, it provides a detailed explanation.

[0715] Specific examples

[0716] For example, if an engine malfunction alarm sounds, the system will:

[0717] 1. The alarm sound identification module identifies the engine abnormality alarm sound.

[0718] 2. The abnormality description module obtains text data stating, "There is an abnormality in the engine. Stop the vehicle immediately and contact a specialist repair shop," and conveys this to the user using a voice synthesis system.

[0719] 3. The emotional state analysis module analyzes the user's facial expressions and detects whether the user is in a panic state.

[0720] 4. The explanation adjustment module simplifies the explanation and says, "There is an abnormality in the engine."

[0721] Prompt Sentence Examples

[0722] Example prompts to input to a generative AI model:

[0723] Design your system to adapt the explanation of the problem depending on the user's emotional state. If the user is panicking, keep the explanation brief, and if the user is calm, provide more detail. For example, if an engine alarm sounds, a panicked user might be told, "There is an engine problem," while a calm user might be told, "There is an engine problem. Stop immediately and contact a professional repair service."

[0724] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0725] Step 1:

[0726] The server receives the warning sound emitted from the vehicle as audio data. The input is the audio data, and the output is the speech recognition result. The server converts the audio data into text using the speech_recognition library. This process identifies what the warning sound means.

[0727] Step 2:

[0728] The server acquires the abnormality content corresponding to the identified alarm sound as text data. The input is the speech recognition result, and the output is the text data of the abnormality content. The server references a dictionary of predefined abnormality content to acquire the corresponding abnormality content.

[0729] Step 3:

[0730] The server receives image data of a user to analyze the user's emotional state. The input is the image data, and the output is the analysis result of the emotional state. The server analyzes the image data using the emotion_recognition library to identify the user's emotional state.

[0731] Step 4:

[0732] The server adjusts the explanation method for the anomaly according to the analyzed emotional state. The input is the analysis result of the emotional state and text data of the anomaly, and the output is the adjusted explanatory text. The server adjusts the level of detail of the explanation based on the emotional state and generates an explanation that is optimal for the user.

[0733] Step 5:

[0734] The server converts the adjusted explanatory text into speech and conveys it to the user. The input is the adjusted explanatory text and the output is the speech data. The server converts the text to speech using the gTTS library and sends the speech data to the user's device.

[0735] Step 6:

[0736] The user's device plays the received audio data and notifies the user of the abnormality. The input is audio data and the output is audio playback. The device uses the playsound library to play the audio data and notify the user of the abnormality.

[0737] Example 3

[0738] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0739] In conventional vehicle systems, when an alarm sound is generated, there are limited means to properly inform the user of the abnormality, making it difficult for the user to take prompt and appropriate action. In addition, there is no means to provide a way to deal with the situation according to the user's emotional state, which often causes anxiety and stress for the user. This leads to issues of reduced safety and convenience for the user.

[0740] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0741] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to the user the details of the abnormality corresponding to the identified alarm sound, means for providing a method of dealing with the abnormality using an internet search tool so that the user can deal with the abnormality, and means for providing a method of dealing with the abnormality according to the user's emotions using an emotion engine that analyzes the user's emotional state. This allows the user to quickly understand the details of the abnormality corresponding to the alarm sound and obtain an appropriate method of dealing with it, and because the optimal method of dealing with it according to the user's emotional state is provided, the user can deal with the abnormality with peace of mind.

[0742] An "alarm sound" is a sound emitted by a vehicle to notify of abnormalities or warnings.

[0743] "Means for identifying" refers to devices or algorithms that analyze the alarm sound and identify what the sound represents.

[0744] The "abnormality content" is the specific content of the vehicle abnormality or problem indicated by the warning sound.

[0745] The "means for explaining to the user" refers to a device or method for communicating the identified abnormality to the user in an easy-to-understand manner.

[0746] An "Internet search tool" is software or a service that allows you to search for information on the Internet and obtain the information you need.

[0747] "Countermeasures" are specific actions or procedures that a user should take in response to an identified abnormality.

[0748] An "emotion engine" is software or algorithms that analyze a user's emotional state and provide information and solutions appropriate to that state.

[0749] "Emotional state" refers to the psychological state such as anxiety, relief, or stress felt by the user.

[0750] MODE FOR CARRYING OUT THE INVENTION

[0751] This invention is a system that identifies warning sounds emitted from a vehicle and provides the user with an appropriate response. The system is realized using hardware and software such as a server, a user terminal, an internet search tool, and an emotion engine.

[0752] Hardware and software used

[0753] Hardware: Servers, user devices (smartphones, tablets, PCs, etc.)

[0754] Software: Internet search tools, sentiment engines, generative AI models

[0755] Specific operation of the system

[0756] 1. Receiving and identifying alarm sounds

[0757] The server receives the alarm sound transmitted from the user terminal.

[0758] The server analyzes the received alarm sound and identifies what the sound represents.

[0759] Specifically, a voice recognition algorithm is used to identify warning sounds such as "tire pressure low."

[0760] 2. Search for anomalies

[0761] The server uses an internet search tool to search for the abnormality content corresponding to the identified alarm sound.

[0762] The server uses search engines such as Google and Bing to gather relevant information.

[0763] Specifically, search for a solution such as "Your tire pressure is low. Please refill the air at the nearest gas station or vehicle repair shop."

[0764] 3. Providing information to users

[0765] The server transmits the search results to the user terminal.

[0766] The user can check how to deal with the problem on the device.

[0767] Specifically, a notification will appear on the user's smartphone with the message, "Your tire pressure is low. Please refill the air at the nearest gas station or vehicle repair shop."

[0768] 4. Emotion analysis using an emotion engine

[0769] The terminal analyzes the user's emotional state using an emotion engine.

[0770] The device reads emotions from the user's voice, facial expressions, input text, etc.

[0771] Specifically, if the user is feeling anxious, the emotion engine analyzes that information and sends it to the server.

[0772] 5. Rediscovering and providing ways to cope with emotions

[0773] The server uses the internet search tool again to search for the most appropriate way to deal with the user's emotional state based on the information from the emotion engine.

[0774] The server searches for detailed instructions and reassuring information and provides it to the user.

[0775] Specifically, the message "Your tire pressure is low. Please refill it at the nearest gas station or repair shop. Regularly checking your tire pressure can also help prevent accidents" will be displayed on the user's smartphone.

[0776] Specific examples

[0777] Example 1: Low tire pressure

[0778] Receiving and identifying the alarm sound: The server receives the "beep, beep" alarm sound sent from the user terminal and identifies it as indicating low tire pressure.

[0779] Search for anomaly: The server searches Google for solutions such as "Your tire pressure is low. Please refill the tire with air at the nearest gas station or repair shop."

[0780] Informing the user: A notification appears on the user's smartphone with the message, "Your tire pressure is low. Please refill it at the nearest gas station or vehicle repair shop."

[0781] Example 2: When a user feels anxious

[0782] Emotion analysis using an emotion engine: The device analyzes the user's voice to determine whether they are feeling anxious and sends that information to the server.

[0783] Re-searching and providing solutions according to the emotion: The server searches for detailed solutions again and displays information on the user's smartphone such as, "Your tire pressure is low. Please refill the air at the nearest gas station or vehicle repair shop. Also, by regularly checking the tire pressure, you can prevent accidents."

[0784] Prompt Sentence Examples

[0785] "What should I do if my tire pressure is low?"

[0786] "Find out more about what to do if your users are worried."

[0787] In this way, the system provides information that enables the user to deal with the abnormality. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0788] Step 1:

[0789] Alarm reception and identification

[0790] Input: Alarm sound data sent from the user terminal

[0791] Specific operation: The server receives the alarm sound sent from the user terminal, analyzes the received alarm sound using a voice recognition algorithm, and identifies what the sound represents.

[0792] Data processing: Analyze the voice data and extract features.

[0793] Output: Identified alarm type (e.g. low tire pressure)

[0794] Step 2:

[0795] Search for anomalies

[0796] Input: Identified alarm type

[0797] Specific operation: The server uses an internet search tool to search for the anomaly corresponding to the identified alarm sound, and collects related information using search engines such as Google and Bing.

[0798] Data processing: generating search queries and performing internet searches.

[0799] Output: Search results with instructions (e.g. "Your tire pressure is low. Please refill it at the nearest gas station or repair shop.")

[0800] Step 3:

[0801] Providing information to users

[0802] Input: What to do as search result

[0803] Specific operation: The server sends the search results to the user's device, where the user can check how to deal with the problem.

[0804] Data processing: Converting search results into a user-friendly format.

[0805] Output: A message to be displayed on the user's device with instructions on what to do (e.g., "Your tire pressure is low. Please refill it at the nearest gas station or repair shop.")

[0806] Step 4:

[0807] Emotion analysis using an emotion engine

[0808] Input: Emotional data such as the user's voice, facial expressions, and typed text

[0809] Specific operation: The device analyzes the user's emotional state using an emotion engine, reading emotions from the user's voice, facial expressions, input text, etc.

[0810] Data processing: Analyze emotion data to identify the user's emotional state.

[0811] Output: User's emotional state (e.g., anxiety)

[0812] Step 5:

[0813] Rediscovering and providing ways to cope with emotions

[0814] Input: User's emotional state, identified alarm sound type

[0815] Specific operation: Based on the information from the emotion engine, the server again uses the internet search tool to search for the most appropriate way to deal with the user's emotional state. It searches for detailed ways to deal with the situation and information that will give a sense of security, and provides it to the user.

[0816] Data processing: Generate search queries based on emotional states and perform internet searches.

[0817] Output: A message displayed on the user's device with advice based on the emotion (e.g., "Your tire pressure is low. Please refill your tires at the nearest gas station or repair shop. Regularly checking your tire pressure can also help prevent accidents.")

[0818] (Application example 3)

[0819] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0820] In conventional vehicle systems, when an alarm sounds, it can be difficult for the user to understand the nature of the abnormality and take appropriate action. Furthermore, since there is no means to provide a way to deal with the situation according to the user's emotional state, the user may feel anxious or stressed. This may result in a loss of safety and comfort for the user.

[0821] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes: means for identifying an alarm sound emitted from a vehicle; a prompt sentence instructing the server to explain to the user the details of an abnormality corresponding to the identified alarm sound; means for explaining to the user the details of the abnormality corresponding to the identified alarm sound using a generative AI model; a prompt sentence instructing the server to explain to the user how to deal with the abnormality; and means for explaining to the user how to deal with the abnormality using the generative AI model. The server also includes means for recognizing the user's emotional state using an emotion engine, and the means for explaining the details of the abnormality to the user includes a prompt sentence instructing the server to explain to the user how to deal with the abnormality based on the recognized emotional state of the user and using the generative AI model. This allows the user to quickly understand the appropriate way to deal with the alarm sound and is further provided with the optimal way to deal with the alarm sound depending on the user's emotional state, thereby improving safety and comfort. The alarm sound includes a warning sound from the vehicle's intelligent key or a warning sound emitted from a collision warning function.

[0822] A "vehicle" is a mechanical device for moving on roads, powered by an engine or motor.

[0823] An "alarm sound" is an audio signal that is generated to warn of an abnormality or danger in the vehicle.

[0824] An "identification means" is a device or software that recognizes a particular sound or signal and determines what it is.

[0825] "Abnormality details" are specific conditions of problems or malfunctions that prevent the vehicle from operating normally.

[0826] A "user" is a person who operates or uses a vehicle.

[0827] "Means for explanation" are devices or software that provide information to users and help them understand.

[0828] "Measures to enable a user to take action" refers to devices or software that provide instructions and support for the user to take appropriate action in response to the abnormality.

[0829] "Emotional state" refers to the user's psychological state or mood.

[0830] A "means for recognizing" is a device or software for detecting and determining a user's emotional state.

[0831] "Countermeasures" are specific actions and procedures to be taken in response to the abnormality.

[0832] A "generative AI model" is an algorithm or system that is trained to perform a specific task using artificial intelligence.

[0833] A "prompt" is text data input into a generative AI model and is an instruction statement to obtain a specific output.

[0834] The following system configuration is used as an embodiment of the present invention.

[0835] System Configuration

[0836] 1. Hardware Configuration

[0837] Vehicle sensor system: Various sensors (e.g., microphones, cameras) installed in the vehicle are used to detect warning sounds and the user's emotional state.

[0838] In-vehicle display: A display that shows the user the nature of the abnormality corresponding to the alarm sound and how to deal with it.

[0839] Smartphone: A device that interacts with the vehicle and provides additional information.

[0840] 2. Software Configuration

[0841] Speech recognition software: Google Cloud Speech-to-Text is used to convert the vehicle's warning sounds into text.

[0842] Internet search engine: Using the Google Search API, search for solutions to the anomaly that corresponds to the identified alarm sound.

[0843] Emotion recognition software (emotion engine): Using the Affectiva SDK, the system recognizes the user's emotional state from facial images.

[0844] Generative AI model: OpenAI (registered trademark) GPT-4 (registered trademark) is used to generate prompts based on the user's emotional state and provide detailed instructions on how to respond.

[0845] Processing flow

[0846] 1. Identifying alarm sounds

[0847] The vehicle's sensor system detects the warning sound and uses voice recognition software to convert the sound into text.

[0848] For example, if a warning beep sounds to indicate low tire pressure, the text "beep beep" is generated. The warning beep includes, for example, a warning beep from an intelligent key or a warning beep from a collision warning function of a vehicle.

[0849] 2. Providing details of the abnormality and how to deal with it

[0850] The server uses a prompt sentence that instructs the user to explain the abnormality corresponding to the identified alarm sound to the user and a generative AI model to explain the abnormality corresponding to the identified alarm sound to the user. Example prompt sentence: "Car model: AA. Alarm sound: beep beep. Use an internet search engine to search for the abnormality that corresponds to the identified alarm sound and explain the abnormality to the user."

[0851] For example, an explanation such as "Your tire pressure is low" may be generated and provided to the user by voice or text display. The server also uses a prompt statement instructing the user to explain how to deal with the abnormality and the generated AI model to explain to the user how to deal with the abnormality. Example prompt sentence: "Error: Tire pressure is low. Please use an internet search engine to find out how to resolve this issue."

[0852] For example, the generated explanation is "Please refill your tires at the nearest gas station or repair shop."

[0853] 3. Recognizing emotional states

[0854] The vehicle's camera captures images of the user's face and emotion recognition software is used to recognize the user's emotional state.

[0855] For example, if the user is feeling anxious, the emotional state "anxious" is recognized.

[0856] 4. Prompt generation

[0857] A generative AI model is used to generate prompt sentences that correspond to the user's emotional state.

[0858] For example, a prompt sentence may be generated that reads, "User's emotional state: anxiety. Solution: refill your air at the nearest gas station or vehicle repair shop. Please adjust the solution instructions to suit the user's emotional state." The server uses the generated prompt sentence and the generative AI model to adjust the explanation of how to deal with the problem.

[0859] 5. Providing solutions

[0860] Users are provided with solutions via the in-car display or smartphone.

[0861] For example, a message might appear saying, "First, stay calm. Then, go to the nearest gas station or repair shop to refill your tires with air."

[0862] Specific examples

[0863] Scenario: A self-driving vehicle is driving when an alarm sounds indicating low tire pressure.

[0864] User emotions: Feeling anxious.

[0865]

[0866] This system enables users to quickly understand the appropriate way to respond to an alarm sound and provides the optimal way to respond depending on their emotional state, thereby improving safety and comfort.

[0867] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0868] Step 1:

[0869] The vehicle's sensor system detects an alarm sound. The input is the audio data generated by the vehicle, and the output is the audio data of the detected alarm sound. Specifically, a microphone mounted on the vehicle collects the surrounding sounds and records them as audio data.

[0870] Step 2:

[0871] The server uses speech recognition software to convert the audio data of the alarm sound into text. The input is the audio data obtained in step 1, and the output is text data representing the alarm sound. Specifically, Google Cloud Speech-to-Text is used to analyze the audio data and generate text such as "beep beep."

[0872] Step 3:

[0873] The server uses a prompt sentence instructing the user to explain the abnormality corresponding to the identified alarm sound and the generative AI model to explain the abnormality corresponding to the identified alarm sound to the user. At this time, the generative AI model uses an internet search engine to search for an explanation of the abnormality corresponding to the identified alarm sound. The input is the text data obtained in step 2, and the output is the search results related to the abnormality. Specifically, the Google Search API is used to send the query "Car model: AA Alarm sound: beep beep Search for the abnormality corresponding to the identified alarm sound" and obtain the search result "Tire pressure is low." The generative AI model generates an explanation based on this search result. The server also uses a prompt sentence that instructs the server to explain to the user how to deal with the abnormality and the generation AI model to generate an explanatory sentence to explain to the user how to deal with the abnormality.

[0874] Step 4:

[0875] The vehicle's camera captures the user's facial image, and the server uses emotion recognition software to recognize the user's emotional state. The input is the user's facial image data, and the output is the user's emotional state (e.g., anxiety). Specifically, the system uses the Affectiva SDK to analyze the facial image and determine whether the user is feeling anxious.

[0876] Step 5:

[0877] The server uses the generative AI model to generate a prompt that instructs the user to adjust the explanation according to their emotional state. The input is the explanation of how to deal with the situation obtained in step 3 and the emotional state obtained in step 4, and the output is the adjusted explanation.

[0878] Step 6:

[0879] The server sends the generated prompt text to the in-vehicle display or smartphone, providing the user with a solution. The input is the explanatory text obtained in step 5, and the output is the message to be displayed to the user. Specifically, the in-vehicle display or smartphone displays the message, "First, please stay calm. Then, go to the nearest gas station or vehicle repair shop to refill your tires."

[0880] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0881] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0882] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0883] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0884] [Second embodiment]

[0885] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0886] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0887] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0888] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0889] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0890] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0891] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0892] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0893] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0894] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0895] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0896] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0897] "Example 1"

[0898] In one embodiment of the present invention, a voice recognition system is used to identify warning sounds emitted from a vehicle. This voice recognition system can learn various warning sounds emitted from the vehicle and identify the meaning of each warning sound. For example, it can identify warning sounds from an intelligent key or a collision warning function.

[0899] "Example 2"

[0900] A speech synthesis system is used as a means for explaining to the user the details of the abnormality corresponding to the identified alarm sound. This speech synthesis system acquires the details of the abnormality corresponding to the identified alarm sound as text data, converts it into voice, and conveys it to the user. For example, when an alarm sound indicating an engine abnormality sounds, the speech synthesis system will convey to the user, "There is an abnormality in the engine. Please stop the vehicle immediately and contact a professional repair shop."

[0901] "Example 3"

[0902] An internet search tool is used as a means for allowing the user to deal with the abnormality. This internet search tool searches the internet for information on the abnormality corresponding to the identified alarm sound and provides the results to the user. For example, if an alarm sound is heard indicating low tire pressure, the internet search tool provides the user with a solution such as "Your tire pressure is low. Please refill the tire with air at the nearest gas station or vehicle repair shop."

[0903] The processing flow of each embodiment will be described below.

[0904] "Example 1"

[0905] Step 1: The warning sound emitted from the vehicle is collected by a microphone.

[0906] Step 2: The collected alarm sounds are input into a speech recognition system, which uses a database of previously trained alarm sounds to identify the meaning of the input alarm sound.

[0907] Step 3: The user is notified of the identification result by, for example, displaying the identification result on a display in the vehicle.

[0908] "Example 2"

[0909] Step 1: Receive the identification result from the speech recognition system.

[0910] Step 2: Obtain the abnormality details corresponding to the identification result as text data. This text data is obtained, for example, from a database.

[0911] Step 3: The acquired text data is input into a speech synthesis system and converted into speech.

[0912] Step 4: The converted audio is output from the speaker to inform the user of the abnormality.

[0913] "Example 3"

[0914] Step 1: Receive the identification result from the speech recognition system.

[0915] Step 2: Use an internet search tool to search for information about the anomaly that corresponds to the identification result.

[0916] Step 3: The search results are notified to the user. This notification is performed, for example, by displaying the search results on the vehicle display.

[0917] Example 1

[0918] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0919] Conventional vehicle alarm sound identification systems have had the problem of making it difficult to quickly and accurately convey the meaning of an alarm sound to the user when it occurs. Furthermore, the information provided to users is insufficient to enable them to understand the meaning of the alarm sound and take appropriate action. This can lead to users being unable to take appropriate action, potentially compromising the safety and convenience of the vehicle.

[0920] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0921] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to a user the details of an abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the details of the abnormality, voice recognition means for analyzing the voice data, and means for notifying the user of the analysis results, thereby making it possible to quickly and accurately convey the meaning of the alarm sound to the user and provide information for the user to take appropriate action.

[0922] A "vehicle" is a mechanical device designed to travel on roads and powered by an engine or motor.

[0923] An "alarm sound" is an audio signal that is generated to notify the user of an abnormality in the vehicle or a situation that requires attention.

[0924] An "identification means" is a technique or device that analyzes a particular audio signal and determines what the audio means.

[0925] "Abnormality details" is information that indicates a state in which the vehicle is deviating from normal operation or a situation that requires attention.

[0926] A "user" is a person who drives or uses a vehicle.

[0927] "Explanatory means" refers to techniques and devices for communicating the details of the identified abnormality to the user in an easily understandable manner.

[0928] The "means for enabling a user to take action" refers to a technology or device that provides information or instructions to the user so that the user can take appropriate action against the identified abnormality.

[0929] "Speech recognition means" refers to technology or devices for analyzing voice data and converting it into text data.

[0930] "Means of notification" refers to the technology or device used to notify the user of the analysis results and abnormality details.

[0931] MODE FOR CARRYING OUT THE INVENTION

[0932] The present invention relates to a system for identifying an alarm sound emitted from a vehicle and notifying a user of the content of the abnormality. Specific embodiments of this system will be described below.

[0933] System Overview

[0934] This system uses voice recognition technology to identify warning sounds emitted by the vehicle. The voice recognition system can learn the various warning sounds emitted by the vehicle and identify the meaning of each warning sound. For example, it can identify the warning sound from the intelligent key or the warning sound from the collision warning function.

[0935] Hardware and software used

[0936] The server hosts the voice recognition system and receives voice data sent from the vehicle. The server analyzes the voice data using voice recognition software (e.g., a general voice recognition API). The analyzed voice data is compared with a database of pre-trained warning sounds.

[0937] When an alarm sound is generated, the terminal (e.g., the vehicle's on-board computer) transmits the sound data to the server. The terminal collects data from various sensors in the vehicle and records the timing and circumstances under which the alarm sound was generated.

[0938] To find out what the vehicle's alarm means, users will receive a notification via a smartphone app or on the vehicle's display, where they can view a detailed explanation of the alarm and how to respond.

[0939] Specific examples

[0940] When an intelligent key warning sound is emitted while the vehicle is in motion, the device sends the audio data to the server. The server uses a common voice recognition API to analyze the audio data and compares the analysis results with an alarm sound database. The server identifies the sound as an intelligent key warning sound and notifies the user's smartphone app. The user receives a warning message through the app stating, "The intelligent key is not in the vehicle."

[0941] Prompt Sentence Examples

[0942] "Design a system to identify warning sounds emitted by a vehicle. The system uses speech recognition technology to analyze the warning sounds and notify the user of what each sound means. As a specific use case, consider a system that can identify the warning sounds from an intelligent key and the warning sounds from a collision warning function."

[0943] In this way, by clarifying the roles of servers, terminals, and users and naming specific hardware and software, it is possible to understand the system's processing in detail.

[0944] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0945] Program processing flow

[0946] Step 1:

[0947] The terminal monitors various sensors inside the vehicle and detects when an alarm is generated. For example, if the intelligent key is not inside the vehicle, an alarm will sound.

[0948] Input: Data from sensors inside the vehicle

[0949] Output: Signal to detect the occurrence of an alarm sound

[0950] Specific operation: The terminal collects data in real time from sensors inside the vehicle and determines whether an alarm has been sounded.

[0951] Step 2:

[0952] The device will record the alarm sound using the built-in microphone, and the recorded audio data will be temporarily stored on the device.

[0953] Input: Alarm sound

[0954] Output: Recorded audio data

[0955] Specific operation: When the device detects an alarm sound, it activates the built-in microphone to record the sound and saves the data in memory.

[0956] Step 3:

[0957] The terminal transmits the recorded voice data to the server using the vehicle's communication module (e.g., LTE module).

[0958] Input: Recorded audio data

[0959] Output: Audio data sent to the server

[0960] Specific operation: The terminal compresses the recorded data and sends it to the server via the vehicle's communication module.

[0961] Step 4:

[0962] The server analyzes the received voice data using a voice recognition API and converts it into text data.

[0963] Input: Audio data sent to the server

[0964] Output: Parsed text data

[0965] Specific operation: The server calls the speech recognition API and converts the speech data into text data.

[0966] Step 5:

[0967] The server compares the converted text data with a pre-trained alarm sound database.

[0968] Input: Parsed text data

[0969] Output: Identified alarm type

[0970] Specific operation: The server compares the text data with the alarm sound database to identify matching alarm sound types.

[0971] Step 6:

[0972] The server notifies the user's smartphone app of the identification result using a push notification service (e.g., a general push notification service).

[0973] Input: identified alarm type

[0974] Output: Notification sent to the user's smartphone

[0975] Specific operation: The server formats the identification results and sends them to the user's smartphone via a push notification service.

[0976] Step 7:

[0977] The user receives a notification via a smartphone app, for example, a message saying "Intelligent key not in vehicle."

[0978] Input: Notification sent to the user's smartphone

[0979] Output: Notification message received by the user

[0980] Specific operation: The user checks the notification on their smartphone and takes action, such as returning the intelligent key to the vehicle, if necessary.

[0981] In this way, by explaining each processing step in detail and clearly indicating the inputs and outputs, the processing flow of the system can be understood in detail.

[0982] (Application example 1)

[0983] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0984] Conventional vehicle alarm sound identification systems are required not only to identify alarm sounds but also to explain the nature of the abnormality to the user and provide countermeasures. However, these systems lack the functionality to notify the user of the type and meaning of the alarm sound in real time and to suggest necessary countermeasures or actions. This makes it difficult for the user to respond quickly and appropriately.

[0985] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0986] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to a user the details of an abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the details of the abnormality, means for notifying the user of the type and meaning of the identified alarm sound, and means for suggesting necessary countermeasures or actions, thereby enabling the user to understand the type and meaning of the alarm sound in real time and to respond promptly and appropriately.

[0987] A "vehicle" is a mechanical device designed to travel on roads and powered by an engine or motor.

[0988] An "alarm sound" is an audio signal emitted by a vehicle and is used to notify of abnormalities or dangers.

[0989] An "identification means" is a device or system that analyzes a specific audio signal and determines its type and meaning.

[0990] "Abnormality details" refers to the specific circumstances or conditions of a problem or malfunction that prevents the vehicle from operating normally.

[0991] The "means for explaining to the user" refers to a device or system for communicating the identified abnormality to the user in an easy-to-understand manner.

[0992] The "means for enabling a user to take action" refers to a device or system that provides information and instructions for the user to take appropriate action against the abnormality.

[0993] The "notification means" is a device or system for informing the user of the type of identified alarm sound and its meaning.

[0994] "Proposed measures" are devices or systems that present specific measures or actions that users should take.

[0995] The system for implementing this invention identifies warning sounds emitted from a vehicle, explains the nature of the abnormality to the user, and provides countermeasures. The system uses a microphone installed in the vehicle, a smartphone, the voice recognition library "SpeechRecognition," and the machine learning library "TensorFlow."

[0996] System configuration

[0997] 1. Hardware

[0998] Vehicle-mounted microphones: collect warning sounds emitted by the vehicle.

[0999] Smartphone: A device for processing voice data and notifying the user.

[1000] 2. Software

[1001] Speech Recognition library: Analyzes collected voice data and identifies alarm sounds.

[1002] TensorFlow machine learning library: Uses an alarm sound identification model to determine the type of alarm sound.

[1003] Processing flow

[1004] 1. Acquisition of audio data: A microphone installed in the vehicle collects the alarm sound and transmits it to a smartphone.

[1005] 2. Preprocessing of audio data: Using the speech recognition library "SpeechRecognition" on the smartphone, the audio data is analyzed and the features of the alarm sound are extracted.

[1006] 3. Alarm sound identification: Using the machine learning library "TensorFlow," audio data is input into a pre-trained alarm sound identification model to identify the type of alarm sound.

[1007] 4. Notification and Suggestion: Inform the user of the identified alarm type and its meaning, and suggest any necessary countermeasures or actions.

[1008] Specific examples

[1009] For example, if the vehicle emits a collision warning sound, the system will identify the sound and notify the smartphone, "There is a risk of a collision. Please apply the brakes."

[1010] Prompt Sentence Examples

[1011] "Develop an application that identifies warning sounds emitted by a vehicle and notifies the user of their type and meaning. The libraries used are SpeechRecognition and TensorFlow. Types of warning sounds include intelligent key warning sounds, collision warning sounds, and engine abnormality warning sounds."

[1012] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1013] Step 1:

[1014] A microphone installed in the vehicle collects the alarm sound and transmits it to a smartphone.

[1015] Input: Vehicle alarm sound

[1016] Output: Audio data sent to your smartphone

[1017] How it works: The vehicle's microphone detects an alarm sound and transmits the audio data in real time to a smartphone, which receives the audio data and proceeds to the next processing step.

[1018] Step 2:

[1019] The voice recognition library "SpeechRecognition" is used on the smartphone to analyze the voice data and extract the characteristics of the alarm sound.

[1020] Input: Voice data sent to smartphone

[1021] Output: Extracted alarm sound feature data

[1022] Specific operation: The smartphone inputs the received voice data into the "SpeechRecognition" library and analyzes the voice data. As a result of the analysis, feature data of the alarm sound is extracted. This feature data is used in the next step.

[1023] Step 3:

[1024] Using the machine learning library "TensorFlow," audio data is input into a pre-trained alarm sound identification model to identify the type of alarm sound.

[1025] Input: Extracted alarm sound feature data

[1026] Output: Identified alarm type

[1027] How it works: The smartphone inputs the extracted feature data into the TensorFlow library and uses the alarm sound classification model to identify the type of alarm sound.

[1028] Step 4:

[1029] The system notifies the user of the type of alarm sound identified and its meaning, and suggests necessary countermeasures or actions.

[1030] Input: Identified alarm type

[1031] Output: The type of alarm that will be notified to the user, its meaning, and any suggested countermeasures or actions.

[1032] Specific behavior: The smartphone notifies the user of the meaning of the identified warning sound based on the type of sound. For example, if a "collision warning sound" is identified, the smartphone will notify the user, "There is a risk of a collision. Please apply the brakes." It will also suggest necessary countermeasures and actions.

[1033] Example 2

[1034] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1035] Conventional vehicle alarm systems have the problem that it is difficult for users to quickly and accurately understand the nature of the abnormality when an alarm sounds. In particular, when an alarm sounds while driving, users must refer to the manual to understand the meaning of the alarm, which can hinder safe driving. In addition, because users cannot immediately know the appropriate way to deal with the abnormality, there is a possibility that they will be slow to respond in situations that require a prompt response.

[1036] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1037] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for acquiring details of an abnormality corresponding to the identified alarm sound as text data, voice synthesis means for converting the acquired text data into voice data, means for notifying the user of the generated voice data, and means for enabling the user to deal with the details of the abnormality. Thus, when an alarm sound is emitted, the user can immediately understand the details of the abnormality through voice and can quickly learn an appropriate way to deal with it.

[1038] An "alarm sound" is an audio signal emitted by a vehicle to notify of an abnormality or warning.

[1039] The "means for identifying" refers to a device or software that analyzes the received alarm sound and identifies its type and content.

[1040] "Anomaly details" is information that indicates the specific problem or situation corresponding to the identified alarm sound.

[1041] "Text data" is data that expresses the content of an abnormality as text information.

[1042] "Voice data" refers to a voice signal generated from text data using voice synthesis technology.

[1043] "Speech synthesis means" refers to a device or software for converting text data into voice data.

[1044] The "notification means" refers to a device or software for notifying the user of the generated voice data.

[1045] "Means for enabling a user to take action" refers to devices or software that provide information and methods for the user to take appropriate action against the abnormality.

[1046] MODE FOR CARRYING OUT THE INVENTION

[1047] The present invention relates to a system for identifying an alarm sound emitted from a vehicle and notifying a user of the content of the abnormality by voice. Specific embodiments of this system will be described below.

[1048] Hardware and software used

[1049] Hardware: Servers, devices (smartphones, tablets, etc.)

[1050] Software: Speech synthesis system (e.g., Google Text-to-Speech API), alarm sound identification system (e.g., speech recognition model using TensorFlow)

[1051] System configuration

[1052] 1. Receiving an alarm sound

[1053] The device records the alarm sounds generated inside the vehicle in real time via a microphone.

[1054] The terminal transmits the recorded voice data to the server.

[1055] 2. Alarm sound identification

[1056] The server inputs the received audio data into the warning sound identification system.

[1057] The server uses an alarm sound identification system to analyze the audio data and identify the type of alarm sound.

[1058] 3. Obtaining the details of the abnormality

[1059] The server retrieves the abnormality content corresponding to the identified alarm sound from the database.

[1060] For example, as the abnormality content corresponding to "engine abnormality," text data such as "There is an abnormality in the engine. Stop the vehicle immediately and contact a specialized repair company" is acquired.

[1061] 4. Generating Audio Data

[1062] The server inputs the acquired text data into a speech synthesis system to generate speech data.

[1063] For example, using the Google Text-to-Speech API, we can generate voice data that says, "There is a problem with the engine. Stop the vehicle immediately and contact a professional repair shop."

[1064] 5. Notice to Users

[1065] The server transmits the generated voice data to the terminal.

[1066] The terminal reproduces the received audio data and notifies the user of the details of the abnormality.

[1067] Specific examples

[1068] For example, if an alarm sounds indicating an engine malfunction while the user is driving a car, the system operates as follows.

[1069] 1. The device records the alarm sound and sends the audio data to the server.

[1070] 2. The server uses the alarm sound identification system to identify the sound as "engine abnormality."

[1071] 3. The server retrieves text data from the database stating, "There is an engine malfunction. Stop the vehicle immediately and contact a professional repair company."

[1072] 4. The server converts the text data into voice data using a voice synthesis system.

[1073] 5. The device plays the generated voice data and informs the user of the abnormality.

[1074] Prompt Sentence Examples

[1075] Input: "Please explain what to do if an engine malfunction alarm sounds."

[1076] Output: "There is an engine malfunction. Stop the vehicle immediately and contact a professional repair service."

[1077] The system aims to ensure user safety by quickly identifying alarm sounds and providing users with audible instructions on how to respond appropriately.

[1078] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1079] Step 1:

[1080] Receiving alarm sounds

[1081] The device records the alarm sounds generated inside the vehicle in real time via a microphone.

[1082] Input: Alarm sound generated inside a car

[1083] Output: Recorded audio data

[1084] What it does: The device's microphone captures the alarm sound and uploads the audio data to a server via the Internet.

[1085] Step 2:

[1086] Alarm sound identification

[1087] The server inputs the received audio data into the warning sound identification system.

[1088] Input: Audio data sent from the device

[1089] Output: Type of alarm identified (e.g. engine abnormality, brake abnormality, etc.)

[1090] Specific operation: The server analyzes the audio data and identifies the type of alarm sound using an alarm sound identification system (e.g., a speech recognition model using TensorFlow).

[1091] Step 3:

[1092] Acquisition of abnormality details

[1093] The server retrieves the abnormality content corresponding to the identified alarm sound from the database.

[1094] Input: Identified alarm type

[1095] Output: Text data about the problem (e.g., "There is a problem with the engine. Stop the vehicle immediately and contact a professional repair shop.")

[1096] Specific operation: The server executes a database query to obtain the corresponding text data of the anomaly.

[1097] Step 4:

[1098] Generate audio data

[1099] The server inputs the acquired text data into a speech synthesis system to generate speech data.

[1100] Input: Text data of the anomaly

[1101] Output: Generated audio data

[1102] Specific operation: The server sends text data to a speech synthesis API (e.g., Google Text-to-Speech API) and receives the generated voice data.

[1103] Step 5:

[1104] User Notification

[1105] The server transmits the generated voice data to the terminal.

[1106] The terminal reproduces the received audio data and notifies the user of the details of the abnormality.

[1107] Input: Generated audio data

[1108] Output: A voice message to be announced to the user

[1109] Specific operation: The device plays audio data over the speaker, telling the user, "There is an engine malfunction. Please stop the vehicle immediately and contact a professional repair company."

[1110] (Application example 2)

[1111] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1112] In conventional vehicles, when an alarm sound is emitted, it is difficult for the user to immediately understand the meaning and take appropriate action. In particular, in autonomous vehicles, users are required to quickly understand the meaning of the alarm sound and take appropriate action, but current systems do not provide this support sufficiently. This may result in a decrease in user safety and vehicle operating efficiency.

[1113] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1114] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to the user the details of the abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the details of the abnormality, and means for audibly communicating the details of the abnormality to the user using a voice synthesis system, thereby enabling the user to immediately understand the meaning of the alarm sound and take appropriate action.

[1115] A "vehicle" is a mechanical device for moving on land, powered by an engine or motor.

[1116] An "alarm sound" is an audio signal that is generated to warn of an abnormality or danger in the vehicle.

[1117] "Discrimination" means distinguishing a particular sound or signal from others and recognizing its content.

[1118] "Abnormality details" is information that indicates a state or problem that deviates from the normal operation of the vehicle.

[1119] "User" refers to a person who operates or uses a vehicle.

[1120] "Explanation" is the act of conveying specific information or content to others so that they can understand it.

[1121] "Response" means taking appropriate action in response to problems or abnormalities that have occurred.

[1122] A "speech synthesis system" is a technology or device that converts text data into speech.

[1123] An "Internet search tool" is software or a service that allows you to search for information on the Internet and obtain the information you need.

[1124] The following system is constructed as an embodiment of this invention. First, a vehicle is equipped with a voice recognition system for identifying warning sounds. This voice recognition system identifies warning sounds generated inside the vehicle in real time and acquires their contents as text data. The speech_recognition library is used for voice recognition.

[1125] Next, a database is prepared to acquire the details of the abnormality corresponding to the identified alarm sound. This database stores the details of the abnormality corresponding to each alarm sound in text format. For example, for an alarm sound that says "engine abnormality," the following abnormality details are stored: "There is an abnormality in the engine. Stop the vehicle immediately and contact a professional repair shop."

[1126] The acquired abnormality information is converted into voice using a voice synthesis system. The gTTS library is used for voice synthesis. This voice synthesis system converts text data into voice data and conveys it to the user. This allows the user to immediately understand the meaning of the alarm sound and take appropriate action.

[1127] As a specific example, consider the case where an alarm sounds indicating an engine malfunction. In this case, the speech recognition system receives the text "engine malfunction" and retrieves the corresponding malfunction details from the database. The speech synthesis system then generates a voice message to the user saying, "There is an engine malfunction. Stop the vehicle immediately and contact a professional repair shop."

[1128] Examples of prompts to input to a generative AI model might include:

[1129] I would like to develop a system that can identify alarm sounds and inform the user of the nature of the abnormality by voice. Specifically, when an alarm indicating an engine abnormality sounds, a voice message should be sent saying, "There is an abnormality in the engine. Please stop the vehicle immediately and contact a professional repair shop." Please provide the Python code to achieve this.

[1130] This system allows users to quickly understand the meaning of the warning sound and take appropriate action, thereby improving vehicle safety and operational efficiency.

[1131] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1132] Step 1:

[1133] The server uses a voice recognition system to identify warning sounds generated inside the vehicle in real time. Specifically, it uses a voice recognition system (speech_recognition library) to convert the warning sounds into text data. The input is the voice data of the warning sounds, and the output is the text data of the identified warning sounds.

[1134] Step 2:

[1135] The server retrieves the corresponding abnormality details from the database based on the text data of the identified alarm sound. Specifically, it searches the database that stores the abnormality details corresponding to the alarm sound and retrieves the text data of the corresponding abnormality details. The input is the text data of the identified alarm sound, and the output is the text data of the corresponding abnormality details.

[1136] Step 3:

[1137] The server converts the acquired text data of the anomaly into voice data using a voice synthesis system. Specifically, it uses a voice synthesis system (gTTS library) to convert the text data into voice data. The input is the text data of the anomaly, and the output is voice data.

[1138] Step 4:

[1139] The terminal transmits the generated voice data to the user. Specifically, the voice data is played through a speaker to inform the user of the abnormality by voice. The input is the voice data, and the output is the voice information that the user hears.

[1140] Step 5:

[1141] The user takes appropriate action based on the details of the abnormality communicated by voice. Specifically, the user follows the instructions communicated by voice and takes appropriate action, such as stopping the vehicle or contacting a repair shop. The input is voice information, and the output is the user's response action.

[1142] Example 3

[1143] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1144] In conventional vehicle systems, when an alarm sound is emitted, it is difficult for the user to quickly and accurately understand the nature of the abnormality and find an appropriate way to deal with it. In particular, when the abnormality is complex or the user does not have technical knowledge, there is a high possibility that appropriate action will be delayed. To solve this problem, a system is needed that can analyze the alarm sound, identify the nature of the abnormality, and quickly provide a way to deal with it.

[1145] The identification process by the identification processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes means for receiving an alarm sound emitted from a vehicle, means for analyzing the received alarm sound to identify the nature of the abnormality, means for searching for a solution from the Internet based on the identified nature of the abnormality, and means for providing the searched solution to the user. This allows the user, when an alarm sound is emitted, to quickly and accurately understand the nature of the abnormality and immediately obtain an appropriate solution.

[1146] A "vehicle" is a mechanical device that uses an engine or motor as a power source for movement, and primarily refers to automobiles that run on roads.

[1147] The "alarm sound" is an audio signal that is generated to alert the user to an abnormality or danger in the vehicle, and is intended to draw the user's attention.

[1148] "Means for receiving" refers to devices or systems for receiving data transmitted from a vehicle, specifically using communication technologies such as Bluetooth or Wi-Fi.

[1149] "Means of analysis" refers to software or algorithms used to process and understand the received data.

[1150] "Abnormality" refers to the specific condition of a problem or malfunction that prevents the vehicle from operating normally.

[1151] "Means of identification" refers to methods or techniques for identifying specific anomalies based on the analysis results.

[1152] "Internet solution search" refers to technology or software that uses Internet resources to find solutions to specific anomalies.

[1153] "Means of providing" refers to the method or device for informing the user of the searched solution, specifically using a display device such as a smartphone or an in-car display.

[1154] MODE FOR CARRYING OUT THE INVENTION

[1155] The present invention is a system for analyzing an alarm sound emitted from a vehicle, identifying the nature of the abnormality, and providing a user with a method for dealing with the abnormality. Specific embodiments of this system will be described below.

[1156] System configuration

[1157] This system mainly uses the following hardware and software:

[1158] Server: The central component that receives, analyzes, searches, and serves data.

[1159] Terminal: A device used by a user, including a smartphone or an in-car display.

[1160] Speech recognition software: Analyzes alarm sounds using the Google Cloud Speech-to-Text API, among others.

[1161] Search engine API: Search for solutions on the Internet using Google Search API, etc.

[1162] Program processing

[1163] The server receives the alarm sound data sent from the user's vehicle. The received data is analyzed using voice recognition software. Based on the analysis results, the server identifies the nature of the abnormality. Based on the identified abnormality, the server uses a search engine API to search for related information from the Internet. The server extracts the most appropriate solution from the search results and provides it to the user.

[1164] Specific examples

[1165] For example, if a user's vehicle emits an alarm sound indicating low tire pressure, the server analyzes the alarm sound and identifies the abnormality as "Your tire pressure is low." The server then uses the Google Search API to search for "What to do if your tire pressure is low" and provides the user with the solution of "Please refill your tires with air at the nearest gas station or vehicle repair shop."

[1166] Prompt Sentence Examples

[1167] Analyze the alarm sound sent from the user's vehicle and identify the abnormality. Based on the identified abnormality, search the Internet for a solution and provide it to the user.

[1168] This system allows the user to quickly deal with any abnormalities in the vehicle. The server performs all processes from analyzing the alarm sound to providing a solution, so the user does not need to perform any complicated operations. This improves the safety of the vehicle and the convenience of the user. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[1169] Step 1: Receiving the alarm sound

[1170] Input: Alarm sound data sent from the user's vehicle

[1171] Specific operation: When the user's vehicle emits an alarm, the audio data is captured by the vehicle's sensors. The device then transmits the captured audio data to the server.

[1172] Output: Alarm sound data sent to the server

[1173] Step 2: Analyze the alarm sound

[1174] Input: Alarm sound data sent to the server

[1175] Specific operation: The server sends the received alarm sound data to speech recognition software (e.g., Google Cloud Speech-to-Text API) and converts it into text data. The server then analyzes the converted text data to identify the content of the alarm sound.

[1176] Output: Parsed text data and alarm sound content

[1177] Step 3: Identifying the anomaly

[1178] Input: Parsed text data and alarm sound content

[1179] Specific operation: The server identifies the abnormality based on the analysis results. For example, it identifies the abnormality as "tire pressure is low."

[1180] Output: Identified anomalies

[1181] Step 4: Internet Search

[1182] Input: Identified anomaly

[1183] What it does: The server uses the Google Search API to search for "What to do when tire pressure is low." The server then extracts the most relevant information from the search results.

[1184] Output: Search results and extracted solutions

[1185] Step 5: Provide solutions

[1186] Input: Search results and extracted solutions

[1187] Specific operation: The server sends the extracted solutions to the user's device. The user checks the solutions displayed on the device and takes appropriate action.

[1188] Output: Actions displayed on the user's device

[1189] In this way, the user can quickly respond to vehicle abnormalities. The server handles everything from analyzing the alarm sound to providing countermeasures, so the user does not need to perform complicated operations. This improves vehicle safety and user convenience.

[1190] (Application example 3)

[1191] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1192] Conventional vehicle systems have a problem in that when an alarm sound is issued, it is difficult for the user to understand the nature of the abnormality and quickly find an appropriate solution. In particular, when the abnormality is complex or the user does not have technical knowledge, appropriate action is often delayed. This leads to issues such as reduced vehicle safety and user convenience.

[1193] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1194] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to a user the nature of an abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the abnormality, means for searching information relating to the abnormality from the Internet, and means for providing the search results to the user. This allows the user to quickly understand the nature of the abnormality when an alarm sound is emitted and to search for and obtain an appropriate solution from the Internet. Furthermore, by notifying the user of the search results by voice, the user can safely receive information while driving.

[1195] A "vehicle" is a mechanical device for moving on roads, powered by an engine or motor.

[1196] An "alarm sound" is an audio signal that is generated to warn of an abnormality or danger in the vehicle.

[1197] An "identification means" is a device or software that detects a particular sound or signal and determines what it means.

[1198] "Abnormality details" refer to the specific condition or cause of a problem or malfunction that prevents the vehicle from operating normally.

[1199] A "user" is a person who operates a vehicle or receives information about the vehicle.

[1200] "Explanatory means" refers to a method or device for helping the user understand the nature of the abnormality.

[1201] The "means for enabling a user to take action" refers to a method or device that supports the user in taking appropriate action in response to the abnormality.

[1202] The "Internet" is an information and communications network that interconnects computer networks around the world.

[1203] A "search means" is a method or device for locating specific information on the Internet.

[1204] "Means for providing" refers to the method or device for conveying search results and information to users.

[1205] "Audio notification" is a method of conveying information using audio.

[1206] The system for carrying out the present invention includes a sensor mounted on a vehicle, a server with an Internet connection function, and a terminal for providing information to a user. The specific configuration and operation of the system will be described below.

[1207] System configuration

[1208] 1. Vehicle sensors

[1209] Vehicles are equipped with various sensors to detect tire pressure, engine temperature, collision warnings, etc. These sensors emit an alarm when they detect an abnormality.

[1210] 2. Server

[1211] The server includes software to identify the warning sounds sent from the vehicle and identify the nature of the abnormality. Specifically, it uses programming languages ​​such as Python to identify the warning sounds and the nature of the abnormality. The server also has the function of searching for information on the abnormality using internet search tools.

[1212] 3. User Device

[1213] The user terminal is a device such as a smartphone or tablet that receives information sent from the server. The user terminal is equipped with a voice notification function that notifies the user of the nature of the abnormality and how to deal with it.

[1214] System Operation

[1215] 1. Abnormality detection and alarm generation

[1216] When the vehicle's sensors detect an abnormality, an alarm sounds, which is transmitted to the server via the vehicle's microphone.

[1217] 2. Identifying alarm sounds and identifying abnormalities

[1218] The server identifies the received alarm sound and determines the abnormality. For example, if the tire pressure is low, the server determines the abnormality as "Tire pressure is low."

[1219] 3. Internet search and information provision

[1220] The server searches the Internet based on the identified anomaly to obtain information on how to deal with the problem, and sends the obtained information to the user's terminal.

[1221] 4. Audio Notifications

[1222] The user terminal notifies the user of the received information by voice, allowing the user to receive information safely while driving.

[1223] Specific examples

[1224] For example, if a vehicle sensor detects an abnormality such as low tire pressure, the system will operate as follows:

[1225] 1. The sensor detects an abnormal sound and sends it to the server.

[1226] 2. The server identifies the abnormality "Tire pressure is low."

[1227] 3. The server performs an internet search and obtains the solution, "Please refill the air at the nearest gas station or vehicle repair shop."

[1228] 4. The user device will give an audio notification and tell the user how to deal with the problem.

[1229] Prompt Sentence Examples

[1230] "Your tire pressure is low. Please refill it at the nearest gas station or vehicle repair shop. Search the internet for solutions."

[1231] In this way, the user can deal with the abnormality promptly and appropriately.

[1232] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1233] Step 1:

[1234] The vehicle's sensors detect an abnormality and sound an alarm.

[1235] Input: Abnormality detection signal from vehicle sensor

[1236] Output: Alarm sound

[1237] Specific operation: If the tire pressure drops, the tire pressure sensor will detect the abnormality and sound an alarm.

[1238] Step 2:

[1239] The warning sound is transmitted to the server through a microphone in the vehicle.

[1240] Input: Alarm

[1241] Output: Audio data to the server

[1242] Specific operation: A microphone inside the vehicle captures the warning sound and sends the audio data to a server.

[1243] Step 3:

[1244] The server identifies the received alarm sound and identifies the nature of the abnormality.

[1245] Input: Audio data

[1246] Output: Abnormality details

[1247] Specific operation: The server uses voice recognition software to analyze the alarm sound and identify the abnormality: "Tire pressure is low."

[1248] Step 4:

[1249] The server performs an internet search based on the identified abnormality and obtains information on how to deal with the problem.

[1250] Input: Abnormality details

[1251] Output: Information on what to do

[1252] Specific operation: The server uses an internet search engine to search for and retrieve the solution, "Your tire pressure is low. Please refill the tire with air at the nearest gas station or vehicle repair shop."

[1253] Step 5:

[1254] The server transmits the acquired information to the user terminal.

[1255] Input: Information on how to proceed

[1256] Output: Sending information to the user's device

[1257] Specific operation: The server sends the acquired information on how to deal with the problem to the user terminal.

[1258] Step 6:

[1259] The user terminal notifies the user of the received information by voice.

[1260] Input: Information on how to proceed

[1261] Output: Audio notification

[1262] Specific operation: The user device uses speech synthesis software to notify the user by voice the message, "Your tire pressure is low. Please refill the tire with air at the nearest gas station or vehicle repair shop."

[1263] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1264] "Example 1"

[1265] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system identifies an alarm sound emitted from a vehicle and explains to the user the nature of the abnormality corresponding to the identified alarm sound. Furthermore, the system allows the user to deal with the abnormality. The system further includes an emotion engine that recognizes the user's emotion. This emotion engine recognizes the user's emotion from, for example, the user's tone of voice, facial expression, choice of words, etc.

[1266] "Example 2"

[1267] The emotion engine adjusts the way it explains the abnormality depending on the user's emotions. For example, if the user is panicking, the emotion engine will explain the abnormality in a concise and clear manner. On the other hand, if the user is calm, the emotion engine will explain the abnormality in more detail. This provides an optimal explanation according to the user's emotional state, allowing the user to deal with the abnormality appropriately.

[1268] "Example 3"

[1269] Furthermore, the emotion engine uses an internet search tool to provide a method for dealing with the abnormality according to the user's emotions. For example, if the user is feeling anxious, the emotion engine searches for detailed methods for dealing with the abnormality and provides them to the user. This provides the optimal method for dealing with the abnormality according to the user's emotional state, allowing the user to deal with the abnormality appropriately.

[1270] The processing flow of each embodiment will be described below.

[1271] "Example 1"

[1272] Step 1: The system identifies the warning sound coming from the vehicle.

[1273] Step 2: The content of the abnormality corresponding to the identified alarm sound is explained to the user.

[1274] Step 3: Activate the emotion engine that recognizes the user's emotions. This emotion engine recognizes the user's emotions from the user's tone of voice, facial expressions, choice of words, etc.

[1275] "Example 2"

[1276] Step 1: The emotion engine recognizes the user's emotion.

[1277] Step 2: The emotion engine adjusts the way it explains the abnormality according to the user's emotions. Step 3: For example, if the user is panicking, the emotion engine will make the explanation of the abnormality concise and clear. If the user is calm, the emotion engine will make the explanation of the abnormality more detailed.

[1278] "Example 3"

[1279] Step 1: The emotion engine recognizes the user's emotion.

[1280] Step 2: The emotion engine uses an internet search tool to provide a solution to the abnormal content according to the user's emotion.

[1281] Step 3: For example, if the user is feeling anxious, the emotion engine will search for detailed coping methods and provide them to the user.

[1282] Example 1

[1283] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1284] In conventional vehicle systems, it was difficult to properly convey the meaning of an alarm sound to the user, making it difficult for the user to take appropriate action. Furthermore, a uniform response was given without considering the user's feelings, which could cause anxiety and confusion for the user. This led to problems such as a loss of vehicle safety and a sense of security for the user.

[1285] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1286] In this invention, the server includes means for identifying an alarm sound emitted from the vehicle, means for explaining to the user the details of the abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the abnormality, means for recognizing the user's emotion, and means for adjusting the explanation of the abnormality based on the recognized emotion. This makes it possible to accurately convey the meaning of the alarm sound to the user and enable the user to deal with the abnormality appropriately. Furthermore, by taking the user's emotion into consideration, it is possible to increase the user's sense of security and improve the safety of the vehicle.

[1287] An "alarm sound" is an audio signal emitted by a vehicle to warn of abnormalities or to alert the driver.

[1288] "Anomaly Description" is a description of the specific problem or situation that corresponds to the identified alarm sound.

[1289] "User" refers to a person who drives or uses a vehicle.

[1290] "Emotion recognition" is a technology that analyzes a user's emotions from their tone of voice, facial expressions, choice of words, etc.

[1291] A "voice recognition system" is a system that analyzes voice data and identifies specific sounds and words.

[1292] An "emotion engine" is software or hardware for analyzing and recognizing a user's emotions.

[1293] An "on-board computer" is a computer system installed in a vehicle that processes and communicates various data.

[1294] An "Internet search tool" is software for searching for information using the Internet.

[1295] A "generative AI model" is a model that uses artificial intelligence to generate new information and content from data.

[1296] This invention is a system that identifies warning sounds emitted by a vehicle, explains the appropriate nature of the abnormality to the user, and adjusts the response by recognizing the user's emotions. This system is realized using hardware such as a microphone, camera, and on-board computer in the vehicle, and software such as a voice recognition system, emotion engine, and generative AI model.

[1297] Hardware and software used

[1298] Hardware: In-vehicle microphones, cameras, on-board computers

[1299] Software: Speech recognition system, emotion engine, generative AI model

[1300] System Operation

[1301] 1. Acquiring audio data

[1302] The terminal (a microphone inside the vehicle) captures voice data generated inside the vehicle in real time.

[1303] The terminal transmits the acquired voice data to the vehicle-mounted computer.

[1304] The server receives the voice data from the on-board computer.

[1305] 2. Alarm sound identification

[1306] The server inputs the received voice data into a voice recognition system.

[1307] The server uses a voice recognition system to analyze the voice data and identify the alarm sound.

[1308] The server identifies the meaning of the identified warning sound, for example, the warning sound from the intelligent key or the warning sound from the collision warning function.

[1309] 3. Generating Anomalous Content

[1310] The server generates anomaly content corresponding to the identified alarm sound.

[1311] The server inputs the abnormality details into the generative AI model and generates a specific message to explain to the user.

[1312] 4. Notification of abnormalities

[1313] The server sends the generated abnormality information to the vehicle-mounted computer.

[1314] The terminal (on-board computer) notifies the user of the abnormality by voice or text.

[1315] The user checks the details of the notified abnormality.

[1316] 5. User Emotion Recognition

[1317] The device (cameras and microphones inside the vehicle) captures the user's tone of voice, facial expressions, and word choice in real time.

[1318] The terminal transmits the acquired data to the vehicle-mounted computer.

[1319] The server receives data obtained from the vehicle's computer and inputs it into the emotion engine.

[1320] The server uses an emotion engine to recognize the user's emotions.

[1321] 6. Emotional responses

[1322] The server adjusts the description of the anomaly based on the perceived user emotion.

[1323] The server generates additional messages according to the emotion and sends them to the vehicle's computer.

[1324] The terminal (on-board computer) notifies the user of additional messages.

[1325] The user sees the additional message and takes appropriate action.

[1326] Specific examples

[1327] Example 1: When the vehicle emits an intelligent key warning sound

[1328] The terminal receives the warning sound and transmits it to the vehicle's computer.

[1329] The server uses a voice recognition system to distinguish the warning sound from the warning sound of the intelligent key.

[1330] The server generates an error message stating "The intelligent key is not inside the vehicle" and sends it to the on-board computer.

[1331] The terminal notifies the user that "the intelligent key is not inside the vehicle."

[1332] The device captures the user's tone of voice and facial expressions and transmits them to the vehicle's computer.

[1333] The server uses an emotion engine to recognize that the user is in a hurry.

[1334] The server generates an additional message saying "Please stay calm. Find your keys" and sends it to the vehicle's computer.

[1335] The terminal will tell the user to "Please stay calm. Find your key."

[1336] Example 2: When the vehicle emits an alarm from the collision warning function

[1337] The terminal captures the alarm sound and transmits it to the vehicle's computer.

[1338] The server uses a voice recognition system to identify the warning sound as a collision warning sound.

[1339] The server generates an abnormality message such as "There is an obstacle ahead. Please apply the brakes," and sends it to the onboard computer.

[1340] The device notifies the user, "There is an obstacle ahead. Please apply the brakes."

[1341] The device captures the user's tone of voice and facial expressions and transmits them to the vehicle's computer.

[1342] The server uses an emotion engine to recognize that the user is surprised.

[1343] The server generates an additional message saying "Please stay calm" and sends it to the vehicle computer.

[1344] The device notifies the user to "remain calm."

[1345] Prompt Sentence Examples

[1346] Prompt 1: "The vehicle's intelligent key is beeping. How would you explain this to the user?"

[1347] Prompt 2: "A vehicle collision warning sound is heard. If the user is startled, how would they respond?"

[1348] The above is an embodiment of the present invention.

[1349] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1350] Step 1:

[1351] Acquiring audio data

[1352] The terminal (a microphone inside the vehicle) captures voice data generated inside the vehicle in real time.

[1353] Input: In-vehicle audio data

[1354] Output: Acquired audio data

[1355] What it does: A microphone captures audio and sends the data to the car's computer.

[1356] Step 2:

[1357] Sending audio data

[1358] The terminal transmits the acquired voice data to the vehicle-mounted computer.

[1359] Input: Acquired audio data

[1360] Output: Audio data sent to the on-board computer

[1361] Specific operation: The onboard computer receives audio data from the microphone and prepares it to be sent to the server.

[1362] Step 3:

[1363] Receiving audio data

[1364] The server receives the voice data from the on-board computer.

[1365] Input: Audio data sent from the on-board computer

[1366] Output: Received audio data

[1367] Specific operation: The server receives the voice data from the vehicle's computer and prepares it for analysis.

[1368] Step 4:

[1369] Alarm sound identification

[1370] The server inputs the received voice data into a voice recognition system.

[1371] Input: Received audio data

[1372] Output: Identified alarm sound

[1373] How it works: The voice recognition system analyzes the voice data and identifies specific warning sounds, such as the intelligent key warning sound or collision warning sound.

[1374] Step 5:

[1375] Anomaly generation

[1376] The server generates anomaly content corresponding to the identified alarm sound.

[1377] Input: Identified alarm sound

[1378] Output: Anomaly details

[1379] Specific operation: Based on the identified alarm sound, the server inputs the abnormal content into the generative AI model and generates a specific message to explain to the user.

[1380] Step 6:

[1381] Notification of abnormality details

[1382] The server sends the generated abnormality information to the vehicle-mounted computer.

[1383] Input: Anomaly

[1384] Output: Anomaly details sent to the onboard computer

[1385] Specific operation: The server sends the generated abnormality information to the on-board computer, which then prepares to notify the user.

[1386] Step 7:

[1387] Display of abnormality details

[1388] The terminal (on-board computer) notifies the user of the abnormality by voice or text.

[1389] Input: Anomaly sent to on-board computer

[1390] Output: The anomaly reported to the user

[1391] Specific operation: The on-board computer notifies the user of the abnormality via an audio output device or display.

[1392] Step 8:

[1393] User Emotion Recognition

[1394] The device (cameras and microphones inside the vehicle) captures the user's tone of voice, facial expressions, and word choice in real time.

[1395] Input: Your tone of voice, facial expressions, and word choice

[1396] Output: Retrieved emotion data

[1397] What it does: Cameras and microphones capture data about the user's emotions and send it to the car's computer.

[1398] Step 9:

[1399] Sending emotional data

[1400] The terminal transmits the acquired emotion data to the vehicle-mounted computer.

[1401] Input: Acquired emotion data

[1402] Output: Emotion data sent to the on-board computer

[1403] Specific operation: The onboard computer receives emotion data from the camera and microphone and prepares it to be sent to the server.

[1404] Step 10:

[1405] Receiving emotion data

[1406] The server receives emotion data from the vehicle computer and inputs it into the emotion engine.

[1407] Input: Emotion data sent from the vehicle's computer

[1408] Output: Emotion data input to the emotion engine

[1409] Specific operation: The server receives emotion data from the vehicle computer and inputs it into the emotion engine.

[1410] Step 11:

[1411] Emotion recognition

[1412] The server uses an emotion engine to recognize the user's emotions.

[1413] Input: Emotion data input to the emotion engine

[1414] Output: Perceived user emotion

[1415] Specific operation: The emotion engine analyzes the emotion data and identifies the user's emotion. For example, it recognizes whether the user is anxious or surprised.

[1416] Step 12:

[1417] Emotion-based responses

[1418] The server adjusts the description of the anomaly based on the perceived user emotion.

[1419] Input: Perceived user emotion

[1420] Output: Adjusted anomaly description

[1421] Specific operation: The server generates an additional message based on the user's emotion and sends it to the vehicle's computer.

[1422] Step 13:

[1423] Additional message notifications

[1424] The terminal (on-board computer) notifies the user of additional messages.

[1425] Input: Additional message sent to the onboard computer

[1426] Output: Additional message notified to the user

[1427] Specific operation: The on-board computer notifies the user of an additional message via an audio output device or display.

[1428] The above is the specific flow of the program processing of this system.

[1429] (Application example 1)

[1430] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1431] In conventional vehicle systems, when an alarm sound is generated, there are limited means to appropriately inform the user of the content of the alarm, which can make it difficult for the user to take prompt action. Furthermore, since information is provided without taking into account the user's emotional state, the user may feel stressed or be unable to take appropriate action. This may result in a loss of vehicle safety and the user's sense of security.

[1432] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1433] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to the user the details of the abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the details of the abnormality, means for recognizing the user's emotions, and means for providing appropriate advice and support according to the user's emotions, thereby making it possible to quickly and accurately convey the details of the alarm sound to the user and provide appropriate support according to the user's emotional state.

[1434] A "vehicle" is a mechanical device designed to travel on roads and powered by an engine or motor.

[1435] An "alarm sound" is an audio signal emitted from the vehicle to notify the user of an abnormality or danger.

[1436] "Discrimination" means recognizing a particular sound or signal and determining what it means.

[1437] "Abnormality" refers to a state or problem that deviates from the normal operation of the vehicle.

[1438] "User" refers to a person operating or riding in a vehicle.

[1439] "Emotion" refers to the user's psychological state or mood, and is determined from the tone of voice, facial expression, etc.

[1440] "Advice" refers to advice or instructions provided to help a user take appropriate action.

[1441] "Support" refers to assistance and help in dealing with problems or anomalies that a user encounters.

[1442] An "Internet search tool" is software or a service that searches for information on the Internet and provides it to users.

[1443] The following system configuration will be described as an embodiment of the present invention.

[1444] System Configuration

[1445] Hardware

[1446] Smartphone: Microphone, camera, processor, memory, storage

[1447] Vehicle: Alarm generator

[1448] software

[1449] Speech recognition library: Google Cloud Speech-to-Text API

[1450] Machine learning model: TensorFlow

[1451] Emotion Recognition Library: Microsoft Azure Emotion API

[1452] Notification system: Firebase Cloud Messaging

[1453] Program processing explanation

[1454] 1. Speech Recognition System

[1455] The smartphone's microphone is used to capture the warning sound emitted from the vehicle.

[1456] The captured audio data is converted into text data using the Google Cloud Speech-to-Text API.

[1457] 2. Alarm sound identification

[1458] The text data obtained from the voice recognition system is input into a machine learning model using TensorFlow to identify the type of alarm sound and the nature of the abnormality.

[1459] 3. Emotion Engine

[1460] It uses the smartphone's camera and microphone to capture the user's tone of voice and facial expressions.

[1461] The captured data is used to analyze user emotions using the Microsoft Azure Emotion API.

[1462] 4. Notice and Support

[1463] Based on the identified alarm sound and user emotion, appropriate notifications and advice are provided to the user using Firebase Cloud Messaging.

[1464] Specific examples

[1465] For example, if a collision warning occurs while the vehicle is moving and the user is startled, the system operates as follows.

[1466] 1. A voice recognition system captures collision warning sounds and converts them into text data.

[1467] 2. The alarm sound identification identifies the collision alarm sound and identifies the abnormality as "A collision alarm has occurred."

[1468] 3. The emotion engine analyzes the user's tone of voice and facial expressions and recognizes that the user is surprised.

[1469] 4. Notification and support provide appropriate advice to the user, such as "A collision warning has occurred. Please apply the brakes. Don't be alarmed, take a deep breath and stay calm."

[1470] Prompt Sentence Examples

[1471] "Your application must be able to identify the warning sounds coming from a vehicle and notify the user of any abnormalities. You must also add a function that recognizes the user's emotions and provides appropriate advice."

[1472] In this way, the embodiments for carrying out the invention have been specifically described.

[1473] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1474] Step 1:

[1475] Audio Capture

[1476] The microphone on the device (smartphone) captures the warning sound emitted from the vehicle. The input is the warning sound from the vehicle, and the output is voice data. This voice data is used for subsequent voice recognition processing.

[1477] Step 2:

[1478] Voice Recognition

[1479] The device sends the captured audio data to the Google Cloud Speech-to-Text API and converts it into text data. The input is audio data, and the output is text data. This text data is used to identify the type of alarm sound.

[1480] Step 3:

[1481] Alarm sound identification

[1482] The server inputs the text data into a machine learning model using TensorFlow to identify the type of alarm sound and the details of the abnormality. The input is text data, and the output is the type of alarm sound and the details of the abnormality. This identification result is used to notify the user.

[1483] Step 4:

[1484] Emotion Capture

[1485] The device's camera and microphone capture the user's tone of voice and facial expressions. The input is the user's voice and facial expressions, and the output is emotional data. This emotional data is used to analyze the user's emotional state.

[1486] Step 5:

[1487] emotion recognition

[1488] The server sends the captured emotion data to the Microsoft Azure Emotion API to analyze the user's emotions. The input is emotion data, and the output is the user's emotional state. This emotional state is used to provide appropriate advice.

[1489] Step 6:

[1490] Notifications and Support

[1491] The server generates appropriate notifications and advice based on the identified alarm type, abnormality details, and the user's emotional state, and sends them to the device using Firebase Cloud Messaging. The input is the alarm type, abnormality details, and the user's emotional state, and the output is a notification message. This notification message provides the user with appropriate countermeasures.

[1492] In this way, the specific operations performed at each processing step and their inputs and outputs have been explained.

[1493] Example 2

[1494] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1495] Conventional vehicle systems have the problem that when an alarm sound is issued, it is difficult for the user to quickly and accurately understand the nature of the abnormality and take appropriate action. In addition, because an appropriate explanation based on the user's emotional state is not provided, users in a panic state may not be able to take appropriate action. This can compromise the safety of the vehicle and the user's sense of security.

[1496] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1497] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for acquiring details of an abnormality corresponding to the identified alarm sound as text data, voice synthesis means for converting the acquired text data into voice and conveying the voice to the user, means for detecting the emotional state of the user, and emotion engine means for adjusting the method of explaining the abnormality according to the emotional state of the user, thereby enabling the user to quickly and accurately understand the details of the abnormality corresponding to the alarm sound and receive an appropriate explanation according to their emotional state.

[1498] An "alarm sound" is a sound emitted by a vehicle to notify of abnormalities or warnings.

[1499] "Means for identifying" refers to devices or software that analyze the alarm sound and identify what the sound indicates.

[1500] The "content of abnormality" is the specific content of the abnormality or problem in the vehicle that corresponds to the warning sound.

[1501] "Text data" is data that expresses the content of an abnormality as text information.

[1502] "Speech synthesis means" refers to a device or software for converting text data into speech.

[1503] A "user" is a person who drives or uses a vehicle.

[1504] "Emotional state" refers to the user's current psychological state or mood.

[1505] The "emotion engine means" is a device or software for detecting the user's emotional state and adjusting the method of explaining the abnormality accordingly.

[1506] MODE FOR CARRYING OUT THE INVENTION

[1507] This invention is a system that identifies warning sounds emitted from a vehicle and provides audible explanations of the abnormality to the user. Furthermore, by adjusting the explanation method according to the user's emotional state, the system allows the user to deal with the abnormality appropriately.

[1508] Hardware and software used

[1509] Hardware: Servers, terminals (user devices)

[1510] Software: speech recognition software, speech synthesis system, emotion engine, database

[1511] System configuration

[1512] 1. Server

[1513] The server receives the warning sounds generated by the vehicle and analyzes them using voice recognition software.

[1514] The abnormality content corresponding to the identified alarm sound is retrieved from the database.

[1515] The acquired abnormality information is temporarily saved as text data.

[1516] An emotion engine is used to adjust the way the anomaly is explained depending on the user's emotional state.

[1517] The adjusted text data is converted into speech using a speech synthesis system.

[1518] 2. Terminal

[1519] The device uses a camera and microphone to detect the user's emotional state.

[1520] The detected emotional state data is transmitted to a server.

[1521] The voice data sent from the server is transmitted to the user through a speaker.

[1522] Specific examples

[1523] For example, if an alarm sounds indicating an engine malfunction, the server identifies the sound as an engine malfunction. Next, the server retrieves text data from the database that reads, "There is an engine malfunction. Stop the vehicle immediately and contact a professional repair shop." The device analyzes the user's tone of voice and facial expression and determines that the user is in a panic. The server uses an emotion engine to adjust the explanation to be concise and clear, and converts it into speech using a speech synthesis system. Finally, the device plays the generated speech over the speaker and conveys it to the user.

[1524] Prompt Sentence Examples

[1525] "When an engine alarm sounds, generate a description for when the user is panicking."

[1526] "When an alarm sounds indicating a brake malfunction, generate an explanation for the user to remain calm."

[1527] In this way, the system can identify alarm sounds and provide an optimal audio explanation based on the user's emotional state, allowing the user to quickly and accurately understand the nature of the anomaly and take appropriate action.

[1528] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1529] Program processing flow

[1530] Step 1: Receiving and identifying the alarm sound

[1531] Input: Vehicle alarm sound

[1532] Process: The server receives the alarm sound and analyzes it using voice recognition software.

[1533] Output: Type of alarm identified (e.g. engine abnormality, brake abnormality)

[1534] What it does: The server receives the alarm sound through a microphone and analyzes it with voice recognition software, for example by detecting specific frequencies or patterns and identifying it as an "engine malfunction."

[1535] Step 2: Obtaining the details of the anomaly

[1536] Input: Identified alarm type

[1537] Processing: The server retrieves the corresponding anomaly details from the database as text data.

[1538] Output: Text data of the abnormality (e.g. "There is an abnormality in the engine. Stop the vehicle immediately and contact a professional repair shop.")

[1539] Specific operation: The server accesses the database, searches for and retrieves the anomaly content corresponding to the identified alarm sound, and temporarily stores the retrieved text data.

[1540] Step 3: Detecting the user's emotional state

[1541] Input: User voice and facial expression data

[1542] Processing: The device detects the user's emotional state using the camera and microphone.

[1543] Output: User's emotional state (e.g., panicked, calm)

[1544] How it works: The device captures the user's facial expressions with a camera and records the user's voice with a microphone. This data is then analyzed to determine whether the user is panicking or calm.

[1545] Step 4: Adjust your explanation

[1546] Input: Textual data of the anomaly, user's emotional state

[1547] Processing: The server uses an emotion engine to adjust the way the anomaly is explained based on the user's emotional state.

[1548] Output: Adjusted text data of the anomaly (e.g., brief description, detailed description)

[1549] Specific behavior: The server uses an emotion engine to adjust the text data based on the user's emotional state. For example, if the user is panicking, the explanation will be concise and clear.

[1550] Step 5: Text-to-Speech

[1551] Input: Adjusted text of the anomaly

[1552] Processing: The server uses a speech synthesis system to convert the text data into speech.

[1553] Output: Audio data describing the problem (e.g., "There is a problem with the engine. Stop the vehicle immediately and contact a professional repair shop.")

[1554] What happens: The server uses speech synthesis software (e.g., a speech synthesis API) to convert the adjusted text data into speech data.

[1555] Step 6: Notify users

[1556] Input: Audio recording of the anomaly

[1557] Processing: The device plays the generated audio to the user.

[1558] Output: The user receives an audio description of the anomaly.

[1559] Specific operation: The device plays audio data through the speaker to inform the user of the abnormality. For example, a voice message may be played from the smartphone saying, "There is an abnormality in the engine. Please stop the vehicle immediately and contact a professional repair shop."

[1560] In this way, the entire system functions by processing input data at each step and generating the output data required for the next step.

[1561] (Application example 2)

[1562] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1563] In conventional vehicle systems, when an alarm sounds, it can be difficult for users to understand the nature of the abnormality and take appropriate action. Furthermore, because an appropriate explanation is not provided that reflects the user's emotional state, the user may panic or miss important information. This poses a challenge, hindering user safety and proper vehicle maintenance.

[1564] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1565] In this invention, the server includes means for identifying an alarm sound emitted from the vehicle, means for explaining to the user the details of the abnormality corresponding to the identified alarm sound, means for analyzing the emotional state of the user, means for adjusting the method of explaining the details of the abnormality in accordance with the analyzed emotional state, and means for enabling the user to deal with the details of the abnormality. This enables the user to deal appropriately with the alarm sound and provides an optimal explanation according to the user's emotional state, thereby ensuring the user's safety and enabling appropriate maintenance of the vehicle.

[1566] "Vehicle" means a mechanical device for moving on land, including automobiles and trucks powered by engines or motors.

[1567] An "alarm sound" is an audio signal that is generated to alert the driver to abnormalities or dangers in the vehicle, and different sounds are generated depending on the situation, such as engine abnormalities or collision warnings.

[1568] "Means for identifying" refers to technology or equipment that recognizes a particular audio signal or data and determines what it means.

[1569] "Abnormality" refers to the specific problem or malfunction that prevents the vehicle from operating normally, including engine failure and flat tires.

[1570] "User" refers to a person who operates or uses a vehicle, including the driver and passengers.

[1571] "Means for explaining" refers to a method or device for conveying specific information to a user, including a voice synthesis system and a display.

[1572] "Emotional state" refers to the user's psychological state and includes emotions such as panic or calm.

[1573] "Analytical tools" refers to techniques or devices used to analyze data and derive specific conclusions or information.

[1574] "Adjusting means" refers to techniques or devices for changing operations or settings in response to specific conditions.

[1575] "Means for enabling a user to cope" refers to a method or device that assists a user in appropriately dealing with a particular problem or situation.

[1576] The system for carrying out the present invention identifies warning sounds emitted from a vehicle, explains to a user the nature of the abnormality corresponding to the identified warning sound, and has the function of analyzing the emotional state of the user and adjusting the method of explaining the abnormality based on the analysis result.

[1577] System configuration

[1578] The system consists of the following main components:

[1579] 1. Warning sound identification module: This module identifies warning sounds emitted by vehicles and uses speech recognition technology. Specifically, it uses the speech_recognition library.

[1580] 2. Anomaly Description Module: This module obtains the anomaly description corresponding to the identified alarm sound as text data and conveys it to the user using a speech synthesis system. Specifically, it uses the gTTS library.

[1581] 3. Emotional State Analysis Module: This module analyzes the user's emotional state and uses image recognition technology. Specifically, it uses the emotion_recognition library.

[1582] 4. Explanation method adjustment module: This module adjusts the explanation method for abnormal content according to the analyzed emotional state.

[1583] System Operation

[1584] The server uses voice recognition technology to identify warning sounds emitted by the vehicle. It obtains the details of the abnormality corresponding to the identified warning sound as text data and communicates this to the user using a voice synthesis system. It analyzes the user's facial expressions using image recognition technology to analyze the user's emotional state. It adjusts the way it explains the abnormality based on the analysis results. For example, if the user is in a panic, it provides a brief explanation, and if the user is calm, it provides a detailed explanation.

[1585] Specific examples

[1586] For example, if an engine malfunction alarm sounds, the system will:

[1587] 1. The alarm sound identification module identifies the engine abnormality alarm sound.

[1588] 2. The abnormality description module obtains text data stating, "There is an abnormality in the engine. Stop the vehicle immediately and contact a specialist repair shop," and conveys this to the user using a voice synthesis system.

[1589] 3. The emotional state analysis module analyzes the user's facial expressions and detects whether the user is in a panic state.

[1590] 4. The explanation adjustment module simplifies the explanation and says, "There is an abnormality in the engine."

[1591] Prompt Sentence Examples

[1592] Example prompts to input to a generative AI model:

[1593] Design your system to adapt the explanation of the problem depending on the user's emotional state. If the user is panicking, keep the explanation brief, and if the user is calm, provide more detail. For example, if an engine alarm sounds, a panicked user might be told, "There is an engine problem," while a calm user might be told, "There is an engine problem. Stop immediately and contact a professional repair service."

[1594] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1595] Step 1:

[1596] The server receives the warning sound emitted from the vehicle as audio data. The input is the audio data, and the output is the speech recognition result. The server converts the audio data into text using the speech_recognition library. This process identifies what the warning sound means.

[1597] Step 2:

[1598] The server acquires the abnormality content corresponding to the identified alarm sound as text data. The input is the speech recognition result, and the output is the text data of the abnormality content. The server references a dictionary of predefined abnormality content to acquire the corresponding abnormality content.

[1599] Step 3:

[1600] The server receives image data of a user to analyze the user's emotional state. The input is the image data, and the output is the analysis result of the emotional state. The server analyzes the image data using the emotion_recognition library to identify the user's emotional state.

[1601] Step 4:

[1602] The server adjusts the explanation method for the anomaly according to the analyzed emotional state. The input is the analysis result of the emotional state and text data of the anomaly, and the output is the adjusted explanatory text. The server adjusts the level of detail of the explanation based on the emotional state and generates an explanation that is optimal for the user.

[1603] Step 5:

[1604] The server converts the adjusted explanatory text into speech and conveys it to the user. The input is the adjusted explanatory text and the output is the speech data. The server converts the text to speech using the gTTS library and sends the speech data to the user's device.

[1605] Step 6:

[1606] The user's device plays the received audio data and notifies the user of the abnormality. The input is audio data and the output is audio playback. The device uses the playsound library to play the audio data and notify the user of the abnormality.

[1607] Example 3

[1608] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1609] In conventional vehicle systems, when an alarm sound is generated, there are limited means to properly inform the user of the abnormality, making it difficult for the user to take prompt and appropriate action. In addition, there is no means to provide a way to deal with the situation according to the user's emotional state, which often causes anxiety and stress for the user. This leads to issues of reduced safety and convenience for the user.

[1610] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1611] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to the user the details of the abnormality corresponding to the identified alarm sound, means for providing a method of dealing with the abnormality using an internet search tool so that the user can deal with the abnormality, and means for providing a method of dealing with the abnormality according to the user's emotions using an emotion engine that analyzes the user's emotional state. This allows the user to quickly understand the details of the abnormality corresponding to the alarm sound and obtain an appropriate method of dealing with it, and because the optimal method of dealing with it according to the user's emotional state is provided, the user can deal with the abnormality with peace of mind.

[1612] An "alarm sound" is a sound emitted by a vehicle to notify of abnormalities or warnings.

[1613] "Means for identifying" refers to devices or algorithms that analyze the alarm sound and identify what the sound represents.

[1614] The "abnormality content" is the specific content of the vehicle abnormality or problem indicated by the warning sound.

[1615] The "means for explaining to the user" refers to a device or method for communicating the identified abnormality to the user in an easy-to-understand manner.

[1616] An "Internet search tool" is software or a service that allows you to search for information on the Internet and obtain the information you need.

[1617] "Countermeasures" are specific actions or procedures that a user should take in response to an identified abnormality.

[1618] An "emotion engine" is software or algorithms that analyze a user's emotional state and provide information and solutions appropriate to that state.

[1619] "Emotional state" refers to the psychological state such as anxiety, relief, or stress felt by the user.

[1620] MODE FOR CARRYING OUT THE INVENTION

[1621] This invention is a system that identifies warning sounds emitted from a vehicle and provides the user with an appropriate response. The system is realized using hardware and software such as a server, a user terminal, an internet search tool, and an emotion engine.

[1622] Hardware and software used

[1623] Hardware: Servers, user devices (smartphones, tablets, PCs, etc.)

[1624] Software: Internet search tools, sentiment engines, generative AI models

[1625] Specific operation of the system

[1626] 1. Receiving and identifying alarm sounds

[1627] The server receives the alarm sound transmitted from the user terminal.

[1628] The server analyzes the received alarm sound and identifies what the sound represents.

[1629] Specifically, a voice recognition algorithm is used to identify warning sounds such as "tire pressure low."

[1630] 2. Search for anomalies

[1631] The server uses an internet search tool to search for the abnormality content corresponding to the identified alarm sound.

[1632] The server uses search engines such as Google and Bing to gather relevant information.

[1633] Specifically, search for a solution such as "Your tire pressure is low. Please refill the air at the nearest gas station or vehicle repair shop."

[1634] 3. Providing information to users

[1635] The server transmits the search results to the user terminal.

[1636] The user can check how to deal with the problem on the device.

[1637] Specifically, a notification will appear on the user's smartphone with the message, "Your tire pressure is low. Please refill the air at the nearest gas station or vehicle repair shop."

[1638] 4. Emotion analysis using an emotion engine

[1639] The terminal analyzes the user's emotional state using an emotion engine.

[1640] The device reads emotions from the user's voice, facial expressions, input text, etc.

[1641] Specifically, if the user is feeling anxious, the emotion engine analyzes that information and sends it to the server.

[1642] 5. Rediscovering and providing ways to cope with emotions

[1643] The server uses the internet search tool again to search for the most appropriate way to deal with the user's emotional state based on the information from the emotion engine.

[1644] The server searches for detailed instructions and reassuring information and provides it to the user.

[1645] Specifically, the message "Your tire pressure is low. Please refill it at the nearest gas station or repair shop. Regularly checking your tire pressure can also help prevent accidents" will be displayed on the user's smartphone.

[1646] Specific examples

[1647] Example 1: Low tire pressure

[1648] Receiving and identifying the alarm sound: The server receives the "beep, beep" alarm sound sent from the user terminal and identifies it as indicating low tire pressure.

[1649] Search for anomaly: The server searches Google for solutions such as "Your tire pressure is low. Please refill the tire with air at the nearest gas station or repair shop."

[1650] Informing the user: A notification appears on the user's smartphone with the message, "Your tire pressure is low. Please refill it at the nearest gas station or vehicle repair shop."

[1651] Example 2: When a user feels anxious

[1652] Emotion analysis using an emotion engine: The device analyzes the user's voice to determine whether they are feeling anxious and sends that information to the server.

[1653] Re-searching and providing solutions according to the emotion: The server searches for detailed solutions again and displays information on the user's smartphone such as, "Your tire pressure is low. Please refill the air at the nearest gas station or vehicle repair shop. Also, by regularly checking the tire pressure, you can prevent accidents."

[1654] Prompt Sentence Examples

[1655] "What should I do if my tire pressure is low?"

[1656] "Find out more about what to do if your users are worried."

[1657] In this way, the system provides information that enables the user to deal with the abnormality. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1658] Step 1:

[1659] Alarm reception and identification

[1660] Input: Alarm sound data sent from the user terminal

[1661] Specific operation: The server receives the alarm sound sent from the user terminal, analyzes the received alarm sound using a voice recognition algorithm, and identifies what the sound represents.

[1662] Data processing: Analyze the voice data and extract features.

[1663] Output: Identified alarm type (e.g. low tire pressure)

[1664] Step 2:

[1665] Search for anomalies

[1666] Input: Identified alarm type

[1667] Specific operation: The server uses an internet search tool to search for the anomaly corresponding to the identified alarm sound, and collects related information using search engines such as Google and Bing.

[1668] Data processing: generating search queries and performing internet searches.

[1669] Output: Search results with instructions (e.g. "Your tire pressure is low. Please refill it at the nearest gas station or repair shop.")

[1670] Step 3:

[1671] Providing information to users

[1672] Input: What to do as search result

[1673] Specific operation: The server sends the search results to the user's device, where the user can check how to deal with the problem.

[1674] Data processing: Converting search results into a user-friendly format.

[1675] Output: A message to be displayed on the user's device with instructions on what to do (e.g., "Your tire pressure is low. Please refill it at the nearest gas station or repair shop.")

[1676] Step 4:

[1677] Emotion analysis using an emotion engine

[1678] Input: Emotional data such as the user's voice, facial expressions, and typed text

[1679] Specific operation: The device analyzes the user's emotional state using an emotion engine, reading emotions from the user's voice, facial expressions, input text, etc.

[1680] Data processing: Analyze emotion data to identify the user's emotional state.

[1681] Output: User's emotional state (e.g., anxiety)

[1682] Step 5:

[1683] Rediscovering and providing ways to cope with emotions

[1684] Input: User's emotional state, identified alarm sound type

[1685] Specific operation: Based on the information from the emotion engine, the server again uses the internet search tool to search for the most appropriate way to deal with the user's emotional state. It searches for detailed ways to deal with the situation and information that will give a sense of security, and provides it to the user.

[1686] Data processing: Generate search queries based on emotional states and perform internet searches.

[1687] Output: A message displayed on the user's device with advice based on the emotion (e.g., "Your tire pressure is low. Please refill your tires at the nearest gas station or repair shop. Regularly checking your tire pressure can also help prevent accidents.")

[1688] (Application example 3)

[1689] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1690] In conventional vehicle systems, when an alarm sounds, it can be difficult for the user to understand the nature of the abnormality and take appropriate action. Furthermore, since there is no means to provide a way to deal with the situation according to the user's emotional state, the user may feel anxious or stressed. This may result in a loss of safety and comfort for the user.

[1691] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes: means for identifying an alarm sound emitted from a vehicle; a prompt sentence instructing the server to explain to the user the details of an abnormality corresponding to the identified alarm sound; means for explaining to the user the details of the abnormality corresponding to the identified alarm sound using a generative AI model; a prompt sentence instructing the server to explain to the user how to deal with the abnormality; and means for explaining to the user how to deal with the abnormality using the generative AI model. The server also includes means for recognizing the user's emotional state using an emotion engine, and the means for explaining the details of the abnormality to the user includes a prompt sentence instructing the server to explain to the user how to deal with the abnormality based on the recognized emotional state of the user and using the generative AI model. This allows the user to quickly understand the appropriate way to deal with the alarm sound and is further provided with the optimal way to deal with the alarm sound depending on the user's emotional state, thereby improving safety and comfort. The alarm sound includes a warning sound from the vehicle's intelligent key or a warning sound emitted from a collision warning function.

[1692] A "vehicle" is a mechanical device for moving on roads, powered by an engine or motor.

[1693] An "alarm sound" is an audio signal that is generated to warn of an abnormality or danger in the vehicle.

[1694] An "identification means" is a device or software that recognizes a particular sound or signal and determines what it is.

[1695] "Abnormality details" are specific conditions of problems or malfunctions that prevent the vehicle from operating normally.

[1696] A "user" is a person who operates or uses a vehicle.

[1697] "Means for explanation" are devices or software that provide information to users and help them understand.

[1698] "Measures to enable a user to take action" refers to devices or software that provide instructions and support for the user to take appropriate action in response to the abnormality.

[1699] "Emotional state" refers to the user's psychological state or mood.

[1700] A "means for recognizing" is a device or software for detecting and determining a user's emotional state.

[1701] "Countermeasures" are specific actions and procedures to be taken in response to the abnormality.

[1702] A "generative AI model" is an algorithm or system that is trained to perform a specific task using artificial intelligence.

[1703] A "prompt" is text data input into a generative AI model and is an instruction statement to obtain a specific output.

[1704] The following system configuration is used as an embodiment of the present invention.

[1705] System Configuration

[1706] 1. Hardware Configuration

[1707] Vehicle sensor system: Various sensors (e.g., microphones, cameras) installed in the vehicle are used to detect warning sounds and the user's emotional state.

[1708] In-vehicle display: A display that shows the user the nature of the abnormality corresponding to the alarm sound and how to deal with it.

[1709] Smartphone: A device that interacts with the vehicle and provides additional information.

[1710] 2. Software Configuration

[1711] Speech recognition software: Google Cloud Speech-to-Text is used to convert the vehicle's warning sounds into text.

[1712] Internet search engine: Using the Google Search API, search for solutions to the anomaly that corresponds to the identified alarm sound.

[1713] Emotion recognition software (emotion engine): Using the Affectiva SDK, the system recognizes the user's emotional state from facial images.

[1714] Generative AI model: OpenAI (registered trademark) GPT-4 (registered trademark) is used to generate prompts based on the user's emotional state and provide detailed instructions on how to respond.

[1715] Processing flow

[1716] 1. Identifying alarm sounds

[1717] The vehicle's sensor system detects the warning sound and uses voice recognition software to convert the sound into text.

[1718] For example, if a warning beep sounds to indicate low tire pressure, the text "beep beep" is generated. The warning beep includes, for example, a warning beep from an intelligent key or a warning beep from a collision warning function of a vehicle.

[1719] 2. Providing details of the abnormality and how to deal with it

[1720] The server uses a prompt sentence that instructs the user to explain the abnormality corresponding to the identified alarm sound to the user and a generative AI model to explain the abnormality corresponding to the identified alarm sound to the user. Example prompt sentence: "Car model: AA. Alarm sound: beep beep. Use an internet search engine to search for the abnormality that corresponds to the identified alarm sound and explain the abnormality to the user."

[1721] For example, an explanation such as "Your tire pressure is low" may be generated and provided to the user by voice or text display. The server also uses a prompt statement instructing the user to explain how to deal with the abnormality and the generated AI model to explain to the user how to deal with the abnormality. Example prompt sentence: "Error: Tire pressure is low. Please use an internet search engine to find out how to resolve this issue."

[1722] For example, the generated explanation is "Please refill your tires at the nearest gas station or repair shop."

[1723] 3. Recognizing emotional states

[1724] The vehicle's camera captures images of the user's face and emotion recognition software is used to recognize the user's emotional state.

[1725] For example, if the user is feeling anxious, the emotional state "anxious" is recognized.

[1726] 4. Prompt generation

[1727] A generative AI model is used to generate prompt sentences that correspond to the user's emotional state.

[1728] For example, a prompt sentence may be generated that reads, "User's emotional state: anxiety. Solution: refill your air at the nearest gas station or vehicle repair shop. Please adjust the solution instructions to suit the user's emotional state." The server uses the generated prompt sentence and the generative AI model to adjust the explanation of how to deal with the problem.

[1729] 5. Providing solutions

[1730] Users are provided with solutions via the in-car display or smartphone.

[1731] For example, a message might appear saying, "First, stay calm. Then, go to the nearest gas station or repair shop to refill your tires with air."

[1732] Specific examples

[1733] Scenario: A self-driving vehicle is driving when an alarm sounds indicating low tire pressure.

[1734] User emotions: Feeling anxious.

[1735]

[1736] This system enables users to quickly understand the appropriate way to respond to an alarm sound and provides the optimal way to respond depending on their emotional state, thereby improving safety and comfort.

[1737] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1738] Step 1:

[1739] The vehicle's sensor system detects an alarm sound. The input is the audio data generated by the vehicle, and the output is the audio data of the detected alarm sound. Specifically, a microphone mounted on the vehicle collects the surrounding sounds and records them as audio data.

[1740] Step 2:

[1741] The server uses speech recognition software to convert the audio data of the alarm sound into text. The input is the audio data obtained in step 1, and the output is text data representing the alarm sound. Specifically, Google Cloud Speech-to-Text is used to analyze the audio data and generate text such as "beep beep."

[1742] Step 3:

[1743] The server uses a prompt sentence instructing the user to explain the abnormality corresponding to the identified alarm sound and the generative AI model to explain the abnormality corresponding to the identified alarm sound to the user. At this time, the generative AI model uses an internet search engine to search for an explanation of the abnormality corresponding to the identified alarm sound. The input is the text data obtained in step 2, and the output is the search results related to the abnormality. Specifically, the Google Search API is used to send the query "Car model: AA Alarm sound: beep beep Search for the abnormality corresponding to the identified alarm sound" and obtain the search result "Tire pressure is low." The generative AI model generates an explanation based on this search result. The server also uses a prompt sentence that instructs the server to explain to the user how to deal with the abnormality and the generation AI model to generate an explanatory sentence to explain to the user how to deal with the abnormality.

[1744] Step 4:

[1745] The vehicle's camera captures the user's facial image, and the server uses emotion recognition software to recognize the user's emotional state. The input is the user's facial image data, and the output is the user's emotional state (e.g., anxiety). Specifically, the system uses the Affectiva SDK to analyze the facial image and determine whether the user is feeling anxious.

[1746] Step 5:

[1747] The server uses the generative AI model to generate a prompt that instructs the user to adjust the explanation according to their emotional state. The input is the explanation of how to deal with the situation obtained in step 3 and the emotional state obtained in step 4, and the output is the adjusted explanation.

[1748] Step 6:

[1749] The server sends the generated prompt text to the in-vehicle display or smartphone, providing the user with a solution. The input is the explanatory text obtained in step 5, and the output is the message to be displayed to the user. Specifically, the in-vehicle display or smartphone displays the message, "First, please stay calm. Then, go to the nearest gas station or vehicle repair shop to refill your tires."

[1750] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1751] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1752] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1753] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1754] [Third embodiment]

[1755] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1756] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1757] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1758] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1759] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1760] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1761] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1762] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1763] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1764] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1765] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1766] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1767] "Example 1"

[1768] In one embodiment of the present invention, a voice recognition system is used to identify warning sounds emitted from a vehicle. This voice recognition system can learn various warning sounds emitted from the vehicle and identify the meaning of each warning sound. For example, it can identify warning sounds from an intelligent key or a collision warning function.

[1769] "Example 2"

[1770] A speech synthesis system is used as a means for explaining to the user the details of the abnormality corresponding to the identified alarm sound. This speech synthesis system acquires the details of the abnormality corresponding to the identified alarm sound as text data, converts it into voice, and conveys it to the user. For example, when an alarm sound indicating an engine abnormality sounds, the speech synthesis system will convey to the user, "There is an abnormality in the engine. Please stop the vehicle immediately and contact a professional repair shop."

[1771] "Example 3"

[1772] An internet search tool is used as a means for allowing the user to deal with the abnormality. This internet search tool searches the internet for information on the abnormality corresponding to the identified alarm sound and provides the results to the user. For example, if an alarm sound is heard indicating low tire pressure, the internet search tool provides the user with a solution such as "Your tire pressure is low. Please refill the tire with air at the nearest gas station or vehicle repair shop."

[1773] The processing flow of each embodiment will be described below.

[1774] "Example 1"

[1775] Step 1: The warning sound emitted from the vehicle is collected by a microphone.

[1776] Step 2: The collected alarm sounds are input into a speech recognition system, which uses a database of previously trained alarm sounds to identify the meaning of the input alarm sound.

[1777] Step 3: The user is notified of the identification result by, for example, displaying the identification result on a display in the vehicle.

[1778] "Example 2"

[1779] Step 1: Receive the identification result from the speech recognition system.

[1780] Step 2: Obtain the abnormality details corresponding to the identification result as text data. This text data is obtained, for example, from a database.

[1781] Step 3: The acquired text data is input into a speech synthesis system and converted into speech.

[1782] Step 4: The converted audio is output from the speaker to inform the user of the abnormality.

[1783] "Example 3"

[1784] Step 1: Receive the identification result from the speech recognition system.

[1785] Step 2: Use an internet search tool to search for information about the anomaly that corresponds to the identification result.

[1786] Step 3: The search results are notified to the user. This notification is performed, for example, by displaying the search results on the vehicle display.

[1787] Example 1

[1788] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1789] Conventional vehicle alarm sound identification systems have had the problem of making it difficult to quickly and accurately convey the meaning of an alarm sound to the user when it occurs. Furthermore, the information provided to users is insufficient to enable them to understand the meaning of the alarm sound and take appropriate action. This can lead to users being unable to take appropriate action, potentially compromising the safety and convenience of the vehicle.

[1790] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1791] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to a user the details of an abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the details of the abnormality, voice recognition means for analyzing the voice data, and means for notifying the user of the analysis results, thereby making it possible to quickly and accurately convey the meaning of the alarm sound to the user and provide information for the user to take appropriate action.

[1792] A "vehicle" is a mechanical device designed to travel on roads and powered by an engine or motor.

[1793] An "alarm sound" is an audio signal that is generated to notify the user of an abnormality in the vehicle or a situation that requires attention.

[1794] An "identification means" is a technique or device that analyzes a particular audio signal and determines what the audio means.

[1795] "Abnormality details" is information that indicates a state in which the vehicle is deviating from normal operation or a situation that requires attention.

[1796] A "user" is a person who drives or uses a vehicle.

[1797] "Explanatory means" refers to techniques and devices for communicating the details of the identified abnormality to the user in an easily understandable manner.

[1798] The "means for enabling a user to take action" refers to a technology or device that provides information or instructions to the user so that the user can take appropriate action against the identified abnormality.

[1799] "Speech recognition means" refers to technology or devices for analyzing voice data and converting it into text data.

[1800] "Means of notification" refers to the technology or device used to notify the user of the analysis results and abnormality details.

[1801] MODE FOR CARRYING OUT THE INVENTION

[1802] The present invention relates to a system for identifying an alarm sound emitted from a vehicle and notifying a user of the content of the abnormality. Specific embodiments of this system will be described below.

[1803] System Overview

[1804] This system uses voice recognition technology to identify warning sounds emitted by the vehicle. The voice recognition system can learn the various warning sounds emitted by the vehicle and identify the meaning of each warning sound. For example, it can identify the warning sound from the intelligent key or the warning sound from the collision warning function.

[1805] Hardware and software used

[1806] The server hosts the voice recognition system and receives voice data sent from the vehicle. The server analyzes the voice data using voice recognition software (e.g., a general voice recognition API). The analyzed voice data is compared with a database of pre-trained warning sounds.

[1807] When an alarm sound is generated, the terminal (e.g., the vehicle's on-board computer) transmits the sound data to the server. The terminal collects data from various sensors in the vehicle and records the timing and circumstances under which the alarm sound was generated.

[1808] To find out what the vehicle's alarm means, users will receive a notification via a smartphone app or on the vehicle's display, where they can view a detailed explanation of the alarm and how to respond.

[1809] Specific examples

[1810] When an intelligent key warning sound is emitted while the vehicle is in motion, the device sends the audio data to the server. The server uses a common voice recognition API to analyze the audio data and compares the analysis results with an alarm sound database. The server identifies the sound as an intelligent key warning sound and notifies the user's smartphone app. The user receives a warning message through the app stating, "The intelligent key is not in the vehicle."

[1811] Prompt Sentence Examples

[1812] "Design a system to identify warning sounds emitted by a vehicle. The system uses speech recognition technology to analyze the warning sounds and notify the user of what each sound means. As a specific use case, consider a system that can identify the warning sounds from an intelligent key and the warning sounds from a collision warning function."

[1813] In this way, by clarifying the roles of servers, terminals, and users and naming specific hardware and software, it is possible to understand the system's processing in detail.

[1814] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1815] Program processing flow

[1816] Step 1:

[1817] The terminal monitors various sensors inside the vehicle and detects when an alarm is generated. For example, if the intelligent key is not inside the vehicle, an alarm will sound.

[1818] Input: Data from sensors inside the vehicle

[1819] Output: Signal to detect the occurrence of an alarm sound

[1820] Specific operation: The terminal collects data in real time from sensors inside the vehicle and determines whether an alarm has been sounded.

[1821] Step 2:

[1822] The device will record the alarm sound using the built-in microphone, and the recorded audio data will be temporarily stored on the device.

[1823] Input: Alarm sound

[1824] Output: Recorded audio data

[1825] Specific operation: When the device detects an alarm sound, it activates the built-in microphone to record the sound and saves the data in memory.

[1826] Step 3:

[1827] The terminal transmits the recorded voice data to the server using the vehicle's communication module (e.g., LTE module).

[1828] Input: Recorded audio data

[1829] Output: Audio data sent to the server

[1830] Specific operation: The terminal compresses the recorded data and sends it to the server via the vehicle's communication module.

[1831] Step 4:

[1832] The server analyzes the received voice data using a voice recognition API and converts it into text data.

[1833] Input: Audio data sent to the server

[1834] Output: Parsed text data

[1835] Specific operation: The server calls the speech recognition API and converts the speech data into text data.

[1836] Step 5:

[1837] The server compares the converted text data with a pre-trained alarm sound database.

[1838] Input: Parsed text data

[1839] Output: Identified alarm type

[1840] Specific operation: The server compares the text data with the alarm sound database to identify matching alarm sound types.

[1841] Step 6:

[1842] The server notifies the user's smartphone app of the identification result using a push notification service (e.g., a general push notification service).

[1843] Input: identified alarm type

[1844] Output: Notification sent to the user's smartphone

[1845] Specific operation: The server formats the identification results and sends them to the user's smartphone via a push notification service.

[1846] Step 7:

[1847] The user receives a notification via a smartphone app, for example, a message saying "Intelligent key not in vehicle."

[1848] Input: Notification sent to the user's smartphone

[1849] Output: Notification message received by the user

[1850] Specific operation: The user checks the notification on their smartphone and takes action if necessary, such as returning the intelligent key to the vehicle.

[1851] In this way, by explaining each processing step in detail and clearly indicating the inputs and outputs, the processing flow of the system can be understood in detail.

[1852] (Application example 1)

[1853] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1854] Conventional vehicle alarm sound identification systems are required not only to identify alarm sounds but also to explain the nature of the abnormality to the user and provide countermeasures. However, these systems lack the functionality to notify the user of the type and meaning of the alarm sound in real time and to suggest necessary countermeasures or actions. This makes it difficult for the user to respond quickly and appropriately.

[1855] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1856] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to a user the details of an abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the details of the abnormality, means for notifying the user of the type and meaning of the identified alarm sound, and means for suggesting necessary countermeasures or actions, thereby enabling the user to understand the type and meaning of the alarm sound in real time and to respond promptly and appropriately.

[1857] A "vehicle" is a mechanical device designed to travel on roads and powered by an engine or motor.

[1858] An "alarm sound" is an audio signal emitted by a vehicle and is used to notify of abnormalities or dangers.

[1859] An "identification means" is a device or system that analyzes a specific audio signal and determines its type and meaning.

[1860] "Abnormality details" refers to the specific circumstances or conditions of a problem or malfunction that prevents the vehicle from operating normally.

[1861] The "means for explaining to the user" refers to a device or system for communicating the identified abnormality to the user in an easy-to-understand manner.

[1862] The "means for enabling a user to take action" refers to a device or system that provides information and instructions for the user to take appropriate action against the abnormality.

[1863] The "notification means" is a device or system for informing the user of the type of identified alarm sound and its meaning.

[1864] "Proposed measures" are devices or systems that present specific measures or actions that users should take.

[1865] The system for implementing this invention identifies warning sounds emitted from a vehicle, explains the nature of the abnormality to the user, and provides countermeasures. The system uses a microphone installed in the vehicle, a smartphone, the voice recognition library "SpeechRecognition," and the machine learning library "TensorFlow."

[1866] System configuration

[1867] 1. Hardware

[1868] Vehicle-mounted microphones: collect warning sounds emitted by the vehicle.

[1869] Smartphone: A device for processing voice data and notifying the user.

[1870] 2. Software

[1871] Speech Recognition library: Analyzes collected voice data and identifies alarm sounds.

[1872] TensorFlow machine learning library: Uses an alarm sound identification model to determine the type of alarm sound.

[1873] Processing flow

[1874] 1. Acquisition of audio data: A microphone installed in the vehicle collects the alarm sound and transmits it to a smartphone.

[1875] 2. Preprocessing of audio data: Using the speech recognition library "SpeechRecognition" on the smartphone, the audio data is analyzed and the features of the alarm sound are extracted.

[1876] 3. Alarm sound identification: Using the machine learning library "TensorFlow," audio data is input into a pre-trained alarm sound identification model to identify the type of alarm sound.

[1877] 4. Notification and Suggestion: Inform the user of the identified alarm type and its meaning, and suggest any necessary countermeasures or actions.

[1878] Specific examples

[1879] For example, if the vehicle emits a collision warning sound, the system will identify the sound and notify the smartphone, "There is a risk of a collision. Please apply the brakes."

[1880] Prompt Sentence Examples

[1881] "Develop an application that identifies warning sounds emitted by a vehicle and notifies the user of their type and meaning. The libraries used are SpeechRecognition and TensorFlow. Types of warning sounds include intelligent key warning sounds, collision warning sounds, and engine abnormality warning sounds."

[1882] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1883] Step 1:

[1884] A microphone installed in the vehicle collects the alarm sound and transmits it to a smartphone.

[1885] Input: Vehicle alarm sound

[1886] Output: Audio data sent to your smartphone

[1887] How it works: The vehicle's microphone detects an alarm sound and transmits the audio data in real time to a smartphone, which receives the audio data and proceeds to the next processing step.

[1888] Step 2:

[1889] The voice recognition library "SpeechRecognition" is used on the smartphone to analyze the voice data and extract the characteristics of the alarm sound.

[1890] Input: Voice data sent to smartphone

[1891] Output: Extracted alarm sound feature data

[1892] Specific operation: The smartphone inputs the received voice data into the "SpeechRecognition" library and analyzes the voice data. As a result of the analysis, feature data of the alarm sound is extracted. This feature data is used in the next step.

[1893] Step 3:

[1894] Using the machine learning library "TensorFlow," audio data is input into a pre-trained alarm sound identification model to identify the type of alarm sound.

[1895] Input: Extracted alarm sound feature data

[1896] Output: Identified alarm type

[1897] How it works: The smartphone inputs the extracted feature data into the TensorFlow library and uses the alarm sound classification model to identify the type of alarm sound.

[1898] Step 4:

[1899] The system notifies the user of the type of alarm sound identified and its meaning, and suggests necessary countermeasures or actions.

[1900] Input: Identified alarm type

[1901] Output: The type of alarm that will be notified to the user, its meaning, and any suggested countermeasures or actions.

[1902] Specific behavior: The smartphone notifies the user of the meaning of the identified warning sound based on the type of sound. For example, if a "collision warning sound" is identified, the smartphone will notify the user, "There is a risk of a collision. Please apply the brakes." It will also suggest necessary countermeasures and actions.

[1903] Example 2

[1904] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1905] Conventional vehicle alarm systems have the problem that it is difficult for users to quickly and accurately understand the nature of the abnormality when an alarm sounds. In particular, when an alarm sounds while driving, users must refer to the manual to understand the meaning of the alarm, which can hinder safe driving. In addition, because users cannot immediately know the appropriate way to deal with the abnormality, there is a possibility that they will be slow to respond in situations that require a prompt response.

[1906] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1907] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for acquiring details of an abnormality corresponding to the identified alarm sound as text data, voice synthesis means for converting the acquired text data into voice data, means for notifying the user of the generated voice data, and means for enabling the user to deal with the details of the abnormality. Thus, when an alarm sound is emitted, the user can immediately understand the details of the abnormality through voice and can quickly learn an appropriate way to deal with it.

[1908] An "alarm sound" is an audio signal emitted by a vehicle to notify of an abnormality or warning.

[1909] The "means for identifying" refers to a device or software that analyzes the received alarm sound and identifies its type and content.

[1910] "Anomaly details" is information that indicates the specific problem or situation corresponding to the identified alarm sound.

[1911] "Text data" is data that expresses the content of an abnormality as text information.

[1912] "Voice data" refers to a voice signal generated from text data using voice synthesis technology.

[1913] "Speech synthesis means" refers to a device or software for converting text data into voice data.

[1914] The "notification means" refers to a device or software for notifying the user of the generated voice data.

[1915] "Means for enabling a user to take action" refers to devices or software that provide information and methods for the user to take appropriate action against the abnormality.

[1916] MODE FOR CARRYING OUT THE INVENTION

[1917] The present invention relates to a system for identifying an alarm sound emitted from a vehicle and notifying a user of the content of the abnormality by voice. Specific embodiments of this system will be described below.

[1918] Hardware and software used

[1919] Hardware: Servers, devices (smartphones, tablets, etc.)

[1920] Software: Speech synthesis system (e.g., Google Text-to-Speech API), alarm sound identification system (e.g., speech recognition model using TensorFlow)

[1921] System configuration

[1922] 1. Receiving an alarm sound

[1923] The device records the alarm sounds generated inside the vehicle in real time via a microphone.

[1924] The terminal transmits the recorded voice data to the server.

[1925] 2. Alarm sound identification

[1926] The server inputs the received audio data into the warning sound identification system.

[1927] The server uses an alarm sound identification system to analyze the audio data and identify the type of alarm sound.

[1928] 3. Obtaining the details of the abnormality

[1929] The server retrieves the abnormality content corresponding to the identified alarm sound from the database.

[1930] For example, as the abnormality content corresponding to "engine abnormality," text data such as "There is an abnormality in the engine. Stop the vehicle immediately and contact a specialized repair company" is acquired.

[1931] 4. Generating Audio Data

[1932] The server inputs the acquired text data into a speech synthesis system to generate speech data.

[1933] For example, using the Google Text-to-Speech API, we can generate voice data that says, "There is a problem with the engine. Stop the vehicle immediately and contact a professional repair shop."

[1934] 5. Notice to Users

[1935] The server transmits the generated voice data to the terminal.

[1936] The terminal reproduces the received audio data and notifies the user of the details of the abnormality.

[1937] Specific examples

[1938] For example, if an alarm sounds indicating an engine malfunction while the user is driving a car, the system operates as follows.

[1939] 1. The device records the alarm sound and sends the audio data to the server.

[1940] 2. The server uses the alarm sound identification system to identify the sound as "engine abnormality."

[1941] 3. The server retrieves text data from the database stating, "There is an engine malfunction. Stop the vehicle immediately and contact a professional repair company."

[1942] 4. The server converts the text data into voice data using a voice synthesis system.

[1943] 5. The device plays the generated voice data and informs the user of the abnormality.

[1944] Prompt Sentence Examples

[1945] Input: "Please explain what to do if an engine malfunction alarm sounds."

[1946] Output: "There is an engine malfunction. Stop the vehicle immediately and contact a professional repair service."

[1947] The system aims to ensure user safety by quickly identifying alarm sounds and providing users with audible instructions on how to respond appropriately.

[1948] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1949] Step 1:

[1950] Receiving alarm sounds

[1951] The device records the alarm sounds generated inside the vehicle in real time via a microphone.

[1952] Input: Alarm sound generated inside a car

[1953] Output: Recorded audio data

[1954] What it does: The device's microphone captures the alarm sound and uploads the audio data to a server via the Internet.

[1955] Step 2:

[1956] Alarm sound identification

[1957] The server inputs the received audio data into the warning sound identification system.

[1958] Input: Audio data sent from the device

[1959] Output: Type of alarm identified (e.g. engine abnormality, brake abnormality, etc.)

[1960] Specific operation: The server analyzes the audio data and identifies the type of alarm sound using an alarm sound identification system (e.g., a speech recognition model using TensorFlow).

[1961] Step 3:

[1962] Acquisition of abnormality details

[1963] The server retrieves the abnormality content corresponding to the identified alarm sound from the database.

[1964] Input: Identified alarm type

[1965] Output: Text data about the problem (e.g., "There is a problem with the engine. Stop the vehicle immediately and contact a professional repair shop.")

[1966] Specific operation: The server executes a database query to obtain the corresponding text data of the anomaly.

[1967] Step 4:

[1968] Generating audio data

[1969] The server inputs the acquired text data into a speech synthesis system to generate speech data.

[1970] Input: Text data of the anomaly

[1971] Output: Generated audio data

[1972] Specific operation: The server sends text data to a speech synthesis API (e.g., Google Text-to-Speech API) and receives the generated speech data.

[1973] Step 5:

[1974] User Notification

[1975] The server transmits the generated voice data to the terminal.

[1976] The terminal reproduces the received audio data and notifies the user of the details of the abnormality.

[1977] Input: Generated audio data

[1978] Output: A voice message to be announced to the user

[1979] Specific operation: The device plays audio data over the speaker, telling the user, "There is an engine malfunction. Please stop the vehicle immediately and contact a professional repair company."

[1980] (Application example 2)

[1981] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1982] In conventional vehicles, when an alarm sound is emitted, it is difficult for the user to immediately understand the meaning and take appropriate action. In particular, in autonomous vehicles, users are required to quickly understand the meaning of the alarm sound and take appropriate action, but current systems do not provide this support sufficiently. This may result in a decrease in user safety and vehicle operating efficiency.

[1983] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1984] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to the user the details of the abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the details of the abnormality, and means for audibly communicating the details of the abnormality to the user using a voice synthesis system, thereby enabling the user to immediately understand the meaning of the alarm sound and take appropriate action.

[1985] A "vehicle" is a mechanical device for moving on land, powered by an engine or motor.

[1986] An "alarm sound" is an audio signal that is generated to warn of an abnormality or danger in the vehicle.

[1987] "Discrimination" means distinguishing a particular sound or signal from others and recognizing its content.

[1988] "Abnormality details" is information that indicates a state or problem that deviates from the normal operation of the vehicle.

[1989] "User" refers to a person who operates or uses a vehicle.

[1990] "Explanation" is the act of conveying specific information or content to others so that they can understand it.

[1991] "Response" means taking appropriate action in response to problems or abnormalities that have occurred.

[1992] A "speech synthesis system" is a technology or device that converts text data into speech.

[1993] An "Internet search tool" is software or a service that allows you to search for information on the Internet and obtain the information you need.

[1994] The following system is constructed as an embodiment of this invention. First, a vehicle is equipped with a voice recognition system for identifying warning sounds. This voice recognition system identifies warning sounds generated inside the vehicle in real time and acquires their contents as text data. The speech_recognition library is used for voice recognition.

[1995] Next, a database is prepared to acquire the details of the abnormality corresponding to the identified alarm sound. This database stores the details of the abnormality corresponding to each alarm sound in text format. For example, for an alarm sound that says "engine abnormality," the following abnormality details are stored: "There is an abnormality in the engine. Stop the vehicle immediately and contact a professional repair shop."

[1996] The acquired abnormality information is converted into voice using a voice synthesis system. The gTTS library is used for voice synthesis. This voice synthesis system converts text data into voice data and conveys it to the user. This allows the user to immediately understand the meaning of the alarm sound and take appropriate action.

[1997] As a specific example, consider the case where an alarm sounds indicating an engine malfunction. In this case, the speech recognition system receives the text "engine malfunction" and retrieves the corresponding malfunction details from the database. The speech synthesis system then generates a voice message to the user saying, "There is an engine malfunction. Stop the vehicle immediately and contact a professional repair shop."

[1998] Examples of prompts to input to a generative AI model might include:

[1999] I would like to develop a system that can identify alarm sounds and inform the user of the nature of the abnormality by voice. Specifically, when an alarm indicating an engine abnormality sounds, a voice message should be sent saying, "There is an abnormality in the engine. Please stop the vehicle immediately and contact a professional repair shop." Please provide the Python code to achieve this.

[2000] This system allows users to quickly understand the meaning of the warning sound and take appropriate action, thereby improving vehicle safety and operational efficiency.

[2001] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2002] Step 1:

[2003] The server uses a voice recognition system to identify warning sounds generated inside the vehicle in real time. Specifically, it uses a voice recognition system (speech_recognition library) to convert the warning sounds into text data. The input is the voice data of the warning sounds, and the output is the text data of the identified warning sounds.

[2004] Step 2:

[2005] The server retrieves the corresponding abnormality details from the database based on the text data of the identified alarm sound. Specifically, it searches the database that stores the abnormality details corresponding to the alarm sound and retrieves the text data of the corresponding abnormality details. The input is the text data of the identified alarm sound, and the output is the text data of the corresponding abnormality details.

[2006] Step 3:

[2007] The server converts the acquired text data of the anomaly into voice data using a voice synthesis system. Specifically, it uses a voice synthesis system (gTTS library) to convert the text data into voice data. The input is the text data of the anomaly, and the output is voice data.

[2008] Step 4:

[2009] The terminal transmits the generated voice data to the user. Specifically, the voice data is played through a speaker to inform the user of the abnormality by voice. The input is the voice data, and the output is the voice information that the user hears.

[2010] Step 5:

[2011] The user takes appropriate action based on the details of the abnormality communicated by voice. Specifically, the user follows the instructions communicated by voice and takes appropriate action, such as stopping the vehicle or contacting a repair shop. The input is voice information, and the output is the user's response action.

[2012] Example 3

[2013] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2014] In conventional vehicle systems, when an alarm sound is emitted, it is difficult for the user to quickly and accurately understand the nature of the abnormality and find an appropriate way to deal with it. In particular, when the abnormality is complex or the user does not have technical knowledge, there is a high possibility that appropriate action will be delayed. To solve this problem, a system is needed that can analyze the alarm sound, identify the nature of the abnormality, and quickly provide a way to deal with it.

[2015] The identification process by the identification processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes means for receiving an alarm sound emitted from a vehicle, means for analyzing the received alarm sound to identify the nature of the abnormality, means for searching for a solution from the Internet based on the identified nature of the abnormality, and means for providing the searched solution to the user. This allows the user, when an alarm sound is emitted, to quickly and accurately understand the nature of the abnormality and immediately obtain an appropriate solution.

[2016] A "vehicle" is a mechanical device that uses an engine or motor as a power source for movement, and primarily refers to automobiles that run on roads.

[2017] The "alarm sound" is an audio signal that is generated to alert the user to an abnormality or danger in the vehicle, and is intended to draw the user's attention.

[2018] "Means for receiving" refers to devices or systems for receiving data transmitted from a vehicle, specifically using communication technologies such as Bluetooth or Wi-Fi.

[2019] "Means of analysis" refers to software or algorithms used to process and understand the received data.

[2020] "Abnormality" refers to the specific condition of a problem or malfunction that prevents the vehicle from operating normally.

[2021] "Means of identification" refers to methods or techniques for identifying specific anomalies based on the analysis results.

[2022] "Internet solution search" refers to technology or software that uses Internet resources to find solutions to specific anomalies.

[2023] "Means of providing" refers to the method or device for informing the user of the searched solution, specifically using a display device such as a smartphone or an in-car display.

[2024] MODE FOR CARRYING OUT THE INVENTION

[2025] The present invention is a system for analyzing an alarm sound emitted from a vehicle, identifying the nature of the abnormality, and providing a user with a method for dealing with the abnormality. Specific embodiments of this system will be described below.

[2026] System configuration

[2027] This system mainly uses the following hardware and software:

[2028] Server: The central component that receives, analyzes, searches, and serves data.

[2029] Terminal: A device used by a user, including a smartphone or an in-car display.

[2030] Speech recognition software: Analyzes alarm sounds using the Google Cloud Speech-to-Text API, among others.

[2031] Search engine API: Search for solutions on the Internet using Google Search API, etc.

[2032] Program processing

[2033] The server receives the alarm sound data sent from the user's vehicle. The received data is analyzed using voice recognition software. Based on the analysis results, the server identifies the nature of the abnormality. Based on the identified abnormality, the server uses a search engine API to search for related information from the Internet. The server extracts the most appropriate solution from the search results and provides it to the user.

[2034] Specific examples

[2035] For example, if a user's vehicle emits an alarm sound indicating low tire pressure, the server analyzes the alarm sound and identifies the abnormality as "Your tire pressure is low." The server then uses the Google Search API to search for "What to do if your tire pressure is low" and provides the user with the solution of "Please refill your tires with air at the nearest gas station or vehicle repair shop."

[2036] Prompt Sentence Examples

[2037] Analyze the alarm sound sent from the user's vehicle and identify the abnormality. Based on the identified abnormality, search the Internet for a solution and provide it to the user.

[2038] This system allows the user to quickly deal with any abnormalities in the vehicle. The server performs all processes from analyzing the alarm sound to providing a solution, so the user does not need to perform any complicated operations. This improves the safety of the vehicle and the convenience of the user. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[2039] Step 1: Receiving the alarm sound

[2040] Input: Alarm sound data sent from the user's vehicle

[2041] Specific operation: When the user's vehicle emits an alarm, the audio data is captured by the vehicle's sensors. The device then transmits the captured audio data to the server.

[2042] Output: Alarm sound data sent to the server

[2043] Step 2: Analyze the alarm sound

[2044] Input: Alarm sound data sent to the server

[2045] Specific operation: The server sends the received alarm sound data to speech recognition software (e.g., Google Cloud Speech-to-Text API) and converts it into text data. The server then analyzes the converted text data to identify the content of the alarm sound.

[2046] Output: Parsed text data and alarm sound content

[2047] Step 3: Identifying the anomaly

[2048] Input: Parsed text data and alarm sound content

[2049] Specific operation: The server identifies the abnormality based on the analysis results. For example, it identifies the abnormality as "tire pressure is low."

[2050] Output: Identified anomalies

[2051] Step 4: Internet Search

[2052] Input: Identified anomaly

[2053] What it does: The server uses the Google Search API to search for "What to do when tire pressure is low." The server then extracts the most relevant information from the search results.

[2054] Output: Search results and extracted solutions

[2055] Step 5: Provide solutions

[2056] Input: Search results and extracted solutions

[2057] Specific operation: The server sends the extracted solutions to the user's device. The user checks the solutions displayed on the device and takes appropriate action.

[2058] Output: Actions displayed on the user's device

[2059] In this way, the user can quickly respond to vehicle abnormalities. The server handles everything from analyzing the alarm sound to providing countermeasures, so the user does not need to perform complicated operations. This improves vehicle safety and user convenience.

[2060] (Application example 3)

[2061] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2062] Conventional vehicle systems have a problem in that when an alarm sound is issued, it is difficult for the user to understand the nature of the abnormality and quickly find an appropriate solution. In particular, when the abnormality is complex or the user does not have technical knowledge, appropriate action is often delayed. This leads to issues such as reduced vehicle safety and user convenience.

[2063] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[2064] In this invention, the server includes means for identifying an alarm sound emitted from a vehicle, means for explaining to a user the nature of an abnormality corresponding to the identified alarm sound, means for enabling the user to deal with the abnormality, means for searching information relating to the abnormality from the Internet, and means for providing the search results to the user. This allows the user to quickly understand the nature of the abnormality when an alarm sound is emitted and to search for and obtain an appropriate solution from the Internet. Furthermore, by notifying the user of the search results by voice, the user can safely receive information while driving.

[2065] A "vehicle" is a mechanical device for moving on roads, powered by an engine or motor.

[2066] An "alarm sound" is an audio signal that is generated to warn of an abnormality or danger in the vehicle.

[2067] An "identification means" is a device or software that detects a particular sound or signal and determines what it means.

[2068] "Abnormality details" refer to the specific condition or cause of a problem or malfunction that prevents the vehicle from operating normally.

[2069] A "user" is a person who operates a vehicle or receives information about the vehicle.

[2070] "Explanatory means" refers to a method or device for helping the user understand the nature of the abnormality.

[2071] The "means for enabling a user to take action" refers to a method or device that supports the user in taking appropriate action in response to the abnormality.

[2072] The "Internet" is an information and communications network that interconnects computer networks around the world.

[2073] A "search means" is a method or device for locating specific information on the Internet.

[2074] "Means for providing" refers to the method or device for conveying search results and information to users.

[2075] "Audio notification" is a method of conveying information using audio.

[2076] The system for carrying out the present invention includes a sensor mounted on a vehicle, a server with an Internet connection function, and a terminal for providing information to a user. The specific configuration and operation of the system will be described below.

[2077] System configuration

[2078] 1. Vehicle sensors

[2079] Vehicles are equipped with various sensors to detect tire pressure, engine temperature, collision warnings, etc. These sensors emit an alarm when they detect an abnormality.

[2080] 2. Server

[2081] The server includes software to identify the warning sounds sent from the vehicle and identify the nature of the abnormality. Specifically, it uses programming languages ​​such as Python to identify the warning sounds and the nature of the abnormality. The server also has the function of searching for information on the abnormality using internet search tools.

[2082] 3. User Device

[2083] The user terminal is a device such as a smartphone or tablet that receives information sent from the server. The user terminal is equipped with a voice notification function that notifies the user of the nature of the abnormality and how to deal with it.

[2084] System Operation

[2085] 1. Abnormality detection and alarm generation

[2086] When the vehicle's sensors detect an abnormality, an alarm sounds, which is transmitted to the server via the vehicle's microphone.

[2087] 2. Identifying alarm sounds and identifying abnormalities

[2088] The server identifies the received alarm sound and determines the abnormality. For example, if the tire pressure is low, the server determines the abnormality as "Tire pressure is low."

[2089] 3. Internet search and information provision

[2090] The server searches the Internet based on the identified anomaly to obtain information on how to deal with the problem, and then transmits the obtained information to the user terminal.

[2091] 4. Audio Notifications

[2092] The user terminal notifies the user of the received information by voice, allowing the user to receive information safely while driving.

[2093] Specific examples

[2094] For example, if a vehicle sensor detects an abnormality such as low tire pressure, the system will operate as follows:

[2095] 1. The sensor detects an abnormal sound and sends it to the server.

[2096] 2. The server identifies the abnormality "Tire pressure is low."

[2097] 3. The server performs an internet search and obtains the solution, "Please refill the air at the nearest gas station or vehicle repair shop."

[2098] 4. The user device will give an audio notification and tell the user how to deal with the problem.

[2099] Prompt Sentence Examples

[2100] "Your tire pressure is low. Please refill it at the nearest gas station or vehicle repair shop. Search the internet for solutions."

[2101] In this way, the user can deal with the abnormality promptly and appropriately.

[2102] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[2103] Step 1:

[2104] The vehicle's sensors detect an abnormality and sound an alarm.

[2105] Input: Abnormality detection signal from vehicle sensor

[2106] Output: Alarm sound

[2107] Specific operation: If the tire pressure drops, the tire pressure sensor will detect the abnormality and sound an alarm.

[2108] Step 2:

[2109] The warning sound is transmitted to the server through a microphone in the vehicle.

[2110] Input: Alarm

[2111] Output: Audio data to the server

[2112] Specific operation: A microphone inside the vehicle captures the warning sound and sends the audio data to a server.

[2113] Step 3:

[2114] The server identifies the received alarm sound and identifies the nature of the abnormality.

[2115] Input: Audio data

[2116] Output: Abnormality details

[2117] Specific operation: The server uses voice recognition software to analyze the alarm sound and identify the abnormality: "Tire pressure is low."

[2118] Step 4:

[2119] The server performs an internet search based on the identified abnormality and obtains information on how to deal with the problem.

[2120] Input: Abnormality details

[2121] Output: Information on what to do

[2122] Specific operation: The server uses an internet search engine to search for and retrieve the solution, "Your tire pressure is low. Please refill the tire with air at the nearest gas station or vehicle repair shop."

[2123] Step 5:

[2124] The server transmits the acquired information to the user terminal.

[2125] Input: Information on how to proceed

[2126] Output: Sending information to the user's terminal

[2127] Specific operation: The server sends the acquired information on how to deal with the problem to the user terminal.

[2128] Step 6:

[2129] The user terminal notifies the user of the received information by voice.

[2130] Input: Information on how to proceed

[2131] Output: Audio notification

[2132] Specific operation: The user device uses speech synthesis software to notify the user by voice the message, "Your tire pressure is low. Please refill the tire with air at the nearest gas station or vehicle repair shop."

[2133] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2134] "Example 1"

[2135] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system identifies an alarm sound emitted from a vehicle and explains to the user the nature of the abnormality corresponding to the identified alarm sound. Furthermore, the system allows the user to deal with the abnormality. The system further includes an emotion engine that recognizes the user's emotion. This emotion engine recognizes the user's emotion from, for example, the user's tone of voice, facial expression, choice of words, etc.

[2136] "Example 2"

[2137] The emotion engine adjusts the way it explains the abnormality depending on the user's emotions. For example, if the user is panicking, the emotion engine will explain the abnormality in a concise and clear manner. On the other hand, if the user is calm, the emotion engine will explain the abn...

Claims

1. A means for identifying an alarm sound generated by a vehicle; A means for explaining to a user the content of an abnormality corresponding to the identified alarm sound by using a prompt sentence that instructs the user to explain the content of an abnormality corresponding to the identified alarm sound and a generation AI model; A means for explaining to the user how to deal with the abnormality using a prompt sentence that instructs the user to explain the abnormality to the user and the generative AI model; A system including:

2. 2. The system of claim 1, wherein the warning sound comprises a warning sound from an intelligent key of the vehicle or a warning sound generated from a collision warning function.

3. means for recognizing an emotional state of the user using an emotion engine; The system according to claim 1, wherein the means for explaining the countermeasure to the user explains to the user how to deal with the abnormality using a prompt sentence that instructs the means to explain to the user how to deal with the abnormality based on the recognized emotional state of the user and the generative AI model.

Images