system

Abstract

The system according to this embodiment aims to enable hearing-impaired individuals to respond appropriately to the sirens of emergency vehicles while driving. [Solution] The system according to the embodiment comprises a detection unit, an analysis unit, a conversion unit, and a display unit. The detection unit detects the siren. The analysis unit analyzes the siren detected by the detection unit. The conversion unit converts the siren analyzed by the analysis unit into sign language. The display unit displays the sign language converted by the conversion unit.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, there is a problem that it is difficult for a hearing-impaired person to appropriately respond to the sirens of emergency vehicles while driving.

[0005] The system according to the embodiment aims to enable a hearing-impaired person to appropriately respond to the sirens of emergency vehicles while driving.

Means for Solving the Problems

[0006] The system according to the embodiment includes a detection unit, an analysis unit, a conversion unit, and a display unit. The detection unit detects a siren. The analysis unit analyzes the siren detected by the detection unit. The conversion unit converts the siren analyzed by the analysis unit into sign language. The display unit displays the sign language converted by the conversion unit. [Effects of the Invention]

[0007] The system according to this embodiment can enable a person with a hearing impairment to respond appropriately to the siren of an emergency vehicle while driving. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit).

[0012] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0013] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] [First Embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] <​​​​The reception device 38 is equipped with a touch panel 38A and a microphone 38B, and accepts user input. The touch panel 38A accepts user input via touch by detecting contact with an object (e.g., a pen or finger). The microphone 38B accepts user input via voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 (see Figure 2) acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user by outputting the data in a form perceptible to the user (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0026] In the smart device 14, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0027] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example of form 1) The driving assistance system according to an embodiment of the present invention is a system that supports the driving of hearing-impaired persons by linking AI and automotive technology. In this driving assistance system, the AI ​​detects the sirens of ambulances and police vehicles, analyzes the audio waveform of the detected sirens, and converts the content into sign language. The converted sign language is displayed on a head-up display, providing visual instructions to the driver. For example, when turning right, the sign language interpreter displays sign language indicating the right direction, prompting the driver to turn right. This system allows hearing-impaired persons to visually recognize the approach of emergency vehicles and take appropriate action. Furthermore, the driving guidance provided by the sign language interpreter makes driving safer and smoother for hearing-impaired persons. In addition, this system contributes to increasing awareness of the hearing-impaired person mark. The instructions from the sign language interpreter displayed on the head-up display are provided in a way that is easy for hearing drivers to understand visually, thus promoting understanding and consideration for hearing-impaired drivers. In this way, by integrating AI and automotive technology to solve potential problems for hearing-impaired persons, the driving environment for hearing-impaired persons is greatly improved. This allows the driver assistance system to visually recognize the approach of an emergency vehicle and take appropriate action.

[0029] The driving assistance system according to this embodiment comprises a detection unit, an analysis unit, a conversion unit, and a display unit. The detection unit detects sirens. Sirens include, but are not limited to, sirens of ambulances, fire trucks, and police vehicles. The detection unit detects sirens using, for example, speech recognition technology. The detection unit can also analyze ambient sounds to identify siren sounds. For example, the detection unit filters out ambient traffic noise and wind noise to detect siren sounds. The analysis unit analyzes the siren detected by the detection unit. The analysis unit identifies the type of siren by, for example, analyzing the frequency and volume of the siren. The analysis unit can also identify the source of the siren. For example, the analysis unit identifies the source of the siren based on the arrival time difference of the siren's sound waves. The conversion unit converts the siren analyzed by the analysis unit into sign language. The conversion unit converts the content of the siren into sign language using, for example, a sign language dictionary. The conversion unit can also display the sign language movements as animation. For example, the conversion unit displays sign language movements as 3D animations, providing visual instructions to the driver. The display unit displays the sign language converted by the conversion unit. The display unit displays the sign language, for example, on a head-up display. The display unit can also adjust the display position and size. For example, the display unit adjusts the display position of the sign language to match the driver's line of sight. This allows the driver assistance system to enable hearing-impaired individuals to visually recognize approaching emergency vehicles and take appropriate action. Some or all of the above processing in the display unit may be performed using AI, for example, or without AI. For example, the display unit can optimize the display position and size of the sign language using an AI model.

[0030] The detection unit detects sirens. Sirens include, but are not limited to, those of ambulances, fire trucks, and police vehicles. The detection unit detects sirens using, for example, speech recognition technology. Specifically, the detection unit is equipped with a high-sensitivity microphone to collect ambient sounds in real time. The collected audio data is processed using a noise reduction algorithm to remove unwanted background noise and highlight the characteristics of the siren sound. Furthermore, speech recognition technology is used to analyze the frequency spectrum and volume changes of the siren sound and detect specific patterns. For example, ambulance sirens change periodically in a specific frequency band, so they can be detected based on this characteristic. The detection unit can also identify siren sounds by analyzing ambient environmental sounds. For example, the detection unit filters out ambient traffic noise and wind noise to detect siren sounds. This involves using machine learning algorithms to learn the characteristics of different ambient sounds and identify siren sounds with high accuracy. As a result, the detection unit can accurately detect siren sounds even in complex environments and proceed to the next processing step.

[0031] The analysis unit analyzes the siren detected by the detection unit. For example, the analysis unit analyzes the frequency and volume of the siren to identify the type of siren. Specifically, the analysis unit analyzes the frequency spectrum of the siren sound in detail and identifies specific frequency bands and volume change patterns. This makes it possible to distinguish between different siren sounds, such as those of ambulances, fire trucks, and police vehicles. The analysis unit can also identify the source of the siren. For example, the analysis unit identifies the source of the siren based on the difference in arrival times of the siren's sound waves. Specifically, it measures the arrival time of sound waves using multiple microphones and calculates the position of the sound source using triangulation. This technology allows the analysis unit to accurately identify the direction of the siren source and provide appropriate information to the driver. Furthermore, the analysis unit can learn siren sound patterns by utilizing past data and statistical information to perform more accurate analysis. For example, based on past siren sound data, it can analyze the frequency and patterns of siren sounds in specific areas and time periods to help with future predictions. This allows the analysis unit to handle not only real-time analysis but also long-term risk assessment and trend analysis, thereby improving the reliability and safety of the entire system.

[0032] The conversion unit converts the siren analyzed by the analysis unit into sign language. The conversion unit converts the content of the siren into sign language using, for example, a sign language dictionary. Specifically, the conversion unit selects sign language actions according to the type and source of the siren and generates an animation to visually communicate it to the driver. The sign language dictionary has sign language actions registered that correspond to various siren sounds, and the conversion unit refers to this to select the appropriate sign language. The conversion unit can also display the sign language movements as animations. For example, the conversion unit can display the sign language movements as 3D animations to provide visual instructions to the driver. The 3D animations are designed to be intuitively understandable to the driver and can display the sign language actions in real time. Furthermore, the conversion unit can adjust the sign language actions to match the driver's gaze and attention range. For example, if the driver is looking straight ahead, the sign language animation can be displayed on the head-up display to provide information without diverting the driver's gaze. This allows the conversion unit to provide information that enables hearing-impaired individuals to visually recognize the approach of an emergency vehicle and take appropriate action.

[0033] The display unit displays the sign language converted by the conversion unit. The display unit displays the sign language, for example, on a head-up display. Specifically, the display unit adjusts the display position of the sign language to match the driver's line of sight, allowing the driver to check the information without moving their eyes significantly. The display unit can also adjust the display position and size. For example, the display unit adjusts the display position of the sign language to match the driver's line of sight. This allows the driver assistance system to enable hearing-impaired individuals to visually recognize the approach of emergency vehicles and take appropriate action. Some or all of the above processing in the display unit may be performed using AI, for example, or without AI. For example, the display unit can optimize the display position and size of the sign language using an AI model. Specifically, the AI ​​model learns the driver's eye-tracking data and past usage history to calculate the optimal display position and size in real time. This allows the display unit to display the sign language in the most visible position for the driver, improving the efficiency of information transmission. Furthermore, the display unit can collect driver feedback and continuously improve the accuracy and effectiveness of the displayed content. For example, by providing feedback from the driver regarding the position and size of the sign language display, the AI ​​model optimizes the display settings based on that information. This allows the display unit to provide information to the driver quickly and reliably, improving the reliability and safety of the driver assistance system.

[0034] The driver assistance system includes an animation unit that displays sign language movements as animations. The animation unit can, for example, display sign language movements as 2D animations. For example, the animation unit can display sign language movements as simple 2D animations to provide visual instructions to the driver. The animation unit can also display sign language movements as 3D animations. For example, the animation unit can display sign language movements as realistic 3D animations to provide visual instructions to the driver. The animation unit can also display sign language movements as animated characters. For example, the animation unit can display sign language movements as animated characters to provide visual instructions to the driver. This makes sign language movements visually easy to understand and helps the driver to comprehend them. Some or all of the above processing in the animation unit may be performed using AI, for example, or without AI. For example, the animation unit can input sign language movements into a generation AI and have the generation AI perform the animation generation.

[0035] The driver assistance system includes an adjustment unit that adjusts the display position and size, enabling it to provide the displayed content in an appropriate position and size for the driver. For example, the adjustment unit can adjust the display position to match the driver's line of sight. For example, the adjustment unit can optimize the display position of sign language based on the driver's eye-tracking data. The adjustment unit can also adjust the display size to match the driver's visibility. For example, the adjustment unit can adjust the display size of sign language to match the driver's eyesight and the display resolution. The adjustment unit can also adjust the layout of the displayed content. For example, the adjustment unit can position and size the sign language display to reduce the driver's visual burden. This enables the display content to be provided in the optimal position and size for the driver. Some or all of the above processing in the adjustment unit may be performed using AI, for example, or without AI. For example, the adjustment unit can input the driver's eye-tracking data into a generating AI and have the generating AI perform the optimization of the display position and size.

[0036] The driver assistance system includes an instruction unit that displays specific instructions. The instruction unit can, for example, display instructions for the direction of travel. For example, the instruction unit can display instructions for turning right or left in sign language, providing visual instructions to the driver. The instruction unit can also display instructions for speed. For example, the instruction unit can display instructions for slowing down or stopping in sign language, providing visual instructions to the driver. The instruction unit can also display instructions for emergencies. For example, the instruction unit can display the approach of an emergency vehicle in sign language, providing visual instructions to the driver. In this way, by providing specific driving instructions to the driver, safe driving can be supported. Some or all of the above processing in the instruction unit may be performed using AI, for example, or without AI. For example, the instruction unit can input instructions for the direction of travel and speed into a generating AI, and have the generating AI perform the generation of sign language.

[0037] The display unit can display sign language on a head-up display. For example, the display unit projects sign language onto the head-up display. For example, the display unit projects sign language within the driver's line of sight so that the driver can see the sign language without moving their eyes significantly. The display unit can also adjust the display range of the head-up display. For example, the display unit adjusts the display range of the sign language to match the driver's visibility. The display unit can also customize the content displayed on the head-up display. For example, the display unit customizes the content of the sign language to the driver's preference. This allows the driver to see the sign language without moving their eyes significantly. Some or all of the above processing in the display unit may be performed using AI, for example, or without AI. For example, the display unit can input the display range and content of the sign language into a generating AI and have the generating AI perform optimization.

[0038] The detection unit can detect the sirens of emergency vehicles. For example, the detection unit can detect the siren of an ambulance. For example, the detection unit can identify the sound of an ambulance siren and issue a warning to the driver. The detection unit can also detect the siren of a police vehicle. For example, the detection unit can identify the sound of a police vehicle siren and issue a warning to the driver. The detection unit can also detect the siren of a fire truck. For example, the detection unit can identify the sound of a fire truck siren and issue a warning to the driver. This ensures reliable detection of approaching emergency vehicles. Some or all of the above processing in the detection unit may be performed using AI, for example, or without AI. For example, the detection unit can input the detection of the siren sound to a generating AI and have the generating AI execute the detection result.

[0039] The detection unit can analyze ambient sounds and perform filtering to prevent false detections when it detects a siren. For example, the detection unit can analyze ambient traffic noise in real time and filter it to distinguish it from the siren. For example, the detection unit can analyze the frequency characteristics of traffic noise to distinguish it from the siren. The detection unit can also consider weather conditions (rain noise, wind noise, etc.) and perform filtering to improve the accuracy of siren detection. For example, the detection unit can analyze the patterns of rain noise and wind noise to identify the difference from the siren. The detection unit can also filter to eliminate sounds inside the vehicle (radio, conversation, etc.) to improve the accuracy of siren detection. For example, the detection unit can analyze audio data inside the vehicle and distinguish it from the siren. This prevents false detections of sirens by considering ambient sounds. Some or all of the above processing in the detection unit may be performed using AI, for example, or without AI. For example, the detection unit can input ambient sound data into a generating AI and have the generating AI perform the filtering process.

[0040] The detection unit can set detection priorities based on the vehicle's speed or location information when it detects a siren. For example, if the vehicle is traveling on a highway, the detection unit can increase the siren detection priority and issue an early warning. For example, the detection unit can set the siren detection priority while traveling on a highway based on the vehicle's speed sensor information. The detection unit can also set the detection priority considering the surrounding traffic conditions when the vehicle is traveling in an urban area. For example, the detection unit can set the siren detection priority based on urban traffic sensor information. Furthermore, the detection unit can set a lower siren detection priority when the vehicle is stopped to prevent false warnings. For example, the detection unit can set the siren detection priority while the vehicle is stopped based on the vehicle's location sensor information. This allows for the optimization of siren detection priorities by considering the vehicle's speed and location information. Some or all of the above processing in the detection unit may be performed using AI, for example, or without AI. For example, the detection unit can input the vehicle's speed and location information into a generating AI and have the generating AI perform the setting of detection priorities.

[0041] The detection unit can select the optimal detection method by referring to the driver's past driving history when detecting a siren. For example, if there is a history of false siren detection in the past, the detection unit will analyze the cause and select the optimal detection method. For example, the detection unit will identify the cause of false detection based on past driving history data and improve the detection algorithm. The detection unit can also prioritize the detection of a particular siren sound if there is a history of it being easily detected in the past. For example, the detection unit will set the detection priority for a particular siren sound based on past driving history data. The detection unit can also optimize the siren detection method for specific times of day or locations based on past driving history. For example, the detection unit will adjust the siren detection algorithm for specific times of day or locations based on past driving history data. In this way, the optimal detection method can be selected by considering past driving history. Some or all of the above processing in the detection unit may be performed using AI, for example, or without AI. For example, the detection unit can input the driver's past driving history data into a generating AI and have the generating AI select the optimal detection method.

[0042] The detection unit can improve the accuracy of siren detection based on the driver's gaze information when detecting a siren. For example, if the driver is looking straight ahead, the detection unit can improve the accuracy of detecting a siren sound from the front. For example, the detection unit can adjust the detection algorithm for siren sounds from the front based on the driver's gaze tracking data. The detection unit can also improve the accuracy of detecting siren sounds from the left and right if the driver is looking left and right. For example, the detection unit can adjust the detection algorithm for siren sounds from the left and right based on the driver's gaze tracking data. The detection unit can also improve the accuracy of detecting a siren sound from behind if the driver is looking behind. For example, the detection unit can adjust the detection algorithm for siren sounds from behind based on the driver's gaze tracking data. In this way, the accuracy of siren detection can be improved by taking the driver's gaze information into consideration. Some or all of the above processing in the detection unit may be performed using AI, for example, or without using AI. For example, the detection unit can input the driver's gaze tracking data into a generating AI and have the generating AI perform the improvement of detection accuracy.

[0043] The analysis unit can optimize its analysis algorithm by referring to past analysis data when analyzing sirens. For example, the analysis unit can extract characteristics of a specific siren sound from past analysis data and optimize the analysis algorithm. For example, the analysis unit can identify characteristics of a specific siren sound based on past siren analysis results and adjust the analysis algorithm. The analysis unit can also identify the cause of misanalysis and improve the algorithm based on past analysis data. For example, the analysis unit can analyze past siren analysis results, identify the cause of misanalysis, and improve the algorithm. The analysis unit can also analyze past analysis data and select the most efficient analysis algorithm. For example, the analysis unit selects the most efficient analysis algorithm based on past siren analysis results. In this way, the analysis algorithm can be optimized by utilizing past analysis data. Some or all of the above processes in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input past analysis data into a generating AI and have the generating AI perform the optimization of the analysis algorithm.

[0044] The analysis unit can set the analysis priority based on the surrounding traffic conditions when analyzing a siren. For example, if the surrounding traffic is congested, the analysis unit will increase the siren analysis priority and provide analysis results quickly. For example, the analysis unit will set the analysis priority during traffic congestion based on traffic sensor information. The analysis unit can also perform analysis with the normal analysis priority when there is little surrounding traffic. For example, the analysis unit will set the analysis priority for when there is little traffic based on traffic sensor information. The analysis unit can also analyze the surrounding traffic conditions in real time and set the optimal analysis priority. For example, the analysis unit will set the optimal analysis priority based on real-time traffic sensor information. This allows the analysis priority to be optimized by considering the surrounding traffic conditions. Some or all of the above processing in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input traffic sensor information into a generating AI and have the generating AI perform the setting of the analysis priority.

[0045] The analysis unit can improve the accuracy of its analysis by referring to past driver response data when analyzing sirens. For example, the analysis unit can analyze how drivers have reacted to specific siren sounds from past response data to improve analysis accuracy. For example, the analysis unit can identify reaction patterns to specific siren sounds based on past driver response data and adjust the analysis algorithm. The analysis unit can also identify the causes of driver misreactions based on past response data and improve the analysis algorithm. For example, the analysis unit can analyze past response data to identify the causes of misreactions and improve the algorithm. The analysis unit can also refer to past response data and provide analysis results that allow the driver to react most appropriately. For example, the analysis unit selects analysis results that allow the driver to react most appropriately based on past response data. In this way, the accuracy of the analysis can be improved by utilizing past response data. Some or all of the above processes in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input past response data into a generating AI and have the generating AI perform the improvement of analysis accuracy.

[0046] The analysis unit can improve the accuracy of its analysis of sirens based on information from the vehicle's internal sensors. For example, the analysis unit can improve the accuracy of siren sound analysis by utilizing the vehicle's speed sensor information. For example, the analysis unit can adjust the siren sound analysis algorithm based on the vehicle's speed sensor information. The analysis unit can also identify the source of the siren sound based on the vehicle's position sensor information and improve the analysis accuracy. For example, the analysis unit can identify the source of the siren sound based on the vehicle's position sensor information and adjust the analysis algorithm. The analysis unit can also acquire information from the vehicle's internal sensors in real time and reflect it in the analysis algorithm. For example, the analysis unit can acquire information from the vehicle's internal sensors in real time and adjust the analysis algorithm. This allows the analysis accuracy to be improved by utilizing the vehicle's internal sensor information. Some or all of the above-described processes in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input the vehicle's internal sensor information into a generating AI and have the generating AI perform the improvement of the analysis accuracy.

[0047] The conversion unit can select the optimal conversion method to reduce the driver's visual burden when converting sign language. For example, the conversion unit can reduce the driver's visual burden by using highly visible colors and fonts when converting sign language. For example, the conversion unit can reduce the driver's visual burden by selecting highly visible colors and fonts for displaying sign language. The conversion unit can also reduce the visual burden by adopting simple and easy-to-understand movements when converting sign language. For example, the conversion unit can reduce the driver's visual burden by displaying sign language movements with simple and easy-to-understand animations. The conversion unit can also select the optimal display position by considering the driver's gaze information when converting sign language. For example, the conversion unit optimizes the display position of sign language based on the driver's eye-tracking data. This reduces the driver's visual burden and helps in understanding sign language. Some or all of the above processing in the conversion unit may be performed using AI, for example, or without AI. For example, the conversion unit can input the driver's eye-tracking data into a generating AI and have the generating AI select the optimal conversion method.

[0048] The conversion unit can improve the accuracy of sign language conversion based on the driver's past sign language comprehension level. For example, the conversion unit provides sign language that is easy for the driver to understand based on past sign language comprehension data. For example, the conversion unit selects the sign language conversion method that is easiest for the driver to understand based on past sign language comprehension test results. The conversion unit can also analyze past sign language comprehension data and improve sign language that is easily misunderstood by the driver. For example, the conversion unit improves easily misunderstood sign language expressions based on past sign language usage history. The conversion unit can also refer to past sign language comprehension data and select the sign language conversion method that is easiest for the driver to understand. For example, the conversion unit selects the optimal sign language conversion method based on past sign language comprehension test results. This improves the accuracy of sign language conversion by considering past sign language comprehension. Some or all of the above processing in the conversion unit may be performed using AI, for example, or without AI. For example, the conversion unit can input past sign language comprehension data into a generating AI and have the generating AI perform the improvement of conversion accuracy.

[0049] The conversion unit can select the optimal display position when converting sign language by utilizing the driver's gaze information. For example, if the driver is looking straight ahead, the conversion unit will display the sign language in front. For example, the conversion unit will display the sign language in front based on the driver's gaze tracking data. The conversion unit can also display the sign language to the left or right if the driver is looking left or right. For example, the conversion unit will display the sign language to the left or right based on the driver's gaze tracking data. The conversion unit can also display the sign language behind if the driver is looking behind. For example, the conversion unit will display the sign language behind based on the driver's gaze tracking data. In this way, the display position of the sign language can be optimized by taking the driver's gaze information into consideration. Some or all of the above processing in the conversion unit may be performed using AI, for example, or without using AI. For example, the conversion unit can input the driver's gaze tracking data into a generating AI and have the generating AI select the optimal display position.

[0050] The conversion unit can set conversion priorities based on the driver's past driving history when converting sign language. For example, the conversion unit can set the priority of sign language in specific situations based on past driving history. For example, the conversion unit can set the priority of sign language in specific situations based on past driving history data. The conversion unit can also prioritize displaying the sign language that the driver needs most, based on past driving history data. For example, the conversion unit can prioritize displaying the sign language that the driver is likely to misunderstand, based on past driving history data. For example, the conversion unit can prioritize sign language that is likely to be misunderstood, based on past driving history data. This allows for the optimization of sign language conversion priorities by considering past driving history. Some or all of the above processing in the conversion unit may be performed using AI, for example, or without AI. For example, the conversion unit can input past driving history data into a generating AI and have the generating AI perform the setting of conversion priorities.

[0051] The display unit can select the optimal display method to reduce the driver's visual burden when displaying sign language. For example, the display unit can use highly visible colors and fonts when displaying sign language to reduce the driver's visual burden. The display unit can also reduce the visual burden by adopting simple and easy-to-understand movements when displaying sign language. For example, the display unit can display sign language movements with simple and easy-to-understand animations to reduce the driver's visual burden. The display unit can also select the optimal display position when displaying sign language, taking into account the driver's eye-tracking information. For example, the display unit can optimize the display position of sign language based on the driver's eye-tracking data. This reduces the driver's visual burden and helps in understanding sign language. Some or all of the above processing in the display unit may be performed using AI, for example, or without AI. For example, the display unit can input the driver's eye-tracking data into a generating AI and have the generating AI select the optimal display method.

[0052] The display unit can improve the accuracy of sign language display based on the driver's past display history. For example, the display unit can provide sign language that is easy for the driver to understand based on past display history. For example, the display unit can select the most easily understandable sign language display method based on past display history data. The display unit can also analyze past display history and improve sign language that is easily misunderstood by the driver. For example, the display unit can improve easily misunderstood sign language expressions based on past display history data. The display unit can also refer to past display history and select the most easily understandable sign language display method for the driver. For example, the display unit can select the optimal sign language display method based on past display history data. In this way, the accuracy of sign language display can be improved by considering past display history. Some or all of the above processing in the display unit may be performed using AI, for example, or without AI. For example, the display unit can input past display history data into a generating AI and have the generating AI perform the improvement of display accuracy.

[0053] The display unit can select the optimal display position for sign language by utilizing the driver's gaze information. For example, if the driver is looking straight ahead, the display unit will display sign language in front of them. For example, the display unit will display sign language in front of them based on the driver's gaze tracking data. The display unit can also display sign language to the left and right if the driver is looking left and right. For example, the display unit will display sign language to the left and right based on the driver's gaze tracking data. The display unit can also display sign language behind the driver if the driver is looking behind them. For example, the display unit will display sign language behind the driver based on the driver's gaze tracking data. In this way, the display position of the sign language can be optimized by taking the driver's gaze information into consideration. Some or all of the above processing in the display unit may be performed using AI, for example, or without using AI. For example, the display unit can input the driver's gaze tracking data into a generating AI and have the generating AI select the optimal display position.

[0054] The display unit can set the display priority based on the driver's past driving history when displaying sign language. For example, the display unit can set the priority of sign language in specific situations based on past driving history. For example, the display unit can set the priority of sign language in specific situations based on past driving history data. The display unit can also prioritize displaying the sign language that the driver needs most, based on past driving history data. For example, the display unit can prioritize displaying the sign language that the driver needs most, based on past driving history data. The display unit can also refer to past driving history and prioritize displaying sign language that the driver is likely to misunderstand. For example, the display unit can prioritize sign language that is likely to be misunderstood based on past driving history data. In this way, the display priority of sign language can be optimized by considering past driving history. Some or all of the above processing in the display unit may be performed using AI, for example, or without AI. For example, the display unit can input past driving history data into a generating AI and have the generating AI perform the setting of display priorities.

[0055] The animation unit can select the optimal animation method to reduce the driver's visual burden when displaying sign language animations. For example, the animation unit can use highly visible colors and fonts when displaying sign language animations to reduce the driver's visual burden. For example, the animation unit can select highly visible colors and fonts for displaying sign language animations to reduce the driver's visual burden. The animation unit can also reduce the visual burden by adopting simple and easy-to-understand movements when displaying sign language animations. For example, the animation unit can display sign language movements with simple and easy-to-understand animations to reduce the driver's visual burden. The animation unit can also consider the driver's eye-tracking information and select the optimal display position when displaying sign language animations. For example, the animation unit optimizes the display position of sign language based on the driver's eye-tracking data. This reduces the driver's visual burden and helps in understanding sign language. Some or all of the above processing in the animation unit may be performed using AI, for example, or without using AI. For example, the animation unit can input driver eye-tracking data into a generating AI and have the AI ​​select the optimal animation method.

[0056] The animation unit can improve the accuracy of sign language animations by referring to the driver's past sign language comprehension level when displaying sign language animations. For example, the animation unit provides animations that are easy for the driver to understand based on past sign language comprehension data. For example, the animation unit selects the animation display method that is easiest for the driver to understand based on past sign language comprehension test results. The animation unit can also analyze past sign language comprehension data and improve animations that are easily misunderstood by the driver. For example, the animation unit improves the expression of animations that are easily misunderstood based on past sign language usage history. The animation unit can also refer to past sign language comprehension data and select the animation display method that is easiest for the driver to understand. For example, the animation unit selects the optimal animation display method based on past sign language comprehension test results. This improves the accuracy of animations by considering past sign language comprehension. Some or all of the above processing in the animation unit may be performed using AI, for example, or without AI. For example, the animation unit can input past sign language comprehension data into a generating AI and have the generating AI perform animation accuracy improvements.

[0057] The animation unit can select the optimal display position for sign language animations by utilizing the driver's gaze information. For example, if the driver is looking straight ahead, the animation unit will display the animation in front. For example, the animation unit will display the animation in front based on the driver's eye-tracking data. The animation unit can also display animations to the left or right if the driver is looking left or right. For example, the animation unit will display animations to the left or right based on the driver's eye-tracking data. The animation unit can also display animations behind if the driver is looking behind. For example, the animation unit will display animations behind based on the driver's eye-tracking data. In this way, the display position of the animation can be optimized by taking the driver's gaze information into consideration. Some or all of the above processing in the animation unit may be performed using AI, for example, or without AI. For example, the animation unit can input the driver's eye-tracking data into a generating AI and have the generating AI select the optimal display position.

[0058] The adjustment unit can select the optimal adjustment method to reduce the driver's visual burden when adjusting the display position and size. For example, the adjustment unit can select a highly visible position and size when adjusting the display position and size. For example, the adjustment unit can select a highly visible position and size based on the driver's eye-tracking data. The adjustment unit can also select a simple and easy-to-understand position and size when adjusting the display position and size. For example, the adjustment unit can select a simple and easy-to-understand position and size based on the driver's eye-tracking data. The adjustment unit can also select the optimal position and size by considering the driver's eye-tracking information when adjusting the display position and size. For example, the adjustment unit can select the optimal position and size based on the driver's eye-tracking data. This reduces the driver's visual burden and helps them understand the displayed content. Some or all of the above processing in the adjustment unit may be performed using AI, for example, or without AI. For example, the adjustment unit can input the driver's eye-tracking data into a generating AI and have the generating AI select the optimal adjustment method.

[0059] The adjustment unit can select the optimal display position by utilizing the driver's gaze information when adjusting the display position and size. For example, if the driver is looking straight ahead, the adjustment unit will display the display in front. For example, the adjustment unit will display the display in front based on the driver's gaze tracking data. The adjustment unit can also display the display to the left or right if the driver is looking left or right. For example, the adjustment unit will display the display to the left or right based on the driver's gaze tracking data. The adjustment unit can also display the display behind if the driver is looking behind. For example, the adjustment unit will display the display behind based on the driver's gaze tracking data. In this way, the display position can be optimized by taking the driver's gaze information into consideration. Some or all of the above processing in the adjustment unit may be performed using AI, for example, or without AI. For example, the adjustment unit can input the driver's gaze tracking data into a generating AI and have the generating AI select the optimal display position.

[0060] The instruction unit can select the optimal display method to reduce the driver's visual burden when displaying instructions. For example, the instruction unit can use highly visible colors and fonts when displaying instructions to reduce the driver's visual burden. The instruction unit can also reduce the driver's visual burden by adopting simple and easy-to-understand movements when displaying instructions. For example, the instruction unit can display the movement of the instructions with simple and easy-to-understand animations to reduce the driver's visual burden. The instruction unit can also select the optimal display position when displaying instructions, taking into account the driver's eye-tracking information. For example, the instruction unit optimizes the display position of the instructions based on the driver's eye-tracking data. This reduces the driver's visual burden and helps them understand the instructions. Some or all of the above processing in the instruction unit may be performed using AI, or not. For example, the instruction unit can input the driver's eye-tracking data into a generating AI and have the generating AI select the optimal display method.

[0061] The instruction unit can improve the accuracy of its display by referring to the driver's past display history when displaying instructions. For example, the instruction unit can provide instructions that are easy for the driver to understand based on past display history. For example, the instruction unit can select the display method for instructions that is easiest for the driver to understand based on past display history data. The instruction unit can also analyze past display history and improve instructions that are easily misunderstood by the driver. For example, the instruction unit can improve the expression of instructions that are easily misunderstood based on past display history data. The instruction unit can also refer to past display history and select the display method for instructions that is easiest for the driver to understand. For example, the instruction unit can select the optimal display method for instructions based on past display history data. In this way, the accuracy of the instruction display can be improved by considering past display history. Some or all of the above processing in the instruction unit may be performed using AI, for example, or without AI. For example, the instruction unit can input past display history data into a generating AI and have the generating AI perform the improvement of display accuracy.

[0062] The instruction unit can select the optimal display position when displaying instructions by utilizing the driver's gaze information. For example, if the driver is looking straight ahead, the instruction unit will display the instructions forward. For example, the instruction unit will display the instructions forward based on the driver's eye-tracking data. The instruction unit can also display instructions to the left or right if the driver is looking left or right. For example, the instruction unit will display instructions to the left or right based on the driver's eye-tracking data. The instruction unit can also display instructions behind if the driver is looking behind. For example, the instruction unit will display instructions behind based on the driver's eye-tracking data. In this way, the display position of the instructions can be optimized by taking the driver's gaze information into consideration. Some or all of the above processing in the instruction unit may be performed using AI, for example, or without AI. For example, the instruction unit can input the driver's eye-tracking data into a generating AI and have the generating AI select the optimal display position.

[0063] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0064] The driver assistance system may also include a health monitoring unit that monitors the driver's health condition. The health monitoring unit can, for example, measure the driver's heart rate and blood pressure in real time and issue a warning if an abnormality is detected. For instance, if the driver's heart rate increases rapidly, the health monitoring unit can display a warning and prompt the driver to take a break. Similarly, if the driver's blood pressure is high, the health monitoring unit can instruct the driver to relax. Furthermore, the health monitoring unit can record the driver's health data and perform regular health checks. For example, based on the driver's health data, it can analyze changes in their health condition and recommend a visit to a medical institution if necessary. This allows for constant monitoring of the driver's health condition and supports safe driving.

[0065] The driver assistance system may also include a driving style monitoring unit that monitors the driver's driving style. The driving style monitoring unit, for example, analyzes the driver's acceleration and braking operations and evaluates their driving style. For instance, if the driver frequently performs sudden acceleration or braking, the driving style monitoring unit may determine that the driving style is rough and display instructions encouraging safe driving. Conversely, if the driver is driving smoothly, the driving style monitoring unit may determine that the driving style is good and display a message praising the driver. Furthermore, the driving style monitoring unit can record the driver's driving style and support long-term improvement of that style. For example, it can analyze changes in the driver's driving style and provide advice for safe driving. This allows for continuous monitoring of the driver's driving style and supports safe driving.

[0066] The driver assistance system may also include a driving history analysis unit that analyzes the driver's driving history. For example, the driving history analysis unit can analyze the driver's driving tendencies based on past driving data and support improvements to their driving style. For instance, if a driver has frequently exceeded the speed limit in the past, the driving history analysis unit can identify this tendency and provide advice to encourage safer driving. Furthermore, if a driver has been involved in an accident in the past, the driving history analysis unit can analyze the cause of the accident and provide advice to prevent recurrence. In addition, the driving history analysis unit can support improvements in the driver's driving skills based on their driving history. For example, it can analyze the driver's driving history and propose a training plan to improve their driving skills. This allows for the use of the driver's driving history to support safer driving.

[0067] The driver assistance system may also include a skill evaluation unit that assesses the driver's driving skills. For example, the skill evaluation unit analyzes the driver's acceleration and braking operations to evaluate their driving skills. If the driver performs smooth acceleration and braking, the skill evaluation unit can determine that their driving skills are high and display a message praising the driver. Conversely, if the driver frequently performs sudden acceleration or braking, the skill evaluation unit can determine that their driving skills are low and provide advice for improvement. Furthermore, the skill evaluation unit can record the driver's driving skills to support long-term skill improvement. For example, it can analyze changes in the driver's driving skills and propose a training plan for skill improvement. This allows for continuous monitoring of the driver's driving skills and supports safe driving.

[0068] The driver assistance system may also include a performance evaluation unit that evaluates the driver's driving performance. For example, the performance evaluation unit analyzes the driver's acceleration and braking operations to assess driving performance. If the driver performs smooth acceleration and braking, the performance evaluation unit can determine that the driving performance is high and display a message praising the driver. Conversely, if the driver frequently performs sudden acceleration or braking, the performance evaluation unit can determine that the driving performance is low and provide advice for improvement. Furthermore, the performance evaluation unit can record the driver's driving performance to support long-term performance improvement. For example, it can analyze changes in the driver's driving performance and propose a training plan for performance improvement. This allows for continuous monitoring of the driver's driving performance and supports safe driving.

[0069] The following briefly describes the processing flow for example form 1.

[0070] Step 1: The detection unit detects sirens. Sirens include those from ambulances, fire trucks, and police vehicles. The detection unit uses voice recognition technology to detect sirens and analyzes surrounding environmental sounds to identify the siren sound. For example, it filters out surrounding traffic noise and wind noise to detect the siren sound. Step 2: The analysis unit analyzes the siren detected by the detection unit. The analysis unit analyzes the frequency and volume of the siren to identify the type of siren. It can also identify the source of the siren based on the time difference in the arrival of the siren's sound waves. Step 3: The conversion unit converts the siren analyzed by the analysis unit into sign language. The conversion unit can also use a sign language dictionary to convert the content of the siren into sign language and display the sign language movements as a 3D animation. Step 4: The display unit displays the sign language converted by the conversion unit. The display unit displays the sign language on the head-up display and can adjust the display position and size. For example, it adjusts the display position of the sign language to match the driver's line of sight. Some or all of the processing in the display unit may be performed using AI.

[0071] (Example of form 2) The driving assistance system according to an embodiment of the present invention is a system that supports the driving of hearing-impaired persons by linking AI and automotive technology. In this driving assistance system, the AI ​​detects the sirens of ambulances and police vehicles, analyzes the audio waveform of the detected sirens, and converts the content into sign language. The converted sign language is displayed on a head-up display, providing visual instructions to the driver. For example, when turning right, the sign language interpreter displays sign language indicating the right direction, prompting the driver to turn right. This system allows hearing-impaired persons to visually recognize the approach of emergency vehicles and take appropriate action. Furthermore, the driving guidance provided by the sign language interpreter makes driving safer and smoother for hearing-impaired persons. In addition, this system contributes to increasing awareness of the hearing-impaired person mark. The instructions from the sign language interpreter displayed on the head-up display are provided in a way that is easy for hearing drivers to understand visually, thus promoting understanding and consideration for hearing-impaired drivers. In this way, by integrating AI and automotive technology to solve potential problems for hearing-impaired persons, the driving environment for hearing-impaired persons is greatly improved. This allows the driver assistance system to visually recognize the approach of an emergency vehicle and take appropriate action.

[0072] The driving assistance system according to this embodiment comprises a detection unit, an analysis unit, a conversion unit, and a display unit. The detection unit detects sirens. Sirens include, but are not limited to, sirens of ambulances, fire trucks, and police vehicles. The detection unit detects sirens using, for example, speech recognition technology. The detection unit can also analyze ambient sounds to identify siren sounds. For example, the detection unit filters out ambient traffic noise and wind noise to detect siren sounds. The analysis unit analyzes the siren detected by the detection unit. The analysis unit identifies the type of siren by, for example, analyzing the frequency and volume of the siren. The analysis unit can also identify the source of the siren. For example, the analysis unit identifies the source of the siren based on the arrival time difference of the siren's sound waves. The conversion unit converts the siren analyzed by the analysis unit into sign language. The conversion unit converts the content of the siren into sign language using, for example, a sign language dictionary. The conversion unit can also display the sign language movements as animation. For example, the conversion unit displays sign language movements as 3D animations, providing visual instructions to the driver. The display unit displays the sign language converted by the conversion unit. The display unit displays the sign language, for example, on a head-up display. The display unit can also adjust the display position and size. For example, the display unit adjusts the display position of the sign language to match the driver's line of sight. This allows the driver assistance system to enable hearing-impaired individuals to visually recognize approaching emergency vehicles and take appropriate action. Some or all of the above processing in the display unit may be performed using AI, for example, or without AI. For example, the display unit can optimize the display position and size of the sign language using an AI model.

[0073] The detection unit detects sirens. Sirens include, but are not limited to, those of ambulances, fire trucks, and police vehicles. The detection unit detects sirens using, for example, speech recognition technology. Specifically, the detection unit is equipped with a high-sensitivity microphone to collect ambient sounds in real time. The collected audio data is processed using a noise reduction algorithm to remove unwanted background noise and highlight the characteristics of the siren sound. Furthermore, speech recognition technology is used to analyze the frequency spectrum and volume changes of the siren sound and detect specific patterns. For example, ambulance sirens change periodically in a specific frequency band, so they can be detected based on this characteristic. The detection unit can also identify siren sounds by analyzing ambient environmental sounds. For example, the detection unit filters out ambient traffic noise and wind noise to detect siren sounds. This involves using machine learning algorithms to learn the characteristics of different ambient sounds and identify siren sounds with high accuracy. As a result, the detection unit can accurately detect siren sounds even in complex environments and proceed to the next processing step.

[0074] The analysis unit analyzes the siren detected by the detection unit. For example, the analysis unit analyzes the frequency and volume of the siren to identify the type of siren. Specifically, the analysis unit analyzes the frequency spectrum of the siren sound in detail and identifies specific frequency bands and volume change patterns. This makes it possible to distinguish between different siren sounds, such as those of ambulances, fire trucks, and police vehicles. The analysis unit can also identify the source of the siren. For example, the analysis unit identifies the source of the siren based on the difference in arrival times of the siren's sound waves. Specifically, it measures the arrival time of sound waves using multiple microphones and calculates the position of the sound source using triangulation. This technology allows the analysis unit to accurately identify the direction of the siren source and provide appropriate information to the driver. Furthermore, the analysis unit can learn siren sound patterns by utilizing past data and statistical information to perform more accurate analysis. For example, based on past siren sound data, it can analyze the frequency and patterns of siren sounds in specific areas and time periods to help with future predictions. This allows the analysis unit to handle not only real-time analysis but also long-term risk assessment and trend analysis, thereby improving the reliability and safety of the entire system.

[0075] The conversion unit converts the siren analyzed by the analysis unit into sign language. The conversion unit converts the content of the siren into sign language using, for example, a sign language dictionary. Specifically, the conversion unit selects sign language actions according to the type and source of the siren and generates an animation to visually communicate it to the driver. The sign language dictionary has sign language actions registered that correspond to various siren sounds, and the conversion unit refers to this to select the appropriate sign language. The conversion unit can also display the sign language movements as animations. For example, the conversion unit can display the sign language movements as 3D animations to provide visual instructions to the driver. The 3D animations are designed to be intuitively understandable to the driver and can display the sign language actions in real time. Furthermore, the conversion unit can adjust the sign language actions to match the driver's gaze and attention range. For example, if the driver is looking straight ahead, the sign language animation can be displayed on the head-up display to provide information without diverting the driver's gaze. This allows the conversion unit to provide information that enables hearing-impaired individuals to visually recognize the approach of an emergency vehicle and take appropriate action.

[0076] The display unit displays the sign language converted by the conversion unit. The display unit displays the sign language, for example, on a head-up display. Specifically, the display unit adjusts the display position of the sign language to match the driver's line of sight, allowing the driver to check the information without moving their eyes significantly. The display unit can also adjust the display position and size. For example, the display unit adjusts the display position of the sign language to match the driver's line of sight. This allows the driver assistance system to enable hearing-impaired individuals to visually recognize the approach of emergency vehicles and take appropriate action. Some or all of the above processing in the display unit may be performed using AI, for example, or without AI. For example, the display unit can optimize the display position and size of the sign language using an AI model. Specifically, the AI ​​model learns the driver's eye-tracking data and past usage history to calculate the optimal display position and size in real time. This allows the display unit to display the sign language in the most visible position for the driver, improving the efficiency of information transmission. Furthermore, the display unit can collect driver feedback and continuously improve the accuracy and effectiveness of the displayed content. For example, by providing feedback from the driver regarding the position and size of the sign language display, the AI ​​model optimizes the display settings based on that information. This allows the display unit to provide information to the driver quickly and reliably, improving the reliability and safety of the driver assistance system.

[0077] The driver assistance system includes an animation unit that displays sign language movements as animations. The animation unit can, for example, display sign language movements as 2D animations. For example, the animation unit can display sign language movements as simple 2D animations to provide visual instructions to the driver. The animation unit can also display sign language movements as 3D animations. For example, the animation unit can display sign language movements as realistic 3D animations to provide visual instructions to the driver. The animation unit can also display sign language movements as animated characters. For example, the animation unit can display sign language movements as animated characters to provide visual instructions to the driver. This makes sign language movements visually easy to understand and helps the driver to comprehend them. Some or all of the above processing in the animation unit may be performed using AI, for example, or without AI. For example, the animation unit can input sign language movements into a generation AI and have the generation AI perform the animation generation.

[0078] The driver assistance system includes an adjustment unit that adjusts the display position and size, enabling it to provide the displayed content in an appropriate position and size for the driver. For example, the adjustment unit can adjust the display position to match the driver's line of sight. For example, the adjustment unit can optimize the display position of sign language based on the driver's eye-tracking data. The adjustment unit can also adjust the display size to match the driver's visibility. For example, the adjustment unit can adjust the display size of sign language to match the driver's eyesight and the display resolution. The adjustment unit can also adjust the layout of the displayed content. For example, the adjustment unit can position and size the sign language display to reduce the driver's visual burden. This enables the display content to be provided in the optimal position and size for the driver. Some or all of the above processing in the adjustment unit may be performed using AI, for example, or without AI. For example, the adjustment unit can input the driver's eye-tracking data into a generating AI and have the generating AI perform the optimization of the display position and size.

[0079] The driver assistance system includes an instruction unit that displays specific instructions. The instruction unit can, for example, display instructions for the direction of travel. For example, the instruction unit can display instructions for turning right or left in sign language, providing visual instructions to the driver. The instruction unit can also display instructions for speed. For example, the instruction unit can display instructions for slowing down or stopping in sign language, providing visual instructions to the driver. The instruction unit can also display instructions for emergencies. For example, the instruction unit can display the approach of an emergency vehicle in sign language, providing visual instructions to the driver. In this way, by providing specific driving instructions to the driver, safe driving can be supported. Some or all of the above processing in the instruction unit may be performed using AI, for example, or without AI. For example, the instruction unit can input instructions for the direction of travel and speed into a generating AI, and have the generating AI perform the generation of sign language.

[0080] The display unit can display sign language on a head-up display. For example, the display unit projects sign language onto the head-up display. For example, the display unit projects sign language within the driver's line of sight so that the driver can see the sign language without moving their eyes significantly. The display unit can also adjust the display range of the head-up display. For example, the display unit adjusts the display range of the sign language to match the driver's visibility. The display unit can also customize the content displayed on the head-up display. For example, the display unit customizes the content of the sign language to the driver's preference. This allows the driver to see the sign language without moving their eyes significantly. Some or all of the above processing in the display unit may be performed using AI, for example, or without AI. For example, the display unit can input the display range and content of the sign language into a generating AI and have the generating AI perform optimization.

[0081] The detection unit can detect the sirens of emergency vehicles. For example, the detection unit can detect the siren of an ambulance. For example, the detection unit can identify the sound of an ambulance siren and issue a warning to the driver. The detection unit can also detect the siren of a police vehicle. For example, the detection unit can identify the sound of a police vehicle siren and issue a warning to the driver. The detection unit can also detect the siren of a fire truck. For example, the detection unit can identify the sound of a fire truck siren and issue a warning to the driver. This ensures reliable detection of approaching emergency vehicles. Some or all of the above processing in the detection unit may be performed using AI, for example, or without AI. For example, the detection unit can input the detection of the siren sound to a generating AI and have the generating AI execute the detection result.

[0082] The detection unit can estimate the driver's emotions and adjust the siren's detection sensitivity based on the estimated emotions. For example, if the driver is tense, the detection unit can increase the siren's detection sensitivity and issue an earlier warning. For example, the detection unit can capture the driver's facial expression with a camera and detect the state of tension using an emotion estimation algorithm. The detection unit can also maintain normal detection sensitivity and issue a warning at an appropriate time if the driver is relaxed. For example, the detection unit can record the driver's voice and detect the state of relaxation using voice analysis technology. The detection unit can also fine-tune the detection sensitivity if the driver is tired, preventing false detections while ensuring a reliable warning. For example, the detection unit can collect the driver's biometric data (heart rate and skin electrical activity) with sensors and detect the state of fatigue using an emotion estimation algorithm. This allows the siren's detection sensitivity to be optimized according to the driver's condition. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processing in the detection unit may be performed using AI, for example, or without AI. For example, the detection unit can input driver emotion data into a generating AI and have the generating AI adjust the siren detection sensitivity.

[0083] The detection unit can analyze ambient sounds and perform filtering to prevent false detections when it detects a siren. For example, the detection unit can analyze ambient traffic noise in real time and filter it to distinguish it from the siren. For example, the detection unit can analyze the frequency characteristics of traffic noise to distinguish it from the siren. The detection unit can also consider weather conditions (rain noise, wind noise, etc.) and perform filtering to improve the accuracy of siren detection. For example, the detection unit can analyze the patterns of rain noise and wind noise to identify the difference from the siren. The detection unit can also filter to eliminate sounds inside the vehicle (radio, conversation, etc.) to improve the accuracy of siren detection. For example, the detection unit can analyze audio data inside the vehicle and distinguish it from the siren. This prevents false detections of sirens by considering ambient sounds. Some or all of the above processing in the detection unit may be performed using AI, for example, or without AI. For example, the detection unit can input ambient sound data into a generating AI and have the generating AI perform the filtering process.

[0084] The detection unit can set detection priorities based on the vehicle's speed or location information when it detects a siren. For example, if the vehicle is traveling on a highway, the detection unit can increase the siren detection priority and issue an early warning. For example, the detection unit can set the siren detection priority while traveling on a highway based on the vehicle's speed sensor information. The detection unit can also set the detection priority considering the surrounding traffic conditions when the vehicle is traveling in an urban area. For example, the detection unit can set the siren detection priority based on urban traffic sensor information. Furthermore, the detection unit can set a lower siren detection priority when the vehicle is stopped to prevent false warnings. For example, the detection unit can set the siren detection priority while the vehicle is stopped based on the vehicle's location sensor information. This allows for the optimization of siren detection priorities by considering the vehicle's speed and location information. Some or all of the above processing in the detection unit may be performed using AI, for example, or without AI. For example, the detection unit can input the vehicle's speed and location information into a generating AI and have the generating AI perform the setting of detection priorities.

[0085] The detection unit can estimate the driver's emotions and select the type of siren to detect based on the estimated emotions. For example, if the driver is tense, the detection unit will prioritize detecting sirens indicating a high level of urgency. For example, the detection unit may capture the driver's facial expression with a camera and use an emotion estimation algorithm to detect the state of tension. The detection unit can also detect normal sirens if the driver is relaxed. For example, the detection unit may record the driver's voice and use voice analysis technology to detect the state of relaxation. Furthermore, if the driver is tired, the detection unit may detect only specific sirens to prevent false detections. For example, the detection unit may collect the driver's biometric data (heart rate and skin electrical activity) with sensors and use an emotion estimation algorithm to detect the state of fatigue. This allows for the optimization of the type of siren detected according to the driver's state. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processing in the detection unit may be performed using AI, for example, or without AI. For example, the detection unit can input driver emotion data into a generating AI and have the generating AI select the type of siren.

[0086] The detection unit can select the optimal detection method by referring to the driver's past driving history when detecting a siren. For example, if there is a history of false siren detection in the past, the detection unit will analyze the cause and select the optimal detection method. For example, the detection unit will identify the cause of false detection based on past driving history data and improve the detection algorithm. The detection unit can also prioritize the detection of a particular siren sound if there is a history of it being easily detected in the past. For example, the detection unit will set the detection priority for a particular siren sound based on past driving history data. The detection unit can also optimize the siren detection method for specific times of day or locations based on past driving history. For example, the detection unit will adjust the siren detection algorithm for specific times of day or locations based on past driving history data. In this way, the optimal detection method can be selected by considering past driving history. Some or all of the above processing in the detection unit may be performed using AI, for example, or without AI. For example, the detection unit can input the driver's past driving history data into a generating AI and have the generating AI select the optimal detection method.

[0087] The detection unit can improve the accuracy of siren detection based on the driver's gaze information when detecting a siren. For example, if the driver is looking straight ahead, the detection unit can improve the accuracy of detecting a siren sound from the front. For example, the detection unit can adjust the detection algorithm for siren sounds from the front based on the driver's gaze tracking data. The detection unit can also improve the accuracy of detecting siren sounds from the left and right if the driver is looking left and right. For example, the detection unit can adjust the detection algorithm for siren sounds from the left and right based on the driver's gaze tracking data. The detection unit can also improve the accuracy of detecting a siren sound from behind if the driver is looking behind. For example, the detection unit can adjust the detection algorithm for siren sounds from behind based on the driver's gaze tracking data. In this way, the accuracy of siren detection can be improved by taking the driver's gaze information into consideration. Some or all of the above processing in the detection unit may be performed using AI, for example, or without using AI. For example, the detection unit can input the driver's gaze tracking data into a generating AI and have the generating AI perform the improvement of detection accuracy.

[0088] The analysis unit can estimate the driver's emotions and adjust the level of detail of the analysis based on the estimated emotions. For example, if the driver is tense, the analysis unit can provide detailed analysis results to reassure the driver. For example, the analysis unit can capture the driver's facial expressions with a camera and detect the state of tension using an emotion estimation algorithm. The analysis unit can also provide normal analysis results if the driver is relaxed. For example, the analysis unit can record the driver's voice and detect the state of relaxation using voice analysis technology. The analysis unit can also provide concise analysis results if the driver is tired, reducing the driver's burden. For example, the analysis unit can collect the driver's biometric data (heart rate and skin electrical activity) with sensors and detect the state of fatigue using an emotion estimation algorithm. This allows the level of detail of the analysis to be optimized according to the driver's state. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input driver emotion data into a generating AI and have the generating AI adjust the level of detail of the analysis.

[0089] The analysis unit can optimize its analysis algorithm by referring to past analysis data when analyzing sirens. For example, the analysis unit can extract characteristics of a specific siren sound from past analysis data and optimize the analysis algorithm. For example, the analysis unit can identify characteristics of a specific siren sound based on past siren analysis results and adjust the analysis algorithm. The analysis unit can also identify the cause of misanalysis and improve the algorithm based on past analysis data. For example, the analysis unit can analyze past siren analysis results, identify the cause of misanalysis, and improve the algorithm. The analysis unit can also analyze past analysis data and select the most efficient analysis algorithm. For example, the analysis unit selects the most efficient analysis algorithm based on past siren analysis results. In this way, the analysis algorithm can be optimized by utilizing past analysis data. Some or all of the above processes in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input past analysis data into a generating AI and have the generating AI perform the optimization of the analysis algorithm.

[0090] The analysis unit can set the analysis priority based on the surrounding traffic conditions when analyzing a siren. For example, if the surrounding traffic is congested, the analysis unit will increase the siren analysis priority and provide analysis results quickly. For example, the analysis unit will set the analysis priority during traffic congestion based on traffic sensor information. The analysis unit can also perform analysis with the normal analysis priority when there is little surrounding traffic. For example, the analysis unit will set the analysis priority for when there is little traffic based on traffic sensor information. The analysis unit can also analyze the surrounding traffic conditions in real time and set the optimal analysis priority. For example, the analysis unit will set the optimal analysis priority based on real-time traffic sensor information. This allows the analysis priority to be optimized by considering the surrounding traffic conditions. Some or all of the above processing in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input traffic sensor information into a generating AI and have the generating AI perform the setting of the analysis priority.

[0091] The analysis unit can estimate the driver's emotions and adjust the display method of the analysis results based on the estimated emotions. For example, if the driver is tense, the analysis unit provides the analysis results in a highly visible display method. For example, the analysis unit captures the driver's facial expression with a camera, detects the state of tension using an emotion estimation algorithm, and selects a highly visible display method. The analysis unit can also display detailed analysis results if the driver is relaxed. For example, the analysis unit records the driver's voice, detects the state of relaxation using voice analysis technology, and displays detailed analysis results. The analysis unit can also provide analysis results in a concise and easy-to-understand display method if the driver is tired. For example, the analysis unit collects the driver's biometric data (heart rate and skin electrical activity) with sensors, detects the state of fatigue using an emotion estimation algorithm, and selects a concise display method. This allows the display method of the analysis results to be optimized according to the driver's state. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. The generating AI may be a text generating AI (e.g., LLM) or a multimodal generating AI, but is not limited to such examples. Some or all of the processing described above in the analysis unit may be performed using AI, or not using AI. For example, the analysis unit may input driver emotion data into the generating AI and have the generating AI adjust the display method.

[0092] The analysis unit can improve the accuracy of its analysis by referring to past driver response data when analyzing sirens. For example, the analysis unit can analyze how drivers have reacted to specific siren sounds from past response data to improve analysis accuracy. For example, the analysis unit can identify reaction patterns to specific siren sounds based on past driver response data and adjust the analysis algorithm. The analysis unit can also identify the causes of driver misreactions based on past response data and improve the analysis algorithm. For example, the analysis unit can analyze past response data to identify the causes of misreactions and improve the algorithm. The analysis unit can also refer to past response data and provide analysis results that allow the driver to react most appropriately. For example, the analysis unit selects analysis results that allow the driver to react most appropriately based on past response data. In this way, the accuracy of the analysis can be improved by utilizing past response data. Some or all of the above processes in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input past response data into a generating AI and have the generating AI perform the improvement of analysis accuracy.

[0093] The analysis unit can improve the accuracy of its analysis of sirens based on information from the vehicle's internal sensors. For example, the analysis unit can improve the accuracy of siren sound analysis by utilizing the vehicle's speed sensor information. For example, the analysis unit can adjust the siren sound analysis algorithm based on the vehicle's speed sensor information. The analysis unit can also identify the source of the siren sound based on the vehicle's position sensor information and improve the analysis accuracy. For example, the analysis unit can identify the source of the siren sound based on the vehicle's position sensor information and adjust the analysis algorithm. The analysis unit can also acquire information from the vehicle's internal sensors in real time and reflect it in the analysis algorithm. For example, the analysis unit can acquire information from the vehicle's internal sensors in real time and adjust the analysis algorithm. This allows the analysis accuracy to be improved by utilizing the vehicle's internal sensor information. Some or all of the above-described processes in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input the vehicle's internal sensor information into a generating AI and have the generating AI perform the improvement of the analysis accuracy.

[0094] The conversion unit can estimate the driver's emotions and adjust the sign language conversion speed based on the estimated emotions. For example, if the driver is tense, the conversion unit can speed up the sign language conversion speed to provide instructions quickly. For example, the conversion unit can capture the driver's facial expressions with a camera, detect the state of tension using an emotion estimation algorithm, and speed up the sign language conversion speed. The conversion unit can also provide sign language at a normal conversion speed if the driver is relaxed. For example, the conversion unit can record the driver's voice, detect the state of relaxation using voice analysis technology, and provide sign language at a normal conversion speed. Furthermore, if the driver is tired, the conversion unit can slow down the sign language conversion speed to improve visibility. For example, the conversion unit can collect the driver's biometric data (heart rate and skin electrical activity) with sensors, detect the state of fatigue using an emotion estimation algorithm, and slow down the sign language conversion speed. This allows the sign language conversion speed to be optimized according to the driver's state. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. The generating AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the conversion unit may be performed using AI, or not using AI. For example, the conversion unit may input driver emotion data into the generating AI and cause the generating AI to adjust the sign language conversion speed.

[0095] The conversion unit can select the optimal conversion method to reduce the driver's visual burden when converting sign language. For example, the conversion unit can reduce the driver's visual burden by using highly visible colors and fonts when converting sign language. For example, the conversion unit can reduce the driver's visual burden by selecting highly visible colors and fonts for displaying sign language. The conversion unit can also reduce the visual burden by adopting simple and easy-to-understand movements when converting sign language. For example, the conversion unit can reduce the driver's visual burden by displaying sign language movements with simple and easy-to-understand animations. The conversion unit can also select the optimal display position by considering the driver's gaze information when converting sign language. For example, the conversion unit optimizes the display position of sign language based on the driver's eye-tracking data. This reduces the driver's visual burden and helps in understanding sign language. Some or all of the above processing in the conversion unit may be performed using AI, for example, or without AI. For example, the conversion unit can input the driver's eye-tracking data into a generating AI and have the generating AI select the optimal conversion method.

[0096] The conversion unit can improve the accuracy of sign language conversion based on the driver's past sign language comprehension level. For example, the conversion unit provides sign language that is easy for the driver to understand based on past sign language comprehension data. For example, the conversion unit selects the sign language conversion method that is easiest for the driver to understand based on past sign language comprehension test results. The conversion unit can also analyze past sign language comprehension data and improve sign language that is easily misunderstood by the driver. For example, the conversion unit improves easily misunderstood sign language expressions based on past sign language usage history. The conversion unit can also refer to past sign language comprehension data and select the sign language conversion method that is easiest for the driver to understand. For example, the conversion unit selects the optimal sign language conversion method based on past sign language comprehension test results. This improves the accuracy of sign language conversion by considering past sign language comprehension. Some or all of the above processing in the conversion unit may be performed using AI, for example, or without AI. For example, the conversion unit can input past sign language comprehension data into a generating AI and have the generating AI perform the improvement of conversion accuracy.

[0097] The conversion unit can estimate the driver's emotions and adjust the sign language expression based on the estimated emotions. For example, if the driver is tense, the conversion unit can provide simple and easily recognizable sign language. For instance, it can capture the driver's facial expression with a camera, detect the state of tension using an emotion estimation algorithm, and provide simple and easily recognizable sign language. The conversion unit can also provide detailed sign language if the driver is relaxed. For example, it can record the driver's voice, detect the state of relaxation using voice analysis technology, and provide detailed sign language. Furthermore, if the driver is tired, the conversion unit can provide concise and easy-to-understand sign language. For example, it can collect the driver's biometric data (heart rate and skin electrical activity) with sensors, detect the state of fatigue using an emotion estimation algorithm, and provide concise and easy-to-understand sign language. This allows for the optimization of sign language expression according to the driver's state. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. The generating AI may be a text generating AI (e.g., LLM) or a multimodal generating AI, but is not limited to such examples. Some or all of the above-described processing in the conversion unit may be performed using AI, or not using AI. For example, the conversion unit may input driver emotion data into the generating AI and cause the generating AI to adjust the sign language expression.

[0098] The conversion unit can select the optimal display position when converting sign language by utilizing the driver's gaze information. For example, if the driver is looking straight ahead, the conversion unit will display the sign language in front. For example, the conversion unit will display the sign language in front based on the driver's gaze tracking data. The conversion unit can also display the sign language to the left or right if the driver is looking left or right. For example, the conversion unit will display the sign language to the left or right based on the driver's gaze tracking data. The conversion unit can also display the sign language behind if the driver is looking behind. For example, the conversion unit will display the sign language behind based on the driver's gaze tracking data. In this way, the display position of the sign language can be optimized by taking the driver's gaze information into consideration. Some or all of the above processing in the conversion unit may be performed using AI, for example, or without using AI. For example, the conversion unit can input the driver's gaze tracking data into a generating AI and have the generating AI select the optimal display position.

[0099] The conversion unit can set conversion priorities based on the driver's past driving history when converting sign language. For example, the conversion unit can set the priority of sign language in specific situations based on past driving history. For example, the conversion unit can set the priority of sign language in specific situations based on past driving history data. The conversion unit can also prioritize displaying the sign language that the driver needs most, based on past driving history data. For example, the conversion unit can prioritize displaying the sign language that the driver is likely to misunderstand, based on past driving history data. For example, the conversion unit can prioritize sign language that is likely to be misunderstood, based on past driving history data. This allows for the optimization of sign language conversion priorities by considering past driving history. Some or all of the above processing in the conversion unit may be performed using AI, for example, or without AI. For example, the conversion unit can input past driving history data into a generating AI and have the generating AI perform the setting of conversion priorities.

[0100] The display unit can estimate the driver's emotions and adjust the level of detail in the displayed content based on the estimated emotions. For example, if the driver is tense, the display unit can provide detailed information to give the driver a sense of security. For example, the display unit can capture the driver's facial expression with a camera, detect the state of tension using an emotion estimation algorithm, and provide detailed information. The display unit can also provide normal information if the driver is relaxed. For example, the display unit can record the driver's voice, detect the state of relaxation using voice analysis technology, and provide normal information. Furthermore, if the driver is tired, the display unit can provide concise information to reduce the driver's burden. For example, the display unit can collect the driver's biometric data (heart rate and skin electrical activity) with sensors, detect the state of fatigue using an emotion estimation algorithm, and provide concise information. This allows the level of detail in the displayed content to be optimized according to the driver's state. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. The generating AI may be a text generating AI (e.g., LLM) or a multimodal generating AI, but is not limited to such examples. Some or all of the processing described above in the display unit may be performed using AI, or not using AI. For example, the display unit may input driver emotion data into the generating AI and have the generating AI adjust the level of detail of the displayed content.

[0101] The display unit can select the optimal display method to reduce the driver's visual burden when displaying sign language. For example, the display unit can use highly visible colors and fonts when displaying sign language to reduce the driver's visual burden. The display unit can also reduce the visual burden by adopting simple and easy-to-understand movements when displaying sign language. For example, the display unit can display sign language movements with simple and easy-to-understand animations to reduce the driver's visual burden. The display unit can also select the optimal display position when displaying sign language, taking into account the driver's eye-tracking information. For example, the display unit can optimize the display position of sign language based on the driver's eye-tracking data. This reduces the driver's visual burden and helps in understanding sign language. Some or all of the above processing in the display unit may be performed using AI, for example, or without AI. For example, the display unit can input the driver's eye-tracking data into a generating AI and have the generating AI select the optimal display method.

[0102] The display unit can improve the accuracy of sign language display based on the driver's past display history. For example, the display unit can provide sign language that is easy for the driver to understand based on past display history. For example, the display unit can select the most easily understandable sign language display method based on past display history data. The display unit can also analyze past display history and improve sign language that is easily misunderstood by the driver. For example, the display unit can improve easily misunderstood sign language expressions based on past display history data. The display unit can also refer to past display history and select the most easily understandable sign language display method for the driver. For example, the display unit can select the optimal sign language display method based on past display history data. In this way, the accuracy of sign language display can be improved by considering past display history. Some or all of the above processing in the display unit may be performed using AI, for example, or without AI. For example, the display unit can input past display history data into a generating AI and have the generating AI perform the improvement of display accuracy.

[0103] The display unit can estimate the driver's emotions and determine the priority of the displayed content based on the estimated emotions. For example, if the driver is tense, the display unit will prioritize displaying important information. For instance, the display unit may capture the driver's facial expression with a camera, detect the state of tension using an emotion estimation algorithm, and prioritize the display of important information. The display unit can also display information with normal priority if the driver is relaxed. For example, the display unit may record the driver's voice, detect the state of relaxation using voice analysis technology, and display information with normal priority. The display unit can also prioritize displaying concise and important information if the driver is tired. For example, the display unit may collect the driver's biometric data (heart rate and skin electrical activity) with sensors, detect the state of fatigue using an emotion estimation algorithm, and prioritize the display of concise and important information. This allows for the optimization of the priority of displayed content according to the driver's state. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the display unit may be performed using AI, for example, or without AI. For example, the display unit can input driver emotion data into a generating AI and have the generating AI determine the priority of the display content.

[0104] The display unit can select the optimal display position for sign language by utilizing the driver's gaze information. For example, if the driver is looking straight ahead, the display unit will display sign language in front of them. For example, the display unit will display sign language in front of them based on the driver's gaze tracking data. The display unit can also display sign language to the left and right if the driver is looking left and right. For example, the display unit will display sign language to the left and right based on the driver's gaze tracking data. The display unit can also display sign language behind the driver if the driver is looking behind them. For example, the display unit will display sign language behind the driver based on the driver's gaze tracking data. In this way, the display position of the sign language can be optimized by taking the driver's gaze information into consideration. Some or all of the above processing in the display unit may be performed using AI, for example, or without using AI. For example, the display unit can input the driver's gaze tracking data into a generating AI and have the generating AI select the optimal display position.

[0105] The display unit can set the display priority based on the driver's past driving history when displaying sign language. For example, the display unit can set the priority of sign language in specific situations based on past driving history. For example, the display unit can set the priority of sign language in specific situations based on past driving history data. The display unit can also prioritize displaying the sign language that the driver needs most, based on past driving history data. For example, the display unit can prioritize displaying the sign language that the driver needs most, based on past driving history data. The display unit can also refer to past driving history and prioritize displaying sign language that the driver is likely to misunderstand. For example, the display unit can prioritize sign language that is likely to be misunderstood based on past driving history data. In this way, the display priority of sign language can be optimized by considering past driving history. Some or all of the above processing in the display unit may be performed using AI, for example, or without AI. For example, the display unit can input past driving history data into a generating AI and have the generating AI perform the setting of display priorities.

[0106] The animation unit can estimate the driver's emotions and adjust the animation speed based on the estimated emotions. For example, if the driver is tense, the animation unit can speed up the animation to provide instructions quickly. For instance, the animation unit can capture the driver's facial expressions with a camera, detect the state of tension using an emotion estimation algorithm, and speed up the animation. The animation unit can also provide animation at a normal speed if the driver is relaxed. For example, the animation unit can record the driver's voice, detect the state of relaxation using voice analysis technology, and provide animation at a normal speed. Furthermore, if the driver is tired, the animation unit can slow down the animation to improve visibility. For example, the animation unit can collect the driver's biometric data (heart rate and skin electrical activity) with sensors, detect the state of fatigue using an emotion estimation algorithm, and slow down the animation. This allows the animation speed to be optimized according to the driver's state. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. The generation AI may be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-described processes in the animation unit may be performed using AI, or not using AI. For example, the animation unit may input driver emotion data into the generation AI and have the generation AI adjust the animation speed.

[0107] The animation unit can select the optimal animation method to reduce the driver's visual burden when displaying sign language animations. For example, the animation unit can use highly visible colors and fonts when displaying sign language animations to reduce the driver's visual burden. For example, the animation unit can select highly visible colors and fonts for displaying sign language animations to reduce the driver's visual burden. The animation unit can also reduce the visual burden by adopting simple and easy-to-understand movements when displaying sign language animations. For example, the animation unit can display sign language movements with simple and easy-to-understand animations to reduce the driver's visual burden. The animation unit can also consider the driver's eye-tracking information and select the optimal display position when displaying sign language animations. For example, the animation unit optimizes the display position of sign language based on the driver's eye-tracking data. This reduces the driver's visual burden and helps in understanding sign language. Some or all of the above processing in the animation unit may be performed using AI, for example, or without using AI. For example, the animation unit can input driver eye-tracking data into a generating AI and have the AI ​​select the optimal animation method.

[0108] The animation unit can improve the accuracy of sign language animations by referring to the driver's past sign language comprehension level when displaying sign language animations. For example, the animation unit provides animations that are easy for the driver to understand based on past sign language comprehension data. For example, the animation unit selects the animation display method that is easiest for the driver to understand based on past sign language comprehension test results. The animation unit can also analyze past sign language comprehension data and improve animations that are easily misunderstood by the driver. For example, the animation unit improves the expression of animations that are easily misunderstood based on past sign language usage history. The animation unit can also refer to past sign language comprehension data and select the animation display method that is easiest for the driver to understand. For example, the animation unit selects the optimal animation display method based on past sign language comprehension test results. This improves the accuracy of animations by considering past sign language comprehension. Some or all of the above processing in the animation unit may be performed using AI, for example, or without AI. For example, the animation unit can input past sign language comprehension data into a generating AI and have the generating AI perform animation accuracy improvements.

[0109] The animation unit can estimate the driver's emotions and adjust the animation's presentation based on the estimated emotions. For example, if the driver is tense, the animation unit can provide a simple and easily understandable animation. For instance, it can capture the driver's facial expressions with a camera, detect the state of tension using an emotion estimation algorithm, and provide a simple and easily understandable animation. The animation unit can also provide a detailed animation if the driver is relaxed. For example, it can record the driver's voice, detect the state of relaxation using voice analysis technology, and provide a detailed animation. The animation unit can also provide a concise and easy-to-understand animation if the driver is tired. For example, it can collect the driver's biometric data (heart rate and skin electrical activity) with sensors, detect the state of fatigue using an emotion estimation algorithm, and provide a concise and easy-to-understand animation. This allows the animation's presentation to be optimized according to the driver's state. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the animation unit may be performed using AI, for example, or without AI. For example, the animation unit can input driver emotion data into a generating AI and have the generating AI adjust the animation's expression method.

[0110] The animation unit can select the optimal display position for sign language animations by utilizing the driver's gaze information. For example, if the driver is looking straight ahead, the animation unit will display the animation in front. For example, the animation unit will display the animation in front based on the driver's eye-tracking data. The animation unit can also display animations to the left or right if the driver is looking left or right. For example, the animation unit will display animations to the left or right based on the driver's eye-tracking data. The animation unit can also display animations behind if the driver is looking behind. For example, the animation unit will display animations behind based on the driver's eye-tracking data. In this way, the display position of the animation can be optimized by taking the driver's gaze information into consideration. Some or all of the above processing in the animation unit may be performed using AI, for example, or without AI. For example, the animation unit can input the driver's eye-tracking data into a generating AI and have the generating AI select the optimal display position.

[0111] The adjustment unit can estimate the driver's emotions and adjust the display position and size based on the estimated emotions. For example, if the driver is tense, the adjustment unit will display in a highly visible position and size. For example, the adjustment unit may capture the driver's facial expression with a camera, detect the state of tension using an emotion estimation algorithm, and display it in a highly visible position and size. The adjustment unit can also display in a normal position and size if the driver is relaxed. For example, the adjustment unit may record the driver's voice, detect the state of relaxation using voice analysis technology, and display it in a normal position and size. Furthermore, if the driver is tired, the adjustment unit can adjust the display position and size to improve visibility. For example, the adjustment unit may collect the driver's biometric data (heart rate and skin electrical activity) with sensors, detect the state of fatigue using an emotion estimation algorithm, and adjust the display position and size to improve visibility. This allows the display position and size to be optimized according to the driver's state. Emotion estimation is achieved using an emotion estimation function, for example, with an emotion engine or generative AI. The generating AI may be, but is not limited to, text generating AI (e.g., LLM) or multimodal generating AI. Some or all of the above-described processing in the adjustment unit may be performed using AI, or not using AI. For example, the adjustment unit may input driver emotion data into the generating AI and cause the generating AI to perform adjustments to the display position and size.

[0112] The adjustment unit can select the optimal adjustment method to reduce the driver's visual burden when adjusting the display position and size. For example, the adjustment unit can select a highly visible position and size when adjusting the display position and size. For example, the adjustment unit can select a highly visible position and size based on the driver's eye-tracking data. The adjustment unit can also select a simple and easy-to-understand position and size when adjusting the display position and size. For example, the adjustment unit can select a simple and easy-to-understand position and size based on the driver's eye-tracking data. The adjustment unit can also select the optimal position and size by considering the driver's eye-tracking information when adjusting the display position and size. For example, the adjustment unit can select the optimal position and size based on the driver's eye-tracking data. This reduces the driver's visual burden and helps them understand the displayed content. Some or all of the above processing in the adjustment unit may be performed using AI, for example, or without AI. For example, the adjustment unit can input the driver's eye-tracking data into a generating AI and have the generating AI select the optimal adjustment method.

[0113] The adjustment unit can estimate the driver's emotions and determine the priority of displayed content based on the estimated emotions. For example, if the driver is tense, the adjustment unit will prioritize displaying important information. For instance, it might capture the driver's facial expression with a camera, detect the state of tension using an emotion estimation algorithm, and prioritize the display of important information. Alternatively, if the driver is relaxed, the adjustment unit can display information with normal priority. For example, it might record the driver's voice, detect the state of relaxation using voice analysis technology, and prioritize the display of information with normal priority. Furthermore, if the driver is tired, the adjustment unit can prioritize displaying concise and important information. For example, it might collect the driver's biometric data (heart rate and skin electrical activity) with sensors, detect the state of fatigue using an emotion estimation algorithm, and prioritize the display of concise and important information. This allows for the optimization of the display content priority according to the driver's state. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the adjustment unit may be performed using AI, for example, or without AI. For example, the adjustment unit can input driver emotion data into a generating AI and have the generating AI determine the priority of the display content.

[0114] The adjustment unit can select the optimal display position by utilizing the driver's gaze information when adjusting the display position and size. For example, if the driver is looking straight ahead, the adjustment unit will display the display in front. For example, the adjustment unit will display the display in front based on the driver's gaze tracking data. The adjustment unit can also display the display to the left or right if the driver is looking left or right. For example, the adjustment unit will display the display to the left or right based on the driver's gaze tracking data. The adjustment unit can also display the display behind if the driver is looking behind. For example, the adjustment unit will display the display behind based on the driver's gaze tracking data. In this way, the display position can be optimized by taking the driver's gaze information into consideration. Some or all of the above processing in the adjustment unit may be performed using AI, for example, or without AI. For example, the adjustment unit can input the driver's gaze tracking data into a generating AI and have the generating AI select the optimal display position.

[0115] The instruction unit can estimate the driver's emotions and adjust the level of detail in the instructions based on the estimated emotions. For example, if the driver is tense, the instruction unit can provide detailed instructions to reassure the driver. For instance, the instruction unit can capture the driver's facial expressions with a camera, detect the state of tension using an emotion estimation algorithm, and provide detailed instructions. The instruction unit can also provide normal instructions if the driver is relaxed. For example, the instruction unit can record the driver's voice, detect the state of relaxation using voice analysis technology, and provide normal instructions. Furthermore, if the driver is tired, the instruction unit can provide concise instructions to reduce the driver's burden. For example, the instruction unit can collect the driver's biometric data (heart rate and skin electrical activity) with sensors, detect the state of fatigue using an emotion estimation algorithm, and provide concise instructions. This allows the level of detail in the instructions to be optimized according to the driver's state. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. The generating AI may be, but is not limited to, text generating AI (e.g., LLM) or multimodal generating AI. Some or all of the processing described above in the instruction unit may be performed using AI, or not using AI. For example, the instruction unit may input driver emotion data into the generating AI and have the generating AI adjust the level of detail of the instructions.

[0116] The instruction unit can select the optimal display method to reduce the driver's visual burden when displaying instructions. For example, the instruction unit can use highly visible colors and fonts when displaying instructions to reduce the driver's visual burden. The instruction unit can also reduce the driver's visual burden by adopting simple and easy-to-understand movements when displaying instructions. For example, the instruction unit can display the movement of the instructions with simple and easy-to-understand animations to reduce the driver's visual burden. The instruction unit can also select the optimal display position when displaying instructions, taking into account the driver's eye-tracking information. For example, the instruction unit optimizes the display position of the instructions based on the driver's eye-tracking data. This reduces the driver's visual burden and helps them understand the instructions. Some or all of the above processing in the instruction unit may be performed using AI, or not. For example, the instruction unit can input the driver's eye-tracking data into a generating AI and have the generating AI select the optimal display method.

[0117] The instruction unit can improve the accuracy of its display by referring to the driver's past display history when displaying instructions. For example, the instruction unit can provide instructions that are easy for the driver to understand based on past display history. For example, the instruction unit can select the display method for instructions that is easiest for the driver to understand based on past display history data. The instruction unit can also analyze past display history and improve instructions that are easily misunderstood by the driver. For example, the instruction unit can improve the expression of instructions that are easily misunderstood based on past display history data. The instruction unit can also refer to past display history and select the display method for instructions that is easiest for the driver to understand. For example, the instruction unit can select the optimal display method for instructions based on past display history data. In this way, the accuracy of the instruction display can be improved by considering past display history. Some or all of the above processing in the instruction unit may be performed using AI, for example, or without AI. For example, the instruction unit can input past display history data into a generating AI and have the generating AI perform the improvement of display accuracy.

[0118] The instruction unit can estimate the driver's emotions and determine the priority of instructions based on the estimated emotions. For example, if the driver is tense, the instruction unit will prioritize displaying important instructions. For instance, the instruction unit may capture the driver's facial expression with a camera, detect the state of tension using an emotion estimation algorithm, and prioritize displaying important instructions. The instruction unit can also display instructions in the normal priority order if the driver is relaxed. For example, the instruction unit may record the driver's voice, detect the state of relaxation using voice analysis technology, and prioritize displaying instructions. Furthermore, if the driver is tired, the instruction unit can prioritize displaying concise and important instructions. For example, the instruction unit may collect the driver's biometric data (heart rate and skin electrical activity) with sensors, detect the state of fatigue using an emotion estimation algorithm, and prioritize displaying concise and important instructions. This allows for the optimization of instruction priorities according to the driver's state. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. The generating AI may be a text generating AI (e.g., LLM) or a multimodal generating AI, but is not limited to such examples. Some or all of the processing described above in the instruction unit may be performed using AI, or not using AI. For example, the instruction unit may input driver emotion data into the generating AI and have the generating AI determine the priority of the instructions.

[0119] The instruction unit can select the optimal display position when displaying instructions by utilizing the driver's gaze information. For example, if the driver is looking straight ahead, the instruction unit will display the instructions forward. For example, the instruction unit will display the instructions forward based on the driver's eye-tracking data. The instruction unit can also display instructions to the left or right if the driver is looking left or right. For example, the instruction unit will display instructions to the left or right based on the driver's eye-tracking data. The instruction unit can also display instructions behind if the driver is looking behind. For example, the instruction unit will display instructions behind based on the driver's eye-tracking data. In this way, the display position of the instructions can be optimized by taking the driver's gaze information into consideration. Some or all of the above processing in the instruction unit may be performed using AI, for example, or without AI. For example, the instruction unit can input the driver's eye-tracking data into a generating AI and have the generating AI select the optimal display position.

[0120] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0121] The driver assistance system may also include a health monitoring unit that monitors the driver's health condition. The health monitoring unit can, for example, measure the driver's heart rate and blood pressure in real time and issue a warning if an abnormality is detected. For instance, if the driver's heart rate increases rapidly, the health monitoring unit can display a warning and prompt the driver to take a break. Similarly, if the driver's blood pressure is high, the health monitoring unit can instruct the driver to relax. Furthermore, the health monitoring unit can record the driver's health data and perform regular health checks. For example, based on the driver's health data, it can analyze changes in their health condition and recommend a visit to a medical institution if necessary. This allows for constant monitoring of the driver's health condition and supports safe driving.

[0122] The driver assistance system may also include a stress monitoring unit that monitors the driver's stress level. The stress monitoring unit can, for example, analyze the driver's facial expressions and voice to estimate the stress level. For instance, if the driver's facial expression is tense, the stress monitoring unit can determine that the stress level is high and display instructions to relax. Similarly, if the driver's voice is agitated, the stress monitoring unit can determine that the stress level is high and display instructions to encourage deep breathing. Furthermore, the stress monitoring unit can record the driver's stress level to support long-term stress management. For example, it can analyze changes in the driver's stress level and provide advice for stress reduction. This allows for continuous monitoring of the driver's stress level and provides a comfortable driving environment.

[0123] The driver assistance system may also include a fatigue monitoring unit that monitors the driver's fatigue level. The fatigue monitoring unit can, for example, analyze the driver's blinking frequency and posture to estimate their fatigue level. For instance, if the driver's blinking frequency increases, the fatigue monitoring unit can determine that fatigue is progressing and display a message prompting them to take a break. Similarly, if the driver's posture is poor, the fatigue monitoring unit can determine that fatigue is progressing and display a message instructing them to correct their posture. Furthermore, the fatigue monitoring unit can record the driver's fatigue level to support long-term fatigue management. For example, it can analyze changes in the driver's fatigue level and suggest appropriate break times. This allows for continuous monitoring of the driver's fatigue level and supports safe driving.

[0124] The driver assistance system may also include a concentration monitoring unit that monitors the driver's concentration. The concentration monitoring unit, for example, analyzes the driver's eye movements and reaction time to estimate their concentration level. For instance, if the driver's gaze frequently wanders, the unit can determine that their concentration is declining and display a warning. Similarly, if the driver's reaction time is slow, the unit can determine that their concentration is declining and display a warning to take a break. Furthermore, the concentration monitoring unit can record the driver's concentration to support long-term concentration management. For example, it can analyze changes in the driver's concentration and provide advice to maintain it. This allows for continuous monitoring of the driver's concentration and supports safe driving.

[0125] The driver assistance system may also include a driving style monitoring unit that monitors the driver's driving style. The driving style monitoring unit, for example, analyzes the driver's acceleration and braking operations and evaluates their driving style. For instance, if the driver frequently performs sudden acceleration or braking, the driving style monitoring unit may determine that the driving style is rough and display instructions encouraging safe driving. Conversely, if the driver is driving smoothly, the driving style monitoring unit may determine that the driving style is good and display a message praising the driver. Furthermore, the driving style monitoring unit can record the driver's driving style and support long-term improvement of that style. For example, it can analyze changes in the driver's driving style and provide advice for safe driving. This allows for continuous monitoring of the driver's driving style and supports safe driving.

[0126] The driver assistance system may also include a driving history analysis unit that analyzes the driver's driving history. For example, the driving history analysis unit can analyze the driver's driving tendencies based on past driving data and support improvements to their driving style. For instance, if a driver has frequently exceeded the speed limit in the past, the driving history analysis unit can identify this tendency and provide advice to encourage safer driving. Furthermore, if a driver has been involved in an accident in the past, the driving history analysis unit can analyze the cause of the accident and provide advice to prevent recurrence. In addition, the driving history analysis unit can support improvements in the driver's driving skills based on their driving history. For example, it can analyze the driver's driving history and propose a training plan to improve their driving skills. This allows for the use of the driver's driving history to support safer driving.

[0127] The driver assistance system may also include an environment adjustment unit that estimates the driver's emotions and adjusts the driving environment based on those emotions. For example, if the driver is tense, the environment adjustment unit may soften the interior lighting to provide a relaxing environment. For instance, it might capture the driver's facial expressions with a camera, detect the state of tension using an emotion estimation algorithm, and adjust the lighting accordingly. It can also return to normal lighting when the driver is relaxed. For example, it might record the driver's voice, detect the state of relaxation using voice analysis technology, and return the lighting to normal. Furthermore, if the driver is tired, the environment adjustment unit can appropriately adjust the temperature inside the vehicle to provide a comfortable environment. For example, it might collect the driver's biometric data (heart rate and skin electrical activity) with sensors, detect the state of fatigue using an emotion estimation algorithm, and adjust the temperature accordingly. This allows the driving environment to be optimized according to the driver's emotions.

[0128] The driver assistance system may also include a skill evaluation unit that assesses the driver's driving skills. For example, the skill evaluation unit analyzes the driver's acceleration and braking operations to evaluate their driving skills. If the driver performs smooth acceleration and braking, the skill evaluation unit can determine that their driving skills are high and display a message praising the driver. Conversely, if the driver frequently performs sudden acceleration or braking, the skill evaluation unit can determine that their driving skills are low and provide advice for improvement. Furthermore, the skill evaluation unit can record the driver's driving skills to support long-term skill improvement. For example, it can analyze changes in the driver's driving skills and propose a training plan for skill improvement. This allows for continuous monitoring of the driver's driving skills and supports safe driving.

[0129] The driver assistance system may further include an instruction adjustment unit that estimates the driver's emotions and adjusts the content of driving instructions based on the estimated emotions. For example, if the driver is tense, the instruction adjustment unit can provide detailed driving instructions to give the driver a sense of security. For example, it can capture the driver's facial expression with a camera, detect the state of tension using an emotion estimation algorithm, and provide detailed driving instructions. It can also provide normal driving instructions if the driver is relaxed. For example, it can record the driver's voice, detect the state of relaxation using voice analysis technology, and provide normal driving instructions. Furthermore, if the driver is tired, it can provide concise driving instructions to reduce the driver's burden. For example, it can collect the driver's biometric data (heart rate and skin electrical activity) with sensors, detect the state of fatigue using an emotion estimation algorithm, and provide concise driving instructions. This allows the content of driving instructions to be optimized according to the driver's emotions.

[0130] The driver assistance system may also include a performance evaluation unit that evaluates the driver's driving performance. For example, the performance evaluation unit analyzes the driver's acceleration and braking operations to assess driving performance. If the driver performs smooth acceleration and braking, the performance evaluation unit can determine that the driving performance is high and display a message praising the driver. Conversely, if the driver frequently performs sudden acceleration or braking, the performance evaluation unit can determine that the driving performance is low and provide advice for improvement. Furthermore, the performance evaluation unit can record the driver's driving performance to support long-term performance improvement. For example, it can analyze changes in the driver's driving performance and propose a training plan for performance improvement. This allows for continuous monitoring of the driver's driving performance and supports safe driving.

[0131] The following briefly describes the processing flow for example form 2.

[0132] Step 1: The detection unit detects sirens. Sirens include those from ambulances, fire trucks, and police vehicles. The detection unit uses voice recognition technology to detect sirens and analyzes surrounding environmental sounds to identify the siren sound. For example, it filters out surrounding traffic noise and wind noise to detect the siren sound. Step 2: The analysis unit analyzes the siren detected by the detection unit. The analysis unit analyzes the frequency and volume of the siren to identify the type of siren. It can also identify the source of the siren based on the time difference in the arrival of the siren's sound waves. Step 3: The conversion unit converts the siren analyzed by the analysis unit into sign language. The conversion unit can also use a sign language dictionary to convert the content of the siren into sign language and display the sign language movements as a 3D animation. Step 4: The display unit displays the sign language converted by the conversion unit. The display unit displays the sign language on the head-up display and can adjust the display position and size. For example, it adjusts the display position of the sign language to match the driver's line of sight. Some or all of the processing in the display unit may be performed using AI.

[0133] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0134] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0135] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0136] For example, the detection unit is implemented by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14. For example, the conversion unit is implemented by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14. For example, the display unit is implemented by the display 40A of the smart device 14 or the specific processing unit 290 of the data processing device 12. The correspondence between each unit and the device or control unit is not limited to the examples described above, and various modifications are possible.

[0137] [Second Embodiment] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0138] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0139] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0140] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0141] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0142] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0143] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0144] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing by the processor 28. The storage 32 stores the specific processing program 56.

[0145] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0146] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0147] In the smart glasses 214, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0148] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0149] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0150] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0151] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0152] For example, the detection unit is implemented by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214. For example, the conversion unit is implemented by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214. For example, the display unit is implemented by the display 40A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. The correspondence between each unit and the device or control unit is not limited to the examples described above, and various modifications are possible.

[0153] [Third Embodiment] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0154] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0155] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0156] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0157] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0158] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0159] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0160] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0161] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0162] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0163] In the headset terminal 314, specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50 and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific program 60 executed on the RAM 48. The headset terminal 314 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0164] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0165] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0166] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0167] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0168] For example, the detection unit is implemented by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314. For example, the conversion unit is implemented by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314. For example, the display unit is implemented by the display 343 of the headset terminal 314 or the specific processing unit 290 of the data processing device 12. The correspondence between each unit and the device or control unit is not limited to the examples described above, and various modifications are possible.

[0169] [Fourth Embodiment] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0170] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0171] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0172] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0173] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0174] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS image sensor or CCD image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0175] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0176] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. The robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0177] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0178] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0179] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0180] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in storage 50. Processor 46 reads the specific program 60 from storage 50 and executes it on RAM 48. The specific processing is achieved by processor 46 acting as a control unit 46A according to the specific program 60 executed on RAM 48. Robot 414 also has data generation model 58 and emotion identification model 59, similar to those of the robot, and can perform processing similar to that of the specific processing unit 290 using these models.

[0181] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0182] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0183] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0184] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or an external device, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0185] For example, the detection unit is implemented by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414. For example, the conversion unit is implemented by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414. For example, the display unit is implemented by the display 40A of the robot 414 or the specific processing unit 290 of the data processing device 12. The correspondence between each unit and the device or control unit is not limited to the examples described above, and various modifications are possible.

[0186] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0187] Figure 9 shows the emotion map 400, in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0188] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0189] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0190] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, and motorcycles, emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0191] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0192] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0193] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0194] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0195] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0196] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0197] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0198] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0199] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0200] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0201] Furthermore, although the above-described examples were divided into four embodiments, some or all of these embodiments may be combined. Also, the smart device 14, smart glasses 214, headset terminal 314, and robot 414 are just examples, and they may be combined, or other devices may be used. Also, although the above-described examples were divided into two embodiments, Embodiment 1 and Embodiment 2, these may be combined.

[0202] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and other things that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0203] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0204] (Note 1) A detection unit that detects the siren, An analysis unit analyzes the siren detected by the aforementioned detection unit, A conversion unit that converts the siren analyzed by the aforementioned analysis unit into sign language, The system includes a display unit that displays the sign language converted by the conversion unit. A system characterized by the following features. (Note 2) It features an animation section that displays sign language movements as animations. The system described in Appendix 1, characterized by the features described herein. (Note 3) It is equipped with an adjustment unit to adjust the display position and size, allowing the display content to be provided in an appropriate position and size for the driver. The system described in Appendix 1, characterized by the features described herein. (Note 4) Equipped with an instruction unit that displays specific instructions. The system described in Appendix 1, characterized by the features described herein. (Note 5) The aforementioned display unit is Displaying sign language on a head-up display The system described in Appendix 1, characterized by the features described herein. (Note 6) The detection unit, Detects emergency vehicle sirens. The system described in Appendix 1, characterized by the features described herein. (Note 7) The detection unit, The system estimates the driver's emotions and adjusts the siren detection sensitivity based on those emotions. The system described in Appendix 1, characterized by the features described herein. (Note 8) The detection unit, When a siren is detected, the system analyzes ambient sounds and performs filtering to prevent false detections. The system described in Appendix 1, characterized by the features described herein. (Note 9) The detection unit, When a siren is detected, the detection priority is set based on the vehicle's speed or location information. The system described in Appendix 1, characterized by the features described herein. (Note 10) The detection unit, The system estimates the driver's emotions and selects the type of siren to detect based on those emotions. The system described in Appendix 1, characterized by the features described herein. (Note 11) The detection unit, When a siren is detected, the system selects the optimal detection method by referring to the driver's past driving history. The system described in Appendix 1, characterized by the features described herein. (Note 12) The detection unit, When detecting a siren, the accuracy of detection is improved based on the driver's gaze information. The system described in Appendix 1, characterized by the features described herein. (Note 13) The aforementioned analysis unit, The system estimates the driver's emotions and adjusts the level of detail in the analysis based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 14) The aforementioned analysis unit, When analyzing sirens, the analysis algorithm is optimized by referring to past analysis data. The system described in Appendix 1, characterized by the features described herein. (Note 15) The aforementioned analysis unit, When analyzing sirens, prioritize the analysis based on the surrounding traffic conditions. The system described in Appendix 1, characterized by the features described herein. (Note 16) The aforementioned analysis unit, The system estimates the driver's emotions and adjusts how the analysis results are displayed based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 17) The aforementioned analysis unit, When analyzing sirens, the accuracy of the analysis is improved by referencing past driver response data. The system described in Appendix 1, characterized by the features described herein. (Note 18) The aforementioned analysis unit, When analyzing sirens, the accuracy of the analysis is improved based on information from the vehicle's internal sensors. The system described in Appendix 1, characterized by the features described herein. (Note 19) The conversion unit is The system estimates the driver's emotions and adjusts the sign language conversion speed based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 20) The conversion unit is When converting to sign language, select the optimal conversion method to reduce the driver's visual burden. The system described in Appendix 1, characterized by the features described herein. (Note 21) The conversion unit is Improve the accuracy of the conversion based on the driver's past sign language comprehension level. The system described in Appendix 1, characterized by the features described herein. (Note 22) The conversion unit is The system estimates the driver's emotions and adjusts the sign language expression based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 23) The conversion unit is When converting to sign language, the driver's eye-tracking information is used to select the optimal display position. The system described in Appendix 1, characterized by the features described herein. (Note 24) The conversion unit is When converting to sign language, the conversion priority is set based on the driver's past driving history. The system described in Appendix 1, characterized by the features described herein. (Note 25) The aforementioned display unit is The system estimates the driver's emotions and adjusts the level of detail displayed based on those emotions. The system described in Appendix 1, characterized by the features described herein. (Note 26) The aforementioned display unit is When displaying sign language, select the optimal display method to reduce the driver's visual burden. The system described in Appendix 1, characterized by the features described herein. (Note 27) The aforementioned display unit is When displaying sign language, the accuracy of the display is improved based on the driver's past display history. The system described in Appendix 1, characterized by the features described herein. (Note 28) The aforementioned display unit is The system estimates the driver's emotions and determines the priority of the displayed content based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 29) The aforementioned display unit is When displaying sign language, the optimal display position is selected by utilizing the driver's eye-tracking information. The system described in Appendix 1, characterized by the features described herein. (Note 30) The aforementioned display unit is When displaying sign language, the display priority is set based on the driver's past driving history. The system described in Appendix 1, characterized by the features described herein. (Note 31) The aforementioned animation unit is It estimates the driver's emotions and adjusts the animation speed based on the estimated emotions. The system described in Appendix 2, characterized by the features described herein. (Note 32) The aforementioned animation unit is When displaying sign language animations, select the optimal animation method to reduce the driver's visual burden. The system described in Appendix 2, characterized by the features described herein. (Note 33) The aforementioned animation unit is When displaying sign language animations, the system improves the accuracy of the animations by referencing the driver's past understanding of sign language. The system described in Appendix 2, characterized by the features described herein. (Note 34) The aforementioned animation unit is The system estimates the driver's emotions and adjusts the animation's presentation based on those estimated emotions. The system described in Appendix 2, characterized by the features described herein. (Note 35) The aforementioned animation unit is When displaying sign language animations, the driver's eye-tracking information is used to select the optimal display position. The system described in Appendix 2, characterized by the features described herein. (Note 36) The adjustment unit is, It estimates the driver's emotions and adjusts the display position and size based on the estimated emotions. The system described in Appendix 3, characterized by the features described herein. (Note 37) The adjustment unit is, When adjusting the display position and size, select the optimal adjustment method to reduce the driver's visual burden. The system described in Appendix 3, characterized by the features described herein. (Note 38) The adjustment unit is, The system estimates the driver's emotions and determines the priority of the displayed content based on those estimated emotions. The system described in Appendix 3, characterized by the features described herein. (Note 39) The adjustment unit is, When adjusting the display position and size, the system utilizes the driver's eye-tracking information to select the optimal display position. The system described in Appendix 3, characterized by the features described herein. (Note 40) The indicator unit is, The system estimates the driver's emotions and adjusts the level of detail in the instructions based on those emotions. The system described in Appendix 4, characterized by the features described herein. (Note 41) The indicator unit is, When displaying instructions, select the optimal display method to reduce the driver's visual burden. The system described in Appendix 4, characterized by the features described herein. (Note 42) The indicator unit is, When displaying instructions, the system improves display accuracy by referencing the driver's past display history. The system described in Appendix 4, characterized by the features described herein. (Note 43) The indicator unit is, The system estimates the driver's emotions and prioritizes instructions based on those emotions. The system described in Appendix 4, characterized by the features described herein. (Note 44) The indicator unit is, When displaying instructions, the system utilizes the driver's eye-tracking information to select the optimal display position. The system described in Appendix 4, characterized by the features described herein. [Explanation of Symbols]

[0205] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. A detection unit that detects the siren, An analysis unit analyzes the siren detected by the aforementioned detection unit, A conversion unit that converts the siren analyzed by the aforementioned analysis unit into sign language, The system includes a display unit that displays the sign language converted by the conversion unit. A system characterized by the following features.

2. It features an animation section that displays sign language movements as animations. The system according to feature 1.

3. It is equipped with an adjustment unit to adjust the display position and size, allowing the display content to be provided in an appropriate position and size for the driver. The system according to feature 1.

4. Equipped with an instruction unit that displays specific instructions. The system according to feature 1.

5. The aforementioned display unit is Displaying sign language on a head-up display The system according to feature 1.

6. The detection unit, Detects emergency vehicle sirens. The system according to feature 1.

7. The detection unit, The system estimates the driver's emotions and adjusts the siren detection sensitivity based on those emotions. The system according to feature 1.

8. The detection unit, When a siren is detected, the system analyzes ambient sounds and performs filtering to prevent false detections. The system according to feature 1.

Images