System

Abstract

To provide safer and more intuitive driving support to a driver.SOLUTION: A driving assistance system includes an AR display device, a data collection unit including a sensor and a camera, a data analysis unit configured to analyze environmental data in real time, an information generation unit configured to generate navigation information and a warning based on an analysis result, a display control unit configured to overlay the information on the AR display device, a voice interaction unit configured to analyze a voice instruction and generate a response, and a route suggestion unit configured to suggest an optimal route based on real-time traffic information and a driving history.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional navigation systems have the problem of forcing drivers to look at the in-car display, distracting them from the road. This can result in reduced safety. Furthermore, the fixed information provided lacks customized assistance based on real-time traffic conditions and individual driving history. This invention aims to solve these problems and provide safer, more intuitive driving assistance to drivers. [Means for solving the problem]

[0005] The present invention provides a driving assistance system including an AR display device, a data collection means equipped with sensors and a camera, a data analysis means for analyzing environmental data in real time, an information generation means for generating navigation information and warnings based on the analysis results, a display control means for overlaying the information on the AR display device, a voice interaction means for analyzing voice instructions and generating responses, and a route suggestion means for suggesting an optimal route based on real-time traffic information and driving history. This system allows the driver to obtain necessary information without taking their eyes off the road, realizing real-time situational awareness and personalized driving assistance.

[0006] An "AR display device" is a device that uses augmented reality technology to overlay digital information onto the real world.

[0007] The "data collection means" is a means for acquiring surrounding environmental data in real time using sensors and cameras.

[0008] The "data analysis means" is a means used to analyze acquired environmental data in real time and grasp the surrounding situation.

[0009] The "information generating means" is a means for generating navigation information and warnings based on the analysis results of the data analyzing means.

[0010] The "display control means" is a means for overlay-displaying the generated navigation information and warnings on the AR display device.

[0011] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating a response in response thereto.

[0012] The "route suggestion means" is a means for suggesting the optimum route to the user based on real-time traffic information and driving history. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention is a driving assistance system that combines an AR display device with data collection means, data analysis means, information generation means, display control means, voice interaction means, and route suggestion means. This system allows drivers to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving.

[0035] Program processing explanation

[0036] 1. Data collection and environmental awareness

[0037] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed.

[0038] The acquired data is sent to the server in real time.

[0039] 2. Data analysis and information generation

[0040] The server analyzes the received environmental data, which includes using image recognition algorithms and traffic prediction models.

[0041] Based on the analysis results, the server determines the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents).

[0042] The server generates optimal navigation information and necessary warning information.

[0043] 3. Information overlay

[0044] The server sends the generated navigation information and warnings to the terminal.

[0045] The device then displays the received information on the AR display device. Specifically, images and text are projected onto the AR glasses' display so that they are overlaid directly on the user's field of vision.

[0046] The user continues driving while visually checking this information.

[0047] 4. Voice Dialogue Interface

[0048] Users can ask questions or give instructions by voice while driving, for example, "Where is the next right turn?"

[0049] The terminal analyzes the user's voice using a voice recognition engine and sends the analysis results to the server as text data.

[0050] The server analyzes the voice instructions and generates appropriate information or responses, which are then sent back to the terminal, which then provides the information to the user as audio or visual information.

[0051] 5. Personalized navigation suggestions

[0052] The server analyzes past driving data and user preferences based on a database, learning, for example, the user's preferred routes and areas to avoid.

[0053] The server calculates and proposes the optimal route in combination with real-time traffic information, allowing users to select the most efficient and safe route at that time.

[0054] The device will overlay the suggested navigation route on the AR display and also provide an audio notification.

[0055] Specific examples

[0056] Scenario 1: Navigation while driving

[0057] After the user sets their destination, they begin driving. If they need to turn left at the next intersection, the server analyzes that information and sends it to the device. Based on the received information, the device overlays the instruction "Turn left at the next intersection" on the AR display device.

[0058] Scenario 2: Sudden change in road conditions

[0059] When the device detects a traffic jam on the road ahead using its camera, the data is sent to the server. The server analyzes the traffic jam information and calculates a new route to avoid it. The new route information is sent to the device, which then displays it as an overlay on the AR display and notifies the user by voice.

[0060] This allows the driving assistance system of the present invention to provide safe and intuitive assistance to the driver in real time.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed.

[0064] Step 2:

[0065] The device processes and temporarily stores acquired environmental data in real time, including digital image processing and sensor data analysis.

[0066] Step 3:

[0067] The data collected by the terminal is sent to the server using a secure, low-latency communication protocol.

[0068] Step 4:

[0069] The server analyzes the received environmental data, using image recognition algorithms and traffic prediction models to understand the surroundings.

[0070] Step 5:

[0071] Based on the analysis results, the server determines the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents).

[0072] Step 6:

[0073] The server generates optimal navigation information and necessary warning information, such as specific instructions such as "turn left at the next intersection."

[0074] Step 7:

[0075] The server generates navigation information and alerts and sends them back to the device. The data is transmitted securely with low latency.

[0076] Step 8:

[0077] The navigation information received by the device is displayed on the AR display device. Specifically, images and text are projected so that they are overlaid directly on the user's field of view.

[0078] Step 9:

[0079] The user can continue driving while visually checking navigation information and warnings through the AR display device.

[0080] Step 10:

[0081] The user asks questions or gives instructions by voice while driving, for example, using voice commands such as "Where is the next right turn?"

[0082] Step 11:

[0083] The device analyzes the user's voice using a voice recognition engine and converts it into text data, which is then sent to the server.

[0084] Step 12:

[0085] The server analyzes the voice instructions and generates appropriate information or responses, such as "The next right turn is 200 meters away."

[0086] Step 13:

[0087] The server transmits the generated information to the terminal, which then provides the information to the user as audio or visual information.

[0088] Step 14:

[0089] The server analyzes past driving data and user preferences based on a database, and combines this with real-time traffic information to calculate the optimal route.

[0090] Step 15:

[0091] The server sends the proposed new navigation route to the terminal, which then displays it as an overlay on the AR display device and notifies the user by voice.

[0092] In this way, the driving assistance system of the present invention realizes real-time environment recognition, data analysis, generation and display of navigation information, and personalized driving assistance through a voice dialogue interface.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] From the perspective of safe driving, it is important for drivers to obtain necessary information without taking their eyes off the road. However, conventional navigation systems and driver assistance systems require drivers to pay attention to their smartphones or in-car displays, resulting in distraction and an increased risk of accidents. Furthermore, it is difficult to provide appropriate navigation information and warnings in real time, making it difficult for drivers to reach their destination safely and efficiently.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes a means for displaying visual information through an AR display device, a means for analyzing a user's voice input with a voice recognition engine and converting the analysis result into text, a means for analyzing the text data and generating an appropriate response, and a means for learning past driving data and personalized route information for the user to provide an optimal route. This allows the driver to obtain navigation information and warnings in real time without taking their eyes off the road, enabling safer and more intuitive driving.

[0098] An "AR display device" is a device that uses augmented reality technology to overlay information on the user's field of vision.

[0099] "Data collection means" refers to means for collecting surrounding environmental data using sensors and cameras.

[0100] "Data analysis means" refers to a means for analyzing collected environmental data in real time and extracting important information.

[0101] The "information generating means" is a means for generating navigation information and warning information based on the analysis results.

[0102] The "display control means" is a means for overlay-displaying the generated information on the AR display device.

[0103] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating an appropriate response.

[0104] The "route suggestion means" is a means for suggesting the optimal route based on real-time traffic information and driving history.

[0105] The "means for displaying visual information" refers to a means for displaying information directly into the user's field of vision through an AR display device.

[0106] A "voice recognition engine" is an engine that analyzes a user's voice input and converts it into text data.

[0107] The "means for analyzing text data" is a means for generating an appropriate response based on the text data generated by the voice recognition engine.

[0108] The "means for learning personalized route information" is a means for providing an optimal route based on past driving data and user preferences.

[0109] This invention is a driving assistance system that combines an AR display device with data collection means, data analysis means, information generation means, display control means, voice interaction means, and route suggestion means. This system allows drivers to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving.

[0110] Hardware and software used

[0111] Terminal (AR display device): Equipped with a camera and sensors (e.g., LiDAR sensor). The camera captures images at, for example, several tens of frames per second, and the LiDAR sensor obtains distance information about surrounding objects.

[0112] Server: A high-performance computer (e.g., a cloud server) analyzes the data using deep learning models (e.g., TensorFlow), image recognition algorithms (e.g., OpenCV), and traffic prediction models (e.g., Kalman filters).

[0113] Communication methods: Using high-speed networks such as 5G and Wi-Fi to transfer data in real time.

[0114] System processing overview

[0115] 1. Data collection and environmental recognition: The device collects data on the surrounding environment using a camera and sensors mounted on the front of the vehicle. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the LiDAR sensor obtains distance information. The device transmits this data to a server in real time.

[0116] 2. Data analysis and information generation: The server analyzes the received data and uses deep learning and traffic prediction models to identify vehicles and signs. Based on the analysis results, it generates navigation information and necessary warnings.

[0117] 3. Information overlay display: The server sends the generated navigation information and warnings to the device. The device displays the received information as an overlay on the AR display device. The user can continue driving while checking the information without taking their eyes off the road.

[0118] 4. Voice dialogue interface: Users can ask questions or give instructions by voice. The device analyzes the voice using a voice recognition engine and sends the results as text data to the server. The server analyzes the text data, generates an appropriate response, and sends it to the device. The device then provides the information to the user as audio or visual information.

[0119] 5. Personalized navigation suggestions: The server analyzes past driving data and user preferences to suggest optimal routes. It calculates the optimal route by combining it with real-time traffic information and sends that information to the device. The device then overlays the suggested route on the AR display and notifies the user via voice.

[0120] Specific examples

[0121] Scenario 1: Navigation while driving

[0122] After the user sets their destination, they begin driving. If they need to turn left at the next intersection, the server analyzes that information and sends it to the device. Based on the received information, the device overlays the instruction "Turn left at the next intersection" on the AR display device.

[0123] Scenario 2: Sudden change in road conditions

[0124] When the device first detects a traffic jam using its camera, the data is sent to the server. The server analyzes the congestion information and calculates a new route to avoid it. The new route information is sent to the device, which then displays it as an overlay on the AR display and notifies the driver via voice. This allows the system to provide safe and intuitive assistance to drivers in real time.

[0125] Prompt Sentence Examples

[0126] "Where is the next right turn?"

[0127] "Please suggest the best route taking traffic information into consideration."

[0128] "Please show us what signs your camera recognizes."

[0129] In this way, the server, terminal, and user each play their respective roles, collecting, analyzing, and displaying data in real time, creating a system that provides safe and intuitive driving assistance.

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Step 1: Data collection

[0132] The device (AR display device) activates a camera and sensors (e.g., LiDAR sensor) mounted on the front of the vehicle to collect environmental data. Specifically, the camera captures images at tens of frames per second, and the LiDAR sensor obtains distance information of surrounding objects.

[0133] Input: Image data of the area in front of the vehicle and distance data of the surrounding area.

[0134] Output: Collected video and distance data.

[0135] Step 2: Send data

[0136] The device transmits the collected environmental data to a server in real time using 5G or Wi-Fi as the communication protocol, ensuring low latency data transfer.

[0137] Input: Collected video data and distance data.

[0138] Output: The raw environment data sent to the server.

[0139] Step 3: Receiving data

[0140] The server receives the environmental data sent from the terminal and stores it in a buffer for analysis. For example, it uses Apache Kafka for data stream management.

[0141] Input: Environmental data sent from the device.

[0142] Output: Received and buffered environmental data.

[0143] Step 4: Data analysis

[0144] The server analyzes the received environmental data, using deep learning models (e.g., TensorFlow) and image recognition algorithms (e.g., OpenCV) to identify vehicles, signs, pedestrians, etc.

[0145] Input: Buffered environmental data.

[0146] Output: Information about recognized objects (vehicles, signs, pedestrians, etc.).

[0147] Step 5: Information Generation

[0148] The server generates optimal navigation and warning information based on the analysis results, and uses traffic prediction models (e.g., Kalman filter) to evaluate real-time traffic conditions.

[0149] Input: Recognized object information and traffic prediction data.

[0150] Output: Navigation and warning information.

[0151] Step 6: Send information

[0152] The server transmits the generated navigation information and warning information to the terminal in real time.

[0153] Input: Navigation and warning information.

[0154] Output: Navigation and warning information sent to the device.

[0155] Step 7: Display information

[0156] The device overlays the received navigation and warning information on the AR display device, displaying arrows and warning text in the user's field of view.

[0157] Input: Navigation and warning information sent from the server.

[0158] Output: Information displayed on the AR display device.

[0159] Step 8: Recognizing voice input

[0160] Users can ask questions or give instructions by voice while driving, such as "Where is the next right turn?"

[0161] Input: User's voice command.

[0162] Output: Audio instructions.

[0163] Step 9: Audio Analysis

[0164] The device analyzes the user's voice using a voice recognition engine (e.g., Google Speech-to-Text or Amazon Alexa) and sends the analysis results to the server as text data.

[0165] Input: User's voice command.

[0166] Output: Parsed text data.

[0167] Step 10: Generate voice response

[0168] The server analyzes the voice instructions and generates appropriate information or responses, such as "The next right turn is 300 meters away."

[0169] Input: Parsed text data.

[0170] Output: Audio and text responses.

[0171] Step 11: Send a voice response

[0172] The server sends the generated voice response to the terminal, which then presents the information to the user as voice.

[0173] Input: The generated speech response.

[0174] Output: The audio response sent to the device.

[0175] Step 12: Personalized navigation suggestions

[0176] The server analyzes a database based on past driving data and user preferences, and combines this with real-time traffic information to suggest the optimal route.

[0177] Inputs: Historical driving data, user preferences, real-time traffic information.

[0178] Output: personalized optimal route information.

[0179] The information collected, analyzed, and generated at each step is provided in real time to improve driver safety and convenience, allowing drivers to obtain the necessary information in a timely manner without taking their eyes off the road.

[0180] (Application example 1)

[0181] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0182] Conventional driver assistance systems have the problem of compromising safety while driving because drivers have to look away from the road to check navigation information and emergency warnings. Furthermore, they lack the ability to suggest optimal routes based on real-time traffic conditions and the user's driving history, and they also lack functionality to reduce user stress and fatigue.

[0183] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0184] In this invention, the server includes an AR display device, a data collection means equipped with sensors and a camera, a data analysis means for analyzing environmental data in real time, an information generation means for generating navigation information and warnings based on the analysis results, a display control means for overlaying the information on the AR display device, a voice dialogue means for analyzing voice instructions and generating responses, a route proposal means for proposing an optimal route based on real-time traffic information and driving history, a voice and visual information provision means for providing the analysis results as visual and voice information through smart glasses, a voice analysis means for analyzing voice using a voice recognition engine and generating appropriate information and responses, and a route calculation means for calculating and proposing a stress-reducing route based on past driving data and preferences. This allows the driver to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving. Furthermore, personalized route proposals based on the user's driving history and preferences enable efficient and stress-free driving.

[0185] - An "AR display device" is a device that displays digital information overlaid on images from the real world.

[0186] A "sensor" is a device for acquiring physical environmental data.

[0187] A "camera" is a device for capturing images or video using an optical device.

[0188] "Data collection means" is a general term for devices and technologies used to collect environmental data.

[0189] A "data analysis means" is a device or method for analyzing collected data and converting it into meaningful information.

[0190] The "information generation means" is a mechanism that generates navigation information and warnings based on the analyzed data.

[0191] The "display control means" is a mechanism for appropriately overlaying and displaying the generated information on the AR display device.

[0192] A "voice interaction means" is a device or system for analyzing voice input and interacting with a user.

[0193] The "route suggestion method" is a system that calculates and suggests the optimal route based on real-time traffic information and driving history.

[0194] The "audio and visual information providing means" is a mechanism for providing the analysis results to the smart glasses as visual and audio information.

[0195] A "voice recognition engine" is software or hardware that converts voice into text data and analyzes it.

[0196] "Speech analysis means" refers to a device or method that uses a speech recognition engine to analyze speech and generate appropriate information or responses.

[0197] A "stress-reducing route" is an optimal driving route designed to reduce stress and fatigue based on the user's driving history and preferences.

[0198] This invention is a driving assistance system that combines an AR display device with data collection means, voice interaction means, and route suggestion means. This system allows users to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving.

[0199] System Configuration

[0200] AR display devices can take the form of smart glasses or head-mounted displays that are worn by the user and overlay navigation information and warnings onto the real-world scene.

[0201] Data collection methods include cameras (e.g., common webcams) and sensors (e.g., Lidar sensors). These devices capture data about the surrounding environment in real time and transmit it to a server.

[0202] As a data analysis method, the server analyzes the received data and uses image recognition algorithms (e.g., OpenCV) and generative AI models to analyze the environmental data.

[0203] The information generating means generates navigation information and warnings based on the analysis results, and the generated information is sent from the server to the terminal.

[0204] The display control means overlays the generated information on the AR display device, making it easier for the user to visually confirm the information.

[0205] For voice interaction, a speech recognition engine (e.g., the transformers library) is used to analyze the user's voice instructions. The server generates appropriate information and responses and provides them to the user as audio and visual information.

[0206] In the route suggestion method, the server calculates the optimal route based on real-time traffic information and driving history, taking into account past driving data and user preferences.

[0207] Program processing explanation

[0208] The server receives environmental data from cameras and sensors and uses image recognition algorithms to identify roads, vehicle signs, pedestrians, etc., in order to analyze the collected data. Based on the analysis results, it generates appropriate navigation information and warnings.

[0209] The generated information is sent to the AR display device and overlaid on the user's field of view, allowing the user to check the information without taking their eyes off the road.

[0210] In addition, by using the voice interaction function, users can receive driving route suggestions to reduce fatigue and stress. For example, when a user says, "Please tell me the shortest route to my next destination" or "What is the safest route from my current location?", the server calculates appropriate route information and suggests it in real time.

[0211] Specific example explanation

[0212] Scenario 1: Navigation while driving

[0213] The user sets their destination and starts driving. If they need to turn left at the next intersection, the server analyzes that information and displays an overlay on the AR display saying, "Turn left at the next intersection."

[0214] Scenario 2: Sudden change in road conditions

[0215] When the device detects a traffic jam on the road ahead using its camera, the data is sent to the server. The server analyzes the traffic jam information and calculates a new route to avoid it. The new route information is sent to the device, which then displays it as an overlay on the AR display and notifies the user by voice.

[0216] Prompt Sentence Examples

[0217] "Please tell me the shortest route to my next destination."

[0218] "What is the safest route from my current location?"

[0219] "Please suggest a detour route to avoid the congestion ahead."

[0220] This allows the driving assistance system of the present invention to provide safe and intuitive assistance to the driver in real time, thereby improving the user's driving experience.

[0221] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0222] Step 1:

[0223] Data collection and environmental awareness

[0224] When a user starts driving, the camera and sensors attached to the device collect data on the surrounding environment. Specifically, the camera captures road conditions, traffic signs, and pedestrians in real time, while the sensors measure the vehicle's speed and distance. This data is sent as input from the device to the server, which then receives it.

[0225] Input: Camera image data, sensor measurement data

[0226] Output: Environment data sent to the server

[0227] Step 2:

[0228] Data analysis and information generation

[0229] The server analyzes the received environmental data, using image recognition algorithms (e.g., OpenCV) and generative AI models to recognize specific objects (e.g., signs, pedestrians, other vehicles) and analyze their location and movement. It also takes into account real-time traffic information and historical driving data. Based on the analysis results, the server generates navigation information and warnings.

[0230] Input: Environmental data (video data, sensor measurement data), real-time traffic information, driving history data

[0231] Output: Analysis results, navigation information, warning information

[0232] Step 3:

[0233] Information overlay

[0234] The generated navigation information and warnings are sent from the server to the device. The device's AR display device overlays this information in the user's field of view. Specifically, guidance such as "Turn left at the next intersection" is displayed on the AR display device's screen. This step allows the user to check the information without taking their eyes off the road.

[0235] Input: Navigation information, warning information

[0236] Output: Navigation information and warning information overlaid on the AR display device

[0237] Step 4:

[0238] Voice Dialogue Interface

[0239] When a user speaks a question or command while driving, the device analyzes the voice using a voice recognition engine. For example, if the user says, "Please tell me the shortest route to my next destination," the voice data is input into the voice recognition engine and converted into text data. The converted data is sent to the server, which generates the appropriate information or response.

[0240] Input: User's voice commands

[0241] Output: Parsed text data, response information

[0242] Step 5:

[0243] Responses and information

[0244] The server analyzes the voice instructions and generates response information, which is then sent to the device. The device then overlays this information on the AR display and simultaneously notifies the user by voice, allowing the user to receive information both visually and audibly.

[0245] Input: Parsed text data, response information

[0246] Output: Response information overlaid on the AR display device, audio notification

[0247] Step 6:

[0248] Personalized route suggestions

[0249] The server calculates a stress-reducing route based on past driving data and the user's preferences. It uses an algorithm that prioritizes routes that avoid traffic jams and routes that the user prefers. The calculated route information is sent to the device, which then displays it on an AR display and notifies the user via voice.

[0250] Input: Driving history data, user preferences, real-time traffic information

[0251] Output: Calculated route information, overlaid route information, audio notification

[0252] As a result, driver assistance systems can provide accurate, real-time navigation information and warnings to support users as they drive, allowing them to continue driving safely and intuitively without taking their eyes off the road.

[0253] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0254] This invention is a driving assistance system that combines an AR display device, data collection means, data analysis means, information generation means, display control means, voice dialogue means, route suggestion means, and an emotion engine. This system allows drivers to receive real-time navigation information and warnings, as well as personalized driving assistance based on emotion recognition, without taking their eyes off the road.

[0255] Program processing explanation

[0256] 1. Data collection and environmental awareness

[0257] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed.

[0258] Furthermore, the device collects biometric data such as the user's voice, facial expressions, and heart rate, and obtains data for analyzing the user's emotional state using an emotion engine.

[0259] The acquired data is sent to the server in real time.

[0260] 2. Data analysis and information generation

[0261] The server analyzes the received environmental data, which includes using image recognition algorithms and traffic prediction models.

[0262] The server uses an emotion engine to analyze the user's emotional state, and based on the analysis results, identifies the user's emotional state, such as whether they are tense or relaxed.

[0263] Based on the analysis results, the server identifies the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents), and adjusts navigation information and warnings according to the user's emotional state.

[0264] The server generates optimal navigation information, necessary warning information, and personalized information based on emotional state.

[0265] 3. Information overlay

[0266] The server transmits the generated navigation information, warnings, and information based on emotion recognition to the terminal.

[0267] The device then displays the received information on the AR display device. Specifically, images and text are projected onto the user's field of view as an overlay.

[0268] The user continues driving while visually checking this information.

[0269] 4. Voice Dialogue Interface

[0270] Users can ask questions or give instructions by voice while driving, for example, "Where is the next right turn?"

[0271] The terminal analyzes the user's voice using a voice recognition engine and sends the analysis results to the server as text data.

[0272] The server analyzes the voice instructions and generates appropriate information or responses, which are then sent back to the terminal, which then provides the information to the user as audio or visual information.

[0273] Through the emotion engine, if the user is tense, the server generates a response that has a relaxation effect.

[0274] 5. Personalized navigation suggestions

[0275] The server analyzes past driving data and user preferences based on a database, learning, for example, the user's preferred routes and areas to avoid.

[0276] The server combines real-time traffic information with the user's emotional state to calculate and suggest the optimal route.

[0277] The device will overlay the suggested navigation route on the AR display and also provide an audio notification.

[0278] Specific examples

[0279] Scenario 1: Navigation while driving

[0280] After setting a destination, the user begins driving. If it is necessary to turn left at the next intersection, the server analyzes that information and sends it to the device. Based on the received information, the device overlays the instruction "Turn left at the next intersection" on the AR display device, and if the user is nervous, displays additional instructions such as "Please drive safely."

[0281] Scenario 2: Sudden change in road conditions

[0282] When the device detects a traffic jam on the road ahead using its camera, the data is sent to the server. The server analyzes the traffic information and calculates a new route to avoid it. The new route information is sent to the device, which then overlays it on the AR display and notifies the user via voice. If the user is feeling stressed, the device will also suggest a relaxing route.

[0283] As a result, the driving assistance system of the present invention can realize real-time environment recognition, data analysis, generation and display of navigation information, and personalized driving assistance through a voice dialogue interface, and can also provide appropriate support based on the user's emotional state.

[0284] The processing flow will be explained below.

[0285] Step 1:

[0286] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed. It also collects biometric data such as the user's voice, facial expression, and heart rate.

[0287] Step 2:

[0288] The device processes and temporarily stores acquired environmental and biometric data in real time, including digital image processing and biometric data analysis.

[0289] Step 3:

[0290] The data collected by the terminal is sent to the server using a secure, low-latency communication protocol.

[0291] Step 4:

[0292] The server analyzes the received environmental data, using image recognition algorithms and traffic prediction models to identify surrounding conditions (e.g., traffic jams, construction zones, accidents).

[0293] Step 5:

[0294] The server uses an emotion engine to analyze the user's biometric data, including voice tone analysis, facial expression recognition, and heart rate analysis, to identify the user's emotional state (e.g., tension, stress, relaxation).

[0295] Step 6:

[0296] Based on the analysis results, the server combines the current location, direction of travel, and surrounding conditions to generate optimal navigation information and necessary warning information. The content of the information and warnings is adjusted according to the user's emotional state.

[0297] Step 7:

[0298] The server generates navigation information, warnings, and personalized information based on emotion recognition, and sends the data securely with low latency to the device.

[0299] Step 8:

[0300] The device displays the received information on the AR display device. Specifically, images and text are projected as an overlay directly onto the user's field of vision. For example, instructions such as "Turn left at the next intersection" or "Drive safely" are included.

[0301] Step 9:

[0302] The user can continue driving while visually checking navigation information and warnings through the AR display device, and can also receive emotionally appropriate advice on how to ease tension as needed.

[0303] Step 10:

[0304] The user asks questions or gives instructions by voice while driving, for example, using voice commands such as "Where is the next right turn?"

[0305] Step 11:

[0306] The device analyzes the user's voice using a voice recognition engine and converts it into text data, which is then sent to the server.

[0307] Step 12:

[0308] The server analyzes the voice instructions and generates appropriate information and responses, such as "The next right turn is 200 meters away." If the emotion engine detects stress or tension in the user, it generates a response with a relaxation effect.

[0309] Step 13:

[0310] The server transmits the generated information to the terminal, which then provides the information to the user as audio or visual information.

[0311] Step 14:

[0312] The server analyzes a database of past driving data and user preferences, and combines real-time traffic information and the user's emotional state to calculate the optimal route, taking into account, for example, scenic routes with a relaxing effect.

[0313] Step 15:

[0314] The server sends the proposed new navigation route to the device, which then displays it as an overlay on the AR display and notifies the user by voice. The guidance may include guidance designed to have a relaxing effect.

[0315] In this way, the driving assistance system of the present invention realizes real-time environment recognition, data analysis, generation and display of navigation information, and personalized driving assistance based on emotional state.

[0316] Example 2

[0317] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0318] While conventional driving assistance systems can provide environmental data and traffic information, they have the problem of being unable to provide personalized assistance that takes into account the user's emotional state. As a result, appropriate assistance may not be provided when the driver is feeling nervous or stressed, which could result in a decrease in driving safety.

[0319] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an emotion recognition means that collects biometric data of the user and analyzes the emotional state, an information generation means that generates personalized information based on the emotional state, and a route suggestion means that suggests an optimal route based on real-time traffic information and driving history. This enables personalized driving assistance that takes the user's emotional state into consideration.

[0320] An "AR display device" is a device that uses augmented reality technology to display information and overlay it directly on a user's field of view.

[0321] "Data collection means" refers to means for acquiring environmental data and biological data using sensors and cameras.

[0322] "Data analysis means" refers to means for analyzing collected data in real time and extracting necessary information.

[0323] The "information generating means" is a means for generating navigation information and warnings based on the results of data analysis.

[0324] The "display control means" is a means for overlay-displaying the generated information on the AR display device.

[0325] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating an appropriate response.

[0326] The "route suggestion means" is a means for suggesting the optimal route based on real-time traffic information and driving history.

[0327] The "emotion recognition means" is a means for collecting biometric data of a user and analyzing the emotional state.

[0328] The "means for generating personalized information" is a means for generating information that is optimal for the user based on the analyzed emotional state.

[0329] The present invention relates to a driving assistance system that provides advanced assistance to drivers by combining an AR display device, data collection means, data analysis means, information generation means, display control means, voice dialogue means, route suggestion means, and emotion recognition means. This system allows drivers to receive real-time navigation information and warnings, as well as personalized driving assistance based on emotion recognition, without having to take their eyes off the road.

[0330] First, the device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures images of roads, vehicles, signs, pedestrians, etc., and the sensors obtain data such as distance and speed in real time. Biometric data such as the user's voice, facial expression, and heart rate are also collected. This provides data for the emotion recognition means (described later) to analyze the user's emotional state.

[0331] The collected data is sent to a server in real time. The server analyzes the received environmental data using image recognition algorithms and traffic prediction models. For example, a deep learning algorithm can be used to identify whether a pedestrian is crossing the street. An emotion engine can also be used to analyze the user's emotional state. This can identify whether the user is tense or relaxed.

[0332] The server uses the analysis results to determine the current location, direction of travel, and surrounding conditions. This includes combining received GPS data with image analysis results to accurately pinpoint the current location. It can also tailor navigation information and warnings based on the user's emotional state. For example, if the user is feeling nervous, it can suggest playing music to reduce stress.

[0333] The generated optimal navigation information and necessary warning information are sent to the device, which then displays it on the AR display and overlays it directly on the user's field of view. For example, an arrow can be displayed indicating a right turn point at the next intersection. The user can visually confirm this information and continue driving safely.

[0334] It also has a voice dialogue interface, allowing users to ask questions or give instructions by voice while driving. For example, it can issue prompts such as, "Where is the next right turn?" These voice instructions are analyzed by a voice recognition engine and sent to the server as text data. The server analyzes the voice instructions, generates appropriate information and responses, and sends them back to the device. The device then provides this information to the user again as audio or visual information. For example, it can provide specific guidance such as, "Turn right at the intersection 300 meters ahead."

[0335] It also uses emotion recognition to generate relaxing responses if the user is feeling tense, such as "Take a deep breath and relax." It can also learn from past driving data and user preferences, and combine this with real-time traffic information to calculate optimal routes. This allows it to provide personalized navigation suggestions based on the user's driving history.

[0336] Prompt Sentence Examples

[0337] "Where is the next right turn?"

[0338] "Show navigation information for turning left at the next intersection."

[0339] "Calculate a new route to avoid traffic jams."

[0340] "Generate a response that relaxes the user when they are tense."

[0341] The system improves driver concentration and safety, while providing personalized driving assistance.

[0342] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0343] Step 1: Data collection and environmental awareness

[0344] The device (AR display device) uses a camera and sensors to scan the surrounding environment. As input data, it uses image data (roads, vehicles, signs, pedestrians) captured by the camera and distance and speed data obtained by the sensors. This provides initial data on road conditions and direction of travel. Biometric data such as the user's voice, facial expression, and heart rate are also collected at the same time. Specifically, the camera captures an image of the road ahead, and the sensors measure the vehicle's speed and position. This data is sent to the server in real time.

[0345] Step 2: Data analysis

[0346] The server analyzes the input environmental and biometric data. For environmental data, it uses an image recognition algorithm to recognize road signs and pedestrians from captured images. Specifically, it uses a deep learning algorithm to identify, for example, whether a pedestrian is crossing a street. For biometric data, it uses an emotion engine to analyze the user's emotional state, such as tension or relaxation, from facial expression data. The results of this analysis are used by the server to generate navigation and warning information.

[0347] Step 3: Information generation

[0348] The server generates navigation information and warnings based on the results of data analysis. Specifically, it generates right-turn instructions at the next intersection and warnings if there are obstacles ahead based on the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents). Depending on the user's emotional state, it also suggests playing relaxing music. This information is sent to the device in the next step.

[0349] Step 4: Information overlay

[0350] The server sends the generated navigation information, warnings, and information based on emotion recognition to the device. Specifically, it sends data that displays an arrow indicating the right turn point at the next intersection. The device then overlays the received information on the AR display device. For example, it displays an instruction ahead saying "Turn right at the next intersection" with an arrow. The user visually confirms this information and continues driving safely.

[0351] Step 5: Voice interaction interface

[0352] While driving, the user issues questions or commands by voice. For example, "Where is the next right turn?" The device analyzes the user's voice using a voice recognition engine and converts it into text data. The server analyzes this voice instruction and generates appropriate information or a response, such as "Turn right at the intersection 300 meters ahead." The generated information is sent to the device, which then provides it to the user as audio or visual information.

[0353] Step 6: Personalized navigation suggestions

[0354] The server analyzes past driving data and user preferences based on a database. For example, it learns data on routes the user has previously chosen and areas they have avoided. It then calculates the optimal route by combining real-time traffic information with the user's emotional state. Specifically, it proposes a new avoidance route by taking into account current traffic congestion information and the user's relaxation state. The device overlays the proposed navigation route on the AR display and notifies the user by voice, saying, "A new route will be displayed."

[0355] (Application example 2)

[0356] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0357] Conventional driver assistance systems primarily aim to provide real-time environmental information and navigation, but they do not take into account the driver's mental state and lack methods to reduce stress and fatigue while driving. Therefore, personalized assistance that takes into account the driver's mental state is needed.

[0358] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data collection means for collecting biometric data of the user and performing emotion analysis, an emotion response generation means for providing personalized driving assistance based on the emotion analysis results, and a data analysis means for analyzing environmental data in real time and generating navigation information and warnings based on the analysis results. This enables personalized driving assistance according to the driver's mental state.

[0359] An "AR display device" is a device that uses augmented reality technology to overlay information onto the user's field of vision.

[0360] A "sensor" is a device for collecting data about the physical environment. Examples include distance sensors and accelerometers.

[0361] A "camera" is a device that captures video data.

[0362] The "data collection means" is a means for collecting environmental data and biometric data of the user.

[0363] "Data analysis means" refers to means for analyzing collected data and generating necessary information.

[0364] The "information generating means" is a means for generating navigation information and warnings based on the analysis results.

[0365] The "display control means" is a means for overlay-displaying the generated information on the AR display device.

[0366] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating a response.

[0367] The "route suggestion means" is a means for suggesting an optimal route based on real-time traffic information and driving history.

[0368] "Biometric data" refers to data related to the user's physical condition, such as heart rate and facial expression.

[0369] "Emotion analysis" is the process of assessing a user's emotional state based on their biometric data.

[0370] The "emotion response generating means" is a means for providing individualized driving assistance based on the emotion analysis results.

[0371] This invention is a driving assistance system that includes an AR display device, a data collection means equipped with a sensor and a camera, a data analysis means that analyzes environmental data in real time, an information generation means that generates navigation information and warnings based on the analysis results, a display control means that overlays and displays the information on the AR display device, a voice dialogue means that analyzes voice instructions and generates responses, a route proposal means that proposes an optimal route based on real-time traffic information and driving history, a data collection means that collects a user's biometric data and performs emotion analysis, and an emotion response generation means that provides personalized driving assistance based on the emotion analysis results.

[0372] System Program Overview

[0373] 1. Data Collection Methods

[0374] The device (smart glasses) uses a camera and sensors to scan the surrounding environment. For example, the camera captures road conditions, and the distance sensor measures the location of nearby vehicles and obstacles. In addition, the device collects biometric data such as gaze, facial expression, voice, and heart rate.

[0375] 2. Data analysis methods

[0376] Data sent from the device is sent in real time to a cloud server, where traffic information is analyzed using an image recognition algorithm (e.g., OpenCV). An emotion engine also analyzes the user's emotional state based on their heart rate and facial expression data.

[0377] 3. Information generation means

[0378] The cloud server generates navigation information and warnings based on the data analysis results, such as traffic congestion information, driving direction instructions, and emergency evacuation routes. Taking into account the emotion analysis results, the system displays relaxation advice if the user is feeling tense.

[0379] 4. Display Control Method

[0380] The generated information is sent to the device and overlaid on the AR display, allowing users to view navigation and warning information displayed directly in their field of view.

[0381] 5. Voice interaction means

[0382] Users can ask questions or give instructions by voice, and the device will analyze them using a voice recognition engine. The analysis results are sent to a cloud server, which generates an appropriate response. For example, in response to a question like, "Where is the next right turn?", the cloud server will generate and send an appropriate response based on the latest traffic information.

[0383] 6. Route suggestion method

[0384] The cloud server calculates the optimal route based on real-time traffic information and past driving history, taking into account the user's preferences and areas they want to avoid. The optimal route is sent to the device and displayed as an overlay on the AR display.

[0385] Hardware and software used

[0386] Hardware: Smart glasses (including camera, distance sensor, microphone, and heart rate sensor), cloud server.

[0387] Software: OpenCV (image analysis), emotion analysis algorithm (emotion engine), speech recognition engine (e.g. Google Speech Recognition API), data communication protocol (e.g. HTTP / HTTPS).

[0388] Specific examples

[0389] Scenario 1: Navigation while driving

[0390] Once the user sets their destination and starts driving, the cloud server analyzes environmental data in real time and generates driving direction and traffic congestion information. For example, instructions such as "Turn left at the next intersection" are overlaid in the smart glasses' field of view. If the user is feeling nervous, advice such as "Please drive safely" is also displayed.

[0391] Scenario 2: Sudden change in road conditions

[0392] When the device detects traffic congestion ahead using its camera, the data is sent to a cloud server. The cloud server then calculates a new route and sends it to the device. The smart glasses then overlay the new route information and provide a voice notification. If stress is detected, the glasses will also suggest a relaxing route.

[0393] Example prompts to input to the generative AI model

[0394] "Based on real-time camera footage and heart rate data, analyze traffic information and the driver's emotional state to generate appropriate driving assistance information."

[0395] This enables the driving assistance system to provide personalized information according to the user's emotional state. Real-time data collection, analysis, and response provide a safe and comfortable driving environment.

[0396] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0397] Step 1:

[0398] The user puts on the smart glasses and starts driving. The device uses cameras and sensors to scan the surrounding environment in real time, collecting data on road conditions, other vehicles, pedestrians, etc. It also collects biometric data such as gaze, facial expressions, voice, and heart rate.

[0399] Input: Environmental data (image, distance), biometric data (heart rate, facial expression, voice)

[0400] Output: Collected environmental and biological data

[0401] Step 2:

[0402] The device transmits the collected environmental and biological data to a cloud server in real time. The device transmits the data using a data communication protocol (e.g., HTTP / HTTPS).

[0403] Input: Collected environmental and biometric data

[0404] Output: Data sent to the cloud server

[0405] Step 3:

[0406] The cloud server analyzes the received environmental data. It uses image recognition algorithms (e.g., OpenCV) to identify and analyze road conditions and traffic information. It also uses an emotion engine to analyze the user's heart rate and facial expression data to evaluate their emotional state.

[0407] Input: Environmental and biological data sent to the cloud server

[0408] Output: Analyzed traffic information and user's emotional state

[0409] Step 4:

[0410] The cloud server uses the analysis results to generate navigation information, warnings, and personalized driving assistance based on the user's emotional state, such as "Turn left at the next intersection" or "Display a message to relax because you are nervous."

[0411] Input: Parsed traffic information and user's emotional state

[0412] Output: Generated navigation information, warnings, and driving assistance information based on emotional state

[0413] Step 5:

[0414] The cloud server sends the generated information to the device, which then overlays it on the smart glasses' AR display, allowing the user to see real-time navigation and warnings within their field of vision.

[0415] Input: Generated navigation information, warnings, and driving assistance information based on emotional state

[0416] Output: Information displayed on the smart glasses

[0417] Step 6:

[0418] Users can ask questions or give instructions by voice while driving. The device uses a voice recognition engine (e.g., Google Speech Recognition API) to analyze the voice data and convert it into text data. This text data is then sent to a cloud server.

[0419] Input: User's voice command

[0420] Output: Text data sent to the cloud server

[0421] Step 7:

[0422] The cloud server generates an appropriate response based on the received text data. For example, in response to a question such as "Where is the next right turn?", it generates a response based on the latest traffic information.

[0423] Input: Text data sent to the cloud server

[0424] Output: The generated response

[0425] Step 8:

[0426] The cloud server sends the generated response to the terminal, which then provides it to the user as audio and visual information, for example, the next right turn point is announced by audio and displayed as an overlay in the field of view.

[0427] Input: The generated response

[0428] Output: Audio and visual information presented to the user

[0429] Through the above steps, the driving assistance system can provide personalized information according to the user's emotional state, enabling safe and comfortable driving.

[0430] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0431] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0432] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0433] [Second embodiment]

[0434] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0435] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0436] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0437] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0438] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0439] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0440] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0441] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0442] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0443] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0444] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0445] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0446] This invention is a driving assistance system that combines an AR display device with data collection means, data analysis means, information generation means, display control means, voice interaction means, and route suggestion means. This system allows drivers to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving.

[0447] Program processing explanation

[0448] 1. Data collection and environmental awareness

[0449] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed.

[0450] The acquired data is sent to the server in real time.

[0451] 2. Data analysis and information generation

[0452] The server analyzes the received environmental data, which includes using image recognition algorithms and traffic prediction models.

[0453] Based on the analysis results, the server determines the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents).

[0454] The server generates optimal navigation information and necessary warning information.

[0455] 3. Information overlay

[0456] The server sends the generated navigation information and warnings to the terminal.

[0457] The device then displays the received information on the AR display device. Specifically, images and text are projected onto the AR glasses' display so that they are overlaid directly on the user's field of vision.

[0458] The user continues driving while visually checking this information.

[0459] 4. Voice Dialogue Interface

[0460] Users can ask questions or give instructions by voice while driving, for example, "Where is the next right turn?"

[0461] The terminal analyzes the user's voice using a voice recognition engine and sends the analysis results to the server as text data.

[0462] The server analyzes the voice instructions and generates appropriate information or responses, which are then sent back to the terminal, which then provides the information to the user as audio or visual information.

[0463] 5. Personalized navigation suggestions

[0464] The server analyzes past driving data and user preferences based on a database, learning, for example, the user's preferred routes and areas to avoid.

[0465] The server calculates and proposes the optimal route in combination with real-time traffic information, allowing users to select the most efficient and safe route at that time.

[0466] The device will overlay the suggested navigation route on the AR display and also provide an audio notification.

[0467] Specific examples

[0468] Scenario 1: Navigation while driving

[0469] After the user sets their destination, they begin driving. If they need to turn left at the next intersection, the server analyzes that information and sends it to the device. Based on the received information, the device overlays the instruction "Turn left at the next intersection" on the AR display device.

[0470] Scenario 2: Sudden change in road conditions

[0471] When the device detects a traffic jam on the road ahead using its camera, the data is sent to the server. The server analyzes the traffic jam information and calculates a new route to avoid it. The new route information is sent to the device, which then displays it as an overlay on the AR display and notifies the user by voice.

[0472] This allows the driving assistance system of the present invention to provide safe and intuitive assistance to the driver in real time.

[0473] The processing flow will be explained below.

[0474] Step 1:

[0475] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed.

[0476] Step 2:

[0477] The device processes and temporarily stores acquired environmental data in real time, including digital image processing and sensor data analysis.

[0478] Step 3:

[0479] The data collected by the terminal is sent to the server using a secure, low-latency communication protocol.

[0480] Step 4:

[0481] The server analyzes the received environmental data, using image recognition algorithms and traffic prediction models to understand the surroundings.

[0482] Step 5:

[0483] Based on the analysis results, the server determines the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents).

[0484] Step 6:

[0485] The server generates optimal navigation information and necessary warning information, such as specific instructions such as "turn left at the next intersection."

[0486] Step 7:

[0487] The server generates navigation information and alerts and sends them back to the device. The data is transmitted securely with low latency.

[0488] Step 8:

[0489] The navigation information received by the device is displayed on the AR display device. Specifically, images and text are projected so that they are overlaid directly on the user's field of view.

[0490] Step 9:

[0491] The user can continue driving while visually checking navigation information and warnings through the AR display device.

[0492] Step 10:

[0493] The user asks questions or gives instructions by voice while driving, for example, using voice commands such as "Where is the next right turn?"

[0494] Step 11:

[0495] The device analyzes the user's voice using a voice recognition engine and converts it into text data, which is then sent to the server.

[0496] Step 12:

[0497] The server analyzes the voice instructions and generates appropriate information or responses, such as "The next right turn is 200 meters away."

[0498] Step 13:

[0499] The server transmits the generated information to the terminal, which then provides the information to the user as audio or visual information.

[0500] Step 14:

[0501] The server analyzes past driving data and user preferences based on a database, and combines this with real-time traffic information to calculate the optimal route.

[0502] Step 15:

[0503] The server sends the proposed new navigation route to the terminal, which then displays it as an overlay on the AR display device and notifies the user by voice.

[0504] In this way, the driving assistance system of the present invention realizes real-time environment recognition, data analysis, generation and display of navigation information, and personalized driving assistance through a voice dialogue interface.

[0505] Example 1

[0506] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0507] From the perspective of safe driving, it is important for drivers to obtain necessary information without taking their eyes off the road. However, conventional navigation systems and driver assistance systems require drivers to pay attention to their smartphones or in-car displays, resulting in distraction and an increased risk of accidents. Furthermore, it is difficult to provide appropriate navigation information and warnings in real time, making it difficult for drivers to reach their destination safely and efficiently.

[0508] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0509] In this invention, the server includes a means for displaying visual information through an AR display device, a means for analyzing a user's voice input with a voice recognition engine and converting the analysis result into text, a means for analyzing the text data and generating an appropriate response, and a means for learning past driving data and personalized route information for the user to provide an optimal route. This allows the driver to obtain navigation information and warnings in real time without taking their eyes off the road, enabling safer and more intuitive driving.

[0510] An "AR display device" is a device that uses augmented reality technology to overlay information on the user's field of vision.

[0511] "Data collection means" refers to means for collecting surrounding environmental data using sensors and cameras.

[0512] "Data analysis means" refers to a means for analyzing collected environmental data in real time and extracting important information.

[0513] The "information generating means" is a means for generating navigation information and warning information based on the analysis results.

[0514] The "display control means" is a means for overlay-displaying the generated information on the AR display device.

[0515] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating an appropriate response.

[0516] The "route suggestion means" is a means for suggesting the optimal route based on real-time traffic information and driving history.

[0517] The "means for displaying visual information" refers to a means for displaying information directly into the user's field of vision through an AR display device.

[0518] A "voice recognition engine" is an engine that analyzes a user's voice input and converts it into text data.

[0519] The "means for analyzing text data" is a means for generating an appropriate response based on the text data generated by the voice recognition engine.

[0520] The "means for learning personalized route information" is a means for providing an optimal route based on past driving data and user preferences.

[0521] This invention is a driving assistance system that combines an AR display device with data collection means, data analysis means, information generation means, display control means, voice interaction means, and route suggestion means. This system allows drivers to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving.

[0522] Hardware and software used

[0523] Terminal (AR display device): Equipped with a camera and sensors (e.g., LiDAR sensor). The camera captures images at, for example, several tens of frames per second, and the LiDAR sensor obtains distance information about surrounding objects.

[0524] Server: A high-performance computer (e.g., a cloud server) analyzes the data using deep learning models (e.g., TensorFlow), image recognition algorithms (e.g., OpenCV), and traffic prediction models (e.g., Kalman filters).

[0525] Communication methods: Using high-speed networks such as 5G and Wi-Fi to transfer data in real time.

[0526] System processing overview

[0527] 1. Data collection and environmental recognition: The device collects data on the surrounding environment using a camera and sensors mounted on the front of the vehicle. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the LiDAR sensor obtains distance information. The device transmits this data to a server in real time.

[0528] 2. Data analysis and information generation: The server analyzes the received data and uses deep learning and traffic prediction models to identify vehicles and signs. Based on the analysis results, it generates navigation information and necessary warnings.

[0529] 3. Information overlay display: The server sends the generated navigation information and warnings to the device. The device displays the received information as an overlay on the AR display device. The user can continue driving while checking the information without taking their eyes off the road.

[0530] 4. Voice dialogue interface: Users can ask questions or give instructions by voice. The device analyzes the voice using a voice recognition engine and sends the results as text data to the server. The server analyzes the text data, generates an appropriate response, and sends it to the device. The device then provides the information to the user as audio or visual information.

[0531] 5. Personalized navigation suggestions: The server analyzes past driving data and user preferences to suggest optimal routes. It calculates the optimal route by combining it with real-time traffic information and sends that information to the device. The device then overlays the suggested route on the AR display and notifies the user via voice.

[0532] Specific examples

[0533] Scenario 1: Navigation while driving

[0534] After the user sets their destination, they begin driving. If they need to turn left at the next intersection, the server analyzes that information and sends it to the device. Based on the received information, the device overlays the instruction "Turn left at the next intersection" on the AR display device.

[0535] Scenario 2: Sudden change in road conditions

[0536] When the device first detects a traffic jam using its camera, the data is sent to the server. The server analyzes the congestion information and calculates a new route to avoid it. The new route information is sent to the device, which then displays it as an overlay on the AR display and notifies the driver via voice. This allows the system to provide safe and intuitive assistance to drivers in real time.

[0537] Prompt Sentence Examples

[0538] "Where is the next right turn?"

[0539] "Please suggest the best route taking traffic information into consideration."

[0540] "Please show us what signs your camera recognizes."

[0541] In this way, the server, terminal, and user each play their respective roles, collecting, analyzing, and displaying data in real time, creating a system that provides safe and intuitive driving assistance.

[0542] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0543] Step 1: Data collection

[0544] The device (AR display device) activates a camera and sensors (e.g., LiDAR sensor) mounted on the front of the vehicle to collect environmental data. Specifically, the camera captures images at tens of frames per second, and the LiDAR sensor obtains distance information of surrounding objects.

[0545] Input: Image data of the area in front of the vehicle and distance data of the surrounding area.

[0546] Output: Collected video and distance data.

[0547] Step 2: Send data

[0548] The device transmits the collected environmental data to a server in real time using 5G or Wi-Fi as the communication protocol, ensuring low latency data transfer.

[0549] Input: Collected video data and distance data.

[0550] Output: The raw environment data sent to the server.

[0551] Step 3: Receiving data

[0552] The server receives the environmental data sent from the terminal and stores it in a buffer for analysis. For example, it uses Apache Kafka for data stream management.

[0553] Input: Environmental data sent from the device.

[0554] Output: Received and buffered environmental data.

[0555] Step 4: Data analysis

[0556] The server analyzes the received environmental data, using deep learning models (e.g., TensorFlow) and image recognition algorithms (e.g., OpenCV) to identify vehicles, signs, pedestrians, etc.

[0557] Input: Buffered environmental data.

[0558] Output: Information about recognized objects (vehicles, signs, pedestrians, etc.).

[0559] Step 5: Information Generation

[0560] The server generates optimal navigation and warning information based on the analysis results, and uses traffic prediction models (e.g., Kalman filter) to evaluate real-time traffic conditions.

[0561] Input: Recognized object information and traffic prediction data.

[0562] Output: Navigation and warning information.

[0563] Step 6: Send information

[0564] The server transmits the generated navigation information and warning information to the terminal in real time.

[0565] Input: Navigation and warning information.

[0566] Output: Navigation and warning information sent to the device.

[0567] Step 7: Display information

[0568] The device overlays the received navigation and warning information on the AR display device, displaying arrows and warning text in the user's field of view.

[0569] Input: Navigation and warning information sent from the server.

[0570] Output: Information displayed on the AR display device.

[0571] Step 8: Recognizing voice input

[0572] Users can ask questions or give instructions by voice while driving, such as "Where is the next right turn?"

[0573] Input: User's voice command.

[0574] Output: Audio instructions.

[0575] Step 9: Audio Analysis

[0576] The device analyzes the user's voice using a voice recognition engine (e.g., Google Speech-to-Text or Amazon Alexa) and sends the analysis results to the server as text data.

[0577] Input: User's voice command.

[0578] Output: Parsed text data.

[0579] Step 10: Generate voice response

[0580] The server analyzes the voice instructions and generates appropriate information or responses, such as "The next right turn is 300 meters away."

[0581] Input: Parsed text data.

[0582] Output: Audio and text responses.

[0583] Step 11: Send a voice response

[0584] The server sends the generated voice response to the terminal, which then presents the information to the user as voice.

[0585] Input: The generated speech response.

[0586] Output: The audio response sent to the device.

[0587] Step 12: Personalized navigation suggestions

[0588] The server analyzes a database based on past driving data and user preferences, and combines this with real-time traffic information to suggest the optimal route.

[0589] Inputs: Historical driving data, user preferences, real-time traffic information.

[0590] Output: personalized optimal route information.

[0591] The information collected, analyzed, and generated at each step is provided in real time to improve driver safety and convenience, allowing drivers to obtain the necessary information in a timely manner without taking their eyes off the road.

[0592] (Application example 1)

[0593] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0594] Conventional driver assistance systems have the problem of compromising safety while driving because drivers have to look away from the road to check navigation information and emergency warnings. Furthermore, they lack the ability to suggest optimal routes based on real-time traffic conditions and the user's driving history, and they also lack functionality to reduce user stress and fatigue.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0596] In this invention, the server includes an AR display device, a data collection means equipped with sensors and a camera, a data analysis means for analyzing environmental data in real time, an information generation means for generating navigation information and warnings based on the analysis results, a display control means for overlaying the information on the AR display device, a voice dialogue means for analyzing voice instructions and generating responses, a route proposal means for proposing an optimal route based on real-time traffic information and driving history, a voice and visual information provision means for providing the analysis results as visual and voice information through smart glasses, a voice analysis means for analyzing voice using a voice recognition engine and generating appropriate information and responses, and a route calculation means for calculating and proposing a stress-reducing route based on past driving data and preferences. This allows the driver to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving. Furthermore, personalized route proposals based on the user's driving history and preferences enable efficient and stress-free driving.

[0597] - An "AR display device" is a device that displays digital information overlaid on images from the real world.

[0598] A "sensor" is a device for acquiring physical environmental data.

[0599] A "camera" is a device for capturing images or video using an optical device.

[0600] "Data collection means" is a general term for devices and technologies used to collect environmental data.

[0601] A "data analysis means" is a device or method for analyzing collected data and converting it into meaningful information.

[0602] The "information generation means" is a mechanism that generates navigation information and warnings based on the analyzed data.

[0603] The "display control means" is a mechanism for appropriately overlaying and displaying the generated information on the AR display device.

[0604] A "voice interaction means" is a device or system for analyzing voice input and interacting with a user.

[0605] The "route suggestion method" is a system that calculates and suggests the optimal route based on real-time traffic information and driving history.

[0606] The "audio and visual information providing means" is a mechanism for providing the analysis results to the smart glasses as visual and audio information.

[0607] A "voice recognition engine" is software or hardware that converts voice into text data and analyzes it.

[0608] "Speech analysis means" refers to a device or method that uses a speech recognition engine to analyze speech and generate appropriate information or responses.

[0609] A "stress-reducing route" is an optimal driving route designed to reduce stress and fatigue based on the user's driving history and preferences.

[0610] This invention is a driving assistance system that combines an AR display device with data collection means, voice interaction means, and route suggestion means. This system allows users to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving.

[0611] System Configuration

[0612] AR display devices can take the form of smart glasses or head-mounted displays that are worn by the user and overlay navigation information and warnings onto the real-world scene.

[0613] Data collection methods include cameras (e.g., common webcams) and sensors (e.g., Lidar sensors). These devices capture data about the surrounding environment in real time and transmit it to a server.

[0614] As a data analysis method, the server analyzes the received data and uses image recognition algorithms (e.g., OpenCV) and generative AI models to analyze the environmental data.

[0615] The information generating means generates navigation information and warnings based on the analysis results, and the generated information is sent from the server to the terminal.

[0616] The display control means overlays the generated information on the AR display device, making it easier for the user to visually confirm the information.

[0617] For voice interaction, a speech recognition engine (e.g., the transformers library) is used to analyze the user's voice instructions. The server generates appropriate information and responses and provides them to the user as audio and visual information.

[0618] In the route suggestion method, the server calculates the optimal route based on real-time traffic information and driving history, taking into account past driving data and user preferences.

[0619] Program processing explanation

[0620] The server receives environmental data from cameras and sensors and uses image recognition algorithms to identify roads, vehicle signs, pedestrians, etc., in order to analyze the collected data. Based on the analysis results, it generates appropriate navigation information and warnings.

[0621] The generated information is sent to the AR display device and overlaid on the user's field of view, allowing the user to check the information without taking their eyes off the road.

[0622] In addition, by using the voice interaction function, users can receive driving route suggestions to reduce fatigue and stress. For example, when a user says, "Please tell me the shortest route to my next destination" or "What is the safest route from my current location?", the server calculates appropriate route information and suggests it in real time.

[0623] Specific example explanation

[0624] Scenario 1: Navigation while driving

[0625] The user sets their destination and starts driving. If they need to turn left at the next intersection, the server analyzes that information and displays an overlay on the AR display saying, "Turn left at the next intersection."

[0626] Scenario 2: Sudden change in road conditions

[0627] When the device detects a traffic jam on the road ahead using its camera, the data is sent to the server. The server analyzes the traffic jam information and calculates a new route to avoid it. The new route information is sent to the device, which then displays it as an overlay on the AR display and notifies the user by voice.

[0628] Prompt Sentence Examples

[0629] "Please tell me the shortest route to my next destination."

[0630] "What is the safest route from my current location?"

[0631] "Please suggest a detour route to avoid the congestion ahead."

[0632] This allows the driving assistance system of the present invention to provide safe and intuitive assistance to the driver in real time, thereby improving the user's driving experience.

[0633] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0634] Step 1:

[0635] Data collection and environmental awareness

[0636] When a user starts driving, the camera and sensors attached to the device collect data on the surrounding environment. Specifically, the camera captures road conditions, traffic signs, and pedestrians in real time, while the sensors measure the vehicle's speed and distance. This data is sent as input from the device to the server, which then receives it.

[0637] Input: Camera image data, sensor measurement data

[0638] Output: Environment data sent to the server

[0639] Step 2:

[0640] Data analysis and information generation

[0641] The server analyzes the received environmental data, using image recognition algorithms (e.g., OpenCV) and generative AI models to recognize specific objects (e.g., signs, pedestrians, other vehicles) and analyze their location and movement. It also takes into account real-time traffic information and historical driving data. Based on the analysis results, the server generates navigation information and warnings.

[0642] Input: Environmental data (video data, sensor measurement data), real-time traffic information, driving history data

[0643] Output: Analysis results, navigation information, warning information

[0644] Step 3:

[0645] Information overlay

[0646] The generated navigation information and warnings are sent from the server to the device. The device's AR display device overlays this information in the user's field of view. Specifically, guidance such as "Turn left at the next intersection" is displayed on the AR display device's screen. This step allows the user to check the information without taking their eyes off the road.

[0647] Input: Navigation information, warning information

[0648] Output: Navigation information and warning information overlaid on the AR display device

[0649] Step 4:

[0650] Voice Dialogue Interface

[0651] When a user speaks a question or command while driving, the device analyzes the voice using a voice recognition engine. For example, if the user says, "Please tell me the shortest route to my next destination," the voice data is input into the voice recognition engine and converted into text data. The converted data is sent to the server, which generates the appropriate information or response.

[0652] Input: User's voice commands

[0653] Output: Parsed text data, response information

[0654] Step 5:

[0655] Responses and information

[0656] The server analyzes the voice instructions and generates response information, which is then sent to the device. The device then overlays this information on the AR display and simultaneously notifies the user by voice, allowing the user to receive information both visually and audibly.

[0657] Input: Parsed text data, response information

[0658] Output: Response information overlaid on the AR display device, audio notification

[0659] Step 6:

[0660] Personalized route suggestions

[0661] The server calculates a stress-reducing route based on past driving data and the user's preferences. It uses an algorithm that prioritizes routes that avoid traffic jams and routes that the user prefers. The calculated route information is sent to the device, which then displays it on an AR display and notifies the user via voice.

[0662] Input: Driving history data, user preferences, real-time traffic information

[0663] Output: Calculated route information, overlaid route information, audio notification

[0664] As a result, driver assistance systems can provide accurate, real-time navigation information and warnings to support users as they drive, allowing them to continue driving safely and intuitively without taking their eyes off the road.

[0665] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0666] This invention is a driving assistance system that combines an AR display device, data collection means, data analysis means, information generation means, display control means, voice dialogue means, route suggestion means, and an emotion engine. This system allows drivers to receive real-time navigation information and warnings, as well as personalized driving assistance based on emotion recognition, without taking their eyes off the road.

[0667] Program processing explanation

[0668] 1. Data collection and environmental awareness

[0669] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed.

[0670] Furthermore, the device collects biometric data such as the user's voice, facial expressions, and heart rate, and obtains data for analyzing the user's emotional state using an emotion engine.

[0671] The acquired data is sent to the server in real time.

[0672] 2. Data analysis and information generation

[0673] The server analyzes the received environmental data, which includes using image recognition algorithms and traffic prediction models.

[0674] The server uses an emotion engine to analyze the user's emotional state, and based on the analysis results, identifies the user's emotional state, such as whether they are tense or relaxed.

[0675] Based on the analysis results, the server identifies the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents), and adjusts navigation information and warnings according to the user's emotional state.

[0676] The server generates optimal navigation information, necessary warning information, and personalized information based on emotional state.

[0677] 3. Information overlay

[0678] The server transmits the generated navigation information, warnings, and information based on emotion recognition to the terminal.

[0679] The device then displays the received information on the AR display device. Specifically, images and text are projected onto the user's field of view as an overlay.

[0680] The user continues driving while visually checking this information.

[0681] 4. Voice Dialogue Interface

[0682] Users can ask questions or give instructions by voice while driving, for example, "Where is the next right turn?"

[0683] The terminal analyzes the user's voice using a voice recognition engine and sends the analysis results to the server as text data.

[0684] The server analyzes the voice instructions and generates appropriate information or responses, which are then sent back to the terminal, which then provides the information to the user as audio or visual information.

[0685] Through the emotion engine, if the user is tense, the server generates a response that has a relaxation effect.

[0686] 5. Personalized navigation suggestions

[0687] The server analyzes past driving data and user preferences based on a database, learning, for example, the user's preferred routes and areas to avoid.

[0688] The server combines real-time traffic information with the user's emotional state to calculate and suggest the optimal route.

[0689] The device will overlay the suggested navigation route on the AR display and also provide an audio notification.

[0690] Specific examples

[0691] Scenario 1: Navigation while driving

[0692] After setting a destination, the user begins driving. If it is necessary to turn left at the next intersection, the server analyzes that information and sends it to the device. Based on the received information, the device overlays the instruction "Turn left at the next intersection" on the AR display device, and if the user is nervous, displays additional instructions such as "Please drive safely."

[0693] Scenario 2: Sudden change in road conditions

[0694] When the device detects a traffic jam on the road ahead using its camera, the data is sent to the server. The server analyzes the traffic information and calculates a new route to avoid it. The new route information is sent to the device, which then overlays it on the AR display and notifies the user via voice. If the user is feeling stressed, the device will also suggest a relaxing route.

[0695] As a result, the driving assistance system of the present invention can realize real-time environment recognition, data analysis, generation and display of navigation information, and personalized driving assistance through a voice dialogue interface, and can also provide appropriate support based on the user's emotional state.

[0696] The processing flow will be explained below.

[0697] Step 1:

[0698] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed. It also collects biometric data such as the user's voice, facial expression, and heart rate.

[0699] Step 2:

[0700] The device processes and temporarily stores acquired environmental and biometric data in real time, including digital image processing and biometric data analysis.

[0701] Step 3:

[0702] The data collected by the terminal is sent to the server using a secure, low-latency communication protocol.

[0703] Step 4:

[0704] The server analyzes the received environmental data, using image recognition algorithms and traffic prediction models to identify surrounding conditions (e.g., traffic jams, construction zones, accidents).

[0705] Step 5:

[0706] The server uses an emotion engine to analyze the user's biometric data, including voice tone analysis, facial expression recognition, and heart rate analysis, to identify the user's emotional state (e.g., tension, stress, relaxation).

[0707] Step 6:

[0708] Based on the analysis results, the server combines the current location, direction of travel, and surrounding conditions to generate optimal navigation information and necessary warning information. The content of the information and warnings is adjusted according to the user's emotional state.

[0709] Step 7:

[0710] The server generates navigation information, warnings, and personalized information based on emotion recognition, and sends the data securely with low latency to the device.

[0711] Step 8:

[0712] The device displays the received information on the AR display device. Specifically, images and text are projected as an overlay directly onto the user's field of vision. For example, instructions such as "Turn left at the next intersection" or "Drive safely" are included.

[0713] Step 9:

[0714] The user can continue driving while visually checking navigation information and warnings through the AR display device, and can also receive emotionally appropriate advice on how to ease tension as needed.

[0715] Step 10:

[0716] The user asks questions or gives instructions by voice while driving, for example, using voice commands such as "Where is the next right turn?"

[0717] Step 11:

[0718] The device analyzes the user's voice using a voice recognition engine and converts it into text data, which is then sent to the server.

[0719] Step 12:

[0720] The server analyzes the voice instructions and generates appropriate information and responses, such as "The next right turn is 200 meters away." If the emotion engine detects stress or tension in the user, it generates a response with a relaxation effect.

[0721] Step 13:

[0722] The server transmits the generated information to the terminal, which then provides the information to the user as audio or visual information.

[0723] Step 14:

[0724] The server analyzes a database of past driving data and user preferences, and combines real-time traffic information and the user's emotional state to calculate the optimal route, taking into account, for example, scenic routes with a relaxing effect.

[0725] Step 15:

[0726] The server sends the proposed new navigation route to the device, which then displays it as an overlay on the AR display and notifies the user by voice. The guidance may include guidance designed to have a relaxing effect.

[0727] In this way, the driving assistance system of the present invention realizes real-time environment recognition, data analysis, generation and display of navigation information, and personalized driving assistance based on emotional state.

[0728] Example 2

[0729] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0730] While conventional driving assistance systems can provide environmental data and traffic information, they have the problem of being unable to provide personalized assistance that takes into account the user's emotional state. As a result, appropriate assistance may not be provided when the driver is feeling nervous or stressed, which could result in a decrease in driving safety.

[0731] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an emotion recognition means that collects biometric data of the user and analyzes the emotional state, an information generation means that generates personalized information based on the emotional state, and a route suggestion means that suggests an optimal route based on real-time traffic information and driving history. This enables personalized driving assistance that takes the user's emotional state into consideration.

[0732] An "AR display device" is a device that uses augmented reality technology to display information and overlay it directly on a user's field of view.

[0733] "Data collection means" refers to means for acquiring environmental data and biological data using sensors and cameras.

[0734] "Data analysis means" refers to means for analyzing collected data in real time and extracting necessary information.

[0735] The "information generating means" is a means for generating navigation information and warnings based on the results of data analysis.

[0736] The "display control means" is a means for overlay-displaying the generated information on the AR display device.

[0737] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating an appropriate response.

[0738] The "route suggestion means" is a means for suggesting the optimal route based on real-time traffic information and driving history.

[0739] The "emotion recognition means" is a means for collecting biometric data of a user and analyzing the emotional state.

[0740] The "means for generating personalized information" is a means for generating information that is optimal for the user based on the analyzed emotional state.

[0741] The present invention relates to a driving assistance system that provides advanced assistance to drivers by combining an AR display device, data collection means, data analysis means, information generation means, display control means, voice dialogue means, route suggestion means, and emotion recognition means. This system allows drivers to receive real-time navigation information and warnings, as well as personalized driving assistance based on emotion recognition, without having to take their eyes off the road.

[0742] First, the device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures images of roads, vehicles, signs, pedestrians, etc., and the sensors obtain data such as distance and speed in real time. Biometric data such as the user's voice, facial expression, and heart rate are also collected. This provides data for the emotion recognition means (described later) to analyze the user's emotional state.

[0743] The collected data is sent to a server in real time. The server analyzes the received environmental data using image recognition algorithms and traffic prediction models. For example, a deep learning algorithm can be used to identify whether a pedestrian is crossing the street. An emotion engine can also be used to analyze the user's emotional state. This can identify whether the user is tense or relaxed.

[0744] The server uses the analysis results to determine the current location, direction of travel, and surrounding conditions. This includes combining received GPS data with image analysis results to accurately pinpoint the current location. It can also tailor navigation information and warnings based on the user's emotional state. For example, if the user is feeling nervous, it can suggest playing music to reduce stress.

[0745] The generated optimal navigation information and necessary warning information are sent to the device, which then displays it on the AR display and overlays it directly on the user's field of view. For example, an arrow can be displayed indicating a right turn point at the next intersection. The user can visually confirm this information and continue driving safely.

[0746] It also has a voice dialogue interface, allowing users to ask questions or give instructions by voice while driving. For example, it can issue prompts such as, "Where is the next right turn?" These voice instructions are analyzed by a voice recognition engine and sent to the server as text data. The server analyzes the voice instructions, generates appropriate information and responses, and sends them back to the device. The device then provides this information to the user again as audio or visual information. For example, it can provide specific guidance such as, "Turn right at the intersection 300 meters ahead."

[0747] It also uses emotion recognition to generate relaxing responses if the user is feeling tense, such as "Take a deep breath and relax." It can also learn from past driving data and user preferences, and combine this with real-time traffic information to calculate optimal routes. This allows it to provide personalized navigation suggestions based on the user's driving history.

[0748] Prompt Sentence Examples

[0749] "Where is the next right turn?"

[0750] "Show navigation information for turning left at the next intersection."

[0751] "Calculate a new route to avoid traffic jams."

[0752] "Generate a response that relaxes the user when they are tense."

[0753] The system improves driver concentration and safety, while providing personalized driving assistance.

[0754] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0755] Step 1: Data collection and environmental awareness

[0756] The device (AR display device) uses a camera and sensors to scan the surrounding environment. As input data, it uses image data (roads, vehicles, signs, pedestrians) captured by the camera and distance and speed data obtained by the sensors. This provides initial data on road conditions and direction of travel. Biometric data such as the user's voice, facial expression, and heart rate are also collected at the same time. Specifically, the camera captures an image of the road ahead, and the sensors measure the vehicle's speed and position. This data is sent to the server in real time.

[0757] Step 2: Data analysis

[0758] The server analyzes the input environmental and biometric data. For environmental data, it uses an image recognition algorithm to recognize road signs and pedestrians from captured images. Specifically, it uses a deep learning algorithm to identify, for example, whether a pedestrian is crossing a street. For biometric data, it uses an emotion engine to analyze the user's emotional state, such as tension or relaxation, from facial expression data. The results of this analysis are used by the server to generate navigation and warning information.

[0759] Step 3: Information generation

[0760] The server generates navigation information and warnings based on the results of data analysis. Specifically, it generates right-turn instructions at the next intersection and warnings if there are obstacles ahead based on the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents). Depending on the user's emotional state, it also suggests playing relaxing music. This information is sent to the device in the next step.

[0761] Step 4: Information overlay

[0762] The server sends the generated navigation information, warnings, and information based on emotion recognition to the device. Specifically, it sends data that displays an arrow indicating the right turn point at the next intersection. The device then overlays the received information on the AR display device. For example, it displays an instruction ahead saying "Turn right at the next intersection" with an arrow. The user visually confirms this information and continues driving safely.

[0763] Step 5: Voice interaction interface

[0764] While driving, the user issues questions or commands by voice. For example, "Where is the next right turn?" The device analyzes the user's voice using a voice recognition engine and converts it into text data. The server analyzes this voice instruction and generates appropriate information or a response, such as "Turn right at the intersection 300 meters ahead." The generated information is sent to the device, which then provides it to the user as audio or visual information.

[0765] Step 6: Personalized navigation suggestions

[0766] The server analyzes past driving data and user preferences based on a database. For example, it learns data on routes the user has previously chosen and areas they have avoided. It then calculates the optimal route by combining real-time traffic information with the user's emotional state. Specifically, it proposes a new avoidance route by taking into account current traffic congestion information and the user's relaxation state. The device overlays the proposed navigation route on the AR display and notifies the user by voice, saying, "A new route will be displayed."

[0767] (Application example 2)

[0768] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0769] Conventional driver assistance systems primarily aim to provide real-time environmental information and navigation, but they do not take into account the driver's mental state and lack methods to reduce stress and fatigue while driving. Therefore, personalized assistance that takes into account the driver's mental state is needed.

[0770] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data collection means for collecting biometric data of the user and performing emotion analysis, an emotion response generation means for providing personalized driving assistance based on the emotion analysis results, and a data analysis means for analyzing environmental data in real time and generating navigation information and warnings based on the analysis results. This enables personalized driving assistance according to the driver's mental state.

[0771] An "AR display device" is a device that uses augmented reality technology to overlay information onto the user's field of vision.

[0772] A "sensor" is a device for collecting data about the physical environment. Examples include distance sensors and accelerometers.

[0773] A "camera" is a device that captures video data.

[0774] The "data collection means" is a means for collecting environmental data and biometric data of the user.

[0775] "Data analysis means" refers to means for analyzing collected data and generating necessary information.

[0776] The "information generating means" is a means for generating navigation information and warnings based on the analysis results.

[0777] The "display control means" is a means for overlay-displaying the generated information on the AR display device.

[0778] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating a response.

[0779] The "route suggestion means" is a means for suggesting an optimal route based on real-time traffic information and driving history.

[0780] "Biometric data" refers to data related to the user's physical condition, such as heart rate and facial expression.

[0781] "Emotion analysis" is the process of assessing a user's emotional state based on their biometric data.

[0782] The "emotion response generating means" is a means for providing individualized driving assistance based on the emotion analysis results.

[0783] This invention is a driving assistance system that includes an AR display device, a data collection means equipped with a sensor and a camera, a data analysis means that analyzes environmental data in real time, an information generation means that generates navigation information and warnings based on the analysis results, a display control means that overlays and displays the information on the AR display device, a voice dialogue means that analyzes voice instructions and generates responses, a route proposal means that proposes an optimal route based on real-time traffic information and driving history, a data collection means that collects a user's biometric data and performs emotion analysis, and an emotion response generation means that provides personalized driving assistance based on the emotion analysis results.

[0784] System Program Overview

[0785] 1. Data Collection Methods

[0786] The device (smart glasses) uses a camera and sensors to scan the surrounding environment. For example, the camera captures road conditions, and the distance sensor measures the location of nearby vehicles and obstacles. In addition, the device collects biometric data such as gaze, facial expression, voice, and heart rate.

[0787] 2. Data analysis methods

[0788] Data sent from the device is sent in real time to a cloud server, where traffic information is analyzed using an image recognition algorithm (e.g., OpenCV). An emotion engine also analyzes the user's emotional state based on their heart rate and facial expression data.

[0789] 3. Information generation means

[0790] The cloud server generates navigation information and warnings based on the data analysis results, such as traffic congestion information, driving direction instructions, and emergency evacuation routes. Taking into account the emotion analysis results, the system displays relaxation advice if the user is feeling tense.

[0791] 4. Display Control Method

[0792] The generated information is sent to the device and overlaid on the AR display, allowing users to view navigation and warning information displayed directly in their field of view.

[0793] 5. Voice interaction means

[0794] Users can ask questions or give instructions by voice, and the device will analyze them using a voice recognition engine. The analysis results are sent to a cloud server, which generates an appropriate response. For example, in response to a question like, "Where is the next right turn?", the cloud server will generate and send an appropriate response based on the latest traffic information.

[0795] 6. Route suggestion method

[0796] The cloud server calculates the optimal route based on real-time traffic information and past driving history, taking into account the user's preferences and areas they want to avoid. The optimal route is sent to the device and displayed as an overlay on the AR display.

[0797] Hardware and software used

[0798] Hardware: Smart glasses (including camera, distance sensor, microphone, and heart rate sensor), cloud server.

[0799] Software: OpenCV (image analysis), emotion analysis algorithm (emotion engine), speech recognition engine (e.g. Google Speech Recognition API), data communication protocol (e.g. HTTP / HTTPS).

[0800] Specific examples

[0801] Scenario 1: Navigation while driving

[0802] Once the user sets their destination and starts driving, the cloud server analyzes environmental data in real time and generates driving direction and traffic congestion information. For example, instructions such as "Turn left at the next intersection" are overlaid in the smart glasses' field of view. If the user is feeling nervous, advice such as "Please drive safely" is also displayed.

[0803] Scenario 2: Sudden change in road conditions

[0804] When the device detects traffic congestion ahead using its camera, the data is sent to a cloud server. The cloud server then calculates a new route and sends it to the device. The smart glasses then overlay the new route information and provide a voice notification. If stress is detected, the glasses will also suggest a relaxing route.

[0805] Example prompts to input to the generative AI model

[0806] "Based on real-time camera footage and heart rate data, analyze traffic information and the driver's emotional state to generate appropriate driving assistance information."

[0807] This enables the driving assistance system to provide personalized information according to the user's emotional state. Real-time data collection, analysis, and response provide a safe and comfortable driving environment.

[0808] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0809] Step 1:

[0810] The user puts on the smart glasses and starts driving. The device uses cameras and sensors to scan the surrounding environment in real time, collecting data on road conditions, other vehicles, pedestrians, etc. It also collects biometric data such as gaze, facial expressions, voice, and heart rate.

[0811] Input: Environmental data (image, distance), biometric data (heart rate, facial expression, voice)

[0812] Output: Collected environmental and biological data

[0813] Step 2:

[0814] The device transmits the collected environmental and biological data to a cloud server in real time. The device transmits the data using a data communication protocol (e.g., HTTP / HTTPS).

[0815] Input: Collected environmental and biometric data

[0816] Output: Data sent to the cloud server

[0817] Step 3:

[0818] The cloud server analyzes the received environmental data. It uses image recognition algorithms (e.g., OpenCV) to identify and analyze road conditions and traffic information. It also uses an emotion engine to analyze the user's heart rate and facial expression data to evaluate their emotional state.

[0819] Input: Environmental and biological data sent to the cloud server

[0820] Output: Analyzed traffic information and user's emotional state

[0821] Step 4:

[0822] The cloud server uses the analysis results to generate navigation information, warnings, and personalized driving assistance based on the user's emotional state, such as "Turn left at the next intersection" or "Display a message to relax because you are nervous."

[0823] Input: Parsed traffic information and user's emotional state

[0824] Output: Generated navigation information, warnings, and driving assistance information based on emotional state

[0825] Step 5:

[0826] The cloud server sends the generated information to the device, which then overlays it on the smart glasses' AR display, allowing the user to see real-time navigation and warnings within their field of vision.

[0827] Input: Generated navigation information, warnings, and driving assistance information based on emotional state

[0828] Output: Information displayed on the smart glasses

[0829] Step 6:

[0830] Users can ask questions or give instructions by voice while driving. The device uses a voice recognition engine (e.g., Google Speech Recognition API) to analyze the voice data and convert it into text data. This text data is then sent to a cloud server.

[0831] Input: User's voice command

[0832] Output: Text data sent to the cloud server

[0833] Step 7:

[0834] The cloud server generates an appropriate response based on the received text data. For example, in response to a question such as "Where is the next right turn?", it generates a response based on the latest traffic information.

[0835] Input: Text data sent to the cloud server

[0836] Output: The generated response

[0837] Step 8:

[0838] The cloud server sends the generated response to the terminal, which then provides it to the user as audio and visual information, for example, the next right turn point is announced by audio and displayed as an overlay in the field of view.

[0839] Input: The generated response

[0840] Output: Audio and visual information presented to the user

[0841] Through the above steps, the driving assistance system can provide personalized information according to the user's emotional state, enabling safe and comfortable driving.

[0842] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0843] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0844] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0845] [Third embodiment]

[0846] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0847] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0848] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0849] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0850] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0851] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0852] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0853] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0854] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0855] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0856] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0857] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0858] This invention is a driving assistance system that combines an AR display device with data collection means, data analysis means, information generation means, display control means, voice interaction means, and route suggestion means. This system allows drivers to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving.

[0859] Program processing explanation

[0860] 1. Data collection and environmental awareness

[0861] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed.

[0862] The acquired data is sent to the server in real time.

[0863] 2. Data analysis and information generation

[0864] The server analyzes the received environmental data, which includes using image recognition algorithms and traffic prediction models.

[0865] Based on the analysis results, the server determines the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents).

[0866] The server generates optimal navigation information and necessary warning information.

[0867] 3. Information overlay

[0868] The server sends the generated navigation information and warnings to the terminal.

[0869] The device then displays the received information on the AR display device. Specifically, images and text are projected onto the AR glasses' display so that they are overlaid directly on the user's field of vision.

[0870] The user continues driving while visually checking this information.

[0871] 4. Voice Dialogue Interface

[0872] Users can ask questions or give instructions by voice while driving, for example, "Where is the next right turn?"

[0873] The terminal analyzes the user's voice using a voice recognition engine and sends the analysis results to the server as text data.

[0874] The server analyzes the voice instructions and generates appropriate information or responses, which are then sent back to the terminal, which then provides the information to the user as audio or visual information.

[0875] 5. Personalized navigation suggestions

[0876] The server analyzes past driving data and user preferences based on a database, learning, for example, the user's preferred routes and areas to avoid.

[0877] The server calculates and proposes the optimal route in combination with real-time traffic information, allowing users to select the most efficient and safe route at that time.

[0878] The device will overlay the suggested navigation route on the AR display and also provide an audio notification.

[0879] Specific examples

[0880] Scenario 1: Navigation while driving

[0881] After the user sets their destination, they begin driving. If they need to turn left at the next intersection, the server analyzes that information and sends it to the device. Based on the received information, the device overlays the instruction "Turn left at the next intersection" on the AR display device.

[0882] Scenario 2: Sudden change in road conditions

[0883] When the device detects a traffic jam on the road ahead using its camera, the data is sent to the server. The server analyzes the traffic jam information and calculates a new route to avoid it. The new route information is sent to the device, which then displays it as an overlay on the AR display and notifies the user by voice.

[0884] This allows the driving assistance system of the present invention to provide safe and intuitive assistance to the driver in real time.

[0885] The processing flow will be explained below.

[0886] Step 1:

[0887] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed.

[0888] Step 2:

[0889] The device processes and temporarily stores acquired environmental data in real time, including digital image processing and sensor data analysis.

[0890] Step 3:

[0891] The data collected by the terminal is sent to the server using a secure, low-latency communication protocol.

[0892] Step 4:

[0893] The server analyzes the received environmental data, using image recognition algorithms and traffic prediction models to understand the surroundings.

[0894] Step 5:

[0895] Based on the analysis results, the server determines the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents).

[0896] Step 6:

[0897] The server generates optimal navigation information and necessary warning information, such as specific instructions such as "turn left at the next intersection."

[0898] Step 7:

[0899] The server generates navigation information and alerts and sends them back to the device. The data is transmitted securely with low latency.

[0900] Step 8:

[0901] The navigation information received by the device is displayed on the AR display device. Specifically, images and text are projected so that they are overlaid directly on the user's field of view.

[0902] Step 9:

[0903] The user can continue driving while visually checking navigation information and warnings through the AR display device.

[0904] Step 10:

[0905] The user asks questions or gives instructions by voice while driving, for example, using voice commands such as "Where is the next right turn?"

[0906] Step 11:

[0907] The device analyzes the user's voice using a voice recognition engine and converts it into text data, which is then sent to the server.

[0908] Step 12:

[0909] The server analyzes the voice instructions and generates appropriate information or responses, such as "The next right turn is 200 meters away."

[0910] Step 13:

[0911] The server transmits the generated information to the terminal, which then provides the information to the user as audio or visual information.

[0912] Step 14:

[0913] The server analyzes past driving data and user preferences based on a database, and combines this with real-time traffic information to calculate the optimal route.

[0914] Step 15:

[0915] The server sends the proposed new navigation route to the terminal, which then displays it as an overlay on the AR display device and notifies the user by voice.

[0916] In this way, the driving assistance system of the present invention realizes real-time environment recognition, data analysis, generation and display of navigation information, and personalized driving assistance through a voice dialogue interface.

[0917] Example 1

[0918] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0919] From the perspective of safe driving, it is important for drivers to obtain necessary information without taking their eyes off the road. However, conventional navigation systems and driver assistance systems require drivers to pay attention to their smartphones or in-car displays, resulting in distraction and an increased risk of accidents. Furthermore, it is difficult to provide appropriate navigation information and warnings in real time, making it difficult for drivers to reach their destination safely and efficiently.

[0920] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0921] In this invention, the server includes a means for displaying visual information through an AR display device, a means for analyzing a user's voice input with a voice recognition engine and converting the analysis result into text, a means for analyzing the text data and generating an appropriate response, and a means for learning past driving data and personalized route information for the user to provide an optimal route. This allows the driver to obtain navigation information and warnings in real time without taking their eyes off the road, enabling safer and more intuitive driving.

[0922] An "AR display device" is a device that uses augmented reality technology to overlay information on the user's field of vision.

[0923] "Data collection means" refers to means for collecting surrounding environmental data using sensors and cameras.

[0924] "Data analysis means" refers to a means for analyzing collected environmental data in real time and extracting important information.

[0925] The "information generating means" is a means for generating navigation information and warning information based on the analysis results.

[0926] The "display control means" is a means for overlay-displaying the generated information on the AR display device.

[0927] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating an appropriate response.

[0928] The "route suggestion means" is a means for suggesting the optimal route based on real-time traffic information and driving history.

[0929] The "means for displaying visual information" refers to a means for displaying information directly into the user's field of vision through an AR display device.

[0930] A "voice recognition engine" is an engine that analyzes a user's voice input and converts it into text data.

[0931] The "means for analyzing text data" is a means for generating an appropriate response based on the text data generated by the voice recognition engine.

[0932] The "means for learning personalized route information" is a means for providing an optimal route based on past driving data and user preferences.

[0933] This invention is a driving assistance system that combines an AR display device with data collection means, data analysis means, information generation means, display control means, voice interaction means, and route suggestion means. This system allows drivers to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving.

[0934] Hardware and software used

[0935] Terminal (AR display device): Equipped with a camera and sensors (e.g., LiDAR sensor). The camera captures images at, for example, several tens of frames per second, and the LiDAR sensor obtains distance information about surrounding objects.

[0936] Server: A high-performance computer (e.g., a cloud server) analyzes the data using deep learning models (e.g., TensorFlow), image recognition algorithms (e.g., OpenCV), and traffic prediction models (e.g., Kalman filters).

[0937] Communication methods: Using high-speed networks such as 5G and Wi-Fi to transfer data in real time.

[0938] System processing overview

[0939] 1. Data collection and environmental recognition: The device collects data on the surrounding environment using a camera and sensors mounted on the front of the vehicle. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the LiDAR sensor obtains distance information. The device transmits this data to a server in real time.

[0940] 2. Data analysis and information generation: The server analyzes the received data and uses deep learning and traffic prediction models to identify vehicles and signs. Based on the analysis results, it generates navigation information and necessary warnings.

[0941] 3. Information overlay display: The server sends the generated navigation information and warnings to the device. The device displays the received information as an overlay on the AR display device. The user can continue driving while checking the information without taking their eyes off the road.

[0942] 4. Voice dialogue interface: Users can ask questions or give instructions by voice. The device analyzes the voice using a voice recognition engine and sends the results as text data to the server. The server analyzes the text data, generates an appropriate response, and sends it to the device. The device then provides the information to the user as audio or visual information.

[0943] 5. Personalized navigation suggestions: The server analyzes past driving data and user preferences to suggest optimal routes. It calculates the optimal route by combining it with real-time traffic information and sends that information to the device. The device then overlays the suggested route on the AR display and notifies the user via voice.

[0944] Specific examples

[0945] Scenario 1: Navigation while driving

[0946] After the user sets their destination, they begin driving. If they need to turn left at the next intersection, the server analyzes that information and sends it to the device. Based on the received information, the device overlays the instruction "Turn left at the next intersection" on the AR display device.

[0947] Scenario 2: Sudden change in road conditions

[0948] When the device first detects a traffic jam using its camera, the data is sent to the server. The server analyzes the congestion information and calculates a new route to avoid it. The new route information is sent to the device, which then displays it as an overlay on the AR display and notifies the driver via voice. This allows the system to provide safe and intuitive assistance to drivers in real time.

[0949] Prompt Sentence Examples

[0950] "Where is the next right turn?"

[0951] "Please suggest the best route taking traffic information into consideration."

[0952] "Please show us what signs your camera recognizes."

[0953] In this way, the server, terminal, and user each play their respective roles, collecting, analyzing, and displaying data in real time, creating a system that provides safe and intuitive driving assistance.

[0954] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0955] Step 1: Data collection

[0956] The device (AR display device) activates a camera and sensors (e.g., LiDAR sensor) mounted on the front of the vehicle to collect environmental data. Specifically, the camera captures images at tens of frames per second, and the LiDAR sensor obtains distance information of surrounding objects.

[0957] Input: Image data of the area in front of the vehicle and distance data of the surrounding area.

[0958] Output: Collected video and distance data.

[0959] Step 2: Send data

[0960] The device transmits the collected environmental data to a server in real time using 5G or Wi-Fi as the communication protocol, ensuring low latency data transfer.

[0961] Input: Collected video data and distance data.

[0962] Output: The raw environment data sent to the server.

[0963] Step 3: Receiving data

[0964] The server receives the environmental data sent from the terminal and stores it in a buffer for analysis. For example, it uses Apache Kafka for data stream management.

[0965] Input: Environmental data sent from the device.

[0966] Output: Received and buffered environmental data.

[0967] Step 4: Data analysis

[0968] The server analyzes the received environmental data, using deep learning models (e.g., TensorFlow) and image recognition algorithms (e.g., OpenCV) to identify vehicles, signs, pedestrians, etc.

[0969] Input: Buffered environmental data.

[0970] Output: Information about recognized objects (vehicles, signs, pedestrians, etc.).

[0971] Step 5: Information Generation

[0972] The server generates optimal navigation and warning information based on the analysis results, and uses traffic prediction models (e.g., Kalman filter) to evaluate real-time traffic conditions.

[0973] Input: Recognized object information and traffic prediction data.

[0974] Output: Navigation and warning information.

[0975] Step 6: Send information

[0976] The server transmits the generated navigation information and warning information to the terminal in real time.

[0977] Input: Navigation and warning information.

[0978] Output: Navigation and warning information sent to the device.

[0979] Step 7: Display information

[0980] The device overlays the received navigation and warning information on the AR display device, displaying arrows and warning text in the user's field of view.

[0981] Input: Navigation and warning information sent from the server.

[0982] Output: Information displayed on the AR display device.

[0983] Step 8: Recognizing voice input

[0984] Users can ask questions or give instructions by voice while driving, such as "Where is the next right turn?"

[0985] Input: User's voice command.

[0986] Output: Audio instructions.

[0987] Step 9: Audio Analysis

[0988] The device analyzes the user's voice using a voice recognition engine (e.g., Google Speech-to-Text or Amazon Alexa) and sends the analysis results to the server as text data.

[0989] Input: User's voice command.

[0990] Output: Parsed text data.

[0991] Step 10: Generate voice response

[0992] The server analyzes the voice instructions and generates appropriate information or responses, such as "The next right turn is 300 meters away."

[0993] Input: Parsed text data.

[0994] Output: Audio and text responses.

[0995] Step 11: Send a voice response

[0996] The server sends the generated voice response to the terminal, which then presents the information to the user as voice.

[0997] Input: The generated speech response.

[0998] Output: The audio response sent to the device.

[0999] Step 12: Personalized navigation suggestions

[1000] The server analyzes a database based on past driving data and user preferences, and combines this with real-time traffic information to suggest the optimal route.

[1001] Inputs: Historical driving data, user preferences, real-time traffic information.

[1002] Output: personalized optimal route information.

[1003] The information collected, analyzed, and generated at each step is provided in real time to improve driver safety and convenience, allowing drivers to obtain the necessary information in a timely manner without taking their eyes off the road.

[1004] (Application example 1)

[1005] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1006] Conventional driver assistance systems have the problem of compromising safety while driving because drivers have to look away from the road to check navigation information and emergency warnings. Furthermore, they lack the ability to suggest optimal routes based on real-time traffic conditions and the user's driving history, and they also lack functionality to reduce user stress and fatigue.

[1007] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1008] In this invention, the server includes an AR display device, a data collection means equipped with sensors and a camera, a data analysis means for analyzing environmental data in real time, an information generation means for generating navigation information and warnings based on the analysis results, a display control means for overlaying the information on the AR display device, a voice dialogue means for analyzing voice instructions and generating responses, a route proposal means for proposing an optimal route based on real-time traffic information and driving history, a voice and visual information provision means for providing the analysis results as visual and voice information through smart glasses, a voice analysis means for analyzing voice using a voice recognition engine and generating appropriate information and responses, and a route calculation means for calculating and proposing a stress-reducing route based on past driving data and preferences. This allows the driver to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving. Furthermore, personalized route proposals based on the user's driving history and preferences enable efficient and stress-free driving.

[1009] - An "AR display device" is a device that displays digital information overlaid on images from the real world.

[1010] A "sensor" is a device for acquiring physical environmental data.

[1011] A "camera" is a device for capturing images or video using an optical device.

[1012] "Data collection means" is a general term for devices and technologies used to collect environmental data.

[1013] A "data analysis means" is a device or method for analyzing collected data and converting it into meaningful information.

[1014] The "information generation means" is a mechanism that generates navigation information and warnings based on the analyzed data.

[1015] The "display control means" is a mechanism for appropriately overlaying and displaying the generated information on the AR display device.

[1016] A "voice interaction means" is a device or system for analyzing voice input and interacting with a user.

[1017] The "route suggestion method" is a system that calculates and suggests the optimal route based on real-time traffic information and driving history.

[1018] The "audio and visual information providing means" is a mechanism for providing the analysis results to the smart glasses as visual and audio information.

[1019] A "voice recognition engine" is software or hardware that converts voice into text data and analyzes it.

[1020] "Speech analysis means" refers to a device or method that uses a speech recognition engine to analyze speech and generate appropriate information or responses.

[1021] A "stress-reducing route" is an optimal driving route designed to reduce stress and fatigue based on the user's driving history and preferences.

[1022] This invention is a driving assistance system that combines an AR display device with data collection means, voice interaction means, and route suggestion means. This system allows users to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving.

[1023] System Configuration

[1024] AR display devices can take the form of smart glasses or head-mounted displays that are worn by the user and overlay navigation information and warnings onto the real-world scene.

[1025] Data collection methods include cameras (e.g., common webcams) and sensors (e.g., Lidar sensors). These devices capture data about the surrounding environment in real time and transmit it to a server.

[1026] As a data analysis method, the server analyzes the received data and uses image recognition algorithms (e.g., OpenCV) and generative AI models to analyze the environmental data.

[1027] The information generating means generates navigation information and warnings based on the analysis results, and the generated information is sent from the server to the terminal.

[1028] The display control means overlays the generated information on the AR display device, making it easier for the user to visually confirm the information.

[1029] For voice interaction, a speech recognition engine (e.g., the transformers library) is used to analyze the user's voice instructions. The server generates appropriate information and responses and provides them to the user as audio and visual information.

[1030] In the route suggestion method, the server calculates the optimal route based on real-time traffic information and driving history, taking into account past driving data and user preferences.

[1031] Program processing explanation

[1032] The server receives environmental data from cameras and sensors and uses image recognition algorithms to identify roads, vehicle signs, pedestrians, etc., in order to analyze the collected data. Based on the analysis results, it generates appropriate navigation information and warnings.

[1033] The generated information is sent to the AR display device and overlaid on the user's field of view, allowing the user to check the information without taking their eyes off the road.

[1034] In addition, by using the voice interaction function, users can receive driving route suggestions to reduce fatigue and stress. For example, when a user says, "Please tell me the shortest route to my next destination" or "What is the safest route from my current location?", the server calculates appropriate route information and suggests it in real time.

[1035] Specific example explanation

[1036] Scenario 1: Navigation while driving

[1037] The user sets their destination and starts driving. If they need to turn left at the next intersection, the server analyzes that information and displays an overlay on the AR display saying, "Turn left at the next intersection."

[1038] Scenario 2: Sudden change in road conditions

[1039] When the device detects a traffic jam on the road ahead using its camera, the data is sent to the server. The server analyzes the traffic jam information and calculates a new route to avoid it. The new route information is sent to the device, which then displays it as an overlay on the AR display and notifies the user by voice.

[1040] Prompt Sentence Examples

[1041] "Please tell me the shortest route to my next destination."

[1042] "What is the safest route from my current location?"

[1043] "Please suggest a detour route to avoid the congestion ahead."

[1044] This allows the driving assistance system of the present invention to provide safe and intuitive assistance to the driver in real time, thereby improving the user's driving experience.

[1045] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1046] Step 1:

[1047] Data collection and environmental awareness

[1048] When a user starts driving, the camera and sensors attached to the device collect data on the surrounding environment. Specifically, the camera captures road conditions, traffic signs, and pedestrians in real time, while the sensors measure the vehicle's speed and distance. This data is sent as input from the device to the server, which then receives it.

[1049] Input: Camera image data, sensor measurement data

[1050] Output: Environment data sent to the server

[1051] Step 2:

[1052] Data analysis and information generation

[1053] The server analyzes the received environmental data, using image recognition algorithms (e.g., OpenCV) and generative AI models to recognize specific objects (e.g., signs, pedestrians, other vehicles) and analyze their location and movement. It also takes into account real-time traffic information and historical driving data. Based on the analysis results, the server generates navigation information and warnings.

[1054] Input: Environmental data (video data, sensor measurement data), real-time traffic information, driving history data

[1055] Output: Analysis results, navigation information, warning information

[1056] Step 3:

[1057] Information overlay

[1058] The generated navigation information and warnings are sent from the server to the device. The device's AR display device overlays this information in the user's field of view. Specifically, guidance such as "Turn left at the next intersection" is displayed on the AR display device's screen. This step allows the user to check the information without taking their eyes off the road.

[1059] Input: Navigation information, warning information

[1060] Output: Navigation information and warning information overlaid on the AR display device

[1061] Step 4:

[1062] Voice Dialogue Interface

[1063] When a user speaks a question or command while driving, the device analyzes the voice using a voice recognition engine. For example, if the user says, "Please tell me the shortest route to my next destination," the voice data is input into the voice recognition engine and converted into text data. The converted data is sent to the server, which generates the appropriate information or response.

[1064] Input: User's voice commands

[1065] Output: Parsed text data, response information

[1066] Step 5:

[1067] Responses and information

[1068] The server analyzes the voice instructions and generates response information, which is then sent to the device. The device then overlays this information on the AR display and simultaneously notifies the user by voice, allowing the user to receive information both visually and audibly.

[1069] Input: Parsed text data, response information

[1070] Output: Response information overlaid on the AR display device, audio notification

[1071] Step 6:

[1072] Personalized route suggestions

[1073] The server calculates a stress-reducing route based on past driving data and the user's preferences. It uses an algorithm that prioritizes routes that avoid traffic jams and routes that the user prefers. The calculated route information is sent to the device, which then displays it on an AR display and notifies the user via voice.

[1074] Input: Driving history data, user preferences, real-time traffic information

[1075] Output: Calculated route information, overlaid route information, audio notification

[1076] As a result, driver assistance systems can provide accurate, real-time navigation information and warnings to support users as they drive, allowing them to continue driving safely and intuitively without taking their eyes off the road.

[1077] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1078] This invention is a driving assistance system that combines an AR display device, data collection means, data analysis means, information generation means, display control means, voice dialogue means, route suggestion means, and an emotion engine. This system allows drivers to receive real-time navigation information and warnings, as well as personalized driving assistance based on emotion recognition, without taking their eyes off the road.

[1079] Program processing explanation

[1080] 1. Data collection and environmental awareness

[1081] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed.

[1082] Furthermore, the device collects biometric data such as the user's voice, facial expressions, and heart rate, and obtains data for analyzing the user's emotional state using an emotion engine.

[1083] The acquired data is sent to the server in real time.

[1084] 2. Data analysis and information generation

[1085] The server analyzes the received environmental data, which includes using image recognition algorithms and traffic prediction models.

[1086] The server uses an emotion engine to analyze the user's emotional state, and based on the analysis results, identifies the user's emotional state, such as whether they are tense or relaxed.

[1087] Based on the analysis results, the server identifies the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents), and adjusts navigation information and warnings according to the user's emotional state.

[1088] The server generates optimal navigation information, necessary warning information, and personalized information based on emotional state.

[1089] 3. Information overlay

[1090] The server transmits the generated navigation information, warnings, and information based on emotion recognition to the terminal.

[1091] The device then displays the received information on the AR display device. Specifically, images and text are projected onto the user's field of view as an overlay.

[1092] The user continues driving while visually checking this information.

[1093] 4. Voice Dialogue Interface

[1094] Users can ask questions or give instructions by voice while driving, for example, "Where is the next right turn?"

[1095] The terminal analyzes the user's voice using a voice recognition engine and sends the analysis results to the server as text data.

[1096] The server analyzes the voice instructions and generates appropriate information or responses, which are then sent back to the terminal, which then provides the information to the user as audio or visual information.

[1097] Through the emotion engine, if the user is tense, the server generates a response that has a relaxation effect.

[1098] 5. Personalized navigation suggestions

[1099] The server analyzes past driving data and user preferences based on a database, learning, for example, the user's preferred routes and areas to avoid.

[1100] The server combines real-time traffic information with the user's emotional state to calculate and suggest the optimal route.

[1101] The device will overlay the suggested navigation route on the AR display and also provide an audio notification.

[1102] Specific examples

[1103] Scenario 1: Navigation while driving

[1104] After setting a destination, the user begins driving. If it is necessary to turn left at the next intersection, the server analyzes that information and sends it to the device. Based on the received information, the device overlays the instruction "Turn left at the next intersection" on the AR display device, and if the user is nervous, displays additional instructions such as "Please drive safely."

[1105] Scenario 2: Sudden change in road conditions

[1106] When the device detects a traffic jam on the road ahead using its camera, the data is sent to the server. The server analyzes the traffic information and calculates a new route to avoid it. The new route information is sent to the device, which then overlays it on the AR display and notifies the user via voice. If the user is feeling stressed, the device will also suggest a relaxing route.

[1107] As a result, the driving assistance system of the present invention can realize real-time environment recognition, data analysis, generation and display of navigation information, and personalized driving assistance through a voice dialogue interface, and can also provide appropriate support based on the user's emotional state.

[1108] The processing flow will be explained below.

[1109] Step 1:

[1110] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed. It also collects biometric data such as the user's voice, facial expression, and heart rate.

[1111] Step 2:

[1112] The device processes and temporarily stores acquired environmental and biometric data in real time, including digital image processing and biometric data analysis.

[1113] Step 3:

[1114] The data collected by the terminal is sent to the server using a secure, low-latency communication protocol.

[1115] Step 4:

[1116] The server analyzes the received environmental data, using image recognition algorithms and traffic prediction models to identify surrounding conditions (e.g., traffic jams, construction zones, accidents).

[1117] Step 5:

[1118] The server uses an emotion engine to analyze the user's biometric data, including voice tone analysis, facial expression recognition, and heart rate analysis, to identify the user's emotional state (e.g., tension, stress, relaxation).

[1119] Step 6:

[1120] Based on the analysis results, the server combines the current location, direction of travel, and surrounding conditions to generate optimal navigation information and necessary warning information. The content of the information and warnings is adjusted according to the user's emotional state.

[1121] Step 7:

[1122] The server generates navigation information, warnings, and personalized information based on emotion recognition, and sends the data securely with low latency to the device.

[1123] Step 8:

[1124] The device displays the received information on the AR display device. Specifically, images and text are projected as an overlay directly onto the user's field of vision. For example, instructions such as "Turn left at the next intersection" or "Drive safely" are included.

[1125] Step 9:

[1126] The user can continue driving while visually checking navigation information and warnings through the AR display device, and can also receive emotionally appropriate advice on how to ease tension as needed.

[1127] Step 10:

[1128] The user asks questions or gives instructions by voice while driving, for example, using voice commands such as "Where is the next right turn?"

[1129] Step 11:

[1130] The device analyzes the user's voice using a voice recognition engine and converts it into text data, which is then sent to the server.

[1131] Step 12:

[1132] The server analyzes the voice instructions and generates appropriate information and responses, such as "The next right turn is 200 meters away." If the emotion engine detects stress or tension in the user, it generates a response with a relaxation effect.

[1133] Step 13:

[1134] The server transmits the generated information to the terminal, which then provides the information to the user as audio or visual information.

[1135] Step 14:

[1136] The server analyzes a database of past driving data and user preferences, and combines real-time traffic information and the user's emotional state to calculate the optimal route, taking into account, for example, scenic routes with a relaxing effect.

[1137] Step 15:

[1138] The server sends the proposed new navigation route to the device, which then displays it as an overlay on the AR display and notifies the user by voice. The guidance may include guidance designed to have a relaxing effect.

[1139] In this way, the driving assistance system of the present invention realizes real-time environment recognition, data analysis, generation and display of navigation information, and personalized driving assistance based on emotional state.

[1140] Example 2

[1141] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1142] While conventional driving assistance systems can provide environmental data and traffic information, they have the problem of being unable to provide personalized assistance that takes into account the user's emotional state. As a result, appropriate assistance may not be provided when the driver is feeling nervous or stressed, which could result in a decrease in driving safety.

[1143] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an emotion recognition means that collects biometric data of the user and analyzes the emotional state, an information generation means that generates personalized information based on the emotional state, and a route suggestion means that suggests an optimal route based on real-time traffic information and driving history. This enables personalized driving assistance that takes the user's emotional state into consideration.

[1144] An "AR display device" is a device that uses augmented reality technology to display information and overlay it directly on a user's field of view.

[1145] "Data collection means" refers to means for acquiring environmental data and biological data using sensors and cameras.

[1146] "Data analysis means" refers to means for analyzing collected data in real time and extracting necessary information.

[1147] The "information generating means" is a means for generating navigation information and warnings based on the results of data analysis.

[1148] The "display control means" is a means for overlay-displaying the generated information on the AR display device.

[1149] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating an appropriate response.

[1150] The "route suggestion means" is a means for suggesting the optimal route based on real-time traffic information and driving history.

[1151] The "emotion recognition means" is a means for collecting biometric data of a user and analyzing the emotional state.

[1152] The "means for generating personalized information" is a means for generating information that is optimal for the user based on the analyzed emotional state.

[1153] The present invention relates to a driving assistance system that provides advanced assistance to drivers by combining an AR display device, data collection means, data analysis means, information generation means, display control means, voice dialogue means, route suggestion means, and emotion recognition means. This system allows drivers to receive real-time navigation information and warnings, as well as personalized driving assistance based on emotion recognition, without having to take their eyes off the road.

[1154] First, the device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures images of roads, vehicles, signs, pedestrians, etc., and the sensors obtain data such as distance and speed in real time. Biometric data such as the user's voice, facial expression, and heart rate are also collected. This provides data for the emotion recognition means (described later) to analyze the user's emotional state.

[1155] The collected data is sent to a server in real time. The server analyzes the received environmental data using image recognition algorithms and traffic prediction models. For example, a deep learning algorithm can be used to identify whether a pedestrian is crossing the street. An emotion engine can also be used to analyze the user's emotional state. This can identify whether the user is tense or relaxed.

[1156] The server uses the analysis results to determine the current location, direction of travel, and surrounding conditions. This includes combining received GPS data with image analysis results to accurately pinpoint the current location. It can also tailor navigation information and warnings based on the user's emotional state. For example, if the user is feeling nervous, it can suggest playing music to reduce stress.

[1157] The generated optimal navigation information and necessary warning information are sent to the device, which then displays it on the AR display and overlays it directly on the user's field of view. For example, an arrow can be displayed indicating a right turn point at the next intersection. The user can visually confirm this information and continue driving safely.

[1158] It also has a voice dialogue interface, allowing users to ask questions or give instructions by voice while driving. For example, it can issue prompts such as, "Where is the next right turn?" These voice instructions are analyzed by a voice recognition engine and sent to the server as text data. The server analyzes the voice instructions, generates appropriate information and responses, and sends them back to the device. The device then provides this information to the user again as audio or visual information. For example, it can provide specific guidance such as, "Turn right at the intersection 300 meters ahead."

[1159] It also uses emotion recognition to generate relaxing responses if the user is feeling tense, such as "Take a deep breath and relax." It can also learn from past driving data and user preferences, and combine this with real-time traffic information to calculate optimal routes. This allows it to provide personalized navigation suggestions based on the user's driving history.

[1160] Prompt Sentence Examples

[1161] "Where is the next right turn?"

[1162] "Show navigation information for turning left at the next intersection."

[1163] "Calculate a new route to avoid traffic jams."

[1164] "Generate a response that relaxes the user when they are tense."

[1165] The system improves driver concentration and safety, while providing personalized driving assistance.

[1166] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1167] Step 1: Data collection and environmental awareness

[1168] The device (AR display device) uses a camera and sensors to scan the surrounding environment. As input data, it uses image data (roads, vehicles, signs, pedestrians) captured by the camera and distance and speed data obtained by the sensors. This provides initial data on road conditions and direction of travel. Biometric data such as the user's voice, facial expression, and heart rate are also collected at the same time. Specifically, the camera captures an image of the road ahead, and the sensors measure the vehicle's speed and position. This data is sent to the server in real time.

[1169] Step 2: Data analysis

[1170] The server analyzes the input environmental and biometric data. For environmental data, it uses an image recognition algorithm to recognize road signs and pedestrians from captured images. Specifically, it uses a deep learning algorithm to identify, for example, whether a pedestrian is crossing a street. For biometric data, it uses an emotion engine to analyze the user's emotional state, such as tension or relaxation, from facial expression data. The results of this analysis are used by the server to generate navigation and warning information.

[1171] Step 3: Information generation

[1172] The server generates navigation information and warnings based on the results of data analysis. Specifically, it generates right-turn instructions at the next intersection and warnings if there are obstacles ahead based on the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents). Depending on the user's emotional state, it also suggests playing relaxing music. This information is sent to the device in the next step.

[1173] Step 4: Information overlay

[1174] The server sends the generated navigation information, warnings, and information based on emotion recognition to the device. Specifically, it sends data that displays an arrow indicating the right turn point at the next intersection. The device then overlays the received information on the AR display device. For example, it displays an instruction ahead saying "Turn right at the next intersection" with an arrow. The user visually confirms this information and continues driving safely.

[1175] Step 5: Voice interaction interface

[1176] While driving, the user issues questions or commands by voice. For example, "Where is the next right turn?" The device analyzes the user's voice using a voice recognition engine and converts it into text data. The server analyzes this voice instruction and generates appropriate information or a response, such as "Turn right at the intersection 300 meters ahead." The generated information is sent to the device, which then provides it to the user as audio or visual information.

[1177] Step 6: Personalized navigation suggestions

[1178] The server analyzes past driving data and user preferences based on a database. For example, it learns data on routes the user has previously chosen and areas they have avoided. It then calculates the optimal route by combining real-time traffic information with the user's emotional state. Specifically, it proposes a new avoidance route by taking into account current traffic congestion information and the user's relaxation state. The device overlays the proposed navigation route on the AR display and notifies the user by voice, saying, "A new route will be displayed."

[1179] (Application example 2)

[1180] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1181] Conventional driver assistance systems primarily aim to provide real-time environmental information and navigation, but they do not take into account the driver's mental state and lack methods to reduce stress and fatigue while driving. Therefore, personalized assistance that takes into account the driver's mental state is needed.

[1182] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data collection means for collecting biometric data of the user and performing emotion analysis, an emotion response generation means for providing personalized driving assistance based on the emotion analysis results, and a data analysis means for analyzing environmental data in real time and generating navigation information and warnings based on the analysis results. This enables personalized driving assistance according to the driver's mental state.

[1183] An "AR display device" is a device that uses augmented reality technology to overlay information onto the user's field of vision.

[1184] A "sensor" is a device for collecting data about the physical environment. Examples include distance sensors and accelerometers.

[1185] A "camera" is a device that captures video data.

[1186] The "data collection means" is a means for collecting environmental data and biometric data of the user.

[1187] "Data analysis means" refers to means for analyzing collected data and generating necessary information.

[1188] The "information generating means" is a means for generating navigation information and warnings based on the analysis results.

[1189] The "display control means" is a means for overlay-displaying the generated information on the AR display device.

[1190] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating a response.

[1191] The "route suggestion means" is a means for suggesting an optimal route based on real-time traffic information and driving history.

[1192] "Biometric data" refers to data related to the user's physical condition, such as heart rate and facial expression.

[1193] "Emotion analysis" is the process of assessing a user's emotional state based on their biometric data.

[1194] The "emotion response generating means" is a means for providing individualized driving assistance based on the emotion analysis results.

[1195] This invention is a driving assistance system that includes an AR display device, a data collection means equipped with a sensor and a camera, a data analysis means that analyzes environmental data in real time, an information generation means that generates navigation information and warnings based on the analysis results, a display control means that overlays and displays the information on the AR display device, a voice dialogue means that analyzes voice instructions and generates responses, a route proposal means that proposes an optimal route based on real-time traffic information and driving history, a data collection means that collects a user's biometric data and performs emotion analysis, and an emotion response generation means that provides personalized driving assistance based on the emotion analysis results.

[1196] System Program Overview

[1197] 1. Data Collection Methods

[1198] The device (smart glasses) uses a camera and sensors to scan the surrounding environment. For example, the camera captures road conditions, and the distance sensor measures the location of nearby vehicles and obstacles. In addition, the device collects biometric data such as gaze, facial expression, voice, and heart rate.

[1199] 2. Data analysis methods

[1200] Data sent from the device is sent in real time to a cloud server, where traffic information is analyzed using an image recognition algorithm (e.g., OpenCV). An emotion engine also analyzes the user's emotional state based on their heart rate and facial expression data.

[1201] 3. Information generation means

[1202] The cloud server generates navigation information and warnings based on the data analysis results, such as traffic congestion information, driving direction instructions, and emergency evacuation routes. Taking into account the emotion analysis results, the system displays relaxation advice if the user is feeling tense.

[1203] 4. Display Control Method

[1204] The generated information is sent to the device and overlaid on the AR display, allowing users to view navigation and warning information displayed directly in their field of view.

[1205] 5. Voice interaction means

[1206] Users can ask questions or give instructions by voice, and the device will analyze them using a voice recognition engine. The analysis results are sent to a cloud server, which generates an appropriate response. For example, in response to a question like, "Where is the next right turn?", the cloud server will generate and send an appropriate response based on the latest traffic information.

[1207] 6. Route suggestion method

[1208] The cloud server calculates the optimal route based on real-time traffic information and past driving history, taking into account the user's preferences and areas they want to avoid. The optimal route is sent to the device and displayed as an overlay on the AR display.

[1209] Hardware and software used

[1210] Hardware: Smart glasses (including camera, distance sensor, microphone, and heart rate sensor), cloud server.

[1211] Software: OpenCV (image analysis), emotion analysis algorithm (emotion engine), speech recognition engine (e.g. Google Speech Recognition API), data communication protocol (e.g. HTTP / HTTPS).

[1212] Specific examples

[1213] Scenario 1: Navigation while driving

[1214] Once the user sets their destination and starts driving, the cloud server analyzes environmental data in real time and generates driving direction and traffic congestion information. For example, instructions such as "Turn left at the next intersection" are overlaid in the smart glasses' field of view. If the user is feeling nervous, advice such as "Please drive safely" is also displayed.

[1215] Scenario 2: Sudden change in road conditions

[1216] When the device detects traffic congestion ahead using its camera, the data is sent to a cloud server. The cloud server then calculates a new route and sends it to the device. The smart glasses then overlay the new route information and provide a voice notification. If stress is detected, the glasses will also suggest a relaxing route.

[1217] Example prompts to input to the generative AI model

[1218] "Based on real-time camera footage and heart rate data, analyze traffic information and the driver's emotional state to generate appropriate driving assistance information."

[1219] This enables the driving assistance system to provide personalized information according to the user's emotional state. Real-time data collection, analysis, and response provide a safe and comfortable driving environment.

[1220] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1221] Step 1:

[1222] The user puts on the smart glasses and starts driving. The device uses cameras and sensors to scan the surrounding environment in real time, collecting data on road conditions, other vehicles, pedestrians, etc. It also collects biometric data such as gaze, facial expressions, voice, and heart rate.

[1223] Input: Environmental data (image, distance), biometric data (heart rate, facial expression, voice)

[1224] Output: Collected environmental and biological data

[1225] Step 2:

[1226] The device transmits the collected environmental and biological data to a cloud server in real time. The device transmits the data using a data communication protocol (e.g., HTTP / HTTPS).

[1227] Input: Collected environmental and biometric data

[1228] Output: Data sent to the cloud server

[1229] Step 3:

[1230] The cloud server analyzes the received environmental data. It uses image recognition algorithms (e.g., OpenCV) to identify and analyze road conditions and traffic information. It also uses an emotion engine to analyze the user's heart rate and facial expression data to evaluate their emotional state.

[1231] Input: Environmental and biological data sent to the cloud server

[1232] Output: Analyzed traffic information and user's emotional state

[1233] Step 4:

[1234] The cloud server uses the analysis results to generate navigation information, warnings, and personalized driving assistance based on the user's emotional state, such as "Turn left at the next intersection" or "Display a message to relax because you are nervous."

[1235] Input: Parsed traffic information and user's emotional state

[1236] Output: Generated navigation information, warnings, and driving assistance information based on emotional state

[1237] Step 5:

[1238] The cloud server sends the generated information to the device, which then overlays it on the smart glasses' AR display, allowing the user to see real-time navigation and warnings within their field of vision.

[1239] Input: Generated navigation information, warnings, and driving assistance information based on emotional state

[1240] Output: Information displayed on the smart glasses

[1241] Step 6:

[1242] Users can ask questions or give instructions by voice while driving. The device uses a voice recognition engine (e.g., Google Speech Recognition API) to analyze the voice data and convert it into text data. This text data is then sent to a cloud server.

[1243] Input: User's voice command

[1244] Output: Text data sent to the cloud server

[1245] Step 7:

[1246] The cloud server generates an appropriate response based on the received text data. For example, in response to a question such as "Where is the next right turn?", it generates a response based on the latest traffic information.

[1247] Input: Text data sent to the cloud server

[1248] Output: The generated response

[1249] Step 8:

[1250] The cloud server sends the generated response to the terminal, which then provides it to the user as audio and visual information, for example, the next right turn point is announced by audio and displayed as an overlay in the field of view.

[1251] Input: The generated response

[1252] Output: Audio and visual information presented to the user

[1253] Through the above steps, the driving assistance system can provide personalized information according to the user's emotional state, enabling safe and comfortable driving.

[1254] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1255] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1256] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1257] [Fourth embodiment]

[1258] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1259] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1260] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1261] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1262] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1263] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1264] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1265] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1266] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1267] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1268] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1269] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1270] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1271] This invention is a driving assistance system that combines an AR display device with data collection means, data analysis means, information generation means, display control means, voice interaction means, and route suggestion means. This system allows drivers to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving.

[1272] Program processing explanation

[1273] 1. Data collection and environmental awareness

[1274] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed.

[1275] The acquired data is sent to the server in real time.

[1276] 2. Data analysis and information generation

[1277] The server analyzes the received environmental data, which includes using image recognition algorithms and traffic prediction models.

[1278] Based on the analysis results, the server determines the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents).

[1279] The server generates optimal navigation information and necessary warning information.

[1280] 3. Information overlay

[1281] The server sends the generated navigation information and warnings to the terminal.

[1282] The device then displays the received information on the AR display device. Specifically, images and text are projected onto the AR glasses' display so that they are overlaid directly on the user's field of vision.

[1283] The user continues driving while visually checking this information.

[1284] 4. Voice Dialogue Interface

[1285] Users can ask questions or give instructions by voice while driving, for example, "Where is the next right turn?"

[1286] The terminal analyzes the user's voice using a voice recognition engine and sends the analysis results to the server as text data.

[1287] The server analyzes the voice instructions and generates appropriate information or responses, which are then sent back to the terminal, which then provides the information to the user as audio or visual information.

[1288] 5. Personalized navigation suggestions

[1289] The server analyzes past driving data and user preferences based on a database, learning, for example, the user's preferred routes and areas to avoid.

[1290] The server calculates and proposes the optimal route in combination with real-time traffic information, allowing users to select the most efficient and safe route at that time.

[1291] The device will overlay the suggested navigation route on the AR display and also provide an audio notification.

[1292] Specific examples

[1293] Scenario 1: Navigation while driving

[1294] After the user sets their destination, they begin driving. If they need to turn left at the next intersection, the server analyzes that information and sends it to the device. Based on the received information, the device overlays the instruction "Turn left at the next intersection" on the AR display device.

[1295] Scenario 2: Sudden change in road conditions

[1296] When the device detects a traffic jam on the road ahead using its camera, the data is sent to the server. The server analyzes the traffic jam information and calculates a new route to avoid it. The new route information is sent to the device, which then displays it as an overlay on the AR display and notifies the user by voice.

[1297] This allows the driving assistance system of the present invention to provide safe and intuitive assistance to the driver in real time.

[1298] The processing flow will be explained below.

[1299] Step 1:

[1300] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed.

[1301] Step 2:

[1302] The device processes and temporarily stores acquired environmental data in real time, including digital image processing and sensor data analysis.

[1303] Step 3:

[1304] The data collected by the terminal is sent to the server using a secure, low-latency communication protocol.

[1305] Step 4:

[1306] The server analyzes the received environmental data, using image recognition algorithms and traffic prediction models to understand the surroundings.

[1307] Step 5:

[1308] Based on the analysis results, the server determines the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents).

[1309] Step 6:

[1310] The server generates optimal navigation information and necessary warning information, such as specific instructions such as "turn left at the next intersection."

[1311] Step 7:

[1312] The server generates navigation information and alerts and sends them back to the device. The data is transmitted securely with low latency.

[1313] Step 8:

[1314] The navigation information received by the device is displayed on the AR display device. Specifically, images and text are projected so that they are overlaid directly on the user's field of view.

[1315] Step 9:

[1316] The user can continue driving while visually checking navigation information and warnings through the AR display device.

[1317] Step 10:

[1318] The user asks questions or gives instructions by voice while driving, for example, using voice commands such as "Where is the next right turn?"

[1319] Step 11:

[1320] The device analyzes the user's voice using a voice recognition engine and converts it into text data, which is then sent to the server.

[1321] Step 12:

[1322] The server analyzes the voice instructions and generates appropriate information or responses, such as "The next right turn is 200 meters away."

[1323] Step 13:

[1324] The server transmits the generated information to the terminal, which then provides the information to the user as audio or visual information.

[1325] Step 14:

[1326] The server analyzes past driving data and user preferences based on a database, and combines this with real-time traffic information to calculate the optimal route.

[1327] Step 15:

[1328] The server sends the proposed new navigation route to the terminal, which then displays it as an overlay on the AR display device and notifies the user by voice.

[1329] In this way, the driving assistance system of the present invention realizes real-time environment recognition, data analysis, generation and display of navigation information, and personalized driving assistance through a voice dialogue interface.

[1330] Example 1

[1331] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1332] From the perspective of safe driving, it is important for drivers to obtain necessary information without taking their eyes off the road. However, conventional navigation systems and driver assistance systems require drivers to pay attention to their smartphones or in-car displays, resulting in distraction and an increased risk of accidents. Furthermore, it is difficult to provide appropriate navigation information and warnings in real time, making it difficult for drivers to reach their destination safely and efficiently.

[1333] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1334] In this invention, the server includes a means for displaying visual information through an AR display device, a means for analyzing a user's voice input with a voice recognition engine and converting the analysis result into text, a means for analyzing the text data and generating an appropriate response, and a means for learning past driving data and personalized route information for the user to provide an optimal route. This allows the driver to obtain navigation information and warnings in real time without taking their eyes off the road, enabling safer and more intuitive driving.

[1335] An "AR display device" is a device that uses augmented reality technology to overlay information on the user's field of vision.

[1336] "Data collection means" refers to means for collecting surrounding environmental data using sensors and cameras.

[1337] "Data analysis means" refers to a means for analyzing collected environmental data in real time and extracting important information.

[1338] The "information generating means" is a means for generating navigation information and warning information based on the analysis results.

[1339] The "display control means" is a means for overlay-displaying the generated information on the AR display device.

[1340] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating an appropriate response.

[1341] The "route suggestion means" is a means for suggesting the optimal route based on real-time traffic information and driving history.

[1342] The "means for displaying visual information" refers to a means for displaying information directly into the user's field of vision through an AR display device.

[1343] A "voice recognition engine" is an engine that analyzes a user's voice input and converts it into text data.

[1344] The "means for analyzing text data" is a means for generating an appropriate response based on the text data generated by the voice recognition engine.

[1345] The "means for learning personalized route information" is a means for providing an optimal route based on past driving data and user preferences.

[1346] This invention is a driving assistance system that combines an AR display device with data collection means, data analysis means, information generation means, display control means, voice interaction means, and route suggestion means. This system allows drivers to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving.

[1347] Hardware and software used

[1348] Terminal (AR display device): Equipped with a camera and sensors (e.g., LiDAR sensor). The camera captures images at, for example, several tens of frames per second, and the LiDAR sensor obtains distance information about surrounding objects.

[1349] Server: A high-performance computer (e.g., a cloud server) analyzes the data using deep learning models (e.g., TensorFlow), image recognition algorithms (e.g., OpenCV), and traffic prediction models (e.g., Kalman filters).

[1350] Communication methods: Using high-speed networks such as 5G and Wi-Fi to transfer data in real time.

[1351] System processing overview

[1352] 1. Data collection and environmental recognition: The device collects data on the surrounding environment using a camera and sensors mounted on the front of the vehicle. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the LiDAR sensor obtains distance information. The device transmits this data to a server in real time.

[1353] 2. Data analysis and information generation: The server analyzes the received data and uses deep learning and traffic prediction models to identify vehicles and signs. Based on the analysis results, it generates navigation information and necessary warnings.

[1354] 3. Information overlay display: The server sends the generated navigation information and warnings to the device. The device displays the received information as an overlay on the AR display device. The user can continue driving while checking the information without taking their eyes off the road.

[1355] 4. Voice dialogue interface: Users can ask questions or give instructions by voice. The device analyzes the voice using a voice recognition engine and sends the results as text data to the server. The server analyzes the text data, generates an appropriate response, and sends it to the device. The device then provides the information to the user as audio or visual information.

[1356] 5. Personalized navigation suggestions: The server analyzes past driving data and user preferences to suggest optimal routes. It calculates the optimal route by combining it with real-time traffic information and sends that information to the device. The device then overlays the suggested route on the AR display and notifies the user via voice.

[1357] Specific examples

[1358] Scenario 1: Navigation while driving

[1359] After the user sets their destination, they begin driving. If they need to turn left at the next intersection, the server analyzes that information and sends it to the device. Based on the received information, the device overlays the instruction "Turn left at the next intersection" on the AR display device.

[1360] Scenario 2: Sudden change in road conditions

[1361] When the device first detects a traffic jam using its camera, the data is sent to the server. The server analyzes the congestion information and calculates a new route to avoid it. The new route information is sent to the device, which then displays it as an overlay on the AR display and notifies the driver via voice. This allows the system to provide safe and intuitive assistance to drivers in real time.

[1362] Prompt Sentence Examples

[1363] "Where is the next right turn?"

[1364] "Please suggest the best route taking traffic information into consideration."

[1365] "Please show us what signs your camera recognizes."

[1366] In this way, the server, terminal, and user each play their respective roles, collecting, analyzing, and displaying data in real time, creating a system that provides safe and intuitive driving assistance.

[1367] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1368] Step 1: Data collection

[1369] The device (AR display device) activates a camera and sensors (e.g., LiDAR sensor) mounted on the front of the vehicle to collect environmental data. Specifically, the camera captures images at tens of frames per second, and the LiDAR sensor obtains distance information of surrounding objects.

[1370] Input: Image data of the area in front of the vehicle and distance data of the surrounding area.

[1371] Output: Collected video and distance data.

[1372] Step 2: Send data

[1373] The device transmits the collected environmental data to a server in real time using 5G or Wi-Fi as the communication protocol, ensuring low latency data transfer.

[1374] Input: Collected video data and distance data.

[1375] Output: The raw environment data sent to the server.

[1376] Step 3: Receiving data

[1377] The server receives the environmental data sent from the terminal and stores it in a buffer for analysis. For example, it uses Apache Kafka for data stream management.

[1378] Input: Environmental data sent from the device.

[1379] Output: Received and buffered environmental data.

[1380] Step 4: Data analysis

[1381] The server analyzes the received environmental data, using deep learning models (e.g., TensorFlow) and image recognition algorithms (e.g., OpenCV) to identify vehicles, signs, pedestrians, etc.

[1382] Input: Buffered environmental data.

[1383] Output: Information about recognized objects (vehicles, signs, pedestrians, etc.).

[1384] Step 5: Information Generation

[1385] The server generates optimal navigation and warning information based on the analysis results, and uses traffic prediction models (e.g., Kalman filter) to evaluate real-time traffic conditions.

[1386] Input: Recognized object information and traffic prediction data.

[1387] Output: Navigation and warning information.

[1388] Step 6: Send information

[1389] The server transmits the generated navigation information and warning information to the terminal in real time.

[1390] Input: Navigation and warning information.

[1391] Output: Navigation and warning information sent to the device.

[1392] Step 7: Display information

[1393] The device overlays the received navigation and warning information on the AR display device, displaying arrows and warning text in the user's field of view.

[1394] Input: Navigation and warning information sent from the server.

[1395] Output: Information displayed on the AR display device.

[1396] Step 8: Recognizing voice input

[1397] Users can ask questions or give instructions by voice while driving, such as "Where is the next right turn?"

[1398] Input: User's voice command.

[1399] Output: Audio instructions.

[1400] Step 9: Audio Analysis

[1401] The device analyzes the user's voice using a voice recognition engine (e.g., Google Speech-to-Text or Amazon Alexa) and sends the analysis results to the server as text data.

[1402] Input: User's voice command.

[1403] Output: Parsed text data.

[1404] Step 10: Generate voice response

[1405] The server analyzes the voice instructions and generates appropriate information or responses, such as "The next right turn is 300 meters away."

[1406] Input: Parsed text data.

[1407] Output: Audio and text responses.

[1408] Step 11: Send a voice response

[1409] The server sends the generated voice response to the terminal, which then presents the information to the user as voice.

[1410] Input: The generated speech response.

[1411] Output: The audio response sent to the device.

[1412] Step 12: Personalized navigation suggestions

[1413] The server analyzes a database based on past driving data and user preferences, and combines this with real-time traffic information to suggest the optimal route.

[1414] Inputs: Historical driving data, user preferences, real-time traffic information.

[1415] Output: personalized optimal route information.

[1416] The information collected, analyzed, and generated at each step is provided in real time to improve driver safety and convenience, allowing drivers to obtain the necessary information in a timely manner without taking their eyes off the road.

[1417] (Application example 1)

[1418] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1419] Conventional driver assistance systems have the problem of compromising safety while driving because drivers have to look away from the road to check navigation information and emergency warnings. Furthermore, they lack the ability to suggest optimal routes based on real-time traffic conditions and the user's driving history, and they also lack functionality to reduce user stress and fatigue.

[1420] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1421] In this invention, the server includes an AR display device, a data collection means equipped with sensors and a camera, a data analysis means for analyzing environmental data in real time, an information generation means for generating navigation information and warnings based on the analysis results, a display control means for overlaying the information on the AR display device, a voice dialogue means for analyzing voice instructions and generating responses, a route proposal means for proposing an optimal route based on real-time traffic information and driving history, a voice and visual information provision means for providing the analysis results as visual and voice information through smart glasses, a voice analysis means for analyzing voice using a voice recognition engine and generating appropriate information and responses, and a route calculation means for calculating and proposing a stress-reducing route based on past driving data and preferences. This allows the driver to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving. Furthermore, personalized route proposals based on the user's driving history and preferences enable efficient and stress-free driving.

[1422] - An "AR display device" is a device that displays digital information overlaid on images from the real world.

[1423] A "sensor" is a device for acquiring physical environmental data.

[1424] A "camera" is a device for capturing images or video using an optical device.

[1425] "Data collection means" is a general term for devices and technologies used to collect environmental data.

[1426] A "data analysis means" is a device or method for analyzing collected data and converting it into meaningful information.

[1427] The "information generation means" is a mechanism that generates navigation information and warnings based on the analyzed data.

[1428] The "display control means" is a mechanism for appropriately overlaying and displaying the generated information on the AR display device.

[1429] A "voice interaction means" is a device or system for analyzing voice input and interacting with a user.

[1430] The "route suggestion method" is a system that calculates and suggests the optimal route based on real-time traffic information and driving history.

[1431] The "audio and visual information providing means" is a mechanism for providing the analysis results to the smart glasses as visual and audio information.

[1432] A "voice recognition engine" is software or hardware that converts voice into text data and analyzes it.

[1433] "Speech analysis means" refers to a device or method that uses a speech recognition engine to analyze speech and generate appropriate information or responses.

[1434] A "stress-reducing route" is an optimal driving route designed to reduce stress and fatigue based on the user's driving history and preferences.

[1435] This invention is a driving assistance system that combines an AR display device with data collection means, voice interaction means, and route suggestion means. This system allows users to obtain real-time navigation information and warnings without taking their eyes off the road, enabling safe and intuitive driving.

[1436] System Configuration

[1437] AR display devices can take the form of smart glasses or head-mounted displays that are worn by the user and overlay navigation information and warnings onto the real-world scene.

[1438] Data collection methods include cameras (e.g., common webcams) and sensors (e.g., Lidar sensors). These devices capture data about the surrounding environment in real time and transmit it to a server.

[1439] As a data analysis method, the server analyzes the received data and uses image recognition algorithms (e.g., OpenCV) and generative AI models to analyze the environmental data.

[1440] The information generating means generates navigation information and warnings based on the analysis results, and the generated information is sent from the server to the terminal.

[1441] The display control means overlays the generated information on the AR display device, making it easier for the user to visually confirm the information.

[1442] For voice interaction, a speech recognition engine (e.g., the transformers library) is used to analyze the user's voice instructions. The server generates appropriate information and responses and provides them to the user as audio and visual information.

[1443] In the route suggestion method, the server calculates the optimal route based on real-time traffic information and driving history, taking into account past driving data and user preferences.

[1444] Program processing explanation

[1445] The server receives environmental data from cameras and sensors and uses image recognition algorithms to identify roads, vehicle signs, pedestrians, etc., in order to analyze the collected data. Based on the analysis results, it generates appropriate navigation information and warnings.

[1446] The generated information is sent to the AR display device and overlaid on the user's field of view, allowing the user to check the information without taking their eyes off the road.

[1447] In addition, by using the voice interaction function, users can receive driving route suggestions to reduce fatigue and stress. For example, when a user says, "Please tell me the shortest route to my next destination" or "What is the safest route from my current location?", the server calculates appropriate route information and suggests it in real time.

[1448] Specific example explanation

[1449] Scenario 1: Navigation while driving

[1450] The user sets their destination and starts driving. If they need to turn left at the next intersection, the server analyzes that information and displays an overlay on the AR display saying, "Turn left at the next intersection."

[1451] Scenario 2: Sudden change in road conditions

[1452] When the device detects a traffic jam on the road ahead using its camera, the data is sent to the server. The server analyzes the traffic jam information and calculates a new route to avoid it. The new route information is sent to the device, which then displays it as an overlay on the AR display and notifies the user by voice.

[1453] Prompt Sentence Examples

[1454] "Please tell me the shortest route to my next destination."

[1455] "What is the safest route from my current location?"

[1456] "Please suggest a detour route to avoid the congestion ahead."

[1457] This allows the driving assistance system of the present invention to provide safe and intuitive assistance to the driver in real time, thereby improving the user's driving experience.

[1458] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1459] Step 1:

[1460] Data collection and environmental awareness

[1461] When a user starts driving, the camera and sensors attached to the device collect data on the surrounding environment. Specifically, the camera captures road conditions, traffic signs, and pedestrians in real time, while the sensors measure the vehicle's speed and distance. This data is sent as input from the device to the server, which then receives it.

[1462] Input: Camera image data, sensor measurement data

[1463] Output: Environment data sent to the server

[1464] Step 2:

[1465] Data analysis and information generation

[1466] The server analyzes the received environmental data, using image recognition algorithms (e.g., OpenCV) and generative AI models to recognize specific objects (e.g., signs, pedestrians, other vehicles) and analyze their location and movement. It also takes into account real-time traffic information and historical driving data. Based on the analysis results, the server generates navigation information and warnings.

[1467] Input: Environmental data (video data, sensor measurement data), real-time traffic information, driving history data

[1468] Output: Analysis results, navigation information, warning information

[1469] Step 3:

[1470] Information overlay

[1471] The generated navigation information and warnings are sent from the server to the device. The device's AR display device overlays this information in the user's field of view. Specifically, guidance such as "Turn left at the next intersection" is displayed on the AR display device's screen. This step allows the user to check the information without taking their eyes off the road.

[1472] Input: Navigation information, warning information

[1473] Output: Navigation information and warning information overlaid on the AR display device

[1474] Step 4:

[1475] Voice Dialogue Interface

[1476] When a user speaks a question or command while driving, the device analyzes the voice using a voice recognition engine. For example, if the user says, "Please tell me the shortest route to my next destination," the voice data is input into the voice recognition engine and converted into text data. The converted data is sent to the server, which generates the appropriate information or response.

[1477] Input: User's voice commands

[1478] Output: Parsed text data, response information

[1479] Step 5:

[1480] Responses and information

[1481] The server analyzes the voice instructions and generates response information, which is then sent to the device. The device then overlays this information on the AR display and simultaneously notifies the user by voice, allowing the user to receive information both visually and audibly.

[1482] Input: Parsed text data, response information

[1483] Output: Response information overlaid on the AR display device, audio notification

[1484] Step 6:

[1485] Personalized route suggestions

[1486] The server calculates a stress-reducing route based on past driving data and the user's preferences. It uses an algorithm that prioritizes routes that avoid traffic jams and routes that the user prefers. The calculated route information is sent to the device, which then displays it on an AR display and notifies the user via voice.

[1487] Input: Driving history data, user preferences, real-time traffic information

[1488] Output: Calculated route information, overlaid route information, audio notification

[1489] As a result, driver assistance systems can provide accurate, real-time navigation information and warnings to support users as they drive, allowing them to continue driving safely and intuitively without taking their eyes off the road.

[1490] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1491] This invention is a driving assistance system that combines an AR display device, data collection means, data analysis means, information generation means, display control means, voice dialogue means, route suggestion means, and an emotion engine. This system allows drivers to receive real-time navigation information and warnings, as well as personalized driving assistance based on emotion recognition, without taking their eyes off the road.

[1492] Program processing explanation

[1493] 1. Data collection and environmental awareness

[1494] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed.

[1495] Furthermore, the device collects biometric data such as the user's voice, facial expressions, and heart rate, and obtains data for analyzing the user's emotional state using an emotion engine.

[1496] The acquired data is sent to the server in real time.

[1497] 2. Data analysis and information generation

[1498] The server analyzes the received environmental data, which includes using image recognition algorithms and traffic prediction models.

[1499] The server uses an emotion engine to analyze the user's emotional state, and based on the analysis results, identifies the user's emotional state, such as whether they are tense or relaxed.

[1500] Based on the analysis results, the server identifies the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents), and adjusts navigation information and warnings according to the user's emotional state.

[1501] The server generates optimal navigation information, necessary warning information, and personalized information based on emotional state.

[1502] 3. Information overlay

[1503] The server transmits the generated navigation information, warnings, and information based on emotion recognition to the terminal.

[1504] The device then displays the received information on the AR display device. Specifically, images and text are projected onto the user's field of view as an overlay.

[1505] The user continues driving while visually checking this information.

[1506] 4. Voice Dialogue Interface

[1507] Users can ask questions or give instructions by voice while driving, for example, "Where is the next right turn?"

[1508] The terminal analyzes the user's voice using a voice recognition engine and sends the analysis results to the server as text data.

[1509] The server analyzes the voice instructions and generates appropriate information or responses, which are then sent back to the terminal, which then provides the information to the user as audio or visual information.

[1510] Through the emotion engine, if the user is tense, the server generates a response that has a relaxation effect.

[1511] 5. Personalized navigation suggestions

[1512] The server analyzes past driving data and user preferences based on a database, learning, for example, the user's preferred routes and areas to avoid.

[1513] The server combines real-time traffic information with the user's emotional state to calculate and suggest the optimal route.

[1514] The device will overlay the suggested navigation route on the AR display and also provide an audio notification.

[1515] Specific examples

[1516] Scenario 1: Navigation while driving

[1517] After setting a destination, the user begins driving. If it is necessary to turn left at the next intersection, the server analyzes that information and sends it to the device. Based on the received information, the device overlays the instruction "Turn left at the next intersection" on the AR display device, and if the user is nervous, displays additional instructions such as "Please drive safely."

[1518] Scenario 2: Sudden change in road conditions

[1519] When the device detects a traffic jam on the road ahead using its camera, the data is sent to the server. The server analyzes the traffic information and calculates a new route to avoid it. The new route information is sent to the device, which then overlays it on the AR display and notifies the user via voice. If the user is feeling stressed, the device will also suggest a relaxing route.

[1520] As a result, the driving assistance system of the present invention can realize real-time environment recognition, data analysis, generation and display of navigation information, and personalized driving assistance through a voice dialogue interface, and can also provide appropriate support based on the user's emotional state.

[1521] The processing flow will be explained below.

[1522] Step 1:

[1523] The device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures roads, vehicles, signs, pedestrians, etc., and the sensors acquire data such as distance and speed. It also collects biometric data such as the user's voice, facial expression, and heart rate.

[1524] Step 2:

[1525] The device processes and temporarily stores acquired environmental and biometric data in real time, including digital image processing and biometric data analysis.

[1526] Step 3:

[1527] The data collected by the terminal is sent to the server using a secure, low-latency communication protocol.

[1528] Step 4:

[1529] The server analyzes the received environmental data, using image recognition algorithms and traffic prediction models to identify surrounding conditions (e.g., traffic jams, construction zones, accidents).

[1530] Step 5:

[1531] The server uses an emotion engine to analyze the user's biometric data, including voice tone analysis, facial expression recognition, and heart rate analysis, to identify the user's emotional state (e.g., tension, stress, relaxation).

[1532] Step 6:

[1533] Based on the analysis results, the server combines the current location, direction of travel, and surrounding conditions to generate optimal navigation information and necessary warning information. The content of the information and warnings is adjusted according to the user's emotional state.

[1534] Step 7:

[1535] The server generates navigation information, warnings, and personalized information based on emotion recognition, and sends the data securely with low latency to the device.

[1536] Step 8:

[1537] The device displays the received information on the AR display device. Specifically, images and text are projected as an overlay directly onto the user's field of vision. For example, instructions such as "Turn left at the next intersection" or "Drive safely" are included.

[1538] Step 9:

[1539] The user can continue driving while visually checking navigation information and warnings through the AR display device, and can also receive emotionally appropriate advice on how to ease tension as needed.

[1540] Step 10:

[1541] The user asks questions or gives instructions by voice while driving, for example, using voice commands such as "Where is the next right turn?"

[1542] Step 11:

[1543] The device analyzes the user's voice using a voice recognition engine and converts it into text data, which is then sent to the server.

[1544] Step 12:

[1545] The server analyzes the voice instructions and generates appropriate information and responses, such as "The next right turn is 200 meters away." If the emotion engine detects stress or tension in the user, it generates a response with a relaxation effect.

[1546] Step 13:

[1547] The server transmits the generated information to the terminal, which then provides the information to the user as audio or visual information.

[1548] Step 14:

[1549] The server analyzes a database of past driving data and user preferences, and combines real-time traffic information and the user's emotional state to calculate the optimal route, taking into account, for example, scenic routes with a relaxing effect.

[1550] Step 15:

[1551] The server sends the proposed new navigation route to the device, which then displays it as an overlay on the AR display and notifies the user by voice. The guidance may include guidance designed to have a relaxing effect.

[1552] In this way, the driving assistance system of the present invention realizes real-time environment recognition, data analysis, generation and display of navigation information, and personalized driving assistance based on emotional state.

[1553] Example 2

[1554] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1555] While conventional driving assistance systems can provide environmental data and traffic information, they have the problem of being unable to provide personalized assistance that takes into account the user's emotional state. As a result, appropriate assistance may not be provided when the driver is feeling nervous or stressed, which could result in a decrease in driving safety.

[1556] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an emotion recognition means that collects biometric data of the user and analyzes the emotional state, an information generation means that generates personalized information based on the emotional state, and a route suggestion means that suggests an optimal route based on real-time traffic information and driving history. This enables personalized driving assistance that takes the user's emotional state into consideration.

[1557] An "AR display device" is a device that uses augmented reality technology to display information and overlay it directly on a user's field of view.

[1558] "Data collection means" refers to means for acquiring environmental data and biological data using sensors and cameras.

[1559] "Data analysis means" refers to means for analyzing collected data in real time and extracting necessary information.

[1560] The "information generating means" is a means for generating navigation information and warnings based on the results of data analysis.

[1561] The "display control means" is a means for overlay-displaying the generated information on the AR display device.

[1562] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating an appropriate response.

[1563] The "route suggestion means" is a means for suggesting the optimal route based on real-time traffic information and driving history.

[1564] The "emotion recognition means" is a means for collecting biometric data of a user and analyzing the emotional state.

[1565] The "means for generating personalized information" is a means for generating information that is optimal for the user based on the analyzed emotional state.

[1566] The present invention relates to a driving assistance system that provides advanced assistance to drivers by combining an AR display device, data collection means, data analysis means, information generation means, display control means, voice dialogue means, route suggestion means, and emotion recognition means. This system allows drivers to receive real-time navigation information and warnings, as well as personalized driving assistance based on emotion recognition, without having to take their eyes off the road.

[1567] First, the device (AR display device) scans the surrounding environment using a camera and sensors. Specifically, the camera captures images of roads, vehicles, signs, pedestrians, etc., and the sensors obtain data such as distance and speed in real time. Biometric data such as the user's voice, facial expression, and heart rate are also collected. This provides data for the emotion recognition means (described later) to analyze the user's emotional state.

[1568] The collected data is sent to a server in real time. The server analyzes the received environmental data using image recognition algorithms and traffic prediction models. For example, a deep learning algorithm can be used to identify whether a pedestrian is crossing the street. An emotion engine can also be used to analyze the user's emotional state. This can identify whether the user is tense or relaxed.

[1569] The server uses the analysis results to determine the current location, direction of travel, and surrounding conditions. This includes combining received GPS data with image analysis results to accurately pinpoint the current location. It can also tailor navigation information and warnings based on the user's emotional state. For example, if the user is feeling nervous, it can suggest playing music to reduce stress.

[1570] The generated optimal navigation information and necessary warning information are sent to the device, which then displays it on the AR display and overlays it directly on the user's field of view. For example, an arrow can be displayed indicating a right turn point at the next intersection. The user can visually confirm this information and continue driving safely.

[1571] It also has a voice dialogue interface, allowing users to ask questions or give instructions by voice while driving. For example, it can issue prompts such as, "Where is the next right turn?" These voice instructions are analyzed by a voice recognition engine and sent to the server as text data. The server analyzes the voice instructions, generates appropriate information and responses, and sends them back to the device. The device then provides this information to the user again as audio or visual information. For example, it can provide specific guidance such as, "Turn right at the intersection 300 meters ahead."

[1572] It also uses emotion recognition to generate relaxing responses if the user is feeling tense, such as "Take a deep breath and relax." It can also learn from past driving data and user preferences, and combine this with real-time traffic information to calculate optimal routes. This allows it to provide personalized navigation suggestions based on the user's driving history.

[1573] Prompt Sentence Examples

[1574] "Where is the next right turn?"

[1575] "Show navigation information for turning left at the next intersection."

[1576] "Calculate a new route to avoid traffic jams."

[1577] "Generate a response that relaxes the user when they are tense."

[1578] The system improves driver concentration and safety, while providing personalized driving assistance.

[1579] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1580] Step 1: Data collection and environmental awareness

[1581] The device (AR display device) uses a camera and sensors to scan the surrounding environment. As input data, it uses image data (roads, vehicles, signs, pedestrians) captured by the camera and distance and speed data obtained by the sensors. This provides initial data on road conditions and direction of travel. Biometric data such as the user's voice, facial expression, and heart rate are also collected at the same time. Specifically, the camera captures an image of the road ahead, and the sensors measure the vehicle's speed and position. This data is sent to the server in real time.

[1582] Step 2: Data analysis

[1583] The server analyzes the input environmental and biometric data. For environmental data, it uses an image recognition algorithm to recognize road signs and pedestrians from captured images. Specifically, it uses a deep learning algorithm to identify, for example, whether a pedestrian is crossing a street. For biometric data, it uses an emotion engine to analyze the user's emotional state, such as tension or relaxation, from facial expression data. The results of this analysis are used by the server to generate navigation and warning information.

[1584] Step 3: Information generation

[1585] The server generates navigation information and warnings based on the results of data analysis. Specifically, it generates right-turn instructions at the next intersection and warnings if there are obstacles ahead based on the current location, direction of travel, and surrounding conditions (e.g., traffic jams, construction zones, accidents). Depending on the user's emotional state, it also suggests playing relaxing music. This information is sent to the device in the next step.

[1586] Step 4: Information overlay

[1587] The server sends the generated navigation information, warnings, and information based on emotion recognition to the device. Specifically, it sends data that displays an arrow indicating the right turn point at the next intersection. The device then overlays the received information on the AR display device. For example, it displays an instruction ahead saying "Turn right at the next intersection" with an arrow. The user visually confirms this information and continues driving safely.

[1588] Step 5: Voice interaction interface

[1589] While driving, the user issues questions or commands by voice. For example, "Where is the next right turn?" The device analyzes the user's voice using a voice recognition engine and converts it into text data. The server analyzes this voice instruction and generates appropriate information or a response, such as "Turn right at the intersection 300 meters ahead." The generated information is sent to the device, which then provides it to the user as audio or visual information.

[1590] Step 6: Personalized navigation suggestions

[1591] The server analyzes past driving data and user preferences based on a database. For example, it learns data on routes the user has previously chosen and areas they have avoided. It then calculates the optimal route by combining real-time traffic information with the user's emotional state. Specifically, it proposes a new avoidance route by taking into account current traffic congestion information and the user's relaxation state. The device overlays the proposed navigation route on the AR display and notifies the user by voice, saying, "A new route will be displayed."

[1592] (Application example 2)

[1593] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1594] Conventional driver assistance systems primarily aim to provide real-time environmental information and navigation, but they do not take into account the driver's mental state and lack methods to reduce stress and fatigue while driving. Therefore, personalized assistance that takes into account the driver's mental state is needed.

[1595] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data collection means for collecting biometric data of the user and performing emotion analysis, an emotion response generation means for providing personalized driving assistance based on the emotion analysis results, and a data analysis means for analyzing environmental data in real time and generating navigation information and warnings based on the analysis results. This enables personalized driving assistance according to the driver's mental state.

[1596] An "AR display device" is a device that uses augmented reality technology to overlay information onto the user's field of vision.

[1597] A "sensor" is a device for collecting data about the physical environment. Examples include distance sensors and accelerometers.

[1598] A "camera" is a device that captures video data.

[1599] The "data collection means" is a means for collecting environmental data and biometric data of the user.

[1600] "Data analysis means" refers to means for analyzing collected data and generating necessary information.

[1601] The "information generating means" is a means for generating navigation information and warnings based on the analysis results.

[1602] The "display control means" is a means for overlay-displaying the generated information on the AR display device.

[1603] The "voice dialogue means" is a means for analyzing the user's voice instructions and generating a response.

[1604] The "route suggestion means" is a means for suggesting an optimal route based on real-time traffic information and driving history.

[1605] "Biometric data" refers to data related to the user's physical condition, such as heart rate and facial expression.

[1606] "Emotion analysis" is the process of assessing a user's emotional state based on their biometric data.

[1607] The "emotion response generating means" is a means for providing individualized driving assistance based on the emotion analysis results.

[1608] This invention is a driving assistance system that includes an AR display device, a data collection means equipped with a sensor and a camera, a data analysis means that analyzes environmental data in real time, an information generation means that generates navigation information and warnings based on the analysis results, a display control means that overlays and displays the information on the AR display device, a voice dialogue means that analyzes voice instructions and generates responses, a route proposal means that proposes an optimal route based on real-time traffic information and driving history, a data collection means that collects a user's biometric data and performs emotion analysis, and an emotion response generation means that provides personalized driving assistance based on the emotion analysis results.

[1609] System Program Overview

[1610] 1. Data Collection Methods

[1611] The device (smart glasses) uses a camera and sensors to scan the surrounding environment. For example, the camera captures road conditions, and the distance sensor measures the location of nearby vehicles and obstacles. In addition, the device collects biometric data such as gaze, facial expression, voice, and heart rate.

[1612] 2. Data analysis methods

[1613] Data sent from the device is sent in real time to a cloud server, where traffic information is analyzed using an image recognition algorithm (e.g., OpenCV). An emotion engine also analyzes the user's emotional state based on their heart rate and facial expression data.

[1614] 3. Information generation means

[1615] The cloud server generates navigation information and warnings based on the data analysis results, such as traffic congestion information, driving direction instructions, and emergency evacuation routes. Taking into account the emotion analysis results, the system displays relaxation advice if the user is feeling tense.

[1616] 4. Display Control Method

[1617] The generated information is sent to the device and overlaid on the AR display, allowing users to view navigation and warning information displayed directly in their field of view.

[1618] 5. Voice interaction means

[1619] Users can ask questions or give instructions by voice, and the device will analyze them using a voice recognition engine. The analysis results are sent to a cloud server, which generates an appropriate response. For example, in response to a question like, "Where is the next right turn?", the cloud server will generate and send an appropriate response based on the latest traffic information.

[1620] 6. Route suggestion method

[1621] The cloud server calculates the optimal route based on real-time traffic information and past driving history, taking into account the user's preferences and areas they want to avoid. The optimal route is sent to the device and displayed as an overlay on the AR display.

[1622] Hardware and software used

[1623] Hardware: Smart glasses (including camera, distance sensor, microphone, and heart rate sensor), cloud server.

[1624] Software: OpenCV (image analysis), emotion analysis algorithm (emotion engine), speech recognition engine (e.g. Google Speech Recognition API), data communication protocol (e.g. HTTP / HTTPS).

[1625] Specific examples

[1626] Scenario 1: Navigation while driving

[1627] Once the user sets their destination and starts driving, the cloud server analyzes environmental data in real time and generates driving direction and traffic congestion information. For example, instructions such as "Turn left at the next intersection" are overlaid in the smart glasses' field of view. If the user is feeling nervous, advice such as "Please drive safely" is also displayed.

[1628] Scenario 2: Sudden change in road conditions

[1629] When the device detects traffic congestion ahead using its camera, the data is sent to a cloud server. The cloud server then calculates a new route and sends it to the device. The smart glasses then overlay the new route information and provide a voice notification. If stress is detected, the glasses will also suggest a relaxing route.

[1630] Example prompts to input to the generative AI model

[1631] "Based on real-time camera footage and heart rate data, analyze traffic information and the driver's emotional state to generate appropriate driving assistance information."

[1632] This enables the driving assistance system to provide personalized information according to the user's emotional state. Real-time data collection, analysis, and response provide a safe and comfortable driving environment.

[1633] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1634] Step 1:

[1635] The user puts on the smart glasses and starts driving. The device uses cameras and sensors to scan the surrounding environment in real time, collecting data on road conditions, other vehicles, pedestrians, etc. It also collects biometric data such as gaze, facial expressions, voice, and heart rate.

[1636] Input: Environmental data (image, distance), biometric data (heart rate, facial expression, voice)

[1637] Output: Collected environmental and biological data

[1638] Step 2:

[1639] The device transmits the collected environmental and biological data to a cloud server in real time. The device transmits the data using a data communication protocol (e.g., HTTP / HTTPS).

[1640] Input: Collected environmental and biometric data

[1641] Output: Data sent to the cloud server

[1642] Step 3:

[1643] The cloud server analyzes the received environmental data. It uses image recognition algorithms (e.g., OpenCV) to identify and analyze road conditions and traffic information. It also uses an emotion engine to analyze the user's heart rate and facial expression data to evaluate their emotional state.

[1644] Input: Environmental and biological data sent to the cloud server

[1645] Output: Analyzed traffic information and user's emotional state

[1646] Step 4:

[1647] The cloud server uses the analysis results to generate navigation information, warnings, and personalized driving assistance based on the user's emotional state, such as "Turn left at the next intersection" or "Display a message to relax because you are nervous."

[1648] Input: Parsed traffic information and user's emotional state

[1649] Output: Generated navigation information, warnings, and driving assistance information based on emotional state

[1650] Step 5:

[1651] The cloud server sends the generated information to the device, which then overlays it on the smart glasses' AR display, allowing the user to see real-time navigation and warnings within their field of vision.

[1652] Input: Generated navigation information, warnings, and driving assistance information based on emotional state

[1653] Output: Information displayed on the smart glasses

[1654] Step 6:

[1655] Users can ask questions or give instructions by voice while driving. The device uses a voice recognition engine (e.g., Google Speech Recognition API) to analyze the voice data and convert it into text data. This text data is then sent to a cloud server.

[1656] Input: User's voice command

[1657] Output: Text data sent to the cloud server

[1658] Step 7:

[1659] The cloud server generates an appropriate response based on the received text data. For example, in response to a question such as "Where is the next right turn?", it generates a response based on the latest traffic information.

[1660] Input: Text data sent to the cloud server

[1661] Output: The generated response

[1662] Step 8:

[1663] The cloud server sends the generated response to the terminal, which then provides it to the user as audio and visual information, for example, the next right turn point is announced by audio and displayed as an overlay in the field of view.

[1664] Input: The generated response

[1665] Output: Audio and visual information presented to the user

[1666] Through the above steps, the driving assistance system can provide personalized information according to the user's emotional state, enabling safe and comfortable driving.

[1667] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1668] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1669] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1670] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1671] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1672] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1673] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1674] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1675] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1676] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1677] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1678] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1679] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1680] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1681] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1682] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1683] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1684] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1685] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1686] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1687] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1688] The following is further disclosed regarding the above embodiment.

[1689] (Claim 1)

[1690] an AR display device;

[1691] a data collection means including a sensor and a camera;

[1692] a data analysis means for analyzing environmental data in real time;

[1693] an information generating means for generating navigation information and warnings based on the analysis results;

[1694] a display control means for overlaying and displaying the information on an AR display device;

[1695] a voice interaction means for analyzing voice instructions and generating responses;

[1696] a route suggestion means for suggesting an optimal route based on real-time traffic information and driving history;

[1697] Driver assistance systems including.

[1698] (Claim 2)

[1699] 2. The driving assistance system according to claim 1, wherein the data analysis means is configured to recognize surrounding traffic conditions and generate appropriate information and warnings.

[1700] (Claim 3)

[1701] 2. The driving assistance system according to claim 1, wherein the voice interaction means generates personalized information based on a voice instruction from the user and displays the personalized information on the AR display device.

[1702] "Example 1"

[1703] (Claim 1)

[1704] an AR display device;

[1705] a data collection means including a sensor and a camera;

[1706] a data analysis means for analyzing environmental data in real time;

[1707] an information generating means for generating navigation information and warnings based on the analysis results;

[1708] a display control means for overlaying and displaying the information on an AR display device;

[1709] a voice interaction means for analyzing voice instructions and generating responses;

[1710] a route suggestion means for suggesting an optimal route based on real-time traffic information and driving history;

[1711] means for displaying visual information via an AR display device;

[1712] A means for analyzing a user's voice input with a voice recognition engine and converting the analysis result into text;

[1713] means for analyzing the text data and generating an appropriate response;

[1714] A means for learning past driving data and user-specific route information to provide an optimal route;

[1715] A system including:

[1716] (Claim 2)

[1717] 2. The system according to claim 1, wherein the data analysis means is configured to recognize surrounding traffic conditions and generate appropriate information and warnings.

[1718] (Claim 3)

[1719] 2. The system according to claim 1, wherein the voice interaction means generates personalized information based on a voice instruction from the user and displays the personalized information on the AR display device.

[1720] "Application Example 1"

[1721] (Claim 1)

[1722] an AR display device;

[1723] a data collection means including a sensor and a camera;

[1724] a data analysis means for analyzing environmental data in real time;

[1725] an information generating means for generating navigation information and warnings based on the analysis results;

[1726] a display control means for overlaying and displaying the information on an AR display device;

[1727] a voice interaction means for analyzing voice instructions and generating responses;

[1728] a route suggestion means for suggesting an optimal route based on real-time traffic information and driving history;

[1729] an audio and visual information providing means for providing the analysis result as visual and audio information by smart glasses;

[1730] a voice analysis means for analyzing voice using a voice recognition engine and generating appropriate information or a response;

[1731] a route calculation means for calculating and suggesting a stress-reducing route based on past driving data and preferences;

[1732] A system including:

[1733] (Claim 2)

[1734] 2. The system according to claim 1, wherein the data analysis means is configured to recognize surrounding traffic conditions and generate appropriate information and warnings.

[1735] (Claim 3)

[1736] 2. The system according to claim 1, wherein the voice interaction means generates personalized information based on a voice instruction from the user and displays the personalized information on the AR display device.

[1737] "Example 2: Combining Emotion Engines"

[1738] (Claim 1)

[1739] an AR display device;

[1740] a data collection means including a sensor and a camera;

[1741] a data analysis means for analyzing environmental data in real time;

[1742] an information generating means for generating navigation information and warnings based on the analysis results;

[1743] a display control means for overlaying and displaying the information on an AR display device;

[1744] a voice interaction means for analyzing voice instructions and generating responses;

[1745] a route suggestion means for suggesting an optimal route based on real-time traffic information and driving history;

[1746] emotion recognition means for collecting biometric data of a user and analyzing the user's emotional state;

[1747] an information generating means for generating personalized information based on an emotional state;

[1748] A system including:

[1749] (Claim 2)

[1750] 2. The system according to claim 1, wherein the data analysis means is configured to recognize surrounding traffic conditions and generate appropriate information and warnings.

[1751] (Claim 3)

[1752] 2. The system according to claim 1, wherein the voice interaction means generates personalized information based on a voice instruction from the user and displays the personalized information on the AR display device.

[1753] "Application example 2 when combining emotion engines"

[1754] (Claim 1)

[1755] an AR display device;

[1756] a data collection means including a sensor and a camera;

[1757] a data analysis means for analyzing environmental data in real time;

[1758] an information generating means for generating navigation information and warnings based on the analysis results;

[1759] a display control means for overlaying and displaying the information on an AR display device;

[1760] a voice interaction means for analyzing voice instructions and generating responses;

[1761] a route suggestion means for suggesting an optimal route based on real-time traffic information and driving history;

[1762] a data collection means for collecting biometric data of a user and performing emotion analysis;

[1763] an emotional response generation means for providing personalized driving assistance based on the emotion analysis result;

[1764] A system including:

[1765] (Claim 2)

[1766] 2. The system according to claim 1, wherein the data analysis means is configured to recognize surrounding traffic conditions and generate appropriate information and warnings.

[1767] (Claim 3)

[1768] 2. The system according to claim 1, wherein the voice interaction means generates personalized information based on a voice instruction from the user and displays the personalized information on the AR display device. [Explanation of symbols]

[1769] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. an AR display device; a data collection means including a sensor and a camera; a data analysis means for analyzing environmental data in real time; an information generating means for generating navigation information and warnings based on the analysis results; a display control means for overlaying and displaying the information on an AR display device; a voice interaction means for analyzing voice instructions and generating responses; a route suggestion means for suggesting an optimal route based on real-time traffic information and driving history; Driver assistance systems including.

2. 2. The driving assistance system according to claim 1, wherein the data analysis means is configured to recognize surrounding traffic conditions and generate appropriate information and warnings.

3. 2. The driving assistance system according to claim 1, wherein the voice interaction means generates personalized information based on a voice instruction from the user and displays the personalized information on the AR display device.

Images