system

Abstract

We provide the system. [Solution] A means of acquiring geographic information in real time and analyzing traffic conditions, A means of monitoring the driver's condition and detecting stress and fatigue, A means of switching driving modes in conjunction with autonomous driving technology, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Anxiety and risks in driving are diverse. Driving in unfamiliar environments or in different cultural circles in particular causes many problems. Specifically, there are risks of traffic accidents, reduced judgment due to human stress and fatigue, and anxiety regarding the transition to autonomous driving technology. It is an issue to improve the current situation where safe and secure driving is hindered and to provide a system capable of accommodating various drivers.

Means for Solving the Problems

[0005] This invention provides optimal driving routes by acquiring geographic information in real time and analyzing traffic conditions. It also ensures driver safety by monitoring the driver's condition, detecting stress and fatigue, and suggesting timely breaks and other appropriate actions. Furthermore, it incorporates a mechanism for switching driving modes in conjunction with autonomous driving technology, reducing driver burden. In addition, it supports smooth driving in multicultural environments by recognizing road signs and providing translation and explanations of their information. By detecting emergencies and immediately executing appropriate backup responses, this system dramatically improves driving safety.

[0006] "Real-time" refers to a technology or function that processes information immediately and provides the results instantly.

[0007] "Geographic information" refers to a dataset associated with a specific geographic location, including map information and location data.

[0008] "Traffic conditions" refers to real-time information regarding the flow of vehicles on roads, congestion levels, and traffic regulations.

[0009] "Driver's condition" refers to the individual's physical and psychological health while driving, including stress and fatigue.

[0010] "Monitoring" is the process of continuously observing a specific object to detect changes or anomalies.

[0011] "Autonomous driving technology" refers to technology that enables vehicles to operate themselves based on their own judgment without human intervention.

[0012] "Switching driving modes" refers to the process of transitioning the vehicle's operating system to a different driving control method.

[0013] "Road signs" are visual symbols or signs installed on roads to indicate traffic regulations and warnings to drivers.

[0014] "Translation and commentary" is the act of adapting and reconstructing information belonging to a specific language or culture to another language or culture, and clarifying its meaning and purpose.

[0015] An "emergency situation" refers to a situation in which an unexpected danger or malfunction occurs while driving, requiring immediate action.

[0016] "Backup response" refers to the process of implementing alternative measures or support systems when the normal system fails to function. [Brief explanation of the drawing]

[0017] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Modes for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the language used in the following description will be explained.

[0020] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0021] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] This invention provides an advanced system for assisting driving. The system mainly consists of a server and terminals, and provides users with real-time driving information and assistance.

[0039] First, the server obtains geographical information from an external map provider and analyzes traffic conditions to calculate the optimal route for the user. This enables smooth travel while avoiding congestion.

[0040] Furthermore, the terminal monitors the driver's condition using cameras and voice recognition functions installed in the vehicle. The server analyzes this data and, if it determines that the driver is experiencing stress or fatigue, suggests a break to the user via the terminal. This function ensures safe driving.

[0041] Furthermore, the device uses its camera to recognize road signs and sends the data to a server. The server analyzes this data, translates it into the user's native language, and provides explanations of necessary traffic rules. This feature is especially useful when driving in a foreign country. The system also integrates with autonomous driving technology and can switch to autonomous driving mode when appropriate. This reduces the burden on the driver and enables safer driving under certain conditions.

[0042] In addition, in the event of an emergency, the system has the capability to immediately detect it and notify emergency services as a backup. For example, if an engine malfunction is detected, the terminal will warn the user, and the server will search for the nearest repair shop and provide guidance information.

[0043] In this way, this system provides multifaceted driving assistance and contributes to enabling users to drive with peace of mind.

[0044] The following describes the processing flow.

[0045] Step 1:

[0046] The server retrieves geographic information in real time from external map data provider APIs. This allows it to store the latest road conditions and traffic information in a database.

[0047] Step 2:

[0048] The server analyzes traffic conditions based on collected geographical information. It identifies congestion levels and accident locations and calculates the optimal route.

[0049] Step 3:

[0050] The device uses cameras and microphones installed inside the vehicle to monitor the driver's facial expressions and voice. This allows the system to understand the driver's psychological and physical state.

[0051] Step 4:

[0052] The server analyzes the driver's status data received from the terminal and determines the degree of stress and fatigue. If necessary, it generates a message to suggest that the user take a rest.

[0053] Step 5:

[0054] The device takes a picture of the road sign using its camera and sends the image data to the server. This prepares the system for recognizing the content of the road sign.

[0055] Step 6:

[0056] The server uses an image recognition algorithm to analyze road signs and translate them into the user's language. The results are returned to the terminal, and an explanation is displayed to the user.

[0057] Step 7:

[0058] The terminal monitors specific driving conditions and the driver's state, and sends instructions to the server when it determines that switching to autonomous driving mode is appropriate.

[0059] Step 8:

[0060] The server initiates control of the autonomous driving technology and notifies the user via the terminal that the vehicle is transitioning to autonomous driving. This reduces the driver's workload.

[0061] Step 9:

[0062] If a malfunction occurs in the vehicle, the terminal immediately sends data from the sensors to the server to notify the server that an emergency has occurred.

[0063] Step 10:

[0064] The server, as a backup measure for emergencies, will take appropriate countermeasures and contact emergency services if necessary. The terminal will provide the user with information about the current situation during this process.

[0065] (Example 1)

[0066] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0067] In today's traffic environment, drivers are required to travel safely and efficiently while dealing with increasing traffic volume and complex road conditions. Furthermore, driving while fatigued or stressed can increase the risk of accidents. Additionally, driving in a foreign country can present challenges due to language barriers and unfamiliarity with road signs. Therefore, there is a need for a system that provides comprehensive driver assistance, reduces the burden on drivers, and improves safety.

[0068] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0069] In this invention, the server includes means for acquiring location information and analyzing traffic conditions, means for monitoring the operator's condition and detecting mental and physical stress, and means for coordinating with machine control technology to select a driving mode. This makes it possible to present the driver with the optimal route in real time, automatically select a driving mode to enhance safety, and even suggest appropriate breaks through monitoring the driver's health condition.

[0070] "Location information" refers to data that indicates a geographical location and is used to identify one's current location or destination.

[0071] "Traffic conditions" refers to information that describes the traffic situation on a particular road or route, such as the degree of congestion or travel time.

[0072] "Operator" refers to the person who operates the vehicle or equipment, in other words, the driver.

[0073] "Mental and physical strain" refers to the mental and physical state and condition of the driver, including stress and fatigue.

[0074] "Mechanical control technology" refers to technologies that assist or replace manual driving, including autonomous driving functions and other automated control systems for vehicles.

[0075] "Visual information" refers to image and video data acquired by cameras and sensors, including road signs and the surrounding environment.

[0076] "Translation processing" is the process of accurately converting information between different languages, for example, translating signage information into a foreign language.

[0077] An "abnormal situation" refers to a situation such as an error or accident that deviates from normal operating conditions, and includes situations that require a rapid response.

[0078] This invention provides a driver assistance system that utilizes a server and terminals to achieve multifaceted functionality. The server acquires geographic information through location information services and uses external information provision services (e.g., geographic information APIs) to analyze traffic data. This allows the server to calculate the optimal route using real-time updated traffic information. The server performs the analysis using a generative AI model, and the calculation method includes the integration of real-time traffic data.

[0079] The terminal monitors the driver's condition using cameras and voice recognition devices installed in the vehicle. Specifically, it detects mental and physical stress by analyzing the driver's facial expressions and voice. The data acquired from the terminal is sent to a server and used to evaluate stress and fatigue levels. This process is important for ensuring driver safety.

[0080] Furthermore, the terminal can work in conjunction with machine control technology to switch the driving mode from manual to automatic operation when appropriate. This function reduces the burden on the driver and contributes to improved safety, especially during long-distance travel.

[0081] As part of visual information analysis, the terminal uses its camera to recognize road signs and transmits that information to a server. The server identifies the content of the signs through image processing, translates them as needed, and provides the information to the user. This ensures that accurate traffic information can be obtained even when driving in a foreign country.

[0082] Furthermore, in the event of an abnormal situation, the terminal monitors data from the vehicle's sensors and immediately notifies the server. The server then takes appropriate action based on the situation, such as providing information on how to get to the nearest maintenance facility.

[0083] A possible example of a specific prompt message would be, "Please describe in detail how the system works to provide safe driving assistance when a user is driving in a foreign country." This invention contributes to enabling drivers to operate vehicles with peace of mind through a variety of support functions.

[0084] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0085] Step 1:

[0086] The server receives the vehicle's current location and destination information from the terminal. Next, it uses location services to obtain real-time geographical information and traffic condition data. Based on the acquired data, the server analyzes the traffic situation and calculates the optimal travel route. During this process, it uses a generative AI model to perform the analysis and make necessary route adjustments to avoid congestion and traffic accidents. The output is the calculated optimal route information.

[0087] Step 2:

[0088] The terminal uses a camera and voice recognition device installed in the vehicle to monitor the driver's facial expressions and tone of voice. The terminal sends this data to a server, which uses a generated AI model to analyze the driver's stress and fatigue levels. Based on this analysis, it determines whether the driver needs a break. The output includes the analysis results and, if necessary, a break suggestion message.

[0089] Step 3:

[0090] The device captures road signs using an external camera and sends the image data to a server. The server analyzes the signs using image recognition technology and translates their content into a language the user can understand. The translated information is presented to the user either as audio or as text on the display. As an export, the translated sign information is generated.

[0091] Step 4:

[0092] The terminal monitors data from various sensors installed in the vehicle and reports any errors or abnormalities to the server. The server activates an emergency response protocol, for example, by searching for the nearest maintenance facility and sending its location information to the terminal. The user can then safely respond by following the instructions provided by the terminal. The output includes response instructions and emergency guidance information.

[0093] Step 5:

[0094] The user provides prompts regarding system operation via console or voice input. Based on these prompts, the server performs the necessary data processing and provides the requested information. As output, customized information tailored to the user's request is returned.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] In recent years, the increase in traffic accidents and problems caused by driver fatigue have become social issues. Furthermore, with the increase in international travel, language barriers when driving in foreign countries are also a challenge. To solve these problems, a system is needed that can understand the driver's psychological state and provide adaptive support. Improvements in safety are also required in autonomous driving.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes means for acquiring spatial information in real time and analyzing the movement situation, means for monitoring the driver's condition and detecting their psychological state and fatigue, means for coordinating with autonomous driving technology to change the driving mode, means for providing information within the driver's field of view using a head-mounted display, and means for understanding and responding to the driver's instructions using voice recognition technology. This enables safe and stress-free driving assistance.

[0100] "Real-time" is a technical concept that describes a state in which data and information are acquired and processed instantly.

[0101] "Spatial information" is a term that refers to information that includes data on geographical location and traffic conditions.

[0102] "Mobility status" is a term that refers to the results of an analysis of traffic flow and congestion levels.

[0103] "Driver's condition" refers to the driver's health, including their psychological and physical state.

[0104] "Psychological state" is a term that describes the internal condition of a driver, indicating their level of stress and tension.

[0105] "Fatigue" refers to physical and mental exhaustion resulting from prolonged driving or excessive concentration.

[0106] "Autonomous driving technology" refers to technology that allows vehicles to make their own decisions and drive themselves without human intervention.

[0107] "Changing the driving mode" refers to a control method that switches between manual and automatic driving depending on the situation.

[0108] A "head-mounted display" refers to a display device that a user wears and which projects digital information into their field of vision.

[0109] "Speech recognition technology" refers to the technology that allows a computer to understand human voices and process them as instructions.

[0110] The system that realizes this invention uses a terminal mounted on the vehicle and a central server. The terminal is equipped with a camera and an audio input device, and provides information to the driver via a head-mounted display.

[0111] The server acquires spatial information via the network and analyzes the latest movement status. It uses external information services such as Google Maps API to calculate the optimal travel route. It also analyzes the driver's voice commands using speech recognition technologies such as Amazon Transcribe to understand the driver's intentions.

[0112] The device analyzes the driver's camera footage in real time and uses a machine learning (ML) model to detect the driver's psychological state and fatigue. If it determines that the driver is fatigued, it displays a message in their field of view suggesting they take a break.

[0113] Furthermore, the device uses its camera to recognize road signs and translates and explains them using OCR technology such as the Google Cloud Vision API. The results are provided on a head-mounted display, providing particularly useful information when driving in a foreign country.

[0114] In the event of an emergency, the device immediately detects the situation and sends a notification to the server. The server includes functions to search for the nearest necessary services and provide appropriate advice to the driver.

[0115] For example, during a family trip, if the driver enters a foreign country, the camera can capture unfamiliar signs. This information is translated via a server, allowing the driver to instantly see the meaning of the sign and the applicable traffic rules on the screen, enabling them to drive with peace of mind.

[0116] An example of a prompt message might be, "Please tell me how to develop a system that uses a head-mounted display to translate foreign road signs in real time and support safe driving."

[0117] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0118] Step 1:

[0119] The server obtains current geographical information using a spatial information provision service. It sends location data as input to the API and receives data on travel routes and traffic conditions as output. This data is analyzed to calculate the optimal travel route.

[0120] Step 2:

[0121] The terminal monitors the driver using a camera installed inside the vehicle and acquires video data. Using the camera images as input, an ML model analyzes the driver's facial expressions and movements to determine fatigue and psychological state. The output determines whether the driver needs a break.

[0122] Step 3:

[0123] When the driver issues a voice command, the terminal receives the voice as input and converts it into text data using voice recognition software. The converted data is then analyzed to understand the driver's instructions and provide the necessary support.

[0124] Step 4:

[0125] The server receives image data of road signs transmitted from the terminal by the client. This data is processed using OCR technology to translate the content of the signs. The server generates translated text data as output and sends it to the terminal.

[0126] Step 5:

[0127] The device displays necessary information to the user through a head-mounted display. For example, it supports driver safety by displaying suggestions for breaks based on the driver's condition or information about road signs along the designated route.

[0128] Step 6:

[0129] When a user detects an emergency, the device sends that information to the server. Based on the information received as input, the server searches for emergency response services, provides the most suitable response as output, and sends guidance information to the device.

[0130] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0131] This invention is a system for driver assistance that provides advanced functions including real-time acquisition of geographic information, analysis of traffic conditions, and recognition of the driver's emotions. The system consists of a server, a terminal, and an emotion engine that evaluates the driver's real-time psychological state.

[0132] First, the server acquires geographic information in real time from external map information providers and constantly monitors the latest traffic conditions. This makes it possible to provide users with the optimal driving route. It can also analyze traffic flow and recommend routes that avoid congestion to users.

[0133] The terminal utilizes data from the vehicle's cameras and microphones to monitor the driver's facial expressions and voice. The emotion engine analyzes this data and evaluates the driver's emotional state in real time. Based on this evaluation, if the server determines that the driver is experiencing high stress levels, it can generate suggestions for breaks to alleviate that stress. The terminal notifies the user via display and audio.

[0134] Furthermore, the emotion engine accumulates emotional data during driving and analyzes it using a learning algorithm to provide more personalized driving assistance. This accumulated data helps to provide assistance and suggestions customized for each user. It also integrates camera recognition of road signs and subsequent translation and explanation functions to support safe driving both domestically and internationally.

[0135] In addition, the system has integration capabilities with autonomous driving technology, allowing for a seamless transition to autonomous driving mode under appropriate conditions. This transition is controlled by a server, and the terminal notifies the user. In the event of an emergency, sensors detect the anomaly and immediately execute backup measures to ensure safety.

[0136] For example, if the emotional engine detects high levels of fatigue while a user is driving a long distance, the server will guide the user to the nearest rest stop and issue a voice alert from the terminal saying, "There is a rest stop nearby. Would you like to take a break?"

[0137] In this way, this system utilizes a variety of sensors and advanced data analysis to provide users with an environment that allows them to drive safely and smoothly.

[0138] The following describes the processing flow.

[0139] Step 1:

[0140] The server obtains real-time geographic information from external map information providers via APIs. It stores this information in a database and continuously updates it with the latest traffic conditions.

[0141] Step 2:

[0142] The server utilizes stored geographical information to analyze traffic conditions and calculate the optimal route. This allows it to prepare recommended routes for users to avoid congestion.

[0143] Step 3:

[0144] The device uses cameras and microphones installed inside the vehicle to continuously capture data of the driver's facial expressions and voice.

[0145] Step 4:

[0146] The emotion engine analyzes facial expressions and voice data transmitted from the device to evaluate the driver's emotional state. It quantitatively determines stress levels and fatigue levels.

[0147] Step 5:

[0148] The server receives the analysis results from the emotion engine and, if the driver's stress or fatigue is high, generates a message suggesting a break.

[0149] Step 6:

[0150] The terminal notifies the user of messages received from the server via voice or display, and suggests the nearest rest area or appropriate relaxation methods.

[0151] Step 7:

[0152] The terminal uses the vehicle's camera to capture images of road signs and sends that data to the server.

[0153] Step 8:

[0154] The server analyzes the transmitted marker data using an image recognition algorithm, translates the information as needed, adds an explanation immediately afterward, and sends it back to the terminal.

[0155] Step 9:

[0156] The terminal provides the driver with translations and explanations from the server, either visually or audibly, to aid in understanding.

[0157] Step 10:

[0158] The terminal continuously monitors the user's driving patterns and status, and determines whether the conditions are met to prompt the server to switch to automatic driving mode.

[0159] Step 11:

[0160] The server controls the autonomous driving technology under appropriate conditions and instructs the terminal to smoothly transition into autonomous driving mode.

[0161] Step 12:

[0162] If an anomaly is detected, the terminal immediately sends the vehicle's sensor data to the server to notify the emergency.

[0163] Step 13:

[0164] The server will execute pre-configured backup procedures in response to an emergency and contact emergency services as necessary. The terminal will endeavor to alleviate user anxiety by providing detailed reports on the situation.

[0165] (Example 2)

[0166] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0167] For car drivers, there is a need for systems that improve traffic safety through the provision of appropriate information and support in real time. However, existing driver assistance systems have the challenge of not being able to adequately integrate and analyze geographical information, traffic conditions, and the driver's psychological state, and respond individually. Furthermore, there is a growing need for an integrated system that can reliably perform smooth transitions to autonomous driving modes and appropriate responses in emergencies.

[0168] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0169] In this invention, the server includes means for obtaining location information from a map information service and calculating a travel route, means for analyzing voice and visual data to evaluate a person's emotional state, and means for switching the vehicle's operating mode using automation technology. This enables the presentation of an optimal route that takes into account the driver's psychological state and real-time traffic conditions, as well as the safe and smooth switching of automated driving modes.

[0170] A "map information service" is a function that provides data related to geographical locations and travel routes.

[0171] "Location information" refers to data that indicates a specific point on Earth, and usually includes latitude and longitude.

[0172] "Means for calculating travel routes" refers to technologies that calculate the optimal route from the current location to the destination.

[0173] "Audio and visual data" is a general term for information expressed through a person's facial expressions and speech sounds.

[0174] "Methods for evaluating a person's emotional state" refer to technologies that analyze data such as voice and facial expressions to estimate a person's emotions.

[0175] "Means of switching the operating mode of a vehicle using automation technology" refers to control technology for transitioning from manual driving to automated driving.

[0176] "Psychological state" is a concept that describes a person's emotions and mental condition.

[0177] "Real-time traffic conditions" refers to information about the current flow of traffic and congestion levels.

[0178] "Presenting the optimal route" refers to the act of showing the user the most efficient or safest travel route under the current conditions.

[0179] "Safe and smooth switching to autonomous driving modes" refers to the process of initiating autonomous driving functions while minimizing risks.

[0180] This invention is an advanced driver assistance system that provides comprehensive support to drivers through the integration of real-time acquisition of geographic information, analysis of traffic conditions, recognition of driver emotions, and autonomous driving technology.

[0181] The server uses an API to obtain location information from a map information service. Specifically, it uses a common map data provider. This allows the server to calculate travel routes in real time and provide the user with the optimal route. For example, it can provide rapid route suggestions based on time of day and weather conditions.

[0182] The terminal uses cameras and microphones installed in the vehicle to collect audio and visual data from the driver. This data is sent to the emotion engine, where a machine learning algorithm evaluates the driver's emotional state. If the emotional state indicates stress or fatigue, the server suggests that the driver take a break.

[0183] Furthermore, the server manages the switching of operating modes from manual to automated driving based on automation technology. This enables safe and smooth automated driving, reducing the burden on the driver. In emergencies, the terminal uses sensors to immediately detect abnormalities and take appropriate action. For example, it can assist with sudden braking or automatically activate emergency lights.

[0184] For example, if the emotional engine detects a high stress level while the user is driving for an extended period, the server will guide the user to an optimal rest stop, and the terminal will issue a voice notification such as, "There is a rest stop ahead. Would you like to take a break?" An example of a prompt message could be, "Please advise what kind of driving support would be best based on the driver's situation."

[0185] In this way, this system combines various data processing technologies and driver assistance technologies to provide drivers with a safer and more comfortable driving experience.

[0186] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0187] Step 1:

[0188] The server obtains location information from a map information service API. Given geographic coordinates as input, it calculates surrounding traffic conditions and route information based on this data. This allows the server to generate data for the optimal route to suggest to the user.

[0189] Step 2:

[0190] The device uses the vehicle's camera and microphone to collect the driver's facial expressions and voice data. This data is used as input for the emotion engine. The emotion engine applies machine learning algorithms to evaluate the driver's emotional state. As output of this process, stress and fatigue levels are obtained as numerical data.

[0191] Step 3:

[0192] The server calculates appropriate support for the driver based on the assessment of their emotional state. For example, if stress levels are high, the system processes data to suggest nearby rest stops to the user. Traffic data and geographical information are used in this calculation, and a notification suggesting a rest stop is generated as output.

[0193] Step 4:

[0194] The terminal receives suggestions from the server and notifies the driver. Specifically, it provides voice guidance and display notifications, conveying concrete messages such as, "There is a rest area ahead. Would you like to take a break?" This allows the user to intuitively accept the system's suggestions.

[0195] Step 5:

[0196] The server evaluates environmental conditions and, if necessary, instructs the vehicle to switch to autonomous driving mode. Inputs include real-time traffic data and weather information, and the system selects the optimal driving mode as its output. This enables driving that balances safety and comfort.

[0197] (Application Example 2)

[0198] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0199] In today's driving environment, drivers often experience stress due to traffic congestion and long hours of driving. Furthermore, failing to take breaks at appropriate times or being unable to smoothly transition to autonomous driving mode can lead to accidents and a decrease in safety. To address this, it is necessary to monitor the driver's emotional state in real time and provide optimal assistance tailored to the driving situation.

[0200] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0201] In this invention, the server includes means for acquiring geographic information in real time and analyzing traffic conditions, means for monitoring the driver's state and detecting stress and fatigue, means for presenting stabilization information based on the driver's emotional state, and means for switching driving modes in cooperation with autonomous driving technology. This makes it possible to reduce driver stress and improve driving safety and efficiency.

[0202] "Methods for acquiring geographic information in real time and analyzing traffic conditions" refers to technologies that instantly acquire location information from external geographic information providers and use that data to analyze current traffic conditions.

[0203] "Means for monitoring the driver's condition and detecting stress and fatigue" refers to a technology that uses in-vehicle sensor devices to observe the driver's physiological indicators and evaluates the driver's psychological and physical burden from that data.

[0204] "Means of presenting stabilizing information based on the driver's emotional state" refers to technology that provides appropriate relaxation content and driving warnings based on the results of an analysis of the driver's emotions.

[0205] "Means of switching driving modes in conjunction with autonomous driving technology" refers to technology that controls the transition from manual driving to autonomous driving at a timing appropriate to the driving situation through information exchange with the autonomous driving system.

[0206] To realize this invention, the system consists of a server, a terminal, and an emotion engine that evaluates the user's emotional state. The server acquires geographic information in real time from an external organization that provides geographic information services. This allows the server to analyze the latest traffic conditions and provide the driver with the optimal route.

[0207] The device uses a camera and microphone mounted in the vehicle to collect the driver's facial expressions and voice, and transmits this data to the emotion engine. The emotion engine analyzes this data and evaluates the driver's emotional state in real time. If the evaluation indicates that the driver is experiencing high levels of stress or fatigue, the server sends relaxation content or warning messages to the device. The device then notifies the driver via voice and display.

[0208] For example, if the emotional engine detects stress in the driver during long drives, the server will play relaxation music and display a message on the terminal screen prompting the driver to take deep breaths.

[0209] In systems using generative AI models, personalized assistance can be provided to the driver by inputting prompts such as, "Assess the driver's emotional state while driving and suggest content to help them relax." This process utilizes emotion recognition libraries such as EmotionEngine and content playback libraries such as ContentPlayer.

[0210] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0211] Step 1:

[0212] The server acquires geographic information from external geographic information providers. The input is the current location, and the output is real-time traffic data. This data serves as the foundation for providing the optimal route.

[0213] Step 2:

[0214] The device uses a camera and microphone installed inside the vehicle to record the driver's facial expressions and voice. Input consists of camera video and audio data, and the output data is transmitted to the emotion engine. This collects basic data for understanding the driver's emotional state.

[0215] Step 3:

[0216] The emotion engine analyzes data transmitted from the terminal and evaluates the emotional state. The input is data related to the driver's facial expressions and voice, and the output determines the driver's emotional state. Based on this result, it is determined whether the driver is experiencing high levels of stress.

[0217] Step 4:

[0218] The server generates appropriate stabilization information based on the driver's emotional state determined by the emotion engine. The input is the driver's emotional state, and the output is relaxation content or warning messages. If the content is relaxation music, a generative AI model is used in the selection process.

[0219] Step 5:

[0220] The terminal receives stabilization information sent from the server and notifies the user. The input is stabilization information from the server, and the output is presented to the user via voice or display. Specifically, relaxation music is played for the driver, and a message encouraging deep breathing is displayed on the screen.

[0221] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0222] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0223] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0224] [Second Embodiment]

[0225] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0226] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0227] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0228] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0229] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0230] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0231] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0232] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0233] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0234] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0235] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0236] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0237] This invention provides an advanced system for assisting driving. The system mainly consists of a server and terminals, and provides users with real-time driving information and assistance.

[0238] First, the server obtains geographical information from an external map provider and analyzes traffic conditions to calculate the optimal route for the user. This enables smooth travel while avoiding congestion.

[0239] Furthermore, the terminal monitors the driver's condition using cameras and voice recognition functions installed in the vehicle. The server analyzes this data and, if it determines that the driver is experiencing stress or fatigue, suggests a break to the user via the terminal. This function ensures safe driving.

[0240] Furthermore, the device uses its camera to recognize road signs and sends the data to a server. The server analyzes this data, translates it into the user's native language, and provides explanations of necessary traffic rules. This feature is especially useful when driving in a foreign country. The system also integrates with autonomous driving technology and can switch to autonomous driving mode when appropriate. This reduces the burden on the driver and enables safer driving under certain conditions.

[0241] In addition, in the event of an emergency, the system has the capability to immediately detect it and notify emergency services as a backup. For example, if an engine malfunction is detected, the terminal will warn the user, and the server will search for the nearest repair shop and provide guidance information.

[0242] In this way, this system provides multifaceted driving assistance and contributes to enabling users to drive with peace of mind.

[0243] The following describes the processing flow.

[0244] Step 1:

[0245] The server retrieves geographic information in real time from external map data provider APIs. This allows it to store the latest road conditions and traffic information in a database.

[0246] Step 2:

[0247] The server analyzes traffic conditions based on collected geographical information. It identifies congestion levels and accident locations and calculates the optimal route.

[0248] Step 3:

[0249] The device uses cameras and microphones installed inside the vehicle to monitor the driver's facial expressions and voice. This allows the system to understand the driver's psychological and physical state.

[0250] Step 4:

[0251] The server analyzes the driver's status data received from the terminal and determines the degree of stress and fatigue. If necessary, it generates a message to suggest that the user take a rest.

[0252] Step 5:

[0253] The device takes a picture of the road sign using its camera and sends the image data to the server. This prepares the system for recognizing the content of the road sign.

[0254] Step 6:

[0255] The server uses an image recognition algorithm to analyze road signs and translate them into the user's language. The results are returned to the terminal, and an explanation is displayed to the user.

[0256] Step 7:

[0257] The terminal monitors specific driving conditions and the driver's state, and sends instructions to the server when it determines that switching to autonomous driving mode is appropriate.

[0258] Step 8:

[0259] The server initiates control of the autonomous driving technology and notifies the user via the terminal that the vehicle is transitioning to autonomous driving. This reduces the driver's workload.

[0260] Step 9:

[0261] If a malfunction occurs in the vehicle, the terminal immediately sends data from the sensors to the server to notify the server that an emergency has occurred.

[0262] Step 10:

[0263] The server, as a backup measure for emergencies, will take appropriate countermeasures and contact emergency services if necessary. The terminal will provide the user with information about the current situation during this process.

[0264] (Example 1)

[0265] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0266] In today's traffic environment, drivers are required to travel safely and efficiently while dealing with increasing traffic volume and complex road conditions. Furthermore, driving while fatigued or stressed can increase the risk of accidents. Additionally, driving in a foreign country can present challenges due to language barriers and unfamiliarity with road signs. Therefore, there is a need for a system that provides comprehensive driver assistance, reduces the burden on drivers, and improves safety.

[0267] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0268] In this invention, the server includes means for acquiring location information and analyzing traffic conditions, means for monitoring the operator's condition and detecting mental and physical stress, and means for coordinating with machine control technology to select a driving mode. This makes it possible to present the driver with the optimal route in real time, automatically select a driving mode to enhance safety, and even suggest appropriate breaks through monitoring the driver's health condition.

[0269] "Location information" refers to data that indicates a geographical location and is used to identify one's current location or destination.

[0270] "Traffic conditions" refers to information that describes the traffic situation on a particular road or route, such as the degree of congestion or travel time.

[0271] "Operator" refers to the person who operates the vehicle or equipment, in other words, the driver.

[0272] "Mental and physical strain" refers to the mental and physical state and condition of the driver, including stress and fatigue.

[0273] "Mechanical control technology" refers to technologies that assist or replace manual driving, including autonomous driving functions and other automated control systems for vehicles.

[0274] "Visual information" refers to image and video data acquired by cameras and sensors, including road signs and the surrounding environment.

[0275] "Translation processing" is the process of accurately converting information between different languages, for example, translating signage information into a foreign language.

[0276] An "abnormal situation" refers to a situation such as an error or accident that deviates from normal operating conditions, and includes situations that require a rapid response.

[0277] This invention provides a driver assistance system that utilizes a server and terminals to achieve multifaceted functionality. The server acquires geographic information through location information services and uses external information provision services (e.g., geographic information APIs) to analyze traffic data. This allows the server to calculate the optimal route using real-time updated traffic information. The server performs the analysis using a generative AI model, and the calculation method includes the integration of real-time traffic data.

[0278] The terminal monitors the driver's condition using cameras and voice recognition devices installed in the vehicle. Specifically, by analyzing the driver's expression and voice, mental and physical loads are detected. The data obtained from the terminal is sent to the server and used to evaluate the levels of stress and fatigue. This process is important for ensuring the safety of the driver.

[0279] Furthermore, the terminal can cooperate with machine control technology to switch the driving mode from manual operation to automatic operation in appropriate situations. This function reduces the driver's burden and contributes to improving safety especially during long-distance travel.

[0280] As an analysis of visual information, the terminal uses a camera to recognize road signs and sends the information to the server. The server identifies the content of the signs through image processing, performs translation processing if necessary, and provides it to the user. Thereby, accurate traffic information can be obtained even when driving in a foreign country.

[0281] Also, when an abnormal situation occurs, the terminal monitors the data from the vehicle sensors and immediately notifies the server. The server takes appropriate actions according to the situation, for example, provides guidance information to the nearest maintenance facility.

[0282] As an example of a specific prompt sentence, a form such as "Please specifically explain the operation of the system for providing safe driving support when the user drives in a foreign country." can be considered. The present invention contributes with various support functions so that the driver can drive the vehicle with confidence.

[0283] The flow of the specific process in Example 1 will be described using FIG. 11.

[0284] Step 1:

[0285] The server receives the current location of the vehicle and destination information from the terminal. Next, it uses the location information service to obtain real-time geographical information and traffic status data. Based on the acquired data, the server analyzes the traffic situation and calculates the optimal travel route. At this time, an AI generation model is used for the analysis, and route adjustments necessary to avoid congestion and traffic accidents are made. As output, the calculated optimal route information is obtained.

[0286] Step 2:

[0287] The terminal monitors the driver's expression and voice tone using the camera and voice recognition device installed in the vehicle. The terminal transmits this data to the server, and the server analyzes the driver's stress and fatigue levels using an AI generation model. Based on this analysis result, it is determined whether the driver needs to take a break. As output, the analysis result and a break proposal message if necessary are generated.

[0288] Step 3:

[0289] The terminal captures road signs using an external camera and transmits the image data to the server. The server analyzes the signs using image recognition technology and translates the content into a language that the user can understand. The translated information is presented to the user as text on the voice or display. As output, the translated sign information is generated.

[0290] Step 4:

[0291] The terminal monitors data from various sensors installed in the vehicle and reports to the server if an error or abnormality is detected. The server activates an emergency processing protocol as an abnormal situation, for example, searches for the nearest maintenance facility and transmits its location information to the terminal. The user can respond safely according to the guidance provided by the terminal. As output, response instructions and emergency guidance information are obtained.

[0292] Step 5:

[0293] The user provides prompts regarding system operation via console or voice input. Based on these prompts, the server performs the necessary data processing and provides the requested information. As output, customized information tailored to the user's request is returned.

[0294] (Application Example 1)

[0295] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0296] In recent years, the increase in traffic accidents and problems caused by driver fatigue have become social issues. Furthermore, with the increase in international travel, language barriers when driving in foreign countries are also a challenge. To solve these problems, a system is needed that can understand the driver's psychological state and provide adaptive support. Improvements in safety are also required in autonomous driving.

[0297] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0298] In this invention, the server includes means for acquiring spatial information in real time and analyzing the movement situation, means for monitoring the driver's condition and detecting their psychological state and fatigue, means for coordinating with autonomous driving technology to change the driving mode, means for providing information within the driver's field of view using a head-mounted display, and means for understanding and responding to the driver's instructions using voice recognition technology. This enables safe and stress-free driving assistance.

[0299] "Real-time" is a technical concept that describes a state in which data and information are acquired and processed instantly.

[0300] "Spatial information" is a term that refers to information that includes data on geographical location and traffic conditions.

[0301] "Moving situation" refers to the term indicating the analysis results regarding traffic flow and congestion conditions.

[0302] "Driver's condition" means the health condition including the driver's psychological and physical conditions.

[0303] "Psychological state" is the term representing the internal situation indicating the driver's stress and tension levels.

[0304] "Fatigue" refers to the physical and mental fatigue caused by long - hour driving or excessive concentration.

[0305] "Autopilot technology" is the technology that enables a vehicle to make self - judgments and drive without human operation.

[0306] "Change of driving mode" refers to the control method of switching between manual driving and autopilot according to the situation.

[0307] "Head - mounted display" means a display device that a user wears and projects digital information into the field of vision.

[0308] "Voice recognition technology" refers to the technology that enables a computer to understand human voices and process them as instructions.

[0309] The system for realizing this invention uses a terminal equipped on the vehicle and a central server. The terminal is equipped with a camera and a voice input device, and provides information to the driver through a head - mounted display.

[0310] The server acquires spatial information via a network, analyzes the latest moving situation, calculates the optimal moving route using external information - providing services such as Google Maps API, and analyzes the driver's voice commands using voice recognition technologies such as Amazon Transcribe to understand the driver's intentions.

[0311] The device analyzes the driver's camera footage in real time and uses a machine learning (ML) model to detect the driver's psychological state and fatigue. If it determines that the driver is fatigued, it displays a message in their field of view suggesting they take a break.

[0312] Furthermore, the device uses its camera to recognize road signs and translates and explains them using OCR technology such as the Google Cloud Vision API. The results are provided on a head-mounted display, providing particularly useful information when driving in a foreign country.

[0313] In the event of an emergency, the device immediately detects the situation and sends a notification to the server. The server includes functions to search for the nearest necessary services and provide appropriate advice to the driver.

[0314] For example, during a family trip, if the driver enters a foreign country, the camera can capture unfamiliar signs. This information is translated via a server, allowing the driver to instantly see the meaning of the sign and the applicable traffic rules on the screen, enabling them to drive with peace of mind.

[0315] An example of a prompt message might be, "Please tell me how to develop a system that uses a head-mounted display to translate foreign road signs in real time and support safe driving."

[0316] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0317] Step 1:

[0318] The server obtains current geographical information using a spatial information provision service. It sends location data as input to the API and receives data on travel routes and traffic conditions as output. This data is analyzed to calculate the optimal travel route.

[0319] Step 2:

[0320] The terminal monitors the driver using a camera installed inside the vehicle and acquires video data. Using the camera images as input, an ML model analyzes the driver's facial expressions and movements to determine fatigue and psychological state. The output determines whether the driver needs a break.

[0321] Step 3:

[0322] When the driver issues a voice command, the terminal receives the voice as input and converts it into text data using voice recognition software. The converted data is then analyzed to understand the driver's instructions and provide the necessary support.

[0323] Step 4:

[0324] The server receives image data of road signs transmitted from the terminal by the client. This data is processed using OCR technology to translate the content of the signs. The server generates translated text data as output and sends it to the terminal.

[0325] Step 5:

[0326] The device displays necessary information to the user through a head-mounted display. For example, it supports driver safety by displaying suggestions for breaks based on the driver's condition or information about road signs along the designated route.

[0327] Step 6:

[0328] When a user detects an emergency, the device sends that information to the server. Based on the information received as input, the server searches for emergency response services, provides the most suitable response as output, and sends guidance information to the device.

[0329] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0330] This invention is a system for driver assistance that provides advanced functions including real-time acquisition of geographic information, analysis of traffic conditions, and recognition of the driver's emotions. The system consists of a server, a terminal, and an emotion engine that evaluates the driver's real-time psychological state.

[0331] First, the server acquires geographic information in real time from external map information providers and constantly monitors the latest traffic conditions. This makes it possible to provide users with the optimal driving route. It can also analyze traffic flow and recommend routes that avoid congestion to users.

[0332] The terminal utilizes data from the vehicle's cameras and microphones to monitor the driver's facial expressions and voice. The emotion engine analyzes this data and evaluates the driver's emotional state in real time. Based on this evaluation, if the server determines that the driver is experiencing high stress levels, it can generate suggestions for breaks to alleviate that stress. The terminal notifies the user via display and audio.

[0333] Furthermore, the emotion engine accumulates emotional data during driving and analyzes it using a learning algorithm to provide more personalized driving assistance. This accumulated data helps to provide assistance and suggestions customized for each user. It also integrates camera recognition of road signs and subsequent translation and explanation functions to support safe driving both domestically and internationally.

[0334] In addition, the system has integration capabilities with autonomous driving technology, allowing for a seamless transition to autonomous driving mode under appropriate conditions. This transition is controlled by a server, and the terminal notifies the user. In the event of an emergency, sensors detect the anomaly and immediately execute backup measures to ensure safety.

[0335] For example, if the emotional engine detects high levels of fatigue while a user is driving a long distance, the server will guide the user to the nearest rest stop and issue a voice alert from the terminal saying, "There is a rest stop nearby. Would you like to take a break?"

[0336] In this way, this system utilizes a variety of sensors and advanced data analysis to provide users with an environment that allows them to drive safely and smoothly.

[0337] The following describes the processing flow.

[0338] Step 1:

[0339] The server obtains real-time geographic information from external map information providers via APIs. It stores this information in a database and continuously updates it with the latest traffic conditions.

[0340] Step 2:

[0341] The server utilizes stored geographical information to analyze traffic conditions and calculate the optimal route. This allows it to prepare recommended routes for users to avoid congestion.

[0342] Step 3:

[0343] The device uses cameras and microphones installed inside the vehicle to continuously capture data of the driver's facial expressions and voice.

[0344] Step 4:

[0345] The emotion engine analyzes facial expressions and voice data transmitted from the device to evaluate the driver's emotional state. It quantitatively determines stress levels and fatigue levels.

[0346] Step 5:

[0347] The server receives the analysis results from the emotion engine and, if the driver's stress or fatigue is high, generates a message suggesting a break.

[0348] Step 6:

[0349] The terminal notifies the user of messages received from the server via voice or display, and suggests the nearest rest area or appropriate relaxation methods.

[0350] Step 7:

[0351] The terminal uses the vehicle's camera to capture images of road signs and sends that data to the server.

[0352] Step 8:

[0353] The server analyzes the transmitted marker data using an image recognition algorithm, translates the information as needed, adds an explanation immediately afterward, and sends it back to the terminal.

[0354] Step 9:

[0355] The terminal provides the driver with translations and explanations from the server, either visually or audibly, to aid in understanding.

[0356] Step 10:

[0357] The terminal continuously monitors the user's driving patterns and status, and determines whether the conditions are met to prompt the server to switch to automatic driving mode.

[0358] Step 11:

[0359] The server controls the autonomous driving technology under appropriate conditions and instructs the terminal to smoothly transition into autonomous driving mode.

[0360] Step 12:

[0361] If an anomaly is detected, the terminal immediately sends the vehicle's sensor data to the server to notify the emergency.

[0362] Step 13:

[0363] The server will execute pre-configured backup procedures in response to an emergency and contact emergency services as necessary. The terminal will endeavor to alleviate user anxiety by providing detailed reports on the situation.

[0364] (Example 2)

[0365] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0366] For car drivers, there is a need for systems that improve traffic safety through the provision of appropriate information and support in real time. However, existing driver assistance systems have the challenge of not being able to adequately integrate and analyze geographical information, traffic conditions, and the driver's psychological state, and respond individually. Furthermore, there is a growing need for an integrated system that can reliably perform smooth transitions to autonomous driving modes and appropriate responses in emergencies.

[0367] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0368] In this invention, the server includes means for obtaining location information from a map information service and calculating a travel route, means for analyzing voice and visual data to evaluate a person's emotional state, and means for switching the vehicle's operating mode using automation technology. This enables the presentation of an optimal route that takes into account the driver's psychological state and real-time traffic conditions, as well as the safe and smooth switching of automated driving modes.

[0369] A "map information service" is a function that provides data related to geographical locations and travel routes.

[0370] "Location information" refers to data that indicates a specific point on Earth, and usually includes latitude and longitude.

[0371] "Means for calculating travel routes" refers to technologies that calculate the optimal route from the current location to the destination.

[0372] "Audio and visual data" is a general term for information expressed through a person's facial expressions and speech sounds.

[0373] "Methods for evaluating a person's emotional state" refer to technologies that analyze data such as voice and facial expressions to estimate a person's emotions.

[0374] "Means of switching the operating mode of a vehicle using automation technology" refers to control technology for transitioning from manual driving to automated driving.

[0375] "Psychological state" is a concept that describes a person's emotions and mental condition.

[0376] "Real-time traffic conditions" refers to information about the current flow of traffic and congestion levels.

[0377] "Presenting the optimal route" refers to the act of showing the user the most efficient or safest travel route under the current conditions.

[0378] "Safe and smooth switching to autonomous driving modes" refers to the process of initiating autonomous driving functions while minimizing risks.

[0379] This invention is an advanced driver assistance system that provides comprehensive support to drivers through the integration of real-time acquisition of geographic information, analysis of traffic conditions, recognition of driver emotions, and autonomous driving technology.

[0380] The server uses an API to obtain location information from a map information service. Specifically, it uses a common map data provider. This allows the server to calculate travel routes in real time and provide the user with the optimal route. For example, it can provide rapid route suggestions based on time of day and weather conditions.

[0381] The terminal uses cameras and microphones installed in the vehicle to collect audio and visual data from the driver. This data is sent to the emotion engine, where a machine learning algorithm evaluates the driver's emotional state. If the emotional state indicates stress or fatigue, the server suggests that the driver take a break.

[0382] Furthermore, the server manages the switching of operating modes from manual to automated driving based on automation technology. This enables safe and smooth automated driving, reducing the burden on the driver. In emergencies, the terminal uses sensors to immediately detect abnormalities and take appropriate action. For example, it can assist with sudden braking or automatically activate emergency lights.

[0383] For example, if the emotional engine detects a high stress level while the user is driving for an extended period, the server will guide the user to an optimal rest stop, and the terminal will issue a voice notification such as, "There is a rest stop ahead. Would you like to take a break?" An example of a prompt message could be, "Please advise what kind of driving support would be best based on the driver's situation."

[0384] In this way, this system combines various data processing technologies and driver assistance technologies to provide drivers with a safer and more comfortable driving experience.

[0385] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0386] Step 1:

[0387] The server obtains location information from a map information service API. Given geographic coordinates as input, it calculates surrounding traffic conditions and route information based on this data. This allows the server to generate data for the optimal route to suggest to the user.

[0388] Step 2:

[0389] The device uses the vehicle's camera and microphone to collect the driver's facial expressions and voice data. This data is used as input for the emotion engine. The emotion engine applies machine learning algorithms to evaluate the driver's emotional state. As output of this process, stress and fatigue levels are obtained as numerical data.

[0390] Step 3:

[0391] The server calculates appropriate support for the driver based on the assessment of their emotional state. For example, if stress levels are high, the system processes data to suggest nearby rest stops to the user. Traffic data and geographical information are used in this calculation, and a notification suggesting a rest stop is generated as output.

[0392] Step 4:

[0393] The terminal receives suggestions from the server and notifies the driver. Specifically, it provides voice guidance and display notifications, conveying concrete messages such as, "There is a rest area ahead. Would you like to take a break?" This allows the user to intuitively accept the system's suggestions.

[0394] Step 5:

[0395] The server evaluates environmental conditions and, if necessary, instructs the vehicle to switch to autonomous driving mode. Inputs include real-time traffic data and weather information, and the system selects the optimal driving mode as its output. This enables driving that balances safety and comfort.

[0396] (Application Example 2)

[0397] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0398] In today's driving environment, drivers often experience stress due to traffic congestion and long hours of driving. Furthermore, failing to take breaks at appropriate times or being unable to smoothly transition to autonomous driving mode can lead to accidents and a decrease in safety. To address this, it is necessary to monitor the driver's emotional state in real time and provide optimal assistance tailored to the driving situation.

[0399] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0400] In this invention, the server includes means for acquiring geographic information in real time and analyzing traffic conditions, means for monitoring the driver's state and detecting stress and fatigue, means for presenting stabilization information based on the driver's emotional state, and means for switching driving modes in cooperation with autonomous driving technology. This makes it possible to reduce driver stress and improve driving safety and efficiency.

[0401] "Methods for acquiring geographic information in real time and analyzing traffic conditions" refers to technologies that instantly acquire location information from external geographic information providers and use that data to analyze current traffic conditions.

[0402] "Means for monitoring the driver's condition and detecting stress and fatigue" refers to a technology that uses in-vehicle sensor devices to observe the driver's physiological indicators and evaluates the driver's psychological and physical burden from that data.

[0403] "Means of presenting stabilizing information based on the driver's emotional state" refers to technology that provides appropriate relaxation content and driving warnings based on the results of an analysis of the driver's emotions.

[0404] "Means of switching driving modes in conjunction with autonomous driving technology" refers to technology that controls the transition from manual driving to autonomous driving at a timing appropriate to the driving situation through information exchange with the autonomous driving system.

[0405] To realize this invention, the system consists of a server, a terminal, and an emotion engine that evaluates the user's emotional state. The server acquires geographic information in real time from an external organization that provides geographic information services. This allows the server to analyze the latest traffic conditions and provide the driver with the optimal route.

[0406] The device uses a camera and microphone mounted in the vehicle to collect the driver's facial expressions and voice, and transmits this data to the emotion engine. The emotion engine analyzes this data and evaluates the driver's emotional state in real time. If the evaluation indicates that the driver is experiencing high levels of stress or fatigue, the server sends relaxation content or warning messages to the device. The device then notifies the driver via voice and display.

[0407] For example, if the emotional engine detects stress in the driver during long drives, the server will play relaxation music and display a message on the terminal screen prompting the driver to take deep breaths.

[0408] In systems using generative AI models, personalized assistance can be provided to the driver by inputting prompts such as, "Assess the driver's emotional state while driving and suggest content to help them relax." This process utilizes emotion recognition libraries such as EmotionEngine and content playback libraries such as ContentPlayer.

[0409] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0410] Step 1:

[0411] The server acquires geographic information from external geographic information providers. The input is the current location, and the output is real-time traffic data. This data serves as the foundation for providing the optimal route.

[0412] Step 2:

[0413] The device uses a camera and microphone installed inside the vehicle to record the driver's facial expressions and voice. Input consists of camera video and audio data, and the output data is transmitted to the emotion engine. This collects basic data for understanding the driver's emotional state.

[0414] Step 3:

[0415] The emotion engine analyzes data transmitted from the terminal and evaluates the emotional state. The input is data related to the driver's facial expressions and voice, and the output determines the driver's emotional state. Based on this result, it is determined whether the driver is experiencing high levels of stress.

[0416] Step 4:

[0417] The server generates appropriate stabilization information based on the driver's emotional state determined by the emotion engine. The input is the driver's emotional state, and the output is relaxation content or warning messages. If the content is relaxation music, a generative AI model is used in the selection process.

[0418] Step 5:

[0419] The terminal receives stabilization information sent from the server and notifies the user. The input is stabilization information from the server, and the output is presented to the user via voice or display. Specifically, relaxation music is played for the driver, and a message encouraging deep breathing is displayed on the screen.

[0420] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0421] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0422] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0423] [Third Embodiment]

[0424] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0425] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0426] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0427] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0428] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0429] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0430] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0431] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0432] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0433] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0434] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0435] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0436] This invention provides an advanced system for assisting driving. The system mainly consists of a server and terminals, and provides users with real-time driving information and assistance.

[0437] First, the server obtains geographical information from an external map provider and analyzes traffic conditions to calculate the optimal route for the user. This enables smooth travel while avoiding congestion.

[0438] Furthermore, the terminal monitors the driver's condition using cameras and voice recognition functions installed in the vehicle. The server analyzes this data and, if it determines that the driver is experiencing stress or fatigue, suggests a break to the user via the terminal. This function ensures safe driving.

[0439] Furthermore, the device uses its camera to recognize road signs and sends the data to a server. The server analyzes this data, translates it into the user's native language, and provides explanations of necessary traffic rules. This feature is especially useful when driving in a foreign country. The system also integrates with autonomous driving technology and can switch to autonomous driving mode when appropriate. This reduces the burden on the driver and enables safer driving under certain conditions.

[0440] In addition, in the event of an emergency, the system has the capability to immediately detect it and notify emergency services as a backup. For example, if an engine malfunction is detected, the terminal will warn the user, and the server will search for the nearest repair shop and provide guidance information.

[0441] In this way, this system provides multifaceted driving assistance and contributes to enabling users to drive with peace of mind.

[0442] The following describes the processing flow.

[0443] Step 1:

[0444] The server retrieves geographic information in real time from external map data provider APIs. This allows it to store the latest road conditions and traffic information in a database.

[0445] Step 2:

[0446] The server analyzes traffic conditions based on collected geographical information. It identifies congestion levels and accident locations and calculates the optimal route.

[0447] Step 3:

[0448] The device uses cameras and microphones installed inside the vehicle to monitor the driver's facial expressions and voice. This allows the system to understand the driver's psychological and physical state.

[0449] Step 4:

[0450] The server analyzes the driver's status data received from the terminal and determines the degree of stress and fatigue. If necessary, it generates a message to suggest that the user take a rest.

[0451] Step 5:

[0452] The device takes a picture of the road sign using its camera and sends the image data to the server. This prepares the system for recognizing the content of the road sign.

[0453] Step 6:

[0454] The server uses an image recognition algorithm to analyze road signs and translate them into the user's language. The results are returned to the terminal, and an explanation is displayed to the user.

[0455] Step 7:

[0456] The terminal monitors specific driving conditions and the driver's state, and sends instructions to the server when it determines that switching to autonomous driving mode is appropriate.

[0457] Step 8:

[0458] The server initiates control of the autonomous driving technology and notifies the user via the terminal that the vehicle is transitioning to autonomous driving. This reduces the driver's workload.

[0459] Step 9:

[0460] If a malfunction occurs in the vehicle, the terminal immediately sends data from the sensors to the server to notify the server that an emergency has occurred.

[0461] Step 10:

[0462] The server, as a backup measure for emergencies, will take appropriate countermeasures and contact emergency services if necessary. The terminal will provide the user with information about the current situation during this process.

[0463] (Example 1)

[0464] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0465] In today's traffic environment, drivers are required to travel safely and efficiently while dealing with increasing traffic volume and complex road conditions. Furthermore, driving while fatigued or stressed can increase the risk of accidents. Additionally, driving in a foreign country can present challenges due to language barriers and unfamiliarity with road signs. Therefore, there is a need for a system that provides comprehensive driver assistance, reduces the burden on drivers, and improves safety.

[0466] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0467] In this invention, the server includes means for acquiring location information and analyzing traffic conditions, means for monitoring the operator's condition and detecting mental and physical stress, and means for coordinating with machine control technology to select a driving mode. This makes it possible to present the driver with the optimal route in real time, automatically select a driving mode to enhance safety, and even suggest appropriate breaks through monitoring the driver's health condition.

[0468] "Location information" refers to data that indicates a geographical location and is used to identify one's current location or destination.

[0469] "Traffic conditions" refers to information that describes the traffic situation on a particular road or route, such as the degree of congestion or travel time.

[0470] "Operator" refers to the person who operates the vehicle or equipment, in other words, the driver.

[0471] "Mental and physical strain" refers to the mental and physical state and condition of the driver, including stress and fatigue.

[0472] "Mechanical control technology" refers to technologies that assist or replace manual driving, including autonomous driving functions and other automated control systems for vehicles.

[0473] "Visual information" refers to image and video data acquired by cameras and sensors, including road signs and the surrounding environment.

[0474] "Translation processing" is the process of accurately converting information between different languages, for example, translating signage information into a foreign language.

[0475] An "abnormal situation" refers to a situation such as an error or accident that deviates from normal operating conditions, and includes situations that require a rapid response.

[0476] This invention provides a driver assistance system that utilizes a server and terminals to achieve multifaceted functionality. The server acquires geographic information through location information services and uses external information provision services (e.g., geographic information APIs) to analyze traffic data. This allows the server to calculate the optimal route using real-time updated traffic information. The server performs the analysis using a generative AI model, and the calculation method includes the integration of real-time traffic data.

[0477] The terminal monitors the driver's condition using cameras and voice recognition devices installed in the vehicle. Specifically, it detects mental and physical stress by analyzing the driver's facial expressions and voice. The data acquired from the terminal is sent to a server and used to evaluate stress and fatigue levels. This process is important for ensuring driver safety.

[0478] Furthermore, the terminal can work in conjunction with machine control technology to switch the driving mode from manual to automatic operation when appropriate. This function reduces the burden on the driver and contributes to improved safety, especially during long-distance travel.

[0479] As part of visual information analysis, the terminal uses its camera to recognize road signs and transmits that information to a server. The server identifies the content of the signs through image processing, translates them as needed, and provides the information to the user. This ensures that accurate traffic information can be obtained even when driving in a foreign country.

[0480] Furthermore, in the event of an abnormal situation, the terminal monitors data from the vehicle's sensors and immediately notifies the server. The server then takes appropriate action based on the situation, such as providing information on how to get to the nearest maintenance facility.

[0481] A possible example of a specific prompt message would be, "Please describe in detail how the system works to provide safe driving assistance when a user is driving in a foreign country." This invention contributes to enabling drivers to operate vehicles with peace of mind through a variety of support functions.

[0482] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0483] Step 1:

[0484] The server receives the vehicle's current location and destination information from the terminal. Next, it uses location services to obtain real-time geographical information and traffic condition data. Based on the acquired data, the server analyzes the traffic situation and calculates the optimal travel route. During this process, it uses a generative AI model to perform the analysis and make necessary route adjustments to avoid congestion and traffic accidents. The output is the calculated optimal route information.

[0485] Step 2:

[0486] The terminal uses a camera and voice recognition device installed in the vehicle to monitor the driver's facial expressions and tone of voice. The terminal sends this data to a server, which uses a generated AI model to analyze the driver's stress and fatigue levels. Based on this analysis, it determines whether the driver needs a break. The output includes the analysis results and, if necessary, a break suggestion message.

[0487] Step 3:

[0488] The device captures road signs using an external camera and sends the image data to a server. The server analyzes the signs using image recognition technology and translates their content into a language the user can understand. The translated information is presented to the user either as audio or as text on the display. As an export, the translated sign information is generated.

[0489] Step 4:

[0490] The terminal monitors data from various sensors installed in the vehicle and reports any errors or abnormalities to the server. The server activates an emergency response protocol, for example, by searching for the nearest maintenance facility and sending its location information to the terminal. The user can then safely respond by following the instructions provided by the terminal. The output includes response instructions and emergency guidance information.

[0491] Step 5:

[0492] The user provides prompts regarding system operation via console or voice input. Based on these prompts, the server performs the necessary data processing and provides the requested information. As output, customized information tailored to the user's request is returned.

[0493] (Application Example 1)

[0494] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0495] In recent years, the increase in traffic accidents and problems caused by driver fatigue have become social issues. Furthermore, with the increase in international travel, language barriers when driving in foreign countries are also a challenge. To solve these problems, a system is needed that can understand the driver's psychological state and provide adaptive support. Improvements in safety are also required in autonomous driving.

[0496] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0497] In this invention, the server includes means for acquiring spatial information in real time and analyzing the movement situation, means for monitoring the driver's condition and detecting their psychological state and fatigue, means for coordinating with autonomous driving technology to change the driving mode, means for providing information within the driver's field of view using a head-mounted display, and means for understanding and responding to the driver's instructions using voice recognition technology. This enables safe and stress-free driving assistance.

[0498] "Real-time" is a technical concept that describes a state in which data and information are acquired and processed instantly.

[0499] "Spatial information" is a term that refers to information that includes data on geographical location and traffic conditions.

[0500] "Mobility status" is a term that refers to the results of an analysis of traffic flow and congestion levels.

[0501] "Driver's condition" refers to the driver's health, including their psychological and physical state.

[0502] "Psychological state" is a term that describes the internal condition of a driver, indicating their level of stress and tension.

[0503] "Fatigue" refers to physical and mental exhaustion resulting from prolonged driving or excessive concentration.

[0504] "Autonomous driving technology" refers to technology that allows vehicles to make their own decisions and drive themselves without human intervention.

[0505] "Changing the driving mode" refers to a control method that switches between manual and automatic driving depending on the situation.

[0506] A "head-mounted display" refers to a display device that a user wears and which projects digital information into their field of vision.

[0507] "Speech recognition technology" refers to the technology that allows a computer to understand human voices and process them as instructions.

[0508] The system that realizes this invention uses a terminal mounted on the vehicle and a central server. The terminal is equipped with a camera and an audio input device, and provides information to the driver via a head-mounted display.

[0509] The server acquires spatial information via the network and analyzes the latest movement status. It uses external information services such as the Google Maps API to calculate the optimal travel route. It also analyzes the driver's voice commands using speech recognition technologies such as Amazon Transcribe to understand the driver's intentions.

[0510] The device analyzes the driver's camera footage in real time and uses a machine learning (ML) model to detect the driver's psychological state and fatigue. If it determines that the driver is fatigued, it displays a message in their field of view suggesting they take a break.

[0511] Furthermore, the device uses its camera to recognize road signs and translates and explains them using OCR technology such as the Google Cloud Vision API. The results are provided on a head-mounted display, providing particularly useful information when driving in a foreign country.

[0512] In the event of an emergency, the device immediately detects the situation and sends a notification to the server. The server includes functions to search for the nearest necessary services and provide appropriate advice to the driver.

[0513] For example, during a family trip, if the driver enters a foreign country, the camera can capture unfamiliar signs. This information is translated via a server, allowing the driver to instantly see the meaning of the sign and the applicable traffic rules on the screen, enabling them to drive with peace of mind.

[0514] An example of a prompt message might be, "Please tell me how to develop a system that uses a head-mounted display to translate foreign road signs in real time and support safe driving."

[0515] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0516] Step 1:

[0517] The server obtains current geographical information using a spatial information provision service. It sends location data as input to the API and receives data on travel routes and traffic conditions as output. This data is analyzed to calculate the optimal travel route.

[0518] Step 2:

[0519] The terminal monitors the driver using a camera installed inside the vehicle and acquires video data. Using the camera images as input, an ML model analyzes the driver's facial expressions and movements to determine fatigue and psychological state. The output determines whether the driver needs a break.

[0520] Step 3:

[0521] When the driver issues a voice command, the terminal receives the voice as input and converts it into text data using voice recognition software. The converted data is then analyzed to understand the driver's instructions and provide the necessary support.

[0522] Step 4:

[0523] The server receives image data of road signs transmitted from the terminal by the client. This data is processed using OCR technology to translate the content of the signs. The server generates translated text data as output and sends it to the terminal.

[0524] Step 5:

[0525] The device displays necessary information to the user through a head-mounted display. For example, it supports driver safety by displaying suggestions for breaks based on the driver's condition or information about road signs along the designated route.

[0526] Step 6:

[0527] When a user detects an emergency, the device sends that information to the server. Based on the information received as input, the server searches for emergency response services, provides the most suitable response as output, and sends guidance information to the device.

[0528] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0529] This invention is a system for driver assistance that provides advanced functions including real-time acquisition of geographic information, analysis of traffic conditions, and recognition of the driver's emotions. The system consists of a server, a terminal, and an emotion engine that evaluates the driver's real-time psychological state.

[0530] First, the server acquires geographic information in real time from external map information providers and constantly monitors the latest traffic conditions. This makes it possible to provide users with the optimal driving route. It can also analyze traffic flow and recommend routes that avoid congestion to users.

[0531] The terminal utilizes data from the vehicle's cameras and microphones to monitor the driver's facial expressions and voice. The emotion engine analyzes this data and evaluates the driver's emotional state in real time. Based on this evaluation, if the server determines that the driver is experiencing high stress levels, it can generate suggestions for breaks to alleviate that stress. The terminal notifies the user via display and audio.

[0532] Furthermore, the emotion engine accumulates emotional data during driving and analyzes it using a learning algorithm to provide more personalized driving assistance. This accumulated data helps to provide assistance and suggestions customized for each user. It also integrates camera recognition of road signs and subsequent translation and explanation functions to support safe driving both domestically and internationally.

[0533] In addition, the system has integration capabilities with autonomous driving technology, allowing for a seamless transition to autonomous driving mode under appropriate conditions. This transition is controlled by a server, and the terminal notifies the user. In the event of an emergency, sensors detect the anomaly and immediately execute backup measures to ensure safety.

[0534] For example, if the emotional engine detects high levels of fatigue while a user is driving a long distance, the server will guide the user to the nearest rest stop and issue a voice alert from the terminal saying, "There is a rest stop nearby. Would you like to take a break?"

[0535] In this way, this system utilizes a variety of sensors and advanced data analysis to provide users with an environment that allows them to drive safely and smoothly.

[0536] The following describes the processing flow.

[0537] Step 1:

[0538] The server obtains real-time geographic information from external map information providers via APIs. It stores this information in a database and continuously updates it with the latest traffic conditions.

[0539] Step 2:

[0540] The server utilizes stored geographical information to analyze traffic conditions and calculate the optimal route. This allows it to prepare recommended routes for users to avoid congestion.

[0541] Step 3:

[0542] The device uses cameras and microphones installed inside the vehicle to continuously capture data of the driver's facial expressions and voice.

[0543] Step 4:

[0544] The emotion engine analyzes facial expressions and voice data transmitted from the device to evaluate the driver's emotional state. It quantitatively determines stress levels and fatigue levels.

[0545] Step 5:

[0546] The server receives the analysis results from the emotion engine and, if the driver's stress or fatigue is high, generates a message suggesting a break.

[0547] Step 6:

[0548] The terminal notifies the user of messages received from the server via voice or display, and suggests the nearest rest area or appropriate relaxation methods.

[0549] Step 7:

[0550] The terminal uses the vehicle's camera to capture images of road signs and sends that data to the server.

[0551] Step 8:

[0552] The server analyzes the transmitted marker data using an image recognition algorithm, translates the information as needed, adds an explanation immediately afterward, and sends it back to the terminal.

[0553] Step 9:

[0554] The terminal provides the driver with translations and explanations from the server, either visually or audibly, to aid in understanding.

[0555] Step 10:

[0556] The terminal continuously monitors the user's driving patterns and status, and determines whether the conditions are met to prompt the server to switch to automatic driving mode.

[0557] Step 11:

[0558] The server controls the autonomous driving technology under appropriate conditions and instructs the terminal to smoothly transition into autonomous driving mode.

[0559] Step 12:

[0560] If an anomaly is detected, the terminal immediately sends the vehicle's sensor data to the server to notify the emergency.

[0561] Step 13:

[0562] The server will execute pre-configured backup procedures in response to an emergency and contact emergency services as necessary. The terminal will endeavor to alleviate user anxiety by providing detailed reports on the situation.

[0563] (Example 2)

[0564] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0565] For car drivers, there is a need for systems that improve traffic safety through the provision of appropriate information and support in real time. However, existing driver assistance systems have the challenge of not being able to adequately integrate and analyze geographical information, traffic conditions, and the driver's psychological state, and respond individually. Furthermore, there is a growing need for an integrated system that can reliably perform smooth transitions to autonomous driving modes and appropriate responses in emergencies.

[0566] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0567] In this invention, the server includes means for obtaining location information from a map information service and calculating a travel route, means for analyzing voice and visual data to evaluate a person's emotional state, and means for switching the vehicle's operating mode using automation technology. This enables the presentation of an optimal route that takes into account the driver's psychological state and real-time traffic conditions, as well as the safe and smooth switching of automated driving modes.

[0568] A "map information service" is a function that provides data related to geographical locations and travel routes.

[0569] "Location information" refers to data that indicates a specific point on Earth, and usually includes latitude and longitude.

[0570] "Means for calculating travel routes" refers to technologies that calculate the optimal route from the current location to the destination.

[0571] "Audio and visual data" is a general term for information expressed through a person's facial expressions and speech sounds.

[0572] "Methods for evaluating a person's emotional state" refer to technologies that analyze data such as voice and facial expressions to estimate a person's emotions.

[0573] "Means of switching the operating mode of a vehicle using automation technology" refers to control technology for transitioning from manual driving to automated driving.

[0574] "Psychological state" is a concept that describes a person's emotions and mental condition.

[0575] "Real-time traffic conditions" refers to information about the current flow of traffic and congestion levels.

[0576] "Presenting the optimal route" refers to the act of showing the user the most efficient or safest travel route under the current conditions.

[0577] "Safe and smooth switching to autonomous driving modes" refers to the process of initiating autonomous driving functions while minimizing risks.

[0578] This invention is an advanced driver assistance system that provides comprehensive support to drivers through the integration of real-time acquisition of geographic information, analysis of traffic conditions, recognition of driver emotions, and autonomous driving technology.

[0579] The server uses an API to obtain location information from a map information service. Specifically, it uses a common map data provider. This allows the server to calculate travel routes in real time and provide the user with the optimal route. For example, it can provide rapid route suggestions based on time of day and weather conditions.

[0580] The terminal uses cameras and microphones installed in the vehicle to collect audio and visual data from the driver. This data is sent to the emotion engine, where a machine learning algorithm evaluates the driver's emotional state. If the emotional state indicates stress or fatigue, the server suggests that the driver take a break.

[0581] Furthermore, the server manages the switching of operating modes from manual to automated driving based on automation technology. This enables safe and smooth automated driving, reducing the burden on the driver. In emergencies, the terminal uses sensors to immediately detect abnormalities and take appropriate action. For example, it can assist with sudden braking or automatically activate emergency lights.

[0582] For example, if the emotional engine detects a high stress level while the user is driving for an extended period, the server will guide the user to an optimal rest stop, and the terminal will issue a voice notification such as, "There is a rest stop ahead. Would you like to take a break?" An example of a prompt message could be, "Please advise what kind of driving support would be best based on the driver's situation."

[0583] In this way, this system combines various data processing technologies and driver assistance technologies to provide drivers with a safer and more comfortable driving experience.

[0584] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0585] Step 1:

[0586] The server obtains location information from a map information service API. Given geographic coordinates as input, it calculates surrounding traffic conditions and route information based on this data. This allows the server to generate data for the optimal route to suggest to the user.

[0587] Step 2:

[0588] The device uses the vehicle's camera and microphone to collect the driver's facial expressions and voice data. This data is used as input for the emotion engine. The emotion engine applies machine learning algorithms to evaluate the driver's emotional state. As output of this process, stress and fatigue levels are obtained as numerical data.

[0589] Step 3:

[0590] The server calculates appropriate support for the driver based on the assessment of their emotional state. For example, if stress levels are high, the system processes data to suggest nearby rest stops to the user. Traffic data and geographical information are used in this calculation, and a notification suggesting a rest stop is generated as output.

[0591] Step 4:

[0592] The terminal receives suggestions from the server and notifies the driver. Specifically, it provides voice guidance and display notifications, conveying concrete messages such as, "There is a rest area ahead. Would you like to take a break?" This allows the user to intuitively accept the system's suggestions.

[0593] Step 5:

[0594] The server evaluates environmental conditions and, if necessary, instructs the vehicle to switch to autonomous driving mode. Inputs include real-time traffic data and weather information, and the system selects the optimal driving mode as its output. This enables driving that balances safety and comfort.

[0595] (Application Example 2)

[0596] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0597] In today's driving environment, drivers often experience stress due to traffic congestion and long hours of driving. Furthermore, failing to take breaks at appropriate times or being unable to smoothly transition to autonomous driving mode can lead to accidents and a decrease in safety. To address this, it is necessary to monitor the driver's emotional state in real time and provide optimal assistance tailored to the driving situation.

[0598] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0599] In this invention, the server includes means for acquiring geographic information in real time and analyzing traffic conditions, means for monitoring the driver's state and detecting stress and fatigue, means for presenting stabilization information based on the driver's emotional state, and means for switching driving modes in cooperation with autonomous driving technology. This makes it possible to reduce driver stress and improve driving safety and efficiency.

[0600] "Methods for acquiring geographic information in real time and analyzing traffic conditions" refers to technologies that instantly acquire location information from external geographic information providers and use that data to analyze current traffic conditions.

[0601] "Means for monitoring the driver's condition and detecting stress and fatigue" refers to a technology that uses in-vehicle sensor devices to observe the driver's physiological indicators and evaluates the driver's psychological and physical burden from that data.

[0602] "Means of presenting stabilizing information based on the driver's emotional state" refers to technology that provides appropriate relaxation content and driving warnings based on the results of an analysis of the driver's emotions.

[0603] "Means of switching driving modes in conjunction with autonomous driving technology" refers to technology that controls the transition from manual driving to autonomous driving at a timing appropriate to the driving situation through information exchange with the autonomous driving system.

[0604] To realize this invention, the system consists of a server, a terminal, and an emotion engine that evaluates the user's emotional state. The server acquires geographic information in real time from an external organization that provides geographic information services. This allows the server to analyze the latest traffic conditions and provide the driver with the optimal route.

[0605] The device uses a camera and microphone mounted in the vehicle to collect the driver's facial expressions and voice, and transmits this data to the emotion engine. The emotion engine analyzes this data and evaluates the driver's emotional state in real time. If the evaluation indicates that the driver is experiencing high levels of stress or fatigue, the server sends relaxation content or warning messages to the device. The device then notifies the driver via voice and display.

[0606] For example, if the emotional engine detects stress in the driver during long drives, the server will play relaxation music and display a message on the terminal screen prompting the driver to take deep breaths.

[0607] In systems using generative AI models, personalized assistance can be provided to the driver by inputting prompts such as, "Assess the driver's emotional state while driving and suggest content to help them relax." This process utilizes emotion recognition libraries such as EmotionEngine and content playback libraries such as ContentPlayer.

[0608] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0609] Step 1:

[0610] The server acquires geographic information from external geographic information providers. The input is the current location, and the output is real-time traffic data. This data serves as the foundation for providing the optimal route.

[0611] Step 2:

[0612] The device uses a camera and microphone installed inside the vehicle to record the driver's facial expressions and voice. Input consists of camera video and audio data, and the output data is transmitted to the emotion engine. This collects basic data for understanding the driver's emotional state.

[0613] Step 3:

[0614] The emotion engine analyzes data transmitted from the terminal and evaluates the emotional state. The input is data related to the driver's facial expressions and voice, and the output determines the driver's emotional state. Based on this result, it is determined whether the driver is experiencing high levels of stress.

[0615] Step 4:

[0616] The server generates appropriate stabilization information based on the driver's emotional state determined by the emotion engine. The input is the driver's emotional state, and the output is relaxation content or warning messages. If the content is relaxation music, a generative AI model is used in the selection process.

[0617] Step 5:

[0618] The terminal receives stabilization information sent from the server and notifies the user. The input is stabilization information from the server, and the output is presented to the user via voice or display. Specifically, relaxation music is played for the driver, and a message encouraging deep breathing is displayed on the screen.

[0619] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0620] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0621] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0622] [Fourth Embodiment]

[0623] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0624] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0625] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0626] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0627] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0628] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0629] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0630] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0631] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0632] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0633] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0634] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0635] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0636] This invention provides an advanced system for assisting driving. The system mainly consists of a server and terminals, and provides users with real-time driving information and assistance.

[0637] First, the server obtains geographical information from an external map provider and analyzes traffic conditions to calculate the optimal route for the user. This enables smooth travel while avoiding congestion.

[0638] Furthermore, the terminal monitors the driver's condition using cameras and voice recognition functions installed in the vehicle. The server analyzes this data and, if it determines that the driver is experiencing stress or fatigue, suggests a break to the user via the terminal. This function ensures safe driving.

[0639] Furthermore, the device uses its camera to recognize road signs and sends the data to a server. The server analyzes this data, translates it into the user's native language, and provides explanations of necessary traffic rules. This feature is especially useful when driving in a foreign country. The system also integrates with autonomous driving technology and can switch to autonomous driving mode when appropriate. This reduces the burden on the driver and enables safer driving under certain conditions.

[0640] In addition, in the event of an emergency, the system has the capability to immediately detect it and notify emergency services as a backup. For example, if an engine malfunction is detected, the terminal will warn the user, and the server will search for the nearest repair shop and provide guidance information.

[0641] In this way, this system provides multifaceted driving assistance and contributes to enabling users to drive with peace of mind.

[0642] The following describes the processing flow.

[0643] Step 1:

[0644] The server retrieves geographic information in real time from external map data provider APIs. This allows it to store the latest road conditions and traffic information in a database.

[0645] Step 2:

[0646] The server analyzes traffic conditions based on collected geographical information. It identifies congestion levels and accident locations and calculates the optimal route.

[0647] Step 3:

[0648] The device uses cameras and microphones installed inside the vehicle to monitor the driver's facial expressions and voice. This allows the system to understand the driver's psychological and physical state.

[0649] Step 4:

[0650] The server analyzes the driver's status data received from the terminal and determines the degree of stress and fatigue. If necessary, it generates a message to suggest that the user take a rest.

[0651] Step 5:

[0652] The device takes a picture of the road sign using its camera and sends the image data to the server. This prepares the system for recognizing the content of the road sign.

[0653] Step 6:

[0654] The server uses an image recognition algorithm to analyze road signs and translate them into the user's language. The results are returned to the terminal, and an explanation is displayed to the user.

[0655] Step 7:

[0656] The terminal monitors specific driving conditions and the driver's state, and sends instructions to the server when it determines that switching to autonomous driving mode is appropriate.

[0657] Step 8:

[0658] The server initiates control of the autonomous driving technology and notifies the user via the terminal that the vehicle is transitioning to autonomous driving. This reduces the driver's workload.

[0659] Step 9:

[0660] If a malfunction occurs in the vehicle, the terminal immediately sends data from the sensors to the server to notify the server that an emergency has occurred.

[0661] Step 10:

[0662] The server, as a backup measure for emergencies, will take appropriate countermeasures and contact emergency services if necessary. The terminal will provide the user with information about the current situation during this process.

[0663] (Example 1)

[0664] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0665] In today's traffic environment, drivers are required to travel safely and efficiently while dealing with increasing traffic volume and complex road conditions. Furthermore, driving while fatigued or stressed can increase the risk of accidents. Additionally, driving in a foreign country can present challenges due to language barriers and unfamiliarity with road signs. Therefore, there is a need for a system that provides comprehensive driver assistance, reduces the burden on drivers, and improves safety.

[0666] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0667] In this invention, the server includes means for acquiring location information and analyzing traffic conditions, means for monitoring the operator's condition and detecting mental and physical stress, and means for coordinating with machine control technology to select a driving mode. This makes it possible to present the driver with the optimal route in real time, automatically select a driving mode to enhance safety, and even suggest appropriate breaks through monitoring the driver's health condition.

[0668] "Location information" refers to data that indicates a geographical location and is used to identify one's current location or destination.

[0669] "Traffic conditions" refers to information that describes the traffic situation on a particular road or route, such as the degree of congestion or travel time.

[0670] "Operator" refers to the person who operates the vehicle or equipment, in other words, the driver.

[0671] "Mental and physical strain" refers to the mental and physical state and condition of the driver, including stress and fatigue.

[0672] "Mechanical control technology" refers to technologies that assist or replace manual driving, including autonomous driving functions and other automated control systems for vehicles.

[0673] "Visual information" refers to image and video data acquired by cameras and sensors, including road signs and the surrounding environment.

[0674] "Translation processing" is the process of accurately converting information between different languages, for example, translating signage information into a foreign language.

[0675] An "abnormal situation" refers to a situation such as an error or accident that deviates from normal operating conditions, and includes situations that require a rapid response.

[0676] This invention provides a driver assistance system that utilizes a server and terminals to achieve multifaceted functionality. The server acquires geographic information through location information services and uses external information provision services (e.g., geographic information APIs) to analyze traffic data. This allows the server to calculate the optimal route using real-time updated traffic information. The server performs the analysis using a generative AI model, and the calculation method includes the integration of real-time traffic data.

[0677] The terminal monitors the driver's condition using cameras and voice recognition devices installed in the vehicle. Specifically, it detects mental and physical stress by analyzing the driver's facial expressions and voice. The data acquired from the terminal is sent to a server and used to evaluate stress and fatigue levels. This process is important for ensuring driver safety.

[0678] Furthermore, the terminal can work in conjunction with machine control technology to switch the driving mode from manual to automatic operation when appropriate. This function reduces the burden on the driver and contributes to improved safety, especially during long-distance travel.

[0679] As part of visual information analysis, the terminal uses its camera to recognize road signs and transmits that information to a server. The server identifies the content of the signs through image processing, translates them as needed, and provides the information to the user. This ensures that accurate traffic information can be obtained even when driving in a foreign country.

[0680] Furthermore, in the event of an abnormal situation, the terminal monitors data from the vehicle's sensors and immediately notifies the server. The server then takes appropriate action based on the situation, such as providing information on how to get to the nearest maintenance facility.

[0681] A possible example of a specific prompt message would be, "Please describe in detail how the system works to provide safe driving assistance when a user is driving in a foreign country." This invention contributes to enabling drivers to operate vehicles with peace of mind through a variety of support functions.

[0682] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0683] Step 1:

[0684] The server receives the vehicle's current location and destination information from the terminal. Next, it uses location services to obtain real-time geographical information and traffic condition data. Based on the acquired data, the server analyzes the traffic situation and calculates the optimal travel route. During this process, it uses a generative AI model to perform the analysis and make necessary route adjustments to avoid congestion and traffic accidents. The output is the calculated optimal route information.

[0685] Step 2:

[0686] The terminal uses a camera and voice recognition device installed in the vehicle to monitor the driver's facial expressions and tone of voice. The terminal sends this data to a server, which uses a generated AI model to analyze the driver's stress and fatigue levels. Based on this analysis, it determines whether the driver needs a break. The output includes the analysis results and, if necessary, a break suggestion message.

[0687] Step 3:

[0688] The device captures road signs using an external camera and sends the image data to a server. The server analyzes the signs using image recognition technology and translates their content into a language the user can understand. The translated information is presented to the user either as audio or as text on the display. As an export, the translated sign information is generated.

[0689] Step 4:

[0690] The terminal monitors data from various sensors installed in the vehicle and reports any errors or abnormalities to the server. The server activates an emergency response protocol, for example, by searching for the nearest maintenance facility and sending its location information to the terminal. The user can then safely respond by following the instructions provided by the terminal. The output includes response instructions and emergency guidance information.

[0691] Step 5:

[0692] The user provides prompts regarding system operation via console or voice input. Based on these prompts, the server performs the necessary data processing and provides the requested information. As output, customized information tailored to the user's request is returned.

[0693] (Application Example 1)

[0694] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0695] In recent years, the increase in traffic accidents and problems caused by driver fatigue have become social issues. Furthermore, with the increase in international travel, language barriers when driving in foreign countries are also a challenge. To solve these problems, a system is needed that can understand the driver's psychological state and provide adaptive support. Improvements in safety are also required in autonomous driving.

[0696] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0697] In this invention, the server includes means for acquiring spatial information in real time and analyzing the movement situation, means for monitoring the driver's condition and detecting their psychological state and fatigue, means for coordinating with autonomous driving technology to change the driving mode, means for providing information within the driver's field of view using a head-mounted display, and means for understanding and responding to the driver's instructions using voice recognition technology. This enables safe and stress-free driving assistance.

[0698] "Real-time" is a technical concept that describes a state in which data and information are acquired and processed instantly.

[0699] "Spatial information" is a term that refers to information that includes data on geographical location and traffic conditions.

[0700] "Mobility status" is a term that refers to the results of an analysis of traffic flow and congestion levels.

[0701] "Driver's condition" refers to the driver's health, including their psychological and physical state.

[0702] "Psychological state" is a term that describes the internal condition of a driver, indicating their level of stress and tension.

[0703] "Fatigue" refers to physical and mental exhaustion resulting from prolonged driving or excessive concentration.

[0704] "Autonomous driving technology" refers to technology that allows vehicles to make their own decisions and drive themselves without human intervention.

[0705] "Changing the driving mode" refers to a control method that switches between manual and automatic driving depending on the situation.

[0706] A "head-mounted display" refers to a display device that a user wears and which projects digital information into their field of vision.

[0707] "Speech recognition technology" refers to the technology that allows a computer to understand human voices and process them as instructions.

[0708] The system that realizes this invention uses a terminal mounted on the vehicle and a central server. The terminal is equipped with a camera and an audio input device, and provides information to the driver via a head-mounted display.

[0709] The server acquires spatial information via the network and analyzes the latest movement status. It uses external information services such as the Google Maps API to calculate the optimal travel route. It also analyzes the driver's voice commands using speech recognition technologies such as Amazon Transcribe to understand the driver's intentions.

[0710] The device analyzes the driver's camera footage in real time and uses a machine learning (ML) model to detect the driver's psychological state and fatigue. If it determines that the driver is fatigued, it displays a message in their field of view suggesting they take a break.

[0711] Furthermore, the device uses its camera to recognize road signs and translates and explains them using OCR technology such as the Google Cloud Vision API. The results are provided on a head-mounted display, providing particularly useful information when driving in a foreign country.

[0712] In the event of an emergency, the device immediately detects the situation and sends a notification to the server. The server includes functions to search for the nearest necessary services and provide appropriate advice to the driver.

[0713] For example, during a family trip, if the driver enters a foreign country, the camera can capture unfamiliar signs. This information is translated via a server, allowing the driver to instantly see the meaning of the sign and the applicable traffic rules on the screen, enabling them to drive with peace of mind.

[0714] An example of a prompt message might be, "Please tell me how to develop a system that uses a head-mounted display to translate foreign road signs in real time and support safe driving."

[0715] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0716] Step 1:

[0717] The server obtains current geographical information using a spatial information provision service. It sends location data as input to the API and receives data on travel routes and traffic conditions as output. This data is analyzed to calculate the optimal travel route.

[0718] Step 2:

[0719] The terminal monitors the driver using a camera installed inside the vehicle and acquires video data. Using the camera images as input, an ML model analyzes the driver's facial expressions and movements to determine fatigue and psychological state. The output determines whether the driver needs a break.

[0720] Step 3:

[0721] When the driver issues a voice command, the terminal receives the voice as input and converts it into text data using voice recognition software. The converted data is then analyzed to understand the driver's instructions and provide the necessary support.

[0722] Step 4:

[0723] The server receives image data of road signs transmitted from the terminal by the client. This data is processed using OCR technology to translate the content of the signs. The server generates translated text data as output and sends it to the terminal.

[0724] Step 5:

[0725] The device displays necessary information to the user through a head-mounted display. For example, it supports driver safety by displaying suggestions for breaks based on the driver's condition or information about road signs along the designated route.

[0726] Step 6:

[0727] When a user detects an emergency, the device sends that information to the server. Based on the information received as input, the server searches for emergency response services, provides the most suitable response as output, and sends guidance information to the device.

[0728] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0729] This invention is a system for driver assistance that provides advanced functions including real-time acquisition of geographic information, analysis of traffic conditions, and recognition of the driver's emotions. The system consists of a server, a terminal, and an emotion engine that evaluates the driver's real-time psychological state.

[0730] First, the server acquires geographic information in real time from external map information providers and constantly monitors the latest traffic conditions. This makes it possible to provide users with the optimal driving route. It can also analyze traffic flow and recommend routes that avoid congestion to users.

[0731] The terminal utilizes data from the vehicle's cameras and microphones to monitor the driver's facial expressions and voice. The emotion engine analyzes this data and evaluates the driver's emotional state in real time. Based on this evaluation, if the server determines that the driver is experiencing high stress levels, it can generate suggestions for breaks to alleviate that stress. The terminal notifies the user via display and audio.

[0732] Furthermore, the emotion engine accumulates emotional data during driving and analyzes it using a learning algorithm to provide more personalized driving assistance. This accumulated data helps to provide assistance and suggestions customized for each user. It also integrates camera recognition of road signs and subsequent translation and explanation functions to support safe driving both domestically and internationally.

[0733] In addition, the system has integration capabilities with autonomous driving technology, allowing for a seamless transition to autonomous driving mode under appropriate conditions. This transition is controlled by a server, and the terminal notifies the user. In the event of an emergency, sensors detect the anomaly and immediately execute backup measures to ensure safety.

[0734] For example, if the emotional engine detects high levels of fatigue while a user is driving a long distance, the server will guide the user to the nearest rest stop and issue a voice alert from the terminal saying, "There is a rest stop nearby. Would you like to take a break?"

[0735] In this way, this system utilizes a variety of sensors and advanced data analysis to provide users with an environment that allows them to drive safely and smoothly.

[0736] The following describes the processing flow.

[0737] Step 1:

[0738] The server obtains real-time geographic information from external map information providers via APIs. It stores this information in a database and continuously updates it with the latest traffic conditions.

[0739] Step 2:

[0740] The server utilizes stored geographical information to analyze traffic conditions and calculate the optimal route. This allows it to prepare recommended routes for users to avoid congestion.

[0741] Step 3:

[0742] The device uses cameras and microphones installed inside the vehicle to continuously capture data of the driver's facial expressions and voice.

[0743] Step 4:

[0744] The emotion engine analyzes facial expressions and voice data transmitted from the device to evaluate the driver's emotional state. It quantitatively determines stress levels and fatigue levels.

[0745] Step 5:

[0746] The server receives the analysis results from the emotion engine and, if the driver's stress or fatigue is high, generates a message suggesting a break.

[0747] Step 6:

[0748] The terminal notifies the user of messages received from the server via voice or display, and suggests the nearest rest area or appropriate relaxation methods.

[0749] Step 7:

[0750] The terminal uses the vehicle's camera to capture images of road signs and sends that data to the server.

[0751] Step 8:

[0752] The server analyzes the transmitted marker data using an image recognition algorithm, translates the information as needed, adds an explanation immediately afterward, and sends it back to the terminal.

[0753] Step 9:

[0754] The terminal provides the driver with translations and explanations from the server, either visually or audibly, to aid in understanding.

[0755] Step 10:

[0756] The terminal continuously monitors the user's driving patterns and status, and determines whether the conditions are met to prompt the server to switch to automatic driving mode.

[0757] Step 11:

[0758] The server controls the autonomous driving technology under appropriate conditions and instructs the terminal to smoothly transition into autonomous driving mode.

[0759] Step 12:

[0760] If an anomaly is detected, the terminal immediately sends the vehicle's sensor data to the server to notify the emergency.

[0761] Step 13:

[0762] The server will execute pre-configured backup procedures in response to an emergency and contact emergency services as necessary. The terminal will endeavor to alleviate user anxiety by providing detailed reports on the situation.

[0763] (Example 2)

[0764] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0765] For car drivers, there is a need for systems that improve traffic safety through the provision of appropriate information and support in real time. However, existing driver assistance systems have the challenge of not being able to adequately integrate and analyze geographical information, traffic conditions, and the driver's psychological state, and respond individually. Furthermore, there is a growing need for an integrated system that can reliably perform smooth transitions to autonomous driving modes and appropriate responses in emergencies.

[0766] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0767] In this invention, the server includes means for obtaining location information from a map information service and calculating a travel route, means for analyzing voice and visual data to evaluate a person's emotional state, and means for switching the vehicle's operating mode using automation technology. This enables the presentation of an optimal route that takes into account the driver's psychological state and real-time traffic conditions, as well as the safe and smooth switching of automated driving modes.

[0768] A "map information service" is a function that provides data related to geographical locations and travel routes.

[0769] "Location information" refers to data that indicates a specific point on Earth, and usually includes latitude and longitude.

[0770] "Means for calculating travel routes" refers to technologies that calculate the optimal route from the current location to the destination.

[0771] "Audio and visual data" is a general term for information expressed through a person's facial expressions and speech sounds.

[0772] "Methods for evaluating a person's emotional state" refer to technologies that analyze data such as voice and facial expressions to estimate a person's emotions.

[0773] "Means of switching the operating mode of a vehicle using automation technology" refers to control technology for transitioning from manual driving to automated driving.

[0774] "Psychological state" is a concept that describes a person's emotions and mental condition.

[0775] "Real-time traffic conditions" refers to information about the current flow of traffic and congestion levels.

[0776] "Presenting the optimal route" refers to the act of showing the user the most efficient or safest travel route under the current conditions.

[0777] "Safe and smooth switching to autonomous driving modes" refers to the process of initiating autonomous driving functions while minimizing risks.

[0778] This invention is an advanced driver assistance system that provides comprehensive support to drivers through the integration of real-time acquisition of geographic information, analysis of traffic conditions, recognition of driver emotions, and autonomous driving technology.

[0779] The server uses an API to obtain location information from a map information service. Specifically, it uses a common map data provider. This allows the server to calculate travel routes in real time and provide the user with the optimal route. For example, it can provide rapid route suggestions based on time of day and weather conditions.

[0780] The terminal uses cameras and microphones installed in the vehicle to collect audio and visual data from the driver. This data is sent to the emotion engine, where a machine learning algorithm evaluates the driver's emotional state. If the emotional state indicates stress or fatigue, the server suggests that the driver take a break.

[0781] Furthermore, the server manages the switching of operating modes from manual to automated driving based on automation technology. This enables safe and smooth automated driving, reducing the burden on the driver. In emergencies, the terminal uses sensors to immediately detect abnormalities and take appropriate action. For example, it can assist with sudden braking or automatically activate emergency lights.

[0782] For example, if the emotional engine detects a high stress level while the user is driving for an extended period, the server will guide the user to an optimal rest stop, and the terminal will issue a voice notification such as, "There is a rest stop ahead. Would you like to take a break?" An example of a prompt message could be, "Please advise what kind of driving support would be best based on the driver's situation."

[0783] In this way, this system combines various data processing technologies and driver assistance technologies to provide drivers with a safer and more comfortable driving experience.

[0784] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0785] Step 1:

[0786] The server obtains location information from a map information service API. Given geographic coordinates as input, it calculates surrounding traffic conditions and route information based on this data. This allows the server to generate data for the optimal route to suggest to the user.

[0787] Step 2:

[0788] The device uses the vehicle's camera and microphone to collect the driver's facial expressions and voice data. This data is used as input for the emotion engine. The emotion engine applies machine learning algorithms to evaluate the driver's emotional state. As output of this process, stress and fatigue levels are obtained as numerical data.

[0789] Step 3:

[0790] The server calculates appropriate support for the driver based on the assessment of their emotional state. For example, if stress levels are high, the system processes data to suggest nearby rest stops to the user. Traffic data and geographical information are used in this calculation, and a notification suggesting a rest stop is generated as output.

[0791] Step 4:

[0792] The terminal receives suggestions from the server and notifies the driver. Specifically, it provides voice guidance and display notifications, conveying concrete messages such as, "There is a rest area ahead. Would you like to take a break?" This allows the user to intuitively accept the system's suggestions.

[0793] Step 5:

[0794] The server evaluates environmental conditions and, if necessary, instructs the vehicle to switch to autonomous driving mode. Inputs include real-time traffic data and weather information, and the system selects the optimal driving mode as its output. This enables driving that balances safety and comfort.

[0795] (Application Example 2)

[0796] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0797] In today's driving environment, drivers often experience stress due to traffic congestion and long hours of driving. Furthermore, failing to take breaks at appropriate times or being unable to smoothly transition to autonomous driving mode can lead to accidents and a decrease in safety. To address this, it is necessary to monitor the driver's emotional state in real time and provide optimal assistance tailored to the driving situation.

[0798] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0799] In this invention, the server includes means for acquiring geographic information in real time and analyzing traffic conditions, means for monitoring the driver's state and detecting stress and fatigue, means for presenting stabilization information based on the driver's emotional state, and means for switching driving modes in cooperation with autonomous driving technology. This makes it possible to reduce driver stress and improve driving safety and efficiency.

[0800] "Methods for acquiring geographic information in real time and analyzing traffic conditions" refers to technologies that instantly acquire location information from external geographic information providers and use that data to analyze current traffic conditions.

[0801] "Means for monitoring the driver's condition and detecting stress and fatigue" refers to a technology that uses in-vehicle sensor devices to observe the driver's physiological indicators and evaluates the driver's psychological and physical burden from that data.

[0802] "Means of presenting stabilizing information based on the driver's emotional state" refers to technology that provides appropriate relaxation content and driving warnings based on the results of an analysis of the driver's emotions.

[0803] "Means of switching driving modes in conjunction with autonomous driving technology" refers to technology that controls the transition from manual driving to autonomous driving at a timing appropriate to the driving situation through information exchange with the autonomous driving system.

[0804] To realize this invention, the system consists of a server, a terminal, and an emotion engine that evaluates the user's emotional state. The server acquires geographic information in real time from an external organization that provides geographic information services. This allows the server to analyze the latest traffic conditions and provide the driver with the optimal route.

[0805] The device uses a camera and microphone mounted in the vehicle to collect the driver's facial expressions and voice, and transmits this data to the emotion engine. The emotion engine analyzes this data and evaluates the driver's emotional state in real time. If the evaluation indicates that the driver is experiencing high levels of stress or fatigue, the server sends relaxation content or warning messages to the device. The device then notifies the driver via voice and display.

[0806] For example, if the emotional engine detects stress in the driver during long drives, the server will play relaxation music and display a message on the terminal screen prompting the driver to take deep breaths.

[0807] In systems using generative AI models, personalized assistance can be provided to the driver by inputting prompts such as, "Assess the driver's emotional state while driving and suggest content to help them relax." This process utilizes emotion recognition libraries such as EmotionEngine and content playback libraries such as ContentPlayer.

[0808] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0809] Step 1:

[0810] The server acquires geographic information from external geographic information providers. The input is the current location, and the output is real-time traffic data. This data serves as the foundation for providing the optimal route.

[0811] Step 2:

[0812] The device uses a camera and microphone installed inside the vehicle to record the driver's facial expressions and voice. Input consists of camera video and audio data, and the output data is transmitted to the emotion engine. This collects basic data for understanding the driver's emotional state.

[0813] Step 3:

[0814] The emotion engine analyzes data transmitted from the terminal and evaluates the emotional state. The input is data related to the driver's facial expressions and voice, and the output determines the driver's emotional state. Based on this result, it is determined whether the driver is experiencing high levels of stress.

[0815] Step 4:

[0816] The server generates appropriate stabilization information based on the driver's emotional state determined by the emotion engine. The input is the driver's emotional state, and the output is relaxation content or warning messages. If the content is relaxation music, a generative AI model is used in the selection process.

[0817] Step 5:

[0818] The terminal receives stabilization information sent from the server and notifies the user. The input is stabilization information from the server, and the output is presented to the user via voice or display. Specifically, relaxation music is played for the driver, and a message encouraging deep breathing is displayed on the screen.

[0819] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0820] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0821] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0822] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0823] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0824] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0825] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0826] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0827] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0828] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0829] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0830] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0831] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0832] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0833] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0834] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0835] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0836] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0837] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0838] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0839] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0840] The following is further disclosed regarding the embodiments described above.

[0841] (Claim 1)

[0842] A means of acquiring geographic information in real time and analyzing traffic conditions,

[0843] A means of monitoring the driver's condition and detecting stress and fatigue,

[0844] A means of switching driving modes in conjunction with autonomous driving technology,

[0845] A system that includes this.

[0846] (Claim 2)

[0847] The system according to claim 1, having means for recognizing road signs and translating and explaining their information.

[0848] (Claim 3)

[0849] The system according to claim 1, comprising means for detecting an emergency and immediately taking appropriate backup action.

[0850] "Example 1"

[0851] (Claim 1)

[0852] A means for acquiring location information and analyzing traffic conditions,

[0853] Means for monitoring the operator's condition and detecting mental and physical strain,

[0854] In conjunction with mechanical control technology, a means for selecting the control mode,

[0855] A means for identifying visual information using image recognition technology and performing translation processing,

[0856] A means to detect abnormal situations and take immediate countermeasures,

[0857] A system that includes this.

[0858] (Claim 2)

[0859] The system according to claim 1, further comprising means for analyzing the aforementioned visual information and providing explanations in multiple languages.

[0860] (Claim 3)

[0861] The system according to claim 1, comprising means for identifying an anomaly and communicating with an external party using communication means.

[0862] "Application Example 1"

[0863] (Claim 1)

[0864] A means of acquiring spatial information in real time and analyzing movement patterns,

[0865] A means of monitoring the driver's condition and detecting their psychological state and fatigue,

[0866] A means of changing the driving mode in conjunction with autonomous driving technology,

[0867] A means of providing information within the field of view using a head-mounted display,

[0868] Voice recognition technology allows the system to understand and respond to driver instructions,

[0869] A system that includes this.

[0870] (Claim 2)

[0871] The system according to claim 1, having means for recognizing road signs and translating and explaining their information.

[0872] (Claim 3)

[0873] The system according to claim 1, comprising means for detecting an emergency and immediately taking appropriate backup action.

[0874] "Example 2 of combining an emotion engine"

[0875] (Claim 1)

[0876] A means of obtaining location information from a map information service and calculating a travel route,

[0877] A means for analyzing audio and visual data to evaluate a person's emotional state,

[0878] A means of switching the vehicle's operating mode using automated technology,

[0879] A method for analyzing human emotional data using a learning algorithm,

[0880] A means for recognizing road signs and translating and explaining their information,

[0881] A means of identifying abnormal conditions using various sensors and automatically taking action,

[0882] ...

[0883] A system that includes this.

[0884] (Claim 2)

[0885] The system according to claim 1, which calculates and provides guidance along the optimal route based on geographical information.

[0886] (Claim 3)

[0887] The system according to claim 1, which generates and notifies suggestions based on a psychological state.

[0888] "Application example 2 of combining emotional engines"

[0889] (Claim 1)

[0890] A means of acquiring geographic information in real time and analyzing traffic conditions,

[0891] A means of monitoring the driver's condition and detecting stress and fatigue,

[0892] A means of presenting stabilization information based on the driver's emotional state,

[0893] A means of switching driving modes in conjunction with autonomous driving technology,

[0894] A system that includes this.

[0895] (Claim 2)

[0896] The system according to claim 1, having means for recognizing road signs and translating and explaining their information.

[0897] (Claim 3)

[0898] The system according to claim 1, comprising means for detecting an emergency and immediately taking appropriate backup action. [Explanation of Symbols]

[0899] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring geographic information in real time and analyzing traffic conditions, A means of monitoring the driver's condition and detecting stress and fatigue, A means of switching driving modes in conjunction with autonomous driving technology, A system that includes this.

2. The system according to claim 1, comprising means for recognizing road signs and translating and explaining their information.

3. The system according to claim 1, comprising means for detecting an emergency and immediately taking appropriate backup action.

Images