System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting traffic information in real time; means for analyzing the collected traffic information and calculating an optimal route; means for transmitting the analysis result to a user terminal; means for visually displaying the received traffic information at the user terminal; means for recognizing a road sign or a pedestrian by using a camera; and means for notifying the user terminal of the recognition result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current in-vehicle navigation systems and applications lack the ability to provide real-time traffic information, putting drivers at high risk of getting caught in traffic jams and accidents due to guidance based on outdated information. Furthermore, incomplete multilingual support makes it difficult for users to use the systems in different countries and language environments. Furthermore, recognition of traffic signs and pedestrians is incomplete, leaving drivers with insufficient support to ensure sufficient visibility and safe driving. There is a need for a system that can solve these problems and provide users with a safe and intuitive driving experience. [Means for solving the problem]

[0005] The system includes a means for collecting traffic information in real time, a means for analyzing the collected traffic information and calculating the optimal route, a means for transmitting the analysis results to a user terminal, a means for visually displaying the received traffic information on the user terminal, a means for recognizing road signs and pedestrians using a camera, and a means for notifying the user terminal of the recognition results. The system also includes a means for translating and providing traffic information and navigation advice in multiple languages ​​and a means for providing the translated information in audio and text based on a user-set language. This allows users to obtain the latest traffic information in real time and receive appropriate navigation even in different language environments. The system also supports safe driving by detecting road signs and pedestrians.

[0006] "Traffic information" is a general term for information on dynamic and static conditions on roads, such as traffic congestion, accidents, and road construction.

[0007] "Collection" is a series of processes for taking in information from outside.

[0008] "Analysis" is the process of interpreting and finding meaning in collected data.

[0009] A "route" refers to the direction of travel or the route from a certain point to a destination.

[0010] "User terminal" refers to a device used by the user to operate the system, such as a smartphone or tablet.

[0011] "Visually displaying" means presenting images and text information in a visible form on a digital screen.

[0012] A "camera" is a device that captures light, converts it into electronic signals, and records images or videos.

[0013] "Road signs" are signs installed on roads to inform drivers of traffic rules and instructions.

[0014] "Pedestrians" is a general term for people walking on the road.

[0015] "Recognition" is the process of analyzing information captured by cameras and sensors and identifying what it is.

[0016] "Notification" refers to the action of conveying information from the system to the user.

[0017] "Multilingual" means the ability to support multiple languages ​​and provide information and instructions in each language.

[0018] "Translation" is the process of converting information written in one language into another.

[0019] "Speech" is a means of conveying information through sounds heard through speakers.

[0020] "Text" is a means of conveying information through written characters or sentences.

[0021] "Navigation advice" refers to instructions or advice given to the driver regarding direction or method of travel. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] The present invention provides a system that supports safe and intuitive driving by collecting traffic information in real time, providing that information in multiple languages, and detecting traffic signs and pedestrians using image recognition technology. An embodiment of the system will be described in detail below.

[0044] 1. Providing real-time traffic information

[0045] Server: The server collects real-time traffic information from road traffic management centers, cloud services, etc. The collected data includes information on traffic congestion, accidents, road construction, etc., and integrates this information and stores it in a database.

[0046] Server: The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms to generate the optimal route based on the user's current location and destination.

[0047] Server: The server transmits the analysis results to the user's device in real time, allowing the user to receive navigation based on the latest traffic conditions.

[0048] Terminal: The terminal visually displays the traffic information received from the server on a map and provides the user with the optimal route. The map is displayed using OpenStreetMap or other map APIs.

[0049] 2. Multilingual support

[0050] Device: The device retrieves the user's language preference, which is then sent to the server, allowing different language settings to be applied for each user.

[0051] Server: The server translates traffic information and navigation advice into the user's language using a global API or other translation services.

[0052] Device: The device provides the translated information to the user in both voice and text. It uses speech synthesis technology to speak navigation advice in the user's language and simultaneously displays the translated text on the screen.

[0053] 3. Image Recognition Technology

[0054] Device: The device uses the smartphone camera to capture real-time images of the surroundings, which are then analyzed using image recognition algorithms (e.g., OpenCV or TensorFlow Lite).

[0055] Device: The device recognizes traffic signs and pedestrians from the captured video. The recognized information is used to assist the user in driving.

[0056] Server: The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time.

[0057] Terminal: The terminal notifies the user of advice from the server, for example providing audio and visual alerts to the user if they exceed the speed limit or if there are pedestrians ahead.

[0058] 4. Specific Examples

[0059] Example 1: The user sets a destination. The device sends current location and destination information to the server and receives the optimal route. The device displays the route on a map and provides route guidance based on traffic information updated in real time. If a traffic jam occurs along the way, the server immediately calculates a new route and sends updated information to the device. As a result, the user can reach the destination smoothly.

[0060] Example 2: When a user is driving in a foreign country, the device provides navigation advice in the user's language. The server translates collected traffic information and recognition results into the user's native language and provides them via voice and text through the device. This allows the user to drive with confidence even in a different language environment.

[0061] Example 3: When the device's camera recognizes a road sign while driving, the information is sent to the server and the necessary warning is generated. For example, if the device recognizes a "Stop" sign, the server analyzes it and sends a warning to the device. The device will notify the user by voice, "There is a stop sign ahead," to help the user take appropriate action.

[0062] As described above, this system provides users with the latest traffic information in real time, supports the driving environment with multilingual support, and uses image recognition technology to support safe driving, thereby reducing driving anxiety and providing a comfortable driving experience.

[0063] The processing flow will be explained below.

[0064] Program processing flow and specific operations

[0065] 1. Providing real-time traffic information

[0066] Step 1: Collect traffic information

[0067] Server: "The server collects traffic data in real time via API from road traffic management centers and cloud services."

[0068] Specific operation: Data such as traffic congestion, accidents, and road construction information is obtained using an API and stored in a database.

[0069] Step 2: Traffic information integration and analysis

[0070] Server: "The server integrates the collected traffic data into a database and uses machine learning algorithms to calculate the optimal route."

[0071] What it does: Analyzes traffic information data and calculates the optimal route to the user's destination.

[0072] Step 3: Distributing traffic information

[0073] Server: "The server sends the analyzed route information to the user terminal in real time."

[0074] Specific operation: The calculated optimal route is sent to the user's device via an HTTP request.

[0075] Step 4: View information

[0076] Terminal: "The terminal displays real-time traffic information received from the server on a map."

[0077] Specific behavior: Uses OpenStreetMap and other map APIs to visually display received route information on a map.

[0078] 2. Multilingual support

[0079] Step 1: Get the language settings

[0080] Device: "The device acquires the language information set by the user and sends it to the server."

[0081] Specific behavior: Get the user's language preference from the app's settings menu and send it to the server.

[0082] Step 2: Perform the translation

[0083] Server: "The server translates traffic information and navigation advice into the user's language."

[0084] What it does: Uses a translation API to translate collected traffic information and navigation advice into the user's language of choice.

[0085] Step 3: Providing multilingual guidance

[0086] Device: "The device provides multilingual, translated information to the user via voice and text."

[0087] Specific operation: Using speech synthesis technology, translated navigation information is output as voice and simultaneously displayed as text on the screen.

[0088] 3. Image Recognition Technology

[0089] Step 1: Acquire camera footage

[0090] Device: "The device captures video from the smartphone camera and processes it for image recognition."

[0091] Specific operation: Uses the camera API to capture real-time video and store it in memory.

[0092] Step 2: Perform image recognition

[0093] Device: "The device applies image recognition algorithms to the captured video to detect traffic signs and pedestrians."

[0094] Specific operation: Using OpenCV and TensorFlow Lite, it analyzes signs and pedestrians in video and detects specific features.

[0095] Step 3: Sending the recognition results

[0096] Device: "The device sends the recognition results to the server."

[0097] Specific operation: Organizes the recognized information and sends it to the server via API.

[0098] Step 4: Generate feedback

[0099] Server: "The server analyzes the recognition results and generates advice and warnings for safe driving."

[0100] What it does: Evaluates the received recognition data and generates the necessary feedback.

[0101] Step 5: Deliver feedback to users

[0102] Terminal: "The terminal notifies the user of advice and warnings from the server."

[0103] Specific behavior: Displays a warning message on the screen and issues an audio alert.

[0104] 4. User Interface Operation and Feedback

[0105] Step 1: Set your destination

[0106] User: "The user enters and sets a destination in the application."

[0107] What it does: Select a destination on the map or enter text in the search bar to specify a destination.

[0108] Step 2: Start route guidance

[0109] Terminal: "The terminal starts navigation based on the route information received from the server."

[0110] What it does: Highlights the route on the map and starts voice guidance.

[0111] Step 3: Receive real-time feedback

[0112] Server: "The server monitors the user's real-time movements and updates the route as needed."

[0113] Specific operation: Constantly monitors the user's current location based on GPS information and recalculates the route according to changes in traffic conditions.

[0114] Step 4: Real-time notifications

[0115] Terminal: "The terminal notifies the user of updates from the server in real time."

[0116] Specific behavior: When a new route is provided, the user is immediately notified by map display and voice.

[0117] This enables the entire system to provide users with real-time, up-to-date and reliable traffic information and safe driving support.

[0118] Example 1

[0119] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0120] This invention relates to a system that efficiently collects and analyzes traffic information updated in real time and provides users with optimal routes. In particular, it aims to support safe and intuitive driving by recognizing the surrounding environment, such as traffic signs and pedestrians, from camera images and notifying the user of this information in multiple languages. Conventional systems have had issues with driver convenience and safety due to slow traffic information updates and insufficient multilingual support and real-time image recognition.

[0121] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0122] In this invention, the server includes a means for collecting traffic information in real time, a means for analyzing the collected traffic information and calculating an optimal route using a machine learning algorithm, and a means for transmitting the analysis results to a user terminal. This enables efficient and prompt provision of an optimal route based on the latest traffic information. The user terminal also includes a means for visually displaying the received traffic information on a map, a means for recognizing road signs and pedestrians using a camera, a means for transmitting the recognition results to the server and notifying the user terminal of analyzed warnings and advice, and a means for providing navigation advice using speech synthesis technology that supports multiple languages. This allows users to receive safe and intuitive driving support in multiple languages ​​in real time.

[0123] "Real-time" refers to processing and providing information instantly the moment it is generated.

[0124] "Traffic information" refers to information relating to traffic conditions on roads, including data on congestion, accidents, road construction, and the like.

[0125] A "machine learning algorithm" is a computational method that automatically learns patterns and rules by analyzing large amounts of data, and makes predictions and classifications.

[0126] "Optimal route" refers to the shortest and most efficient route for a user to reach a destination.

[0127] A "user terminal" is a device operated by a user, including a smartphone, tablet, car navigation system, etc.

[0128] "Visually displaying on a map" refers to visually displaying traffic and route information using a map application or navigation system.

[0129] "Camera" means a device for capturing video or images.

[0130] "Road sign" means a sign placed on a road to provide direction or information to motorists.

[0131] "Pedestrian" refers to a person who moves on foot on a road.

[0132] "Recognizing" refers to detecting and identifying specific objects (such as road signs or pedestrians) from the image captured by the camera.

[0133] "Analyze" refers to the detailed analysis of collected data and recognition results to derive meaningful information and predictions.

[0134] "Warning" means providing information to draw attention to a particular situation.

[0135] "Advice" means providing instructions or suggestions for specific action.

[0136] "Speech synthesis technology" is a technology that generates and plays back voices that resemble human voices from text data.

[0137] "Multilingual" means supporting multiple different languages.

[0138] "Translate" refers to converting text written in one language into another language.

[0139] The present invention provides a system that supports safe and intuitive driving by collecting traffic information in real time, providing that information in multiple languages, and detecting traffic signs and pedestrians using image recognition technology. An embodiment of the system will be described in detail below.

[0140] 1. Providing real-time traffic information

[0141] server:

[0142] The server continuously obtains real-time traffic information from the road traffic management center API or cloud services (e.g., general traffic information API), including information on traffic congestion, accidents, road construction, etc. The server stores this information in a database such as MongoDB and generates an integrated dataset.

[0143] server:

[0144] The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms (e.g., Scikit-learn or TensorFlow), and sends the results of this calculation to the user's device in real time.

[0145] Device:

[0146] The device visually displays the traffic information received from the server on a map and provides the user with the optimal route, using external services such as OpenStreetMap and Google Maps API.

[0147] 2. Multilingual navigation available

[0148] Device:

[0149] The terminal obtains the user's language setting and sends that information to the server.

[0150] server:

[0151] The server uses the Global Translate API (e.g., Google Translate API) to translate traffic information and navigation instructions into the user's preferred language.

[0152] Device:

[0153] The device provides the translated information to the user in real time using speech synthesis technology (e.g., Amazon Polly), and also displays the corresponding text information on a map.

[0154] 3. Traffic Sign and Pedestrian Detection Using Image Recognition Technology

[0155] Device:

[0156] The device uses its built-in camera to capture real-time images of its surroundings, which are then analyzed by image recognition algorithms (e.g., OpenCV or TensorFlow Lite) running on the device.

[0157] Device:

[0158] The device recognizes traffic signs and pedestrians in the video and sends the recognition results to the server in real time.

[0159] server:

[0160] The server analyzes the received recognition data and notifies the user's device of warnings and advice necessary for driving.

[0161] Device:

[0162] The device provides the user with audio and visual warnings and advice from the server, such as "There is a pedestrian ahead" or "You are exceeding the speed limit."

[0163] Specific examples

[0164] Example 1:

[0165] When a user inputs their current location and destination, the device sends the information to the server, which calculates the optimal route and sends it back to the device. The device then displays the route on a map and provides route guidance based on traffic information updated in real time. If traffic congestion occurs, the server calculates a new route and sends updated information to the device, allowing the user to reach their destination smoothly.

[0166] Example 2:

[0167] When a user is driving in a foreign country, the device will provide navigation advice in the user's language of choice. The server will translate traffic information and recognition results into the user's native language and provide them via voice and text via the device, allowing users to drive with confidence in different language environments.

[0168] Example 3:

[0169] When the device's camera recognizes a road sign while driving, the information is sent to the server, which generates the necessary warning. For example, if a "Stop" sign is recognized, the server analyzes it and sends a warning to the device. The device then notifies the user by voice, saying "There is a stop sign ahead," helping the user to take appropriate action.

[0170] Prompt Sentence Examples

[0171] 1. Give an example of a program that calculates the optimal route based on traffic information when a user inputs their current location and destination.

[0172] 2. Describe the process flow of a system that provides multilingual voice and text navigation based on real-time traffic data.

[0173] 3. Explain the process of a system that uses a smartphone camera and image recognition algorithms to detect traffic signs and pedestrians and alert the driver.

[0174] This invention allows users to utilize the latest traffic information in real time and easily implements a system that supports the driving environment in multiple languages, thereby enabling safe and efficient driving, reducing driving anxiety, and realizing a comfortable driving experience.

[0175] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0176] Step 1:

[0177] Real-time traffic information collection

[0178] Server: The server obtains real-time traffic information from the Road Traffic Management Center API and cloud services.

[0179] Input: Traffic information data received from the Road Traffic Management Center API and cloud services.

[0180] Data processing: The acquired traffic information is converted into JSON format and stored in a database.

[0181] Output: Updated traffic information dataset.

[0182] What it does: The server periodically sends API requests to retrieve new traffic information and stores it in MongoDB.

[0183] Step 2:

[0184] Traffic information analysis and optimal route calculation

[0185] Server: The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms (e.g., Scikit-learn, TensorFlow).

[0186] Input: Traffic information data stored in a database.

[0187] Data calculations: Using machine learning algorithms to calculate the optimal route based on the user's current location and destination.

[0188] Output: Optimal route information provided to the user.

[0189] Specific operation: The server applies Dijkstra's algorithm based on map data and traffic information to calculate the shortest route.

[0190] Step 3:

[0191] Sending optimal route information

[0192] Server: The server transmits the calculated optimal route information to the user's device in real time.

[0193] Input: Calculated optimal route information.

[0194] Data calculation: If necessary, convert the route information into JSON format so that it can be sent.

[0195] Output: Optimal route information sent to the user terminal.

[0196] Specific operation: The server sends JSON format data to the user terminal via the HTTP protocol.

[0197] Step 4:

[0198] Displaying traffic and route information

[0199] Terminal: The terminal visually displays the traffic and route information received from the server on a map.

[0200] Input: Traffic and route information received from the server.

[0201] Data processing: Data conversion to overlay information on a map.

[0202] Output: Map and route information visually displayed to the user.

[0203] Specific operation: The device displays a map using the Google Maps API and overlays the received route information on it.

[0204] Step 5:

[0205] Multilingual navigation provided

[0206] Device: The device obtains the user's language preference and sends it to the server.

[0207] Input: User's language preference information.

[0208] Output: Language preference data sent to the server.

[0209] Specific behavior: The device obtains the user's selected language from the language settings screen.

[0210] Server: The server translates traffic information and navigation instructions into the user's preferred language.

[0211] Input: Traffic information, navigation instructions, and user language preference data.

[0212] Data Calculation: Translate data into a specified language using the Global Translate API.

[0213] Output: Translated traffic and navigation instructions.

[0214] Specific operation: The server calls the Google Translate API to translate the data and sends the results to the device.

[0215] Terminal: The terminal synthesizes the translated information and provides it to the user.

[0216] Input: Translated traffic and navigation instructions.

[0217] Data processing: Generate voice data using a speech synthesis application (e.g., Amazon Polly).

[0218] Output: Audio and text information provided to the user.

[0219] Specific operation: The device uses Amazon Polly to play a voice message saying "Go straight to your destination" and also displays the text on the map.

[0220] Step 6:

[0221] Traffic sign and pedestrian detection using image recognition

[0222] Device: The device uses its built-in camera to capture real-time images of its surroundings.

[0223] Input: Video data captured by the camera.

[0224] Output: Video data that is passed to the image recognition algorithm.

[0225] Specific operation: The device starts the camera and acquires streaming video data.

[0226] Terminal: The terminal analyzes the video data using image recognition algorithms (e.g., OpenCV or TensorFlow Lite).

[0227] Input: Video data captured by the camera.

[0228] Data calculations: Image recognition algorithms identify traffic signs and pedestrians.

[0229] Output: Information on recognized traffic signs and pedestrians.

[0230] Specific operation: The device uses OpenCV to recognize stop signs and pedestrians.

[0231] Device: Sends the recognition results to the server.

[0232] Input: The recognized information.

[0233] Output: Recognition result data sent to the server.

[0234] Specific operation: The device converts the recognized data into JSON format and sends it to the server.

[0235] Server: The server analyzes the received recognition data and generates any necessary warnings or advice.

[0236] Input: Recognition data sent from the device.

[0237] Data calculations: Analyze perception data to generate warnings and advice.

[0238] Output: Warnings and advice sent to the user's terminal.

[0239] Specific operation: Based on the analysis results, the server generates a warning such as "There is a pedestrian ahead" and sends it to the device.

[0240] Terminal: The terminal notifies the user of warnings and advice from the server by voice and display.

[0241] Input: Any warnings or advice received from the server.

[0242] Output: The audio and visual information provided to the user.

[0243] Specific operation: The device will announce "There is a pedestrian ahead" and display an icon on the screen.

[0244] (Application example 1)

[0245] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0246] Conventional navigation systems only provide traffic information and are limited in their ability to respond to changes in traffic conditions in real time. Furthermore, they lack driving assistance in multilingual environments and the ability to recognize road signs and pedestrians in real time, making them inadequate for supporting safe driving. There is a need to solve these issues and provide safer and more efficient driving assistance.

[0247] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0248] In this invention, the server includes means for collecting traffic information in real time, means for analyzing the collected traffic information and calculating the optimal route, and means for transmitting the analysis results to the user terminal, thereby enabling the server to respond to changes in traffic conditions in real time and provide the optimal route.

[0249] Furthermore, this invention provides safe and intuitive driving assistance by incorporating a means for recognizing road signs and pedestrians using a camera, a means for notifying a user terminal of the recognition results, a means for providing audio advice based on the recognition results, and a means having a multilingual translation function based on local languages.

[0250] "Means of collecting traffic information in real time" refers to the function of instantly obtaining the latest traffic information, such as information on traffic congestion, accidents, and road construction, from traffic management centers and cloud services.

[0251] "Means of analyzing collected traffic information and calculating the optimal route" refers to a function that uses a machine learning algorithm to generate the optimal route taking into account traffic conditions based on the user's current location and destination.

[0252] The "means for transmitting the analysis results to the user terminal" is a function for transferring optimal route information and traffic conditions calculated by the server to the user terminal in real time.

[0253] "Means for visually displaying received traffic information on a user terminal" refers to a function that graphically displays maps and route guidance on the user's terminal, allowing the user to intuitively understand traffic information.

[0254] "Means of recognizing road signs and pedestrians using cameras" refers to a function that analyzes camera footage in real time and detects traffic signs and pedestrians with high accuracy.

[0255] "Means for notifying the user terminal of the recognition results" is a function that immediately transmits information about detected signs and pedestrians to the user terminal and notifies the driver of the situation.

[0256] "Means for providing voice advice based on the recognition results" is a function that provides appropriate driving advice by voice based on the recognized information, supporting the driver in safe driving.

[0257] "Means having a multilingual translation function based on local languages" refers to a function that automatically translates traffic information and navigation advice into different languages ​​according to user settings.

[0258] "Means for translating and providing traffic information and navigation advice in multiple languages" refers to a function that automatically translates and provides collected and analyzed traffic information and driving advice based on the user's language settings.

[0259] "Means for providing translated information in voice and text format based on the user's language settings" refers to a function that provides translated traffic information and route guidance in voice and text format according to the language settings registered by the user.

[0260] This invention will be explained using the smartphone application "SmartDrive Assist" installed in autonomous vehicles as a concrete example. This application collects traffic information in real time, provides it in multiple languages, and utilizes image recognition technology to monitor the external environment and support safe driving.

[0261] First, the server collects traffic information in real time from road traffic management centers and cloud services. This traffic information includes information on traffic congestion, accidents, road construction, and more, and stores it in a database. Next, the collected traffic information is analyzed and the optimal route is calculated using a machine learning algorithm. This calculation uses map APIs such as Google Maps API and OpenStreetMap. Finally, the analysis results are sent to the user's device in real time.

[0262] The device visually displays traffic information received from the server. Maps are displayed using OpenStreetMap and other map APIs. In addition, the device uses the smartphone camera to capture real-time images of the surrounding area. This image is analyzed using image recognition algorithms (e.g., OpenCV or TensorFlow Lite) to recognize road signs and pedestrians. The recognized information is used to assist the user's driving.

[0263] The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time. The device notifies the user of the advice from the server. For example, if the device recognizes a "Stop" sign, the server analyzes it and sends a warning to the device. The device then notifies the user by voice, saying "There is a stop sign ahead," helping the user to take appropriate action.

[0264] The device also obtains the user's language preference and sends this information to the server. The server translates traffic information and navigation advice into the user's language using a global API or other translation services. The device then provides the translated information to the user in voice and text. It uses speech synthesis technology to output navigation advice in the user's language preference and displays the translated information in text on the screen.

[0265] As a concrete example, consider a scenario in which a user is traveling by car from Tokyo to Los Angeles. First, the user launches the "SmartDrive Assist" app on their smartphone and sets their destination (Los Angeles). The app retrieves real-time traffic information from the server and provides voice advice translated into multiple languages. It uses the smartphone's camera to detect road signs and pedestrians and provides appropriate voice warnings to the user.

[0266] Example prompt sentence:

[0267] Consider a scenario where a user is using the "SmartDrive Assist" app to drive from Tokyo to Los Angeles. The app collects real-time traffic information, provides it in multiple languages, and detects traffic signs and pedestrians through camera footage.

[0268] In this way, the present invention provides users with the latest traffic information in real time, supports the driving environment with multilingual support, and uses image recognition technology to support safe driving, thereby reducing driving anxiety and providing a comfortable driving experience.

[0269] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0270] Step 1:

[0271] The server collects real-time traffic information from road traffic management centers and cloud services. It obtains information on traffic congestion, accidents, road construction, etc. through the road traffic management center and cloud service APIs, and receives this data in JSON format. This allows it to accumulate the latest traffic situation data.

[0272] Input: Traffic information from the traffic management center and cloud service API

[0273] Data processing: Receive traffic information in JSON format

[0274] Output: Real-time traffic information database

[0275] Step 2:

[0276] The server analyzes the collected traffic information and calculates the optimal route. It uses machine learning algorithms to generate the optimal route based on the user's current location and destination. The algorithms used include linear regression models and neural networks. Map APIs such as Google Maps API and OpenStreetMap are also used for route calculation.

[0277] Input: Collected traffic information, user's current location, user's destination

[0278] Data processing: Analyzing traffic information and calculating optimal routes using machine learning algorithms

[0279] Output: Optimal route information

[0280] Step 3:

[0281] The server transmits the analysis results to the user's device in real time, using communication protocols such as HTTP requests to transmit the generated optimal route information and traffic condition data to the user's device.

[0282] Input: Optimal route information, traffic condition data

[0283] Data processing: Sending data via HTTP request

[0284] Output: Optimal route information sent to the user terminal

[0285] Step 4:

[0286] The device visually displays the traffic information received from the server. It uses a map API (e.g., OpenStreetMap or Google Maps) to display the received traffic information and the optimal route on a map. This operation uses the display function of the map API.

[0287] Input: Optimal route information and traffic condition data sent from the server

[0288] Data processing: Traffic information and route mapping

[0289] Output: Visual map display

[0290] Step 5:

[0291] The device uses the smartphone camera to capture images of the surroundings in real time. When the user activates the device's camera, real-time images are captured.

[0292] Input: Real-time camera image data

[0293] Data processing: Video capture and real-time acquisition

[0294] Output: Real-time camera footage

[0295] Step 6:

[0296] The device recognizes traffic signs and pedestrians using the captured video. It uses an image recognition algorithm (e.g., OpenCV or TensorFlow Lite) to analyze the captured video and extract information about detected traffic signs and pedestrians.

[0297] Input: Captured camera footage

[0298] Data processing: Traffic sign and pedestrian detection using image recognition algorithms

[0299] Output: Detected traffic signs and pedestrian information

[0300] Step 7:

[0301] The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. For example, if a "Stop" sign is detected, the server analyzes it and generates appropriate driving advice.

[0302] Input: Traffic sign and pedestrian information sent from the device

[0303] Data processing: Analyzing perception data and generating driving advice

[0304] Output: Driving advice

[0305] Step 8:

[0306] The terminal notifies the user of the advice from the server by generating appropriate voice advice and notifying the user using speech synthesis technology (e.g., gTTS), and also by displaying it in text format on the screen.

[0307] Input: Driving advice sent from the server

[0308] Data processing: speech synthesis and text display

[0309] Output: Voice and text driving advice notifications

[0310] Step 9:

[0311] The server obtains the user's language preference and translates traffic information and navigation advice. It uses a translation service (e.g., Google Translate API) to translate the information based on the user's language preference.

[0312] Input: User-defined language information, traffic information, navigation advice

[0313] Data processing: Multilingual translation using translation services

[0314] Output: Translated traffic information and navigation advice

[0315] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0316] The present invention is a system that collects traffic information in real time and provides that information in multiple languages, and by combining it with an emotion engine, recognizes the driver's emotional state and supports safe and comfortable driving. An embodiment of the system will be described in detail below.

[0317] 1. Providing real-time traffic information

[0318] Server: The server collects traffic data in real time via API from road traffic management centers, cloud services, etc. The collected data includes information on traffic congestion, accidents, road construction, etc., and integrates this information and stores it in a database.

[0319] Server: The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms to generate the optimal route based on the user's current location and destination.

[0320] Server: The server transmits the analysis results to the user's device in real time, allowing the user to receive navigation based on the latest traffic conditions.

[0321] Terminal: The terminal visually displays the traffic information received from the server on a map and provides the user with the optimal route. The map is displayed using OpenStreetMap or other map APIs.

[0322] 2. Multilingual support

[0323] Device: The device retrieves the user's language preference, which is then sent to the server, allowing different language settings to be applied for each user.

[0324] Server: The server translates traffic information and navigation advice into the user's language using a global API or other translation services.

[0325] Device: The device provides the translated information to the user in both voice and text. It uses speech synthesis technology to speak navigation advice in the user's language and simultaneously displays the translated text on the screen.

[0326] 3. Image Recognition Technology

[0327] Device: The device uses the smartphone camera to capture real-time images of the surroundings, which are then analyzed using image recognition algorithms (e.g., OpenCV or TensorFlow Lite).

[0328] Device: The device recognizes traffic signs and pedestrians from the captured video. The recognized information is used to assist the user in driving.

[0329] Server: The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time.

[0330] Terminal: The terminal notifies the user of advice from the server, for example providing audio and visual alerts to the user if they exceed the speed limit or if there are pedestrians ahead.

[0331] 4. Emotion Engine

[0332] Device: The device uses a smartphone or in-car camera to recognize emotions from the user's facial expressions, tone of voice, heart rate, etc. This emotional data is collected and evaluated in real time.

[0333] Server: The server analyzes the emotion data sent from the emotion engine and evaluates the user's driving state, for example, to detect if the driver is nervous or tired.

[0334] Device: The device will adjust navigation advice based on the emotion engine's analysis. For example, if the user is nervous, it will provide calming advice or relaxing music.

[0335] Server: The server reconstructs a navigation plan according to the user's emotional state and sends it to the device, thereby providing the optimal route and advice for the user's mental state.

[0336] Device: The device provides users with navigation advice and music tailored by the emotion engine, thereby reducing the user's driving stress and supporting safe driving.

[0337] 5. Specific Examples

[0338] Example 1: When a user sets a destination, the device sends current location and destination information to the server and receives the optimal route. The emotion engine analyzes the user's facial expressions and tone of voice to evaluate their state of mind to drive safely. If the user is nervous, the device plays relaxing music and provides gentle voice guidance.

[0339] Example 2: When a user is driving for a long time, the emotion engine detects fatigue and notifies the server. The server recalculates the route and suggests places where users can take a break. The device notifies the user by voice, saying, "There is a rest area ahead. Would you like to take a short break?"

[0340] Example 3: For users who are nervous, the device provides information on recognized traffic signs and pedestrians in real time. For example, if the camera recognizes a "Stop" sign, the device will give a voice command saying, "There is a stop sign. Please stop," giving the user a sense of security.

[0341] As described above, this system combines real-time traffic information, multilingual support, image recognition technology, and an emotion engine to provide users with a safe and comfortable driving experience. By appropriately assessing the user's emotional state and adjusting navigation advice based on that assessment, the system achieves a less stressful driving experience.

[0342] The processing flow will be explained below.

[0343] Program processing flow and specific operations (including emotion engine)

[0344] 1. Providing real-time traffic information

[0345] Step 1: Collecting traffic information

[0346] Server: "The server collects traffic data in real time via API from road traffic management centers and cloud services."

[0347] Specific operation: Data such as traffic congestion, accidents, and road construction information is obtained using an API and stored in a database.

[0348] Step 2: Traffic information integration and analysis

[0349] Server: "The server integrates the collected traffic data into a database and uses machine learning algorithms to calculate the optimal route."

[0350] What it does: Analyzes traffic information data and calculates the optimal route to the user's destination.

[0351] Step 3: Distributing traffic information

[0352] Server: "The server sends the analyzed route information to the user terminal in real time."

[0353] Specific operation: The calculated optimal route is sent to the user's device via an HTTP request.

[0354] Step 4: Viewing information

[0355] Terminal: "The terminal displays real-time traffic information received from the server on a map."

[0356] Specific behavior: Uses OpenStreetMap and other map APIs to visually display received route information on a map.

[0357] 2. Multilingual support

[0358] Step 1: Get the language settings

[0359] Device: "The device acquires the language information set by the user and sends it to the server."

[0360] Specific behavior: Get the user's language preference from the app's settings menu and send it to the server.

[0361] Step 2: Performing the translation

[0362] Server: "The server translates traffic information and navigation advice into the user's language."

[0363] What it does: Uses a translation API to translate collected traffic information and navigation advice into the user's language of choice.

[0364] Step 3: Providing multilingual guidance

[0365] Device: "The device provides multilingual, translated information to the user via voice and text."

[0366] Specific operation: Using speech synthesis technology, translated navigation information is output as voice and simultaneously displayed as text on the screen.

[0367] 3. Image Recognition Technology

[0368] Step 1: Acquire camera footage

[0369] Device: "The device captures video from the smartphone camera and processes it for image recognition."

[0370] Specific operation: Uses the camera API to capture real-time video and store it in memory.

[0371] Step 2: Perform image recognition

[0372] Device: "The device applies image recognition algorithms to the captured video to detect traffic signs and pedestrians."

[0373] Specific operation: Using OpenCV and TensorFlow Lite, it analyzes signs and pedestrians in video and detects specific features.

[0374] Step 3: Sending the recognition results

[0375] Device: "The device sends the recognition results to the server."

[0376] Specific operation: Organizes the recognized information and sends it to the server via API.

[0377] Step 4: Generate feedback

[0378] Server: "The server analyzes the recognition results and generates advice and warnings for safe driving."

[0379] What it does: Evaluates the received recognition data and generates the necessary feedback.

[0380] Step 5: Deliver feedback to users

[0381] Terminal: "The terminal notifies the user of advice and warnings from the server."

[0382] Specific behavior: Displays a warning message on the screen and issues an audio alert.

[0383] 4. Emotion Engine

[0384] Step 1: Collecting emotion data

[0385] Device: "The device uses a smartphone or in-car camera to capture emotional data such as the user's facial expressions, tone of voice, and heart rate."

[0386] What it does: It uses facial recognition APIs, voice analysis APIs, and heart rate sensors to capture emotion-related data and evaluate it in real time.

[0387] Step 2: Sending emotion data

[0388] Device: "The device sends the collected emotion data to the server."

[0389] Specific operation: Organize emotion data and send it to the server via API.

[0390] Step 3: Analyze the sentiment data

[0391] Server: "The server uses the emotion engine to analyze the received emotion data and evaluate the user's driving state."

[0392] What it does: It uses analytics algorithms to identify the user's emotional state, such as whether they are nervous or tired.

[0393] Step 4: Adjust navigation based on emotions

[0394] Device: "The device will adjust its navigation advice based on the emotion engine's analysis."

[0395] Specific actions: For example, if the user is nervous, give gentle advice and play relaxing music.

[0396] Step 5: Emotionally recalculated route

[0397] Server: "The server reconstructs the optimal navigation plan based on the user's emotional state and sends it to the device."

[0398] What it does: Calculates a new route and provides the best route for the user's current mental state.

[0399] Step 6: Provide emotion-based feedback

[0400] Device: "The device provides users with navigation advice and music tailored by the emotion engine."

[0401] Specific behavior: Provides optimal feedback to the user through voice notifications and on-screen displays, supporting stress-free driving.

[0402] In this way, this system combines real-time traffic information, multilingual support, image recognition technology, and an emotion engine to provide users with a safe and comfortable driving experience. By properly recognizing the user's emotional state and adjusting navigation advice accordingly, it further reduces driving stress and supports safe driving.

[0403] Example 2

[0404] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0405] In today's traffic environment, drivers are required to select the optimal route based on traffic information updated in real time. However, systems that do not properly reflect traffic information, lack multilingual support, or do not take the driver's emotional state into account make it difficult to support safe and comfortable driving. This raises concerns about increased traffic accidents and driving stress. Therefore, the present invention aims to solve these issues and provide drivers with a safe and comfortable driving experience.

[0406] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0407] In this invention, the server includes means for collecting traffic information in real time from road traffic management agencies and cloud services, means for analyzing the collected traffic information using a machine learning algorithm to calculate an optimal route, and means for transmitting the analysis results to a user terminal. This allows the user to receive optimal navigation based on the latest traffic conditions. It also includes means for recognizing road signs and pedestrians using a camera and means for notifying the driver of warnings and advice based on the recognition results. This allows the user to receive appropriate warnings and advice in real time while driving. Furthermore, the system also includes means for collecting the user's facial expressions, tone of voice, and heart rate to analyze the user's emotional state, and means for adjusting navigation advice based on the emotional state, thereby supporting safe and comfortable driving according to the driver's emotional state.

[0408] "Real-time traffic information" refers to information that is obtained immediately after a traffic situation occurs, and includes current road conditions, congestion, accident information, road construction information, and the like.

[0409] A "machine learning algorithm" is a computational method that learns patterns from data and makes future decisions and predictions based on the results of that learning.

[0410] The "optimal route" is the route determined to be the most efficient from the user's starting point to the destination, taking into consideration factors such as time, distance, and traffic conditions.

[0411] A "user terminal" is an information terminal used by a driver, and includes a smartphone, tablet, in-vehicle display, etc.

[0412] "Visually displaying" means presenting information in a graphical format on a display or screen, including displays using maps, icons, text, etc.

[0413] A "camera" is a device used to capture images, such as a camera built into a smartphone or a camera attached to a vehicle.

[0414] "Road signs" are signs that indicate traffic rules and road conditions, and include stop signs, speed limit signs, no entry signs, and the like.

[0415] "Pedestrian" refers to a person crossing a road or walking on the side of the road.

[0416] "Emotional state" refers to the driver's psychological and emotional state, including states such as tension, fatigue, and relief.

[0417] "Navigation advice" is route guidance information provided to a user, including instructions in audio or text format.

[0418] "Warnings and advice" refers to alerts and instructions for actions to the user, including audible and visual alerts to warn of danger, and advice to encourage improved driving.

[0419] The present invention provides a system that collects traffic information in real time, provides the information in multiple languages, and recognizes the emotional state of the driver to support safe and comfortable driving. Detailed embodiments of the system are described below.

[0420] 1. Providing real-time traffic information

[0421] Server: The server collects real-time traffic data from road traffic management agencies and cloud services via API. The collected data includes information on traffic congestion, accidents, road construction, etc., and consolidates this information and stores it in a database.

[0422] Example: A server sends an API request every minute to get new traffic data and automatically stores the data in a database.

[0423] Server: The server analyzes the collected traffic information using machine learning algorithms to calculate the optimal route, using machine learning libraries such as TensorFlow and Scikit-learn.

[0424] Example: The server takes the user's current location and destination as input, runs an algorithm to calculate the optimal route, and stores the results in a database.

[0425] Server: The server sends the calculation results to the user's device in real time.

[0426] Example: The server sends a push notification to the user's device, providing a data packet containing optimal route information.

[0427] Terminal: The terminal visually displays the traffic information received from the server on a map and suggests the optimal route to the user. The map is displayed using OpenStreetMap (OSM) or Google Maps API.

[0428] Example: The device uses the Google Maps API to display a map and highlight the best route.

[0429] 2. Multilingual support

[0430] Terminal: The terminal acquires the language information set by the user and sends it to the server.

[0431] Example: A device reads the language information selected in the language setting menu and sends that information to a server.

[0432] Server: The server translates traffic information and navigation advice into the user's language using Google Cloud Translation API and DeepL.

[0433] Example: The server sends the acquired traffic information to a translation service via API and receives the translation results.

[0434] Terminal: The terminal provides the translated information to the user in voice and text format. It uses speech synthesis technology to convert the translated information into voice and display it on the display.

[0435] Example: The device starts a speech synthesis engine, converts the translated text into speech, and outputs it through the speaker, while displaying the text on the display.

[0436] 3. Image Recognition Technology

[0437] Device: The device uses the smartphone camera to capture real-time images of the surroundings, which are then analyzed using image recognition algorithms (e.g., OpenCV, TensorFlow Lite).

[0438] Example: The device activates the camera and starts the video stream.

[0439] Device: The device uses image recognition algorithms to recognize traffic signs and pedestrians from the captured video.

[0440] Example: The device runs image recognition algorithms on each video frame to detect signs and pedestrians.

[0441] Server: The server analyzes the recognition data sent from the device and generates warnings and advice for the driver.

[0442] Example: The server analyzes the received data and, if necessary, generates warning messages or driving advice and sends them to the terminal.

[0443] Terminal: The terminal notifies the user of advice from the server, for example providing audio and visual alerts to the driver if the speed limit is exceeded or if there is a pedestrian ahead.

[0444] Example: A device uses a speech synthesis engine to generate a warning message and output it over the speaker, and also displays the warning text on the screen.

[0445] 4. Emotion Engine

[0446] Device: The device uses a smartphone or in-car camera and microphone to capture the user's facial expressions, tone of voice, heart rate, and other data in real time.

[0447] Example: The device activates the camera and microphone to collect facial expressions, voice data, and data from a heart rate monitor.

[0448] Server: The server analyzes the emotion data sent from the emotion engine and evaluates the user's driving state, for example, detecting whether the user is nervous or tired.

[0449] Example: The server uses a machine learning model to analyze emotion data and evaluate the driver's state.

[0450] Device: The device will adjust its navigation advice based on the emotion engine's analysis.

[0451] Example: Based on the analysis received from the server, the device adjusts the settings of the navigation system, plays relaxing music, or provides gentle voice guidance.

[0452] Server: The server reconstructs the navigation plan according to the user's emotional state and sends it to the device.

[0453] Example: The server creates a recalculated navigation plan based on the results of an evaluation of the user's emotional state and sends it to the device.

[0454] Device: The device provides users with navigation advice and music tailored based on the emotion engine, thereby supporting safe driving.

[0455] Example: The device plays relaxing music selected from a playlist and outputs gentle voice prompts through the speaker while displaying instructions on the screen.

[0456] Prompt Sentence Examples

[0457] 1. Example 1: When a user sets a destination, the device sends current location and destination information to the server and receives the optimal route. The emotion engine analyzes the user's facial expressions and tone of voice to evaluate their level of confidence in driving. If the user is nervous, the device plays relaxing music and provides gentle voice guidance.

[0458] Example prompt: "Describe a scenario in which a user receives the best route to a destination and the emotion engine suggests relaxing music."

[0459] 2. Example 2: When a user is driving for a long time, the emotion engine detects fatigue and notifies the server. The server recalculates the route and suggests places where you can take a break. The device notifies the user by voice, "There is a rest area up ahead. Would you like to take a short break?"

[0460] Example prompt: "Describe a scenario where a user is driving for a long period of time, the emotion engine detects their fatigue, and the server suggests places where they can take a break."

[0461] 3. Example 3: For users who are nervous, the device provides information on recognized traffic signs and pedestrians in real time. For example, if the camera recognizes a "Stop" sign, the device will give a voice command saying, "There is a stop sign. Please stop," giving the user a sense of security.

[0462] Example prompt: "Describe a scenario in which a user feels anxious and is provided with real-time information about recognized traffic signs and pedestrians to promote safety."

[0463] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0464] Step 1: Collecting traffic data

[0465] Server: The server collects traffic data from road traffic management agencies and cloud services via API.

[0466] Input: API request

[0467] Data processing: The acquired data is analyzed in JSON format to extract information such as traffic congestion, accidents, and road construction information.

[0468] Output: Traffic information stored in a database

[0469] What it does: The server sends an API request every minute to get new traffic data, parses the data, and stores it in a database.

[0470] Step 2: Analyze the collected data

[0471] Server: The server uses machine learning algorithms to analyze the collected traffic information and calculate the optimal route.

[0472] Input: Traffic information data and user's current location and destination

[0473] Data processing: Calculate the optimal route using machine learning models (e.g., TensorFlow, Scikit-learn).

[0474] Output: Optimal route information

[0475] Specific operation: The server receives the user's current location and destination as input, calculates the optimal route through an algorithm, and stores the results in a database.

[0476] Step 3: Sending route information

[0477] Server: The server sends the calculation results to the user's device in real time.

[0478] Input: Optimal route information

[0479] Data processing: None

[0480] Output: Route information sent to the user terminal

[0481] Specific operation: The server sends a push notification to the user's device and provides a data packet containing optimal route information.

[0482] Step 4: View traffic information

[0483] Terminal: The terminal visually displays the traffic information received from the server on a map and presents the optimal route to the user.

[0484] Input: Route information sent

[0485] Data processing: Analyze the received data and display it on a map using a map API (e.g., OpenStreetMap, Google Maps API).

[0486] Output: Visual representation of the route

[0487] Specific operation: The device uses the map API to display a map and highlight the optimal route.

[0488] Step 5: Get and send language settings

[0489] Terminal: The terminal obtains the language information set by the user and sends it to the server.

[0490] Input: User's language preference

[0491] Data processing: None

[0492] Output: Language preference information sent to the server

[0493] Specific operation: The device reads the language information selected in the language setting menu and sends that information to the server.

[0494] Step 6: Translating traffic information and advice

[0495] Server: The server translates traffic information and navigation advice into the user's language.

[0496] Input: Traffic and navigation advice, language preference information

[0497] Data processing: Translation is performed using a translation API (e.g., Google Cloud Translation API, DeepL).

[0498] Output: Translated traffic information and navigation advice

[0499] Specific operation: The server sends the acquired traffic information and navigation advice to the translation service via API and receives the translation results.

[0500] Step 7: Provide translation information

[0501] Terminal: The terminal provides the translated information to the user in voice and text.

[0502] Input: Translated traffic and navigation advice

[0503] Data processing: Convert the data into speech using speech synthesis technology (e.g., speech synthesis engine), and display the text on the screen.

[0504] Output: Spoken and textual translation information

[0505] What happens: The device will launch its speech synthesis engine, convert the translated text into speech and output it through the speaker, and display the text on the display.

[0506] Step 8: Acquire and analyze footage

[0507] Device: The device uses the smartphone camera to capture real-time images of the surrounding area and analyzes them using image recognition algorithms.

[0508] Input: Camera image

[0509] Data processing: Run an image recognition algorithm (e.g., OpenCV, TensorFlow Lite) on each video frame to detect signs and pedestrians.

[0510] Output: Recognized traffic signs and pedestrian information

[0511] What happens: The device turns on the camera and starts streaming video. It then performs image recognition on each video frame to detect signs and pedestrians.

[0512] Step 9: Sending Recognition Data

[0513] Device: The device sends the recognized traffic signs and pedestrian information to the server.

[0514] Input: Recognized signs and pedestrian information

[0515] Data processing: None

[0516] Output: Identification information sent to the server

[0517] Specific operation: The device includes the recognized information in a data packet and sends it to the server.

[0518] Step 10: Generate warnings and advice

[0519] Server: The server analyzes the recognition data sent from the device and generates real-time warnings and advice for the driver.

[0520] Input: Recognition data

[0521] Data processing: Generate warnings and advice based on the recognition data.

[0522] Output: any warnings or advice generated

[0523] Specific operation: The server analyzes the received data and, if necessary, generates warning messages or driving advice and sends them to the terminal.

[0524] Step 11: Notification of warnings and advice

[0525] Terminal: The terminal notifies the user of warnings and advice from the server.

[0526] Input: Warnings and advice

[0527] Data processing: Converts data into speech using speech synthesis technology and displays the text on the screen.

[0528] Output: Audio and visual warnings and advice

[0529] What happens: The device uses a speech synthesis engine to generate a warning message and outputs it over the speaker, and also displays the warning text on the screen.

[0530] Step 12: Obtaining Emotion Data

[0531] Device: The device uses a smartphone, in-car camera, or microphone to capture the user's facial expressions, tone of voice, heart rate, and other information in real time.

[0532] Input: Camera video and audio data, heart rate monitor data

[0533] Data processing: Analyze the acquired data and identify the emotional state.

[0534] Output: Emotional state data

[0535] Specific operation: The device activates the camera and microphone to collect facial expressions, voice data, and heart rate monitor data.

[0536] Step 13: Analyze the sentiment data

[0537] Server: The server analyzes the emotion data sent from the emotion engine and evaluates the user's driving state.

[0538] Input: Emotional state data

[0539] Data processing: Using machine learning models to analyze emotion data and assess the driver's state.

[0540] Output: Driving condition evaluation data

[0541] Specific operation: The server uses a machine learning model to analyze emotion data and evaluate the driver's state.

[0542] Step 14: Adjusting Navigation Advice

[0543] Device: The device will adjust its navigation advice based on the emotion engine's analysis.

[0544] Input: Driving condition evaluation data

[0545] Data manipulation: Adjust navigation settings, change tips and music.

[0546] Output: Adjusted navigation advice

[0547] What it does: Based on the analysis received from the server, the device adjusts navigation settings, plays relaxing music, and provides gentle voice advice.

[0548] Step 15: Reconstructing the navigation plan

[0549] Server: The server reconstructs the navigation plan according to the user's emotional state and sends it to the device.

[0550] Input: Emotional state assessment data

[0551] Data transformation: Generate a new navigation plan.

[0552] Output: Reconstructed navigation plan

[0553] Specific operation: The server generates a recalculated navigation plan based on the results of the evaluation of the user's emotional state and sends it to the device.

[0554] Step 16: Offering advice and music

[0555] Device: The device provides users with navigation advice and music tailored to their needs based on the emotion engine, thereby supporting safe driving.

[0556] Input: Reconstructed navigation plan

[0557] Data processing: Provides voice advice and music based on the navigation plan.

[0558] Output: Audio advice and music provided

[0559] What it does: The device plays relaxing music selected from a playlist, emits gentle voice prompts through the speaker, and displays instructions on the screen.

[0560] (Application example 2)

[0561] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0562] In modern society, traffic congestion and accidents occur on a daily basis, and drivers need real-time traffic information. However, there have been no navigation systems that support multiple languages ​​or take into account the driver's emotional state. In addition, driver stress and fatigue can affect driving safety, and an effective system to mitigate these effects has been sought. Therefore, an object of the present invention is to provide a system that provides real-time traffic information and evaluates the driver's emotional state to improve driving safety and comfort.

[0563] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0564] In this invention, the server includes a means for collecting traffic information in real time, a means for analyzing the collected traffic information and calculating the optimal route, and a means for transmitting the analysis results to a user terminal. This allows the driver to receive optimal navigation based on the latest traffic conditions. Furthermore, by adding a means for collecting and recognizing the driver's emotional data, such as facial expressions, tone of voice, and heart rate, it is possible to provide relaxing music or gentle voice guidance when the driver is feeling tense or tired. Furthermore, by combining a means for recognizing road signs and pedestrians using a camera and notifying the user terminal of the recognition results, safety can be further improved.

[0565] "Real-time" refers to a method of processing data and providing results in accordance with the actual progress of time.

[0566] "Traffic information" refers to information relating to public transportation and vehicle traffic, such as road congestion, accidents, and construction work.

[0567] The term "route" refers to the route or path from a starting point to a destination.

[0568] A "user terminal" is an electronic device carried by a user, such as a smartphone, tablet, or in-vehicle terminal.

[0569] "Visually displaying" means showing data or information on a screen in the form of images or text.

[0570] A "camera" is a device for taking images or videos.

[0571] "Road signs" are signs installed to inform drivers and pedestrians of road safety and rules.

[0572] "Pedestrians" are people who walk on the road.

[0573] "Facial expressions" are facial muscle movements and expressions that indicate a person's emotional state.

[0574] "Tone of voice" refers to the timbre or tone of spoken language.

[0575] "Heart rate" is the number of times the heart beats within a certain period of time.

[0576] "Emotional data" means data collected to indicate the emotional state of a driver.

[0577] "Emotion recognition" means analyzing and evaluating a person's emotional state.

[0578] "Navigation advice" is advice to guide the driver to appropriate routes and important points to note.

[0579] "Music" refers to an art form that combines rhythm and melody of sounds.

[0580] "Voice guidance" refers to providing instructions and information audibly through a speaker.

[0581] "Means" refers to the tools, methods, and equipment used to achieve a specific purpose.

[0582] The present invention is a system that collects traffic information in real time and provides that information in multiple languages, and by combining it with an emotion engine, recognizes the driver's emotional state and supports safe and comfortable driving. An embodiment of the system will be described in detail below.

[0583] 1. Providing real-time traffic information

[0584] The server collects traffic data in real time via APIs from road traffic management centers, cloud services, etc. The collected data includes information on traffic congestion, accidents, road construction, etc., and consolidates this information and stores it in a database. The server analyzes the collected traffic information and calculates the optimal route. A machine learning algorithm is used for the calculation, generating the optimal route based on the user's current location and destination. The server sends the analysis results to the user's device in real time, allowing the user to receive navigation based on the latest traffic conditions. The user's device visually displays the traffic information received from the server on a map and provides the user with the optimal route. The map is displayed using OpenStreetMap or other map APIs.

[0585] 2. Multilingual support

[0586] The user device obtains the language information set by the user. This information is sent to the server, which applies different language settings to each user. The server translates traffic information and navigation advice into the user's language using a global API or other translation service. The user device then provides the translated information to the user in voice and text. Using speech synthesis technology, the navigation advice is output as voice in the user's language. At the same time, the translated information is displayed in text on the screen.

[0587] 3. Image Recognition Technology

[0588] The user device uses the smartphone camera to capture images of the surroundings in real time. This image is analyzed using an image recognition algorithm (e.g., OpenCV or TensorFlow Lite). The user device recognizes traffic signs and pedestrians from the captured image. The recognized information is used to assist the user's driving. The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time. The user device notifies the user of the advice from the server. For example, if the speed limit is exceeded or if a pedestrian is ahead, the user is provided with audio and visual alerts.

[0589] 4. Emotion Engine

[0590] The user device recognizes emotions from the user's facial expressions, tone of voice, heart rate, etc. using a smartphone or in-vehicle camera. This emotional data is collected and evaluated in real time. The server analyzes the emotional data sent from the emotion engine and evaluates the user's driving state. For example, it detects if the driver is nervous or tired. The user device adjusts navigation advice based on the emotion engine's analysis results. For example, if the user is nervous, it provides calming advice or relaxing music. The server reconstructs a navigation plan according to the user's emotional state and sends it to the device. This makes it possible to provide routes and advice optimal for the user's mental state. The user device provides the user with navigation advice and music adjusted by the emotion engine. This reduces the user's driving stress and supports safe driving.

[0591] 5. Specific Examples

[0592] Example 1: When a user sets a destination, the device sends current location and destination information to the server and receives the optimal route. The emotion engine analyzes the user's facial expressions and tone of voice to evaluate their state of mind to drive safely. If the user is nervous, the device plays relaxing music and provides gentle voice guidance.

[0593] Example 2: When a user is driving for a long time, the emotion engine detects fatigue and notifies the server. The server recalculates the route and suggests places where users can take a break. The device notifies the user by voice, saying, "There is a rest area ahead. Would you like to take a short break?"

[0594] Example 3: For users who are nervous, the device provides information on recognized traffic signs and pedestrians in real time. For example, if the camera recognizes a "Stop" sign, the device will give a voice command saying, "There is a stop sign. Please stop," to reassure the user.

[0595] The overall processing flow of this system involves the server and user terminal working together to provide traffic information and advice based on the driver's emotional state, thereby achieving safe and comfortable driving.

[0596] Example prompt: "Describe an application that recognizes a driver's emotions based on their facial expression, tone of voice, and heart rate to help them drive safely."

[0597] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0598] Step 1:

[0599] The server collects traffic data in real time via API from road traffic management centers, cloud services, etc. The input is traffic information (traffic congestion, accidents, road construction information, etc.) obtained from road traffic management centers and cloud services. This information is stored in a database and processed into a format that can be matched with the user's current location and destination information. The output is integrated traffic information data.

[0600] Step 2:

[0601] The server analyzes the collected traffic information using machine learning algorithms and calculates the optimal route. The inputs are the collected traffic information, the user's current location, and destination information. The server performs data calculations to generate the optimal route. The output is the optimized route information.

[0602] Step 3:

[0603] The server transmits the analysis result, the optimal route information, to the user terminal in real time. The input is the optimized route information, and the output is the navigation data transmitted to the user terminal.

[0604] Step 4:

[0605] The user device visually displays the optimal route information received from the server on a map. It uses OpenStreetMap and other map APIs to obtain map data and display the optimal route to the user. The input is the route information sent from the server and the map data obtained from the map API, and the output is a visual map display.

[0606] Step 5:

[0607] The user device uses the smartphone camera to capture images of the surroundings in real time and uses image recognition algorithms (OpenCV or TensorFlow Lite) to recognize traffic signs and pedestrians. The input is image data from the camera, which is analyzed using image recognition. The output is recognized traffic signs and pedestrian information.

[0608] Step 6:

[0609] The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. The input is traffic signs and pedestrian information, and by processing the data, it generates information that the driver should pay attention to. The output is a warning or advice sent to the user's device.

[0610] Step 7:

[0611] The user terminal notifies the user of warnings and advice from the server in real time. The input is the warnings and advice sent from the server and provided to the user as audio and visual alerts. The output is the notification of the warnings and advice given to the user.

[0612] Step 8:

[0613] The user device collects emotion data from the user's facial expressions, tone of voice, heart rate, etc. using a smartphone or in-car camera, and analyzes it with an emotion engine. The input is the user's facial expressions, tone of voice, and heart rate data, which are analyzed by the emotion engine. The output is the recognized emotional state.

[0614] Step 9:

[0615] The server analyzes the emotion data sent from the emotion engine and evaluates the user's driving state. The input is the emotional state data, and the server performs data calculations to evaluate the user's driving state. The output is navigation advice and music suggestions based on the emotional state to the user's device.

[0616] Step 10:

[0617] The user device provides navigation advice and relaxing music based on the analysis results of the emotion engine. The input is navigation advice and music data based on the emotional state, which are provided to the user visually and audibly. The output is support for safe and comfortable driving for the user.

[0618] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0619] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0620] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0621] [Second embodiment]

[0622] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0623] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0624] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0625] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0626] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0627] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0628] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0629] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0630] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0631] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0632] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0633] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0634] The present invention provides a system that supports safe and intuitive driving by collecting traffic information in real time, providing that information in multiple languages, and detecting traffic signs and pedestrians using image recognition technology. An embodiment of the system will be described in detail below.

[0635] 1. Providing real-time traffic information

[0636] Server: The server collects real-time traffic information from road traffic management centers, cloud services, etc. The collected data includes information on traffic congestion, accidents, road construction, etc., and integrates this information and stores it in a database.

[0637] Server: The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms to generate the optimal route based on the user's current location and destination.

[0638] Server: The server transmits the analysis results to the user's device in real time, allowing the user to receive navigation based on the latest traffic conditions.

[0639] Terminal: The terminal visually displays the traffic information received from the server on a map and provides the user with the optimal route. The map is displayed using OpenStreetMap or other map APIs.

[0640] 2. Multilingual support

[0641] Device: The device retrieves the user's language preference, which is then sent to the server, allowing different language settings to be applied for each user.

[0642] Server: The server translates traffic information and navigation advice into the user's language using a global API or other translation services.

[0643] Device: The device provides the translated information to the user in both voice and text. It uses speech synthesis technology to speak navigation advice in the user's language and simultaneously displays the translated text on the screen.

[0644] 3. Image Recognition Technology

[0645] Device: The device uses the smartphone camera to capture real-time images of the surroundings, which are then analyzed using image recognition algorithms (e.g., OpenCV or TensorFlow Lite).

[0646] Device: The device recognizes traffic signs and pedestrians from the captured video. The recognized information is used to assist the user in driving.

[0647] Server: The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time.

[0648] Terminal: The terminal notifies the user of advice from the server, for example providing audio and visual alerts to the user if they exceed the speed limit or if there are pedestrians ahead.

[0649] 4. Specific Examples

[0650] Example 1: The user sets a destination. The device sends current location and destination information to the server and receives the optimal route. The device displays the route on a map and provides route guidance based on traffic information updated in real time. If a traffic jam occurs along the way, the server immediately calculates a new route and sends updated information to the device. As a result, the user can reach the destination smoothly.

[0651] Example 2: When a user is driving in a foreign country, the device provides navigation advice in the user's language. The server translates collected traffic information and recognition results into the user's native language and provides them via voice and text through the device. This allows the user to drive with confidence even in a different language environment.

[0652] Example 3: When the device's camera recognizes a road sign while driving, the information is sent to the server and the necessary warning is generated. For example, if the device recognizes a "Stop" sign, the server analyzes it and sends a warning to the device. The device will notify the user by voice, "There is a stop sign ahead," to help the user take appropriate action.

[0653] As described above, this system provides users with the latest traffic information in real time, supports the driving environment with multilingual support, and uses image recognition technology to support safe driving, thereby reducing driving anxiety and providing a comfortable driving experience.

[0654] The processing flow will be explained below.

[0655] Program processing flow and specific operations

[0656] 1. Providing real-time traffic information

[0657] Step 1: Collect traffic information

[0658] Server: "The server collects traffic data in real time via API from road traffic management centers and cloud services."

[0659] Specific operation: Data such as traffic congestion, accidents, and road construction information is obtained using an API and stored in a database.

[0660] Step 2: Traffic information integration and analysis

[0661] Server: "The server integrates the collected traffic data into a database and uses machine learning algorithms to calculate the optimal route."

[0662] What it does: Analyzes traffic information data and calculates the optimal route to the user's destination.

[0663] Step 3: Distributing traffic information

[0664] Server: "The server sends the analyzed route information to the user terminal in real time."

[0665] Specific operation: The calculated optimal route is sent to the user's device via an HTTP request.

[0666] Step 4: View information

[0667] Terminal: "The terminal displays real-time traffic information received from the server on a map."

[0668] Specific behavior: Uses OpenStreetMap and other map APIs to visually display received route information on a map.

[0669] 2. Multilingual support

[0670] Step 1: Get the language settings

[0671] Device: "The device acquires the language information set by the user and sends it to the server."

[0672] Specific behavior: Get the user's language preference from the app's settings menu and send it to the server.

[0673] Step 2: Perform the translation

[0674] Server: "The server translates traffic information and navigation advice into the user's language."

[0675] What it does: Uses a translation API to translate collected traffic information and navigation advice into the user's language of choice.

[0676] Step 3: Providing multilingual guidance

[0677] Device: "The device provides multilingual, translated information to the user via voice and text."

[0678] Specific operation: Using speech synthesis technology, translated navigation information is output as voice and simultaneously displayed as text on the screen.

[0679] 3. Image Recognition Technology

[0680] Step 1: Acquire camera footage

[0681] Device: "The device captures video from the smartphone camera and processes it for image recognition."

[0682] Specific operation: Uses the camera API to capture real-time video and store it in memory.

[0683] Step 2: Perform image recognition

[0684] Device: "The device applies image recognition algorithms to the captured video to detect traffic signs and pedestrians."

[0685] Specific operation: Using OpenCV and TensorFlow Lite, it analyzes signs and pedestrians in video and detects specific features.

[0686] Step 3: Sending the recognition results

[0687] Device: "The device sends the recognition results to the server."

[0688] Specific operation: Organizes the recognized information and sends it to the server via API.

[0689] Step 4: Generate feedback

[0690] Server: "The server analyzes the recognition results and generates advice and warnings for safe driving."

[0691] What it does: Evaluates the received recognition data and generates the necessary feedback.

[0692] Step 5: Deliver feedback to users

[0693] Terminal: "The terminal notifies the user of advice and warnings from the server."

[0694] Specific behavior: Displays a warning message on the screen and issues an audio alert.

[0695] 4. User Interface Operation and Feedback

[0696] Step 1: Set your destination

[0697] User: "The user enters and sets a destination in the application."

[0698] What it does: Select a destination on the map or enter text in the search bar to specify a destination.

[0699] Step 2: Start route guidance

[0700] Terminal: "The terminal starts navigation based on the route information received from the server."

[0701] What it does: Highlights the route on the map and starts voice guidance.

[0702] Step 3: Receive real-time feedback

[0703] Server: "The server monitors the user's real-time movements and updates the route as needed."

[0704] Specific operation: Constantly monitors the user's current location based on GPS information and recalculates the route according to changes in traffic conditions.

[0705] Step 4: Real-time notifications

[0706] Terminal: "The terminal notifies the user of updates from the server in real time."

[0707] Specific behavior: When a new route is provided, the user is immediately notified by map display and voice.

[0708] This enables the entire system to provide users with real-time, up-to-date and reliable traffic information and safe driving support.

[0709] Example 1

[0710] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0711] This invention relates to a system that efficiently collects and analyzes traffic information updated in real time and provides users with optimal routes. In particular, it aims to support safe and intuitive driving by recognizing the surrounding environment, such as traffic signs and pedestrians, from camera images and notifying the user of this information in multiple languages. Conventional systems have had issues with driver convenience and safety due to slow traffic information updates and insufficient multilingual support and real-time image recognition.

[0712] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0713] In this invention, the server includes a means for collecting traffic information in real time, a means for analyzing the collected traffic information and calculating an optimal route using a machine learning algorithm, and a means for transmitting the analysis results to a user terminal. This enables efficient and prompt provision of an optimal route based on the latest traffic information. The user terminal also includes a means for visually displaying the received traffic information on a map, a means for recognizing road signs and pedestrians using a camera, a means for transmitting the recognition results to the server and notifying the user terminal of analyzed warnings and advice, and a means for providing navigation advice using speech synthesis technology that supports multiple languages. This allows users to receive safe and intuitive driving support in multiple languages ​​in real time.

[0714] "Real-time" refers to processing and providing information instantly the moment it is generated.

[0715] "Traffic information" refers to information relating to traffic conditions on roads, including data on congestion, accidents, road construction, and the like.

[0716] A "machine learning algorithm" is a computational method that automatically learns patterns and rules by analyzing large amounts of data, and makes predictions and classifications.

[0717] "Optimal route" refers to the shortest and most efficient route for a user to reach a destination.

[0718] A "user terminal" is a device operated by a user, including a smartphone, tablet, car navigation system, etc.

[0719] "Visually displaying on a map" refers to visually displaying traffic and route information using a map application or navigation system.

[0720] "Camera" means a device for capturing video or images.

[0721] "Road sign" means a sign placed on a road to provide direction or information to motorists.

[0722] "Pedestrian" refers to a person who moves on foot on a road.

[0723] "Recognizing" refers to detecting and identifying specific objects (such as road signs or pedestrians) from the image captured by the camera.

[0724] "Analyze" refers to the detailed analysis of collected data and recognition results to derive meaningful information and predictions.

[0725] "Warning" means providing information to draw attention to a particular situation.

[0726] "Advice" means providing instructions or suggestions for specific action.

[0727] "Speech synthesis technology" is a technology that generates and plays back voices that resemble human voices from text data.

[0728] "Multilingual" means supporting multiple different languages.

[0729] "Translate" refers to converting text written in one language into another language.

[0730] The present invention provides a system that supports safe and intuitive driving by collecting traffic information in real time, providing that information in multiple languages, and detecting traffic signs and pedestrians using image recognition technology. An embodiment of the system will be described in detail below.

[0731] 1. Providing real-time traffic information

[0732] server:

[0733] The server continuously obtains real-time traffic information from the road traffic management center API or cloud services (e.g., general traffic information API), including information on traffic congestion, accidents, road construction, etc. The server stores this information in a database such as MongoDB and generates an integrated dataset.

[0734] server:

[0735] The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms (e.g., Scikit-learn or TensorFlow), and sends the results of this calculation to the user's device in real time.

[0736] Device:

[0737] The device visually displays the traffic information received from the server on a map and provides the user with the optimal route, using external services such as OpenStreetMap and Google Maps API.

[0738] 2. Multilingual navigation available

[0739] Device:

[0740] The terminal obtains the user's language setting and sends that information to the server.

[0741] server:

[0742] The server uses the Global Translate API (e.g., Google Translate API) to translate traffic information and navigation instructions into the user's preferred language.

[0743] Device:

[0744] The device provides the translated information to the user in real time using speech synthesis technology (e.g., Amazon Polly), and also displays the corresponding text information on a map.

[0745] 3. Traffic Sign and Pedestrian Detection Using Image Recognition Technology

[0746] Device:

[0747] The device uses its built-in camera to capture real-time images of its surroundings, which are then analyzed by image recognition algorithms (e.g., OpenCV or TensorFlow Lite) running on the device.

[0748] Device:

[0749] The device recognizes traffic signs and pedestrians in the video and sends the recognition results to the server in real time.

[0750] server:

[0751] The server analyzes the received recognition data and notifies the user's device of warnings and advice necessary for driving.

[0752] Device:

[0753] The device provides the user with audio and visual warnings and advice from the server, such as "There is a pedestrian ahead" or "You are exceeding the speed limit."

[0754] Specific examples

[0755] Example 1:

[0756] When a user inputs their current location and destination, the device sends the information to the server, which calculates the optimal route and sends it back to the device. The device then displays the route on a map and provides route guidance based on traffic information updated in real time. If traffic congestion occurs, the server calculates a new route and sends updated information to the device, allowing the user to reach their destination smoothly.

[0757] Example 2:

[0758] When a user is driving in a foreign country, the device will provide navigation advice in the user's language of choice. The server will translate traffic information and recognition results into the user's native language and provide them via voice and text via the device, allowing users to drive with confidence in different language environments.

[0759] Example 3:

[0760] When the device's camera recognizes a road sign while driving, the information is sent to the server, which generates the necessary warning. For example, if a "Stop" sign is recognized, the server analyzes it and sends a warning to the device. The device then notifies the user by voice, saying "There is a stop sign ahead," helping the user to take appropriate action.

[0761] Prompt Sentence Examples

[0762] 1. Give an example of a program that calculates the optimal route based on traffic information when a user inputs their current location and destination.

[0763] 2. Describe the process flow of a system that provides multilingual voice and text navigation based on real-time traffic data.

[0764] 3. Explain the process of a system that uses a smartphone camera and image recognition algorithms to detect traffic signs and pedestrians and alert the driver.

[0765] This invention allows users to utilize the latest traffic information in real time and easily implements a system that supports the driving environment in multiple languages, thereby enabling safe and efficient driving, reducing driving anxiety, and realizing a comfortable driving experience.

[0766] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0767] Step 1:

[0768] Real-time traffic information collection

[0769] Server: The server obtains real-time traffic information from the Road Traffic Management Center API and cloud services.

[0770] Input: Traffic information data received from the Road Traffic Management Center API and cloud services.

[0771] Data processing: The acquired traffic information is converted into JSON format and stored in a database.

[0772] Output: Updated traffic information dataset.

[0773] What it does: The server periodically sends API requests to retrieve new traffic information and stores it in MongoDB.

[0774] Step 2:

[0775] Traffic information analysis and optimal route calculation

[0776] Server: The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms (e.g., Scikit-learn, TensorFlow).

[0777] Input: Traffic information data stored in a database.

[0778] Data calculations: Using machine learning algorithms to calculate the optimal route based on the user's current location and destination.

[0779] Output: Optimal route information provided to the user.

[0780] Specific operation: The server applies Dijkstra's algorithm based on map data and traffic information to calculate the shortest route.

[0781] Step 3:

[0782] Sending optimal route information

[0783] Server: The server transmits the calculated optimal route information to the user's device in real time.

[0784] Input: Calculated optimal route information.

[0785] Data calculation: If necessary, convert the route information into JSON format so that it can be sent.

[0786] Output: Optimal route information sent to the user terminal.

[0787] Specific operation: The server sends JSON format data to the user terminal via the HTTP protocol.

[0788] Step 4:

[0789] Displaying traffic and route information

[0790] Terminal: The terminal visually displays the traffic and route information received from the server on a map.

[0791] Input: Traffic and route information received from the server.

[0792] Data processing: Data conversion to overlay information on a map.

[0793] Output: Map and route information visually displayed to the user.

[0794] Specific operation: The device displays a map using the Google Maps API and overlays the received route information on it.

[0795] Step 5:

[0796] Multilingual navigation provided

[0797] Device: The device obtains the user's language preference and sends it to the server.

[0798] Input: User's language preference information.

[0799] Output: Language preference data sent to the server.

[0800] Specific behavior: The device obtains the user's selected language from the language settings screen.

[0801] Server: The server translates traffic information and navigation instructions into the user's preferred language.

[0802] Input: Traffic information, navigation instructions, and user language preference data.

[0803] Data Calculation: Translate data into a specified language using the Global Translate API.

[0804] Output: Translated traffic and navigation instructions.

[0805] Specific operation: The server calls the Google Translate API to translate the data and sends the results to the device.

[0806] Terminal: The terminal synthesizes the translated information and provides it to the user.

[0807] Input: Translated traffic and navigation instructions.

[0808] Data processing: Generate voice data using a speech synthesis application (e.g., Amazon Polly).

[0809] Output: Audio and text information provided to the user.

[0810] Specific operation: The device uses Amazon Polly to play a voice message saying "Go straight to your destination" and also displays the text on the map.

[0811] Step 6:

[0812] Traffic sign and pedestrian detection using image recognition

[0813] Device: The device uses its built-in camera to capture real-time images of its surroundings.

[0814] Input: Video data captured by the camera.

[0815] Output: Video data that is passed to the image recognition algorithm.

[0816] Specific operation: The device starts the camera and acquires streaming video data.

[0817] Terminal: The terminal analyzes the video data using image recognition algorithms (e.g., OpenCV or TensorFlow Lite).

[0818] Input: Video data captured by the camera.

[0819] Data calculations: Image recognition algorithms identify traffic signs and pedestrians.

[0820] Output: Information on recognized traffic signs and pedestrians.

[0821] Specific operation: The device uses OpenCV to recognize stop signs and pedestrians.

[0822] Device: Sends the recognition results to the server.

[0823] Input: The recognized information.

[0824] Output: Recognition result data sent to the server.

[0825] Specific operation: The device converts the recognized data into JSON format and sends it to the server.

[0826] Server: The server analyzes the received recognition data and generates any necessary warnings or advice.

[0827] Input: Recognition data sent from the device.

[0828] Data calculations: Analyze perception data to generate warnings and advice.

[0829] Output: Warnings and advice sent to the user's terminal.

[0830] Specific operation: Based on the analysis results, the server generates a warning such as "There is a pedestrian ahead" and sends it to the device.

[0831] Terminal: The terminal notifies the user of warnings and advice from the server by voice and display.

[0832] Input: Any warnings or advice received from the server.

[0833] Output: The audio and visual information provided to the user.

[0834] Specific operation: The device will announce "There is a pedestrian ahead" and display an icon on the screen.

[0835] (Application example 1)

[0836] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0837] Conventional navigation systems only provide traffic information and are limited in their ability to respond to changes in traffic conditions in real time. Furthermore, they lack driving assistance in multilingual environments and the ability to recognize road signs and pedestrians in real time, making them inadequate for supporting safe driving. There is a need to solve these issues and provide safer and more efficient driving assistance.

[0838] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0839] In this invention, the server includes means for collecting traffic information in real time, means for analyzing the collected traffic information and calculating the optimal route, and means for transmitting the analysis results to the user terminal, thereby enabling the server to respond to changes in traffic conditions in real time and provide the optimal route.

[0840] Furthermore, this invention provides safe and intuitive driving assistance by incorporating a means for recognizing road signs and pedestrians using a camera, a means for notifying a user terminal of the recognition results, a means for providing audio advice based on the recognition results, and a means having a multilingual translation function based on local languages.

[0841] "Means of collecting traffic information in real time" refers to the function of instantly obtaining the latest traffic information, such as information on traffic congestion, accidents, and road construction, from traffic management centers and cloud services.

[0842] "Means of analyzing collected traffic information and calculating the optimal route" refers to a function that uses a machine learning algorithm to generate the optimal route taking into account traffic conditions based on the user's current location and destination.

[0843] The "means for transmitting the analysis results to the user terminal" is a function for transferring optimal route information and traffic conditions calculated by the server to the user terminal in real time.

[0844] "Means for visually displaying received traffic information on a user terminal" refers to a function that graphically displays maps and route guidance on the user's terminal, allowing the user to intuitively understand traffic information.

[0845] "Means of recognizing road signs and pedestrians using cameras" refers to a function that analyzes camera footage in real time and detects traffic signs and pedestrians with high accuracy.

[0846] "Means for notifying the user terminal of the recognition results" is a function that immediately transmits information about detected signs and pedestrians to the user terminal and notifies the driver of the situation.

[0847] "Means for providing voice advice based on the recognition results" is a function that provides appropriate driving advice by voice based on the recognized information, supporting the driver in safe driving.

[0848] "Means having a multilingual translation function based on local languages" refers to a function that automatically translates traffic information and navigation advice into different languages ​​according to user settings.

[0849] "Means for translating and providing traffic information and navigation advice in multiple languages" refers to a function that automatically translates and provides collected and analyzed traffic information and driving advice based on the user's language settings.

[0850] "Means for providing translated information in voice and text format based on the user's language settings" refers to a function that provides translated traffic information and route guidance in voice and text format according to the language settings registered by the user.

[0851] This invention will be explained using the smartphone application "SmartDrive Assist" installed in autonomous vehicles as a concrete example. This application collects traffic information in real time, provides it in multiple languages, and utilizes image recognition technology to monitor the external environment and support safe driving.

[0852] First, the server collects traffic information in real time from road traffic management centers and cloud services. This traffic information includes information on traffic congestion, accidents, road construction, and more, and stores it in a database. Next, the collected traffic information is analyzed and the optimal route is calculated using a machine learning algorithm. This calculation uses map APIs such as Google Maps API and OpenStreetMap. Finally, the analysis results are sent to the user's device in real time.

[0853] The device visually displays traffic information received from the server. Maps are displayed using OpenStreetMap and other map APIs. In addition, the device uses the smartphone camera to capture real-time images of the surrounding area. This image is analyzed using image recognition algorithms (e.g., OpenCV or TensorFlow Lite) to recognize road signs and pedestrians. The recognized information is used to assist the user's driving.

[0854] The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time. The device notifies the user of the advice from the server. For example, if the device recognizes a "Stop" sign, the server analyzes it and sends a warning to the device. The device then notifies the user by voice, saying "There is a stop sign ahead," helping the user to take appropriate action.

[0855] The device also obtains the user's language preference and sends this information to the server. The server translates traffic information and navigation advice into the user's language using a global API or other translation services. The device then provides the translated information to the user in voice and text. It uses speech synthesis technology to output navigation advice in the user's language preference and displays the translated information in text on the screen.

[0856] As a concrete example, consider a scenario in which a user is traveling by car from Tokyo to Los Angeles. First, the user launches the "SmartDrive Assist" app on their smartphone and sets their destination (Los Angeles). The app retrieves real-time traffic information from the server and provides voice advice translated into multiple languages. It uses the smartphone's camera to detect road signs and pedestrians and provides appropriate voice warnings to the user.

[0857] Example prompt sentence:

[0858] Consider a scenario where a user is using the "SmartDrive Assist" app to drive from Tokyo to Los Angeles. The app collects real-time traffic information, provides it in multiple languages, and detects traffic signs and pedestrians through camera footage.

[0859] In this way, the present invention provides users with the latest traffic information in real time, supports the driving environment with multilingual support, and uses image recognition technology to support safe driving, thereby reducing driving anxiety and providing a comfortable driving experience.

[0860] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0861] Step 1:

[0862] The server collects real-time traffic information from road traffic management centers and cloud services. It obtains information on traffic congestion, accidents, road construction, etc. through the road traffic management center and cloud service APIs, and receives this data in JSON format. This allows it to accumulate the latest traffic situation data.

[0863] Input: Traffic information from the traffic management center and cloud service API

[0864] Data processing: Receive traffic information in JSON format

[0865] Output: Real-time traffic information database

[0866] Step 2:

[0867] The server analyzes the collected traffic information and calculates the optimal route. It uses machine learning algorithms to generate the optimal route based on the user's current location and destination. The algorithms used include linear regression models and neural networks. Map APIs such as Google Maps API and OpenStreetMap are also used for route calculation.

[0868] Input: Collected traffic information, user's current location, user's destination

[0869] Data processing: Analyzing traffic information and calculating optimal routes using machine learning algorithms

[0870] Output: Optimal route information

[0871] Step 3:

[0872] The server transmits the analysis results to the user's device in real time, using communication protocols such as HTTP requests to transmit the generated optimal route information and traffic condition data to the user's device.

[0873] Input: Optimal route information, traffic condition data

[0874] Data processing: Sending data via HTTP request

[0875] Output: Optimal route information sent to the user terminal

[0876] Step 4:

[0877] The device visually displays the traffic information received from the server. It uses a map API (e.g., OpenStreetMap or Google Maps) to display the received traffic information and the optimal route on a map. This operation uses the display function of the map API.

[0878] Input: Optimal route information and traffic condition data sent from the server

[0879] Data processing: Traffic information and route mapping

[0880] Output: Visual map display

[0881] Step 5:

[0882] The device uses the smartphone camera to capture images of the surroundings in real time. When the user activates the device's camera, real-time images are captured.

[0883] Input: Real-time camera image data

[0884] Data processing: Video capture and real-time acquisition

[0885] Output: Real-time camera footage

[0886] Step 6:

[0887] The device recognizes traffic signs and pedestrians using the captured video. It uses an image recognition algorithm (e.g., OpenCV or TensorFlow Lite) to analyze the captured video and extract information about detected traffic signs and pedestrians.

[0888] Input: Captured camera footage

[0889] Data processing: Traffic sign and pedestrian detection using image recognition algorithms

[0890] Output: Detected traffic signs and pedestrian information

[0891] Step 7:

[0892] The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. For example, if a "Stop" sign is detected, the server analyzes it and generates appropriate driving advice.

[0893] Input: Traffic sign and pedestrian information sent from the device

[0894] Data processing: Analyzing perception data and generating driving advice

[0895] Output: Driving advice

[0896] Step 8:

[0897] The terminal notifies the user of the advice from the server by generating appropriate voice advice and notifying the user using speech synthesis technology (e.g., gTTS), and also by displaying it in text format on the screen.

[0898] Input: Driving advice sent from the server

[0899] Data processing: speech synthesis and text display

[0900] Output: Voice and text driving advice notifications

[0901] Step 9:

[0902] The server obtains the user's language preference and translates traffic information and navigation advice. It uses a translation service (e.g., Google Translate API) to translate the information based on the user's language preference.

[0903] Input: User-defined language information, traffic information, navigation advice

[0904] Data processing: Multilingual translation using translation services

[0905] Output: Translated traffic information and navigation advice

[0906] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0907] The present invention is a system that collects traffic information in real time and provides that information in multiple languages, and by combining it with an emotion engine, recognizes the driver's emotional state and supports safe and comfortable driving. An embodiment of the system will be described in detail below.

[0908] 1. Providing real-time traffic information

[0909] Server: The server collects traffic data in real time via API from road traffic management centers, cloud services, etc. The collected data includes information on traffic congestion, accidents, road construction, etc., and integrates this information and stores it in a database.

[0910] Server: The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms to generate the optimal route based on the user's current location and destination.

[0911] Server: The server transmits the analysis results to the user's device in real time, allowing the user to receive navigation based on the latest traffic conditions.

[0912] Terminal: The terminal visually displays the traffic information received from the server on a map and provides the user with the optimal route. The map is displayed using OpenStreetMap or other map APIs.

[0913] 2. Multilingual support

[0914] Device: The device retrieves the user's language preference, which is then sent to the server, allowing different language settings to be applied for each user.

[0915] Server: The server translates traffic information and navigation advice into the user's language using a global API or other translation services.

[0916] Device: The device provides the translated information to the user in both voice and text. It uses speech synthesis technology to speak navigation advice in the user's language and simultaneously displays the translated text on the screen.

[0917] 3. Image Recognition Technology

[0918] Device: The device uses the smartphone camera to capture real-time images of the surroundings, which are then analyzed using image recognition algorithms (e.g., OpenCV or TensorFlow Lite).

[0919] Device: The device recognizes traffic signs and pedestrians from the captured video. The recognized information is used to assist the user in driving.

[0920] Server: The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time.

[0921] Terminal: The terminal notifies the user of advice from the server, for example providing audio and visual alerts to the user if they exceed the speed limit or if there are pedestrians ahead.

[0922] 4. Emotion Engine

[0923] Device: The device uses a smartphone or in-car camera to recognize emotions from the user's facial expressions, tone of voice, heart rate, etc. This emotional data is collected and evaluated in real time.

[0924] Server: The server analyzes the emotion data sent from the emotion engine and evaluates the user's driving state, for example, to detect if the driver is nervous or tired.

[0925] Device: The device will adjust navigation advice based on the emotion engine's analysis. For example, if the user is nervous, it will provide calming advice or relaxing music.

[0926] Server: The server reconstructs a navigation plan according to the user's emotional state and sends it to the device, thereby providing the optimal route and advice for the user's mental state.

[0927] Device: The device provides users with navigation advice and music tailored by the emotion engine, thereby reducing the user's driving stress and supporting safe driving.

[0928] 5. Specific Examples

[0929] Example 1: When a user sets a destination, the device sends current location and destination information to the server and receives the optimal route. The emotion engine analyzes the user's facial expressions and tone of voice to evaluate their state of mind to drive safely. If the user is nervous, the device plays relaxing music and provides gentle voice guidance.

[0930] Example 2: When a user is driving for a long time, the emotion engine detects fatigue and notifies the server. The server recalculates the route and suggests places where users can take a break. The device notifies the user by voice, saying, "There is a rest area ahead. Would you like to take a short break?"

[0931] Example 3: For users who are nervous, the device provides information on recognized traffic signs and pedestrians in real time. For example, if the camera recognizes a "Stop" sign, the device will give a voice command saying, "There is a stop sign. Please stop," giving the user a sense of security.

[0932] As described above, this system combines real-time traffic information, multilingual support, image recognition technology, and an emotion engine to provide users with a safe and comfortable driving experience. By appropriately assessing the user's emotional state and adjusting navigation advice based on that assessment, the system achieves a less stressful driving experience.

[0933] The processing flow will be explained below.

[0934] Program processing flow and specific operations (including emotion engine)

[0935] 1. Providing real-time traffic information

[0936] Step 1: Collecting traffic information

[0937] Server: "The server collects traffic data in real time via API from road traffic management centers and cloud services."

[0938] Specific operation: Data such as traffic congestion, accidents, and road construction information is obtained using an API and stored in a database.

[0939] Step 2: Traffic information integration and analysis

[0940] Server: "The server integrates the collected traffic data into a database and uses machine learning algorithms to calculate the optimal route."

[0941] What it does: Analyzes traffic information data and calculates the optimal route to the user's destination.

[0942] Step 3: Distributing traffic information

[0943] Server: "The server sends the analyzed route information to the user terminal in real time."

[0944] Specific operation: The calculated optimal route is sent to the user's device via an HTTP request.

[0945] Step 4: Viewing information

[0946] Terminal: "The terminal displays real-time traffic information received from the server on a map."

[0947] Specific behavior: Uses OpenStreetMap and other map APIs to visually display received route information on a map.

[0948] 2. Multilingual support

[0949] Step 1: Get the language settings

[0950] Device: "The device acquires the language information set by the user and sends it to the server."

[0951] Specific behavior: Get the user's language preference from the app's settings menu and send it to the server.

[0952] Step 2: Performing the translation

[0953] Server: "The server translates traffic information and navigation advice into the user's language."

[0954] What it does: Uses a translation API to translate collected traffic information and navigation advice into the user's language of choice.

[0955] Step 3: Providing multilingual guidance

[0956] Device: "The device provides multilingual, translated information to the user via voice and text."

[0957] Specific operation: Using speech synthesis technology, translated navigation information is output as voice and simultaneously displayed as text on the screen.

[0958] 3. Image Recognition Technology

[0959] Step 1: Acquire camera footage

[0960] Device: "The device captures video from the smartphone camera and processes it for image recognition."

[0961] Specific operation: Uses the camera API to capture real-time video and store it in memory.

[0962] Step 2: Perform image recognition

[0963] Device: "The device applies image recognition algorithms to the captured video to detect traffic signs and pedestrians."

[0964] Specific operation: Using OpenCV and TensorFlow Lite, it analyzes signs and pedestrians in video and detects specific features.

[0965] Step 3: Sending the recognition results

[0966] Device: "The device sends the recognition results to the server."

[0967] Specific operation: Organizes the recognized information and sends it to the server via API.

[0968] Step 4: Generate feedback

[0969] Server: "The server analyzes the recognition results and generates advice and warnings for safe driving."

[0970] What it does: Evaluates the received recognition data and generates the necessary feedback.

[0971] Step 5: Deliver feedback to users

[0972] Terminal: "The terminal notifies the user of advice and warnings from the server."

[0973] Specific behavior: Displays a warning message on the screen and issues an audio alert.

[0974] 4. Emotion Engine

[0975] Step 1: Collecting emotion data

[0976] Device: "The device uses a smartphone or in-car camera to capture emotional data such as the user's facial expressions, tone of voice, and heart rate."

[0977] What it does: It uses facial recognition APIs, voice analysis APIs, and heart rate sensors to capture emotion-related data and evaluate it in real time.

[0978] Step 2: Sending emotion data

[0979] Device: "The device sends the collected emotion data to the server."

[0980] Specific operation: Organize emotion data and send it to the server via API.

[0981] Step 3: Analyze the sentiment data

[0982] Server: "The server uses the emotion engine to analyze the received emotion data and evaluate the user's driving state."

[0983] What it does: It uses analytics algorithms to identify the user's emotional state, such as whether they are nervous or tired.

[0984] Step 4: Adjust navigation based on emotions

[0985] Device: "The device will adjust its navigation advice based on the emotion engine's analysis."

[0986] Specific actions: For example, if the user is nervous, give gentle advice and play relaxing music.

[0987] Step 5: Emotionally recalculated route

[0988] Server: "The server reconstructs the optimal navigation plan based on the user's emotional state and sends it to the device."

[0989] What it does: Calculates a new route and provides the best route for the user's current mental state.

[0990] Step 6: Provide emotion-based feedback

[0991] Device: "The device provides users with navigation advice and music tailored by the emotion engine."

[0992] Specific behavior: Provides optimal feedback to the user through voice notifications and on-screen displays, supporting stress-free driving.

[0993] In this way, this system combines real-time traffic information, multilingual support, image recognition technology, and an emotion engine to provide users with a safe and comfortable driving experience. By properly recognizing the user's emotional state and adjusting navigation advice accordingly, it further reduces driving stress and supports safe driving.

[0994] Example 2

[0995] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0996] In today's traffic environment, drivers are required to select the optimal route based on traffic information updated in real time. However, systems that do not properly reflect traffic information, lack multilingual support, or do not take the driver's emotional state into account make it difficult to support safe and comfortable driving. This raises concerns about increased traffic accidents and driving stress. Therefore, the present invention aims to solve these issues and provide drivers with a safe and comfortable driving experience.

[0997] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0998] In this invention, the server includes means for collecting traffic information in real time from road traffic management agencies and cloud services, means for analyzing the collected traffic information using a machine learning algorithm to calculate an optimal route, and means for transmitting the analysis results to a user terminal. This allows the user to receive optimal navigation based on the latest traffic conditions. It also includes means for recognizing road signs and pedestrians using a camera and means for notifying the driver of warnings and advice based on the recognition results. This allows the user to receive appropriate warnings and advice in real time while driving. Furthermore, the system also includes means for collecting the user's facial expressions, tone of voice, and heart rate to analyze the user's emotional state, and means for adjusting navigation advice based on the emotional state, thereby supporting safe and comfortable driving according to the driver's emotional state.

[0999] "Real-time traffic information" refers to information that is obtained immediately after a traffic situation occurs, and includes current road conditions, congestion, accident information, road construction information, and the like.

[1000] A "machine learning algorithm" is a computational method that learns patterns from data and makes future decisions and predictions based on the results of that learning.

[1001] The "optimal route" is the route determined to be the most efficient from the user's starting point to the destination, taking into consideration factors such as time, distance, and traffic conditions.

[1002] A "user terminal" is an information terminal used by a driver, and includes a smartphone, tablet, in-vehicle display, etc.

[1003] "Visually displaying" means presenting information in a graphical format on a display or screen, including displays using maps, icons, text, etc.

[1004] A "camera" is a device used to capture images, such as a camera built into a smartphone or a camera attached to a vehicle.

[1005] "Road signs" are signs that indicate traffic rules and road conditions, and include stop signs, speed limit signs, no entry signs, and the like.

[1006] "Pedestrian" refers to a person crossing a road or walking on the side of the road.

[1007] "Emotional state" refers to the driver's psychological and emotional state, including states such as tension, fatigue, and relief.

[1008] "Navigation advice" is route guidance information provided to a user, including instructions in audio or text format.

[1009] "Warnings and advice" refers to alerts and instructions for actions to the user, including audible and visual alerts to warn of danger, and advice to encourage improved driving.

[1010] The present invention provides a system that collects traffic information in real time, provides the information in multiple languages, and recognizes the emotional state of the driver to support safe and comfortable driving. Detailed embodiments of the system are described below.

[1011] 1. Providing real-time traffic information

[1012] Server: The server collects real-time traffic data from road traffic management agencies and cloud services via API. The collected data includes information on traffic congestion, accidents, road construction, etc., and consolidates this information and stores it in a database.

[1013] Example: A server sends an API request every minute to get new traffic data and automatically stores the data in a database.

[1014] Server: The server analyzes the collected traffic information using machine learning algorithms to calculate the optimal route, using machine learning libraries such as TensorFlow and Scikit-learn.

[1015] Example: The server takes the user's current location and destination as input, runs an algorithm to calculate the optimal route, and stores the results in a database.

[1016] Server: The server sends the calculation results to the user's device in real time.

[1017] Example: The server sends a push notification to the user's device, providing a data packet containing optimal route information.

[1018] Terminal: The terminal visually displays the traffic information received from the server on a map and suggests the optimal route to the user. The map is displayed using OpenStreetMap (OSM) or Google Maps API.

[1019] Example: The device uses the Google Maps API to display a map and highlight the best route.

[1020] 2. Multilingual support

[1021] Terminal: The terminal acquires the language information set by the user and sends it to the server.

[1022] Example: A device reads the language information selected in the language setting menu and sends that information to a server.

[1023] Server: The server translates traffic information and navigation advice into the user's language using Google Cloud Translation API and DeepL.

[1024] Example: The server sends the acquired traffic information to a translation service via API and receives the translation results.

[1025] Terminal: The terminal provides the translated information to the user in voice and text format. It uses speech synthesis technology to convert the translated information into voice and display it on the display.

[1026] Example: The device starts a speech synthesis engine, converts the translated text into speech, and outputs it through the speaker, while displaying the text on the display.

[1027] 3. Image Recognition Technology

[1028] Device: The device uses the smartphone camera to capture real-time images of the surroundings, which are then analyzed using image recognition algorithms (e.g., OpenCV, TensorFlow Lite).

[1029] Example: The device activates the camera and starts the video stream.

[1030] Device: The device uses image recognition algorithms to recognize traffic signs and pedestrians from the captured video.

[1031] Example: The device runs image recognition algorithms on each video frame to detect signs and pedestrians.

[1032] Server: The server analyzes the recognition data sent from the device and generates warnings and advice for the driver.

[1033] Example: The server analyzes the received data and, if necessary, generates warning messages or driving advice and sends them to the terminal.

[1034] Terminal: The terminal notifies the user of advice from the server, for example providing audio and visual alerts to the driver if the speed limit is exceeded or if there is a pedestrian ahead.

[1035] Example: A device uses a speech synthesis engine to generate a warning message and output it over the speaker, and also displays the warning text on the screen.

[1036] 4. Emotion Engine

[1037] Device: The device uses a smartphone or in-car camera and microphone to capture the user's facial expressions, tone of voice, heart rate, and other data in real time.

[1038] Example: The device activates the camera and microphone to collect facial expressions, voice data, and data from a heart rate monitor.

[1039] Server: The server analyzes the emotion data sent from the emotion engine and evaluates the user's driving state, for example, detecting whether the user is nervous or tired.

[1040] Example: The server uses a machine learning model to analyze emotion data and evaluate the driver's state.

[1041] Device: The device will adjust its navigation advice based on the emotion engine's analysis.

[1042] Example: Based on the analysis received from the server, the device adjusts the settings of the navigation system, plays relaxing music, or provides gentle voice guidance.

[1043] Server: The server reconstructs the navigation plan according to the user's emotional state and sends it to the device.

[1044] Example: The server creates a recalculated navigation plan based on the results of an evaluation of the user's emotional state and sends it to the device.

[1045] Device: The device provides users with navigation advice and music tailored based on the emotion engine, thereby supporting safe driving.

[1046] Example: The device plays relaxing music selected from a playlist and outputs gentle voice prompts through the speaker while displaying instructions on the screen.

[1047] Prompt Sentence Examples

[1048] 1. Example 1: When a user sets a destination, the device sends current location and destination information to the server and receives the optimal route. The emotion engine analyzes the user's facial expressions and tone of voice to evaluate their level of confidence in driving. If the user is nervous, the device plays relaxing music and provides gentle voice guidance.

[1049] Example prompt: "Describe a scenario in which a user receives the best route to a destination and the emotion engine suggests relaxing music."

[1050] 2. Example 2: When a user is driving for a long time, the emotion engine detects fatigue and notifies the server. The server recalculates the route and suggests places where you can take a break. The device notifies the user by voice, "There is a rest area up ahead. Would you like to take a short break?"

[1051] Example prompt: "Describe a scenario where a user is driving for a long period of time, the emotion engine detects their fatigue, and the server suggests places where they can take a break."

[1052] 3. Example 3: For users who are nervous, the device provides information on recognized traffic signs and pedestrians in real time. For example, if the camera recognizes a "Stop" sign, the device will give a voice command saying, "There is a stop sign. Please stop," giving the user a sense of security.

[1053] Example prompt: "Describe a scenario in which a user feels anxious and is provided with real-time information about recognized traffic signs and pedestrians to promote safety."

[1054] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1055] Step 1: Collecting traffic data

[1056] Server: The server collects traffic data from road traffic management agencies and cloud services via API.

[1057] Input: API request

[1058] Data processing: The acquired data is analyzed in JSON format to extract information such as traffic congestion, accidents, and road construction information.

[1059] Output: Traffic information stored in a database

[1060] What it does: The server sends an API request every minute to get new traffic data, parses the data, and stores it in a database.

[1061] Step 2: Analyze the collected data

[1062] Server: The server uses machine learning algorithms to analyze the collected traffic information and calculate the optimal route.

[1063] Input: Traffic information data and user's current location and destination

[1064] Data processing: Calculate the optimal route using machine learning models (e.g., TensorFlow, Scikit-learn).

[1065] Output: Optimal route information

[1066] Specific operation: The server receives the user's current location and destination as input, calculates the optimal route through an algorithm, and stores the results in a database.

[1067] Step 3: Sending route information

[1068] Server: The server sends the calculation results to the user's device in real time.

[1069] Input: Optimal route information

[1070] Data processing: None

[1071] Output: Route information sent to the user terminal

[1072] Specific operation: The server sends a push notification to the user's device and provides a data packet containing optimal route information.

[1073] Step 4: View traffic information

[1074] Terminal: The terminal visually displays the traffic information received from the server on a map and presents the optimal route to the user.

[1075] Input: Route information sent

[1076] Data processing: Analyze the received data and display it on a map using a map API (e.g., OpenStreetMap, Google Maps API).

[1077] Output: Visual representation of the route

[1078] Specific operation: The device uses the map API to display a map and highlight the optimal route.

[1079] Step 5: Get and send language settings

[1080] Terminal: The terminal obtains the language information set by the user and sends it to the server.

[1081] Input: User's language preference

[1082] Data processing: None

[1083] Output: Language preference information sent to the server

[1084] Specific operation: The device reads the language information selected in the language setting menu and sends that information to the server.

[1085] Step 6: Translating traffic information and advice

[1086] Server: The server translates traffic information and navigation advice into the user's language.

[1087] Input: Traffic and navigation advice, language preference information

[1088] Data processing: Translation is performed using a translation API (e.g., Google Cloud Translation API, DeepL).

[1089] Output: Translated traffic information and navigation advice

[1090] Specific operation: The server sends the acquired traffic information and navigation advice to the translation service via API and receives the translation results.

[1091] Step 7: Provide translation information

[1092] Terminal: The terminal provides the translated information to the user in voice and text.

[1093] Input: Translated traffic and navigation advice

[1094] Data processing: Convert the data into speech using speech synthesis technology (e.g., speech synthesis engine), and display the text on the screen.

[1095] Output: Spoken and textual translation information

[1096] What happens: The device will launch its speech synthesis engine, convert the translated text into speech and output it through the speaker, and display the text on the display.

[1097] Step 8: Acquire and analyze footage

[1098] Device: The device uses the smartphone camera to capture real-time images of the surrounding area and analyzes them using image recognition algorithms.

[1099] Input: Camera image

[1100] Data processing: Run an image recognition algorithm (e.g., OpenCV, TensorFlow Lite) on each video frame to detect signs and pedestrians.

[1101] Output: Recognized traffic signs and pedestrian information

[1102] What happens: The device turns on the camera and starts streaming video. It then performs image recognition on each video frame to detect signs and pedestrians.

[1103] Step 9: Sending Recognition Data

[1104] Device: The device sends the recognized traffic signs and pedestrian information to the server.

[1105] Input: Recognized signs and pedestrian information

[1106] Data processing: None

[1107] Output: Identification information sent to the server

[1108] Specific operation: The device includes the recognized information in a data packet and sends it to the server.

[1109] Step 10: Generate warnings and advice

[1110] Server: The server analyzes the recognition data sent from the device and generates real-time warnings and advice for the driver.

[1111] Input: Recognition data

[1112] Data processing: Generate warnings and advice based on the recognition data.

[1113] Output: any warnings or advice generated

[1114] Specific operation: The server analyzes the received data and, if necessary, generates warning messages or driving advice and sends them to the terminal.

[1115] Step 11: Notification of warnings and advice

[1116] Terminal: The terminal notifies the user of warnings and advice from the server.

[1117] Input: Warnings and advice

[1118] Data processing: Converts data into speech using speech synthesis technology and displays the text on the screen.

[1119] Output: Audio and visual warnings and advice

[1120] What happens: The device uses a speech synthesis engine to generate a warning message and outputs it over the speaker, and also displays the warning text on the screen.

[1121] Step 12: Obtaining Emotion Data

[1122] Device: The device uses a smartphone, in-car camera, or microphone to capture the user's facial expressions, tone of voice, heart rate, and other information in real time.

[1123] Input: Camera video and audio data, heart rate monitor data

[1124] Data processing: Analyze the acquired data and identify the emotional state.

[1125] Output: Emotional state data

[1126] Specific operation: The device activates the camera and microphone to collect facial expressions, voice data, and heart rate monitor data.

[1127] Step 13: Analyze the sentiment data

[1128] Server: The server analyzes the emotion data sent from the emotion engine and evaluates the user's driving state.

[1129] Input: Emotional state data

[1130] Data processing: Using machine learning models to analyze emotion data and assess the driver's state.

[1131] Output: Driving condition evaluation data

[1132] Specific operation: The server uses a machine learning model to analyze emotion data and evaluate the driver's state.

[1133] Step 14: Adjusting Navigation Advice

[1134] Device: The device will adjust its navigation advice based on the emotion engine's analysis.

[1135] Input: Driving condition evaluation data

[1136] Data manipulation: Adjust navigation settings, change tips and music.

[1137] Output: Adjusted navigation advice

[1138] What it does: Based on the analysis received from the server, the device adjusts navigation settings, plays relaxing music, and provides gentle voice advice.

[1139] Step 15: Reconstructing the navigation plan

[1140] Server: The server reconstructs the navigation plan according to the user's emotional state and sends it to the device.

[1141] Input: Emotional state assessment data

[1142] Data transformation: Generate a new navigation plan.

[1143] Output: Reconstructed navigation plan

[1144] Specific operation: The server generates a recalculated navigation plan based on the results of the evaluation of the user's emotional state and sends it to the device.

[1145] Step 16: Offering advice and music

[1146] Device: The device provides users with navigation advice and music tailored to their needs based on the emotion engine, thereby supporting safe driving.

[1147] Input: Reconstructed navigation plan

[1148] Data processing: Provides voice advice and music based on the navigation plan.

[1149] Output: Audio advice and music provided

[1150] What it does: The device plays relaxing music selected from a playlist, emits gentle voice prompts through the speaker, and displays instructions on the screen.

[1151] (Application example 2)

[1152] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1153] In modern society, traffic congestion and accidents occur on a daily basis, and drivers need real-time traffic information. However, there have been no navigation systems that support multiple languages ​​or take into account the driver's emotional state. In addition, driver stress and fatigue can affect driving safety, and an effective system to mitigate these effects has been sought. Therefore, an object of the present invention is to provide a system that provides real-time traffic information and evaluates the driver's emotional state to improve driving safety and comfort.

[1154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1155] In this invention, the server includes a means for collecting traffic information in real time, a means for analyzing the collected traffic information and calculating the optimal route, and a means for transmitting the analysis results to a user terminal. This allows the driver to receive optimal navigation based on the latest traffic conditions. Furthermore, by adding a means for collecting and recognizing the driver's emotional data, such as facial expressions, tone of voice, and heart rate, it is possible to provide relaxing music or gentle voice guidance when the driver is feeling tense or tired. Furthermore, by combining a means for recognizing road signs and pedestrians using a camera and notifying the user terminal of the recognition results, safety can be further improved.

[1156] "Real-time" refers to a method of processing data and providing results in accordance with the actual progress of time.

[1157] "Traffic information" refers to information relating to public transportation and vehicle traffic, such as road congestion, accidents, and construction work.

[1158] The term "route" refers to the route or path from a starting point to a destination.

[1159] A "user terminal" is an electronic device carried by a user, such as a smartphone, tablet, or in-vehicle terminal.

[1160] "Visually displaying" means showing data or information on a screen in the form of images or text.

[1161] A "camera" is a device for taking images or videos.

[1162] "Road signs" are signs installed to inform drivers and pedestrians of road safety and rules.

[1163] "Pedestrians" are people who walk on the road.

[1164] "Facial expressions" are facial muscle movements and expressions that indicate a person's emotional state.

[1165] "Tone of voice" refers to the timbre or tone of spoken language.

[1166] "Heart rate" is the number of times the heart beats within a certain period of time.

[1167] "Emotional data" means data collected to indicate the emotional state of a driver.

[1168] "Emotion recognition" means analyzing and evaluating a person's emotional state.

[1169] "Navigation advice" is advice to guide the driver to appropriate routes and important points to note.

[1170] "Music" refers to an art form that combines rhythm and melody of sounds.

[1171] "Voice guidance" refers to providing instructions and information audibly through a speaker.

[1172] "Means" refers to the tools, methods, and equipment used to achieve a specific purpose.

[1173] The present invention is a system that collects traffic information in real time and provides that information in multiple languages, and by combining it with an emotion engine, recognizes the driver's emotional state and supports safe and comfortable driving. An embodiment of the system will be described in detail below.

[1174] 1. Providing real-time traffic information

[1175] The server collects traffic data in real time via APIs from road traffic management centers, cloud services, etc. The collected data includes information on traffic congestion, accidents, road construction, etc., and consolidates this information and stores it in a database. The server analyzes the collected traffic information and calculates the optimal route. A machine learning algorithm is used for the calculation, generating the optimal route based on the user's current location and destination. The server sends the analysis results to the user's device in real time, allowing the user to receive navigation based on the latest traffic conditions. The user's device visually displays the traffic information received from the server on a map and provides the user with the optimal route. The map is displayed using OpenStreetMap or other map APIs.

[1176] 2. Multilingual support

[1177] The user device obtains the language information set by the user. This information is sent to the server, which applies different language settings to each user. The server translates traffic information and navigation advice into the user's language using a global API or other translation service. The user device then provides the translated information to the user in voice and text. Using speech synthesis technology, the navigation advice is output as voice in the user's language. At the same time, the translated information is displayed in text on the screen.

[1178] 3. Image Recognition Technology

[1179] The user device uses the smartphone camera to capture images of the surroundings in real time. This image is analyzed using an image recognition algorithm (e.g., OpenCV or TensorFlow Lite). The user device recognizes traffic signs and pedestrians from the captured image. The recognized information is used to assist the user's driving. The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time. The user device notifies the user of the advice from the server. For example, if the speed limit is exceeded or if a pedestrian is ahead, the user is provided with audio and visual alerts.

[1180] 4. Emotion Engine

[1181] The user device recognizes emotions from the user's facial expressions, tone of voice, heart rate, etc. using a smartphone or in-vehicle camera. This emotional data is collected and evaluated in real time. The server analyzes the emotional data sent from the emotion engine and evaluates the user's driving state. For example, it detects if the driver is nervous or tired. The user device adjusts navigation advice based on the emotion engine's analysis results. For example, if the user is nervous, it provides calming advice or relaxing music. The server reconstructs a navigation plan according to the user's emotional state and sends it to the device. This makes it possible to provide routes and advice optimal for the user's mental state. The user device provides the user with navigation advice and music adjusted by the emotion engine. This reduces the user's driving stress and supports safe driving.

[1182] 5. Specific Examples

[1183] Example 1: When a user sets a destination, the device sends current location and destination information to the server and receives the optimal route. The emotion engine analyzes the user's facial expressions and tone of voice to evaluate their state of mind to drive safely. If the user is nervous, the device plays relaxing music and provides gentle voice guidance.

[1184] Example 2: When a user is driving for a long time, the emotion engine detects fatigue and notifies the server. The server recalculates the route and suggests places where users can take a break. The device notifies the user by voice, saying, "There is a rest area ahead. Would you like to take a short break?"

[1185] Example 3: For users who are nervous, the device provides information on recognized traffic signs and pedestrians in real time. For example, if the camera recognizes a "Stop" sign, the device will give a voice command saying, "There is a stop sign. Please stop," to reassure the user.

[1186] The overall processing flow of this system involves the server and user terminal working together to provide traffic information and advice based on the driver's emotional state, thereby achieving safe and comfortable driving.

[1187] Example prompt: "Describe an application that recognizes a driver's emotions based on their facial expression, tone of voice, and heart rate to help them drive safely."

[1188] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1189] Step 1:

[1190] The server collects traffic data in real time via API from road traffic management centers, cloud services, etc. The input is traffic information (traffic congestion, accidents, road construction information, etc.) obtained from road traffic management centers and cloud services. This information is stored in a database and processed into a format that can be matched with the user's current location and destination information. The output is integrated traffic information data.

[1191] Step 2:

[1192] The server analyzes the collected traffic information using machine learning algorithms and calculates the optimal route. The inputs are the collected traffic information, the user's current location, and destination information. The server performs data calculations to generate the optimal route. The output is the optimized route information.

[1193] Step 3:

[1194] The server transmits the analysis result, the optimal route information, to the user terminal in real time. The input is the optimized route information, and the output is the navigation data transmitted to the user terminal.

[1195] Step 4:

[1196] The user device visually displays the optimal route information received from the server on a map. It uses OpenStreetMap and other map APIs to obtain map data and display the optimal route to the user. The input is the route information sent from the server and the map data obtained from the map API, and the output is a visual map display.

[1197] Step 5:

[1198] The user device uses the smartphone camera to capture images of the surroundings in real time and uses image recognition algorithms (OpenCV or TensorFlow Lite) to recognize traffic signs and pedestrians. The input is image data from the camera, which is analyzed using image recognition. The output is recognized traffic signs and pedestrian information.

[1199] Step 6:

[1200] The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. The input is traffic signs and pedestrian information, and by processing the data, it generates information that the driver should pay attention to. The output is a warning or advice sent to the user's device.

[1201] Step 7:

[1202] The user terminal notifies the user of warnings and advice from the server in real time. The input is the warnings and advice sent from the server and provided to the user as audio and visual alerts. The output is the notification of the warnings and advice given to the user.

[1203] Step 8:

[1204] The user device collects emotion data from the user's facial expressions, tone of voice, heart rate, etc. using a smartphone or in-car camera, and analyzes it with an emotion engine. The input is the user's facial expressions, tone of voice, and heart rate data, which are analyzed by the emotion engine. The output is the recognized emotional state.

[1205] Step 9:

[1206] The server analyzes the emotion data sent from the emotion engine and evaluates the user's driving state. The input is the emotional state data, and the server performs data calculations to evaluate the user's driving state. The output is navigation advice and music suggestions based on the emotional state to the user's device.

[1207] Step 10:

[1208] The user device provides navigation advice and relaxing music based on the analysis results of the emotion engine. The input is navigation advice and music data based on the emotional state, which are provided to the user visually and audibly. The output is support for safe and comfortable driving for the user.

[1209] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1210] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1211] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1212] [Third embodiment]

[1213] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1214] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1215] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1216] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1217] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1218] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1219] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1220] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1221] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1222] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1223] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1224] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1225] The present invention provides a system that supports safe and intuitive driving by collecting traffic information in real time, providing that information in multiple languages, and detecting traffic signs and pedestrians using image recognition technology. An embodiment of the system will be described in detail below.

[1226] 1. Providing real-time traffic information

[1227] Server: The server collects real-time traffic information from road traffic management centers, cloud services, etc. The collected data includes information on traffic congestion, accidents, road construction, etc., and integrates this information and stores it in a database.

[1228] Server: The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms to generate the optimal route based on the user's current location and destination.

[1229] Server: The server transmits the analysis results to the user's device in real time, allowing the user to receive navigation based on the latest traffic conditions.

[1230] Terminal: The terminal visually displays the traffic information received from the server on a map and provides the user with the optimal route. The map is displayed using OpenStreetMap or other map APIs.

[1231] 2. Multilingual support

[1232] Device: The device retrieves the user's language preference, which is then sent to the server, allowing different language settings to be applied for each user.

[1233] Server: The server translates traffic information and navigation advice into the user's language using a global API or other translation services.

[1234] Device: The device provides the translated information to the user in both voice and text. It uses speech synthesis technology to speak navigation advice in the user's language and simultaneously displays the translated text on the screen.

[1235] 3. Image Recognition Technology

[1236] Device: The device uses the smartphone camera to capture real-time images of the surroundings, which are then analyzed using image recognition algorithms (e.g., OpenCV or TensorFlow Lite).

[1237] Device: The device recognizes traffic signs and pedestrians from the captured video. The recognized information is used to assist the user in driving.

[1238] Server: The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time.

[1239] Terminal: The terminal notifies the user of advice from the server, for example providing audio and visual alerts to the user if they exceed the speed limit or if there are pedestrians ahead.

[1240] 4. Specific Examples

[1241] Example 1: The user sets a destination. The device sends current location and destination information to the server and receives the optimal route. The device displays the route on a map and provides route guidance based on traffic information updated in real time. If a traffic jam occurs along the way, the server immediately calculates a new route and sends updated information to the device. As a result, the user can reach the destination smoothly.

[1242] Example 2: When a user is driving in a foreign country, the device provides navigation advice in the user's language. The server translates collected traffic information and recognition results into the user's native language and provides them via voice and text through the device. This allows the user to drive with confidence even in a different language environment.

[1243] Example 3: When the device's camera recognizes a road sign while driving, the information is sent to the server and the necessary warning is generated. For example, if the device recognizes a "Stop" sign, the server analyzes it and sends a warning to the device. The device will notify the user by voice, "There is a stop sign ahead," to help the user take appropriate action.

[1244] As described above, this system provides users with the latest traffic information in real time, supports the driving environment with multilingual support, and uses image recognition technology to support safe driving, thereby reducing driving anxiety and providing a comfortable driving experience.

[1245] The processing flow will be explained below.

[1246] Program processing flow and specific operations

[1247] 1. Providing real-time traffic information

[1248] Step 1: Collect traffic information

[1249] Server: "The server collects traffic data in real time via API from road traffic management centers and cloud services."

[1250] Specific operation: Data such as traffic congestion, accidents, and road construction information is obtained using an API and stored in a database.

[1251] Step 2: Traffic information integration and analysis

[1252] Server: "The server integrates the collected traffic data into a database and uses machine learning algorithms to calculate the optimal route."

[1253] What it does: Analyzes traffic information data and calculates the optimal route to the user's destination.

[1254] Step 3: Distributing traffic information

[1255] Server: "The server sends the analyzed route information to the user terminal in real time."

[1256] Specific operation: The calculated optimal route is sent to the user's device via an HTTP request.

[1257] Step 4: View information

[1258] Terminal: "The terminal displays real-time traffic information received from the server on a map."

[1259] Specific behavior: Uses OpenStreetMap and other map APIs to visually display received route information on a map.

[1260] 2. Multilingual support

[1261] Step 1: Get the language settings

[1262] Device: "The device acquires the language information set by the user and sends it to the server."

[1263] Specific behavior: Get the user's language preference from the app's settings menu and send it to the server.

[1264] Step 2: Perform the translation

[1265] Server: "The server translates traffic information and navigation advice into the user's language."

[1266] What it does: Uses a translation API to translate collected traffic information and navigation advice into the user's language of choice.

[1267] Step 3: Providing multilingual guidance

[1268] Device: "The device provides multilingual, translated information to the user via voice and text."

[1269] Specific operation: Using speech synthesis technology, translated navigation information is output as voice and simultaneously displayed as text on the screen.

[1270] 3. Image Recognition Technology

[1271] Step 1: Acquire camera footage

[1272] Device: "The device captures video from the smartphone camera and processes it for image recognition."

[1273] Specific operation: Uses the camera API to capture real-time video and store it in memory.

[1274] Step 2: Perform image recognition

[1275] Device: "The device applies image recognition algorithms to the captured video to detect traffic signs and pedestrians."

[1276] Specific operation: Using OpenCV and TensorFlow Lite, it analyzes signs and pedestrians in video and detects specific features.

[1277] Step 3: Sending the recognition results

[1278] Device: "The device sends the recognition results to the server."

[1279] Specific operation: Organizes the recognized information and sends it to the server via API.

[1280] Step 4: Generate feedback

[1281] Server: "The server analyzes the recognition results and generates advice and warnings for safe driving."

[1282] What it does: Evaluates the received recognition data and generates the necessary feedback.

[1283] Step 5: Deliver feedback to users

[1284] Terminal: "The terminal notifies the user of advice and warnings from the server."

[1285] Specific behavior: Displays a warning message on the screen and issues an audio alert.

[1286] 4. User Interface Operation and Feedback

[1287] Step 1: Set your destination

[1288] User: "The user enters and sets a destination in the application."

[1289] What it does: Select a destination on the map or enter text in the search bar to specify a destination.

[1290] Step 2: Start route guidance

[1291] Terminal: "The terminal starts navigation based on the route information received from the server."

[1292] What it does: Highlights the route on the map and starts voice guidance.

[1293] Step 3: Receive real-time feedback

[1294] Server: "The server monitors the user's real-time movements and updates the route as needed."

[1295] Specific operation: Constantly monitors the user's current location based on GPS information and recalculates the route according to changes in traffic conditions.

[1296] Step 4: Real-time notifications

[1297] Terminal: "The terminal notifies the user of updates from the server in real time."

[1298] Specific behavior: When a new route is provided, the user is immediately notified by map display and voice.

[1299] This enables the entire system to provide users with real-time, up-to-date and reliable traffic information and safe driving support.

[1300] Example 1

[1301] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1302] This invention relates to a system that efficiently collects and analyzes traffic information updated in real time and provides users with optimal routes. In particular, it aims to support safe and intuitive driving by recognizing the surrounding environment, such as traffic signs and pedestrians, from camera images and notifying the user of this information in multiple languages. Conventional systems have had issues with driver convenience and safety due to slow traffic information updates and insufficient multilingual support and real-time image recognition.

[1303] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1304] In this invention, the server includes a means for collecting traffic information in real time, a means for analyzing the collected traffic information and calculating an optimal route using a machine learning algorithm, and a means for transmitting the analysis results to a user terminal. This enables efficient and prompt provision of an optimal route based on the latest traffic information. The user terminal also includes a means for visually displaying the received traffic information on a map, a means for recognizing road signs and pedestrians using a camera, a means for transmitting the recognition results to the server and notifying the user terminal of analyzed warnings and advice, and a means for providing navigation advice using speech synthesis technology that supports multiple languages. This allows users to receive safe and intuitive driving support in multiple languages ​​in real time.

[1305] "Real-time" refers to processing and providing information instantly the moment it is generated.

[1306] "Traffic information" refers to information relating to traffic conditions on roads, including data on congestion, accidents, road construction, and the like.

[1307] A "machine learning algorithm" is a computational method that automatically learns patterns and rules by analyzing large amounts of data, and makes predictions and classifications.

[1308] "Optimal route" refers to the shortest and most efficient route for a user to reach a destination.

[1309] A "user terminal" is a device operated by a user, including a smartphone, tablet, car navigation system, etc.

[1310] "Visually displaying on a map" refers to visually displaying traffic and route information using a map application or navigation system.

[1311] "Camera" means a device for capturing video or images.

[1312] "Road sign" means a sign placed on a road to provide direction or information to motorists.

[1313] "Pedestrian" refers to a person who moves on foot on a road.

[1314] "Recognizing" refers to detecting and identifying specific objects (such as road signs or pedestrians) from the image captured by the camera.

[1315] "Analyze" refers to the detailed analysis of collected data and recognition results to derive meaningful information and predictions.

[1316] "Warning" means providing information to draw attention to a particular situation.

[1317] "Advice" means providing instructions or suggestions for specific action.

[1318] "Speech synthesis technology" is a technology that generates and plays back voices that resemble human voices from text data.

[1319] "Multilingual" means supporting multiple different languages.

[1320] "Translate" refers to converting text written in one language into another language.

[1321] The present invention provides a system that supports safe and intuitive driving by collecting traffic information in real time, providing that information in multiple languages, and detecting traffic signs and pedestrians using image recognition technology. An embodiment of the system will be described in detail below.

[1322] 1. Providing real-time traffic information

[1323] server:

[1324] The server continuously obtains real-time traffic information from the road traffic management center API or cloud services (e.g., general traffic information API), including information on traffic congestion, accidents, road construction, etc. The server stores this information in a database such as MongoDB and generates an integrated dataset.

[1325] server:

[1326] The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms (e.g., Scikit-learn or TensorFlow), and sends the results of this calculation to the user's device in real time.

[1327] Device:

[1328] The device visually displays the traffic information received from the server on a map and provides the user with the optimal route, using external services such as OpenStreetMap and Google Maps API.

[1329] 2. Multilingual navigation available

[1330] Device:

[1331] The terminal obtains the user's language setting and sends that information to the server.

[1332] server:

[1333] The server uses the Global Translate API (e.g., Google Translate API) to translate traffic information and navigation instructions into the user's preferred language.

[1334] Device:

[1335] The device provides the translated information to the user in real time using speech synthesis technology (e.g., Amazon Polly), and also displays the corresponding text information on a map.

[1336] 3. Traffic Sign and Pedestrian Detection Using Image Recognition Technology

[1337] Device:

[1338] The device uses its built-in camera to capture real-time images of its surroundings, which are then analyzed by image recognition algorithms (e.g., OpenCV or TensorFlow Lite) running on the device.

[1339] Device:

[1340] The device recognizes traffic signs and pedestrians in the video and sends the recognition results to the server in real time.

[1341] server:

[1342] The server analyzes the received recognition data and notifies the user's device of warnings and advice necessary for driving.

[1343] Device:

[1344] The device provides the user with audio and visual warnings and advice from the server, such as "There is a pedestrian ahead" or "You are exceeding the speed limit."

[1345] Specific examples

[1346] Example 1:

[1347] When a user inputs their current location and destination, the device sends the information to the server, which calculates the optimal route and sends it back to the device. The device then displays the route on a map and provides route guidance based on traffic information updated in real time. If traffic congestion occurs, the server calculates a new route and sends updated information to the device, allowing the user to reach their destination smoothly.

[1348] Example 2:

[1349] When a user is driving in a foreign country, the device will provide navigation advice in the user's language of choice. The server will translate traffic information and recognition results into the user's native language and provide them via voice and text via the device, allowing users to drive with confidence in different language environments.

[1350] Example 3:

[1351] When the device's camera recognizes a road sign while driving, the information is sent to the server, which generates the necessary warning. For example, if a "Stop" sign is recognized, the server analyzes it and sends a warning to the device. The device then notifies the user by voice, saying "There is a stop sign ahead," helping the user to take appropriate action.

[1352] Prompt Sentence Examples

[1353] 1. Give an example of a program that calculates the optimal route based on traffic information when a user inputs their current location and destination.

[1354] 2. Describe the process flow of a system that provides multilingual voice and text navigation based on real-time traffic data.

[1355] 3. Explain the process of a system that uses a smartphone camera and image recognition algorithms to detect traffic signs and pedestrians and alert the driver.

[1356] This invention allows users to utilize the latest traffic information in real time and easily implements a system that supports the driving environment in multiple languages, thereby enabling safe and efficient driving, reducing driving anxiety, and realizing a comfortable driving experience.

[1357] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1358] Step 1:

[1359] Real-time traffic information collection

[1360] Server: The server obtains real-time traffic information from the Road Traffic Management Center API and cloud services.

[1361] Input: Traffic information data received from the Road Traffic Management Center API and cloud services.

[1362] Data processing: The acquired traffic information is converted into JSON format and stored in a database.

[1363] Output: Updated traffic information dataset.

[1364] What it does: The server periodically sends API requests to retrieve new traffic information and stores it in MongoDB.

[1365] Step 2:

[1366] Traffic information analysis and optimal route calculation

[1367] Server: The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms (e.g., Scikit-learn, TensorFlow).

[1368] Input: Traffic information data stored in a database.

[1369] Data calculations: Using machine learning algorithms to calculate the optimal route based on the user's current location and destination.

[1370] Output: Optimal route information provided to the user.

[1371] Specific operation: The server applies Dijkstra's algorithm based on map data and traffic information to calculate the shortest route.

[1372] Step 3:

[1373] Sending optimal route information

[1374] Server: The server transmits the calculated optimal route information to the user's device in real time.

[1375] Input: Calculated optimal route information.

[1376] Data calculation: If necessary, convert the route information into JSON format so that it can be sent.

[1377] Output: Optimal route information sent to the user terminal.

[1378] Specific operation: The server sends JSON format data to the user terminal via the HTTP protocol.

[1379] Step 4:

[1380] Displaying traffic and route information

[1381] Terminal: The terminal visually displays the traffic and route information received from the server on a map.

[1382] Input: Traffic and route information received from the server.

[1383] Data processing: Data conversion to overlay information on a map.

[1384] Output: Map and route information visually displayed to the user.

[1385] Specific operation: The device displays a map using the Google Maps API and overlays the received route information on it.

[1386] Step 5:

[1387] Multilingual navigation provided

[1388] Device: The device obtains the user's language preference and sends it to the server.

[1389] Input: User's language preference information.

[1390] Output: Language preference data sent to the server.

[1391] Specific behavior: The device obtains the user's selected language from the language settings screen.

[1392] Server: The server translates traffic information and navigation instructions into the user's preferred language.

[1393] Input: Traffic information, navigation instructions, and user language preference data.

[1394] Data Calculation: Translate data into a specified language using the Global Translate API.

[1395] Output: Translated traffic and navigation instructions.

[1396] Specific operation: The server calls the Google Translate API to translate the data and sends the results to the device.

[1397] Terminal: The terminal synthesizes the translated information and provides it to the user.

[1398] Input: Translated traffic and navigation instructions.

[1399] Data processing: Generate voice data using a speech synthesis application (e.g., Amazon Polly).

[1400] Output: Audio and text information provided to the user.

[1401] Specific operation: The device uses Amazon Polly to play a voice message saying "Go straight to your destination" and also displays the text on the map.

[1402] Step 6:

[1403] Traffic sign and pedestrian detection using image recognition

[1404] Device: The device uses its built-in camera to capture real-time images of its surroundings.

[1405] Input: Video data captured by the camera.

[1406] Output: Video data that is passed to the image recognition algorithm.

[1407] Specific operation: The device starts the camera and acquires streaming video data.

[1408] Terminal: The terminal analyzes the video data using image recognition algorithms (e.g., OpenCV or TensorFlow Lite).

[1409] Input: Video data captured by the camera.

[1410] Data calculations: Image recognition algorithms identify traffic signs and pedestrians.

[1411] Output: Information on recognized traffic signs and pedestrians.

[1412] Specific operation: The device uses OpenCV to recognize stop signs and pedestrians.

[1413] Device: Sends the recognition results to the server.

[1414] Input: The recognized information.

[1415] Output: Recognition result data sent to the server.

[1416] Specific operation: The device converts the recognized data into JSON format and sends it to the server.

[1417] Server: The server analyzes the received recognition data and generates any necessary warnings or advice.

[1418] Input: Recognition data sent from the device.

[1419] Data calculations: Analyze perception data to generate warnings and advice.

[1420] Output: Warnings and advice sent to the user's terminal.

[1421] Specific operation: Based on the analysis results, the server generates a warning such as "There is a pedestrian ahead" and sends it to the device.

[1422] Terminal: The terminal notifies the user of warnings and advice from the server by voice and display.

[1423] Input: Any warnings or advice received from the server.

[1424] Output: The audio and visual information provided to the user.

[1425] Specific operation: The device will announce "There is a pedestrian ahead" and display an icon on the screen.

[1426] (Application example 1)

[1427] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1428] Conventional navigation systems only provide traffic information and are limited in their ability to respond to changes in traffic conditions in real time. Furthermore, they lack driving assistance in multilingual environments and the ability to recognize road signs and pedestrians in real time, making them inadequate for supporting safe driving. There is a need to solve these issues and provide safer and more efficient driving assistance.

[1429] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1430] In this invention, the server includes means for collecting traffic information in real time, means for analyzing the collected traffic information and calculating the optimal route, and means for transmitting the analysis results to the user terminal, thereby enabling the server to respond to changes in traffic conditions in real time and provide the optimal route.

[1431] Furthermore, this invention provides safe and intuitive driving assistance by incorporating a means for recognizing road signs and pedestrians using a camera, a means for notifying a user terminal of the recognition results, a means for providing audio advice based on the recognition results, and a means having a multilingual translation function based on local languages.

[1432] "Means of collecting traffic information in real time" refers to the function of instantly obtaining the latest traffic information, such as information on traffic congestion, accidents, and road construction, from traffic management centers and cloud services.

[1433] "Means of analyzing collected traffic information and calculating the optimal route" refers to a function that uses a machine learning algorithm to generate the optimal route taking into account traffic conditions based on the user's current location and destination.

[1434] The "means for transmitting the analysis results to the user terminal" is a function for transferring optimal route information and traffic conditions calculated by the server to the user terminal in real time.

[1435] "Means for visually displaying received traffic information on a user terminal" refers to a function that graphically displays maps and route guidance on the user's terminal, allowing the user to intuitively understand traffic information.

[1436] "Means of recognizing road signs and pedestrians using cameras" refers to a function that analyzes camera footage in real time and detects traffic signs and pedestrians with high accuracy.

[1437] "Means for notifying the user terminal of the recognition results" is a function that immediately transmits information about detected signs and pedestrians to the user terminal and notifies the driver of the situation.

[1438] "Means for providing voice advice based on the recognition results" is a function that provides appropriate driving advice by voice based on the recognized information, supporting the driver in safe driving.

[1439] "Means having a multilingual translation function based on local languages" refers to a function that automatically translates traffic information and navigation advice into different languages ​​according to user settings.

[1440] "Means for translating and providing traffic information and navigation advice in multiple languages" refers to a function that automatically translates and provides collected and analyzed traffic information and driving advice based on the user's language settings.

[1441] "Means for providing translated information in voice and text format based on the user's language settings" refers to a function that provides translated traffic information and route guidance in voice and text format according to the language settings registered by the user.

[1442] This invention will be explained using the smartphone application "SmartDrive Assist" installed in autonomous vehicles as a concrete example. This application collects traffic information in real time, provides it in multiple languages, and utilizes image recognition technology to monitor the external environment and support safe driving.

[1443] First, the server collects traffic information in real time from road traffic management centers and cloud services. This traffic information includes information on traffic congestion, accidents, road construction, and more, and stores it in a database. Next, the collected traffic information is analyzed and the optimal route is calculated using a machine learning algorithm. This calculation uses map APIs such as Google Maps API and OpenStreetMap. Finally, the analysis results are sent to the user's device in real time.

[1444] The device visually displays traffic information received from the server. Maps are displayed using OpenStreetMap and other map APIs. In addition, the device uses the smartphone camera to capture real-time images of the surrounding area. This image is analyzed using image recognition algorithms (e.g., OpenCV or TensorFlow Lite) to recognize road signs and pedestrians. The recognized information is used to assist the user's driving.

[1445] The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time. The device notifies the user of the advice from the server. For example, if the device recognizes a "Stop" sign, the server analyzes it and sends a warning to the device. The device then notifies the user by voice, saying "There is a stop sign ahead," helping the user to take appropriate action.

[1446] The device also obtains the user's language preference and sends this information to the server. The server translates traffic information and navigation advice into the user's language using a global API or other translation services. The device then provides the translated information to the user in voice and text. It uses speech synthesis technology to output navigation advice in the user's language preference and displays the translated information in text on the screen.

[1447] As a concrete example, consider a scenario in which a user is traveling by car from Tokyo to Los Angeles. First, the user launches the "SmartDrive Assist" app on their smartphone and sets their destination (Los Angeles). The app retrieves real-time traffic information from the server and provides voice advice translated into multiple languages. It uses the smartphone's camera to detect road signs and pedestrians and provides appropriate voice warnings to the user.

[1448] Example prompt sentence:

[1449] Consider a scenario where a user is using the "SmartDrive Assist" app to drive from Tokyo to Los Angeles. The app collects real-time traffic information, provides it in multiple languages, and detects traffic signs and pedestrians through camera footage.

[1450] In this way, the present invention provides users with the latest traffic information in real time, supports the driving environment with multilingual support, and uses image recognition technology to support safe driving, thereby reducing driving anxiety and providing a comfortable driving experience.

[1451] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1452] Step 1:

[1453] The server collects real-time traffic information from road traffic management centers and cloud services. It obtains information on traffic congestion, accidents, road construction, etc. through the road traffic management center and cloud service APIs, and receives this data in JSON format. This allows it to accumulate the latest traffic situation data.

[1454] Input: Traffic information from the traffic management center and cloud service API

[1455] Data processing: Receive traffic information in JSON format

[1456] Output: Real-time traffic information database

[1457] Step 2:

[1458] The server analyzes the collected traffic information and calculates the optimal route. It uses machine learning algorithms to generate the optimal route based on the user's current location and destination. The algorithms used include linear regression models and neural networks. Map APIs such as Google Maps API and OpenStreetMap are also used for route calculation.

[1459] Input: Collected traffic information, user's current location, user's destination

[1460] Data processing: Analyzing traffic information and calculating optimal routes using machine learning algorithms

[1461] Output: Optimal route information

[1462] Step 3:

[1463] The server transmits the analysis results to the user's device in real time, using communication protocols such as HTTP requests to transmit the generated optimal route information and traffic condition data to the user's device.

[1464] Input: Optimal route information, traffic condition data

[1465] Data processing: Sending data via HTTP request

[1466] Output: Optimal route information sent to the user terminal

[1467] Step 4:

[1468] The device visually displays the traffic information received from the server. It uses a map API (e.g., OpenStreetMap or Google Maps) to display the received traffic information and the optimal route on a map. This operation uses the display function of the map API.

[1469] Input: Optimal route information and traffic condition data sent from the server

[1470] Data processing: Traffic information and route mapping

[1471] Output: Visual map display

[1472] Step 5:

[1473] The device uses the smartphone camera to capture images of the surroundings in real time. When the user activates the device's camera, real-time images are captured.

[1474] Input: Real-time camera image data

[1475] Data processing: Video capture and real-time acquisition

[1476] Output: Real-time camera footage

[1477] Step 6:

[1478] The device recognizes traffic signs and pedestrians using the captured video. It uses an image recognition algorithm (e.g., OpenCV or TensorFlow Lite) to analyze the captured video and extract information about detected traffic signs and pedestrians.

[1479] Input: Captured camera footage

[1480] Data processing: Traffic sign and pedestrian detection using image recognition algorithms

[1481] Output: Detected traffic signs and pedestrian information

[1482] Step 7:

[1483] The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. For example, if a "Stop" sign is detected, the server analyzes it and generates appropriate driving advice.

[1484] Input: Traffic sign and pedestrian information sent from the device

[1485] Data processing: Analyzing perception data and generating driving advice

[1486] Output: Driving advice

[1487] Step 8:

[1488] The terminal notifies the user of the advice from the server by generating appropriate voice advice and notifying the user using speech synthesis technology (e.g., gTTS), and also by displaying it in text format on the screen.

[1489] Input: Driving advice sent from the server

[1490] Data processing: speech synthesis and text display

[1491] Output: Voice and text driving advice notifications

[1492] Step 9:

[1493] The server obtains the user's language preference and translates traffic information and navigation advice. It uses a translation service (e.g., Google Translate API) to translate the information based on the user's language preference.

[1494] Input: User-defined language information, traffic information, navigation advice

[1495] Data processing: Multilingual translation using translation services

[1496] Output: Translated traffic information and navigation advice

[1497] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1498] The present invention is a system that collects traffic information in real time and provides that information in multiple languages, and by combining it with an emotion engine, recognizes the driver's emotional state and supports safe and comfortable driving. An embodiment of the system will be described in detail below.

[1499] 1. Providing real-time traffic information

[1500] Server: The server collects traffic data in real time via API from road traffic management centers, cloud services, etc. The collected data includes information on traffic congestion, accidents, road construction, etc., and integrates this information and stores it in a database.

[1501] Server: The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms to generate the optimal route based on the user's current location and destination.

[1502] Server: The server transmits the analysis results to the user's device in real time, allowing the user to receive navigation based on the latest traffic conditions.

[1503] Terminal: The terminal visually displays the traffic information received from the server on a map and provides the user with the optimal route. The map is displayed using OpenStreetMap or other map APIs.

[1504] 2. Multilingual support

[1505] Device: The device retrieves the user's language preference, which is then sent to the server, allowing different language settings to be applied for each user.

[1506] Server: The server translates traffic information and navigation advice into the user's language using a global API or other translation services.

[1507] Device: The device provides the translated information to the user in both voice and text. It uses speech synthesis technology to speak navigation advice in the user's language and simultaneously displays the translated text on the screen.

[1508] 3. Image Recognition Technology

[1509] Device: The device uses the smartphone camera to capture real-time images of the surroundings, which are then analyzed using image recognition algorithms (e.g., OpenCV or TensorFlow Lite).

[1510] Device: The device recognizes traffic signs and pedestrians from the captured video. The recognized information is used to assist the user in driving.

[1511] Server: The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time.

[1512] Terminal: The terminal notifies the user of advice from the server, for example providing audio and visual alerts to the user if they exceed the speed limit or if there are pedestrians ahead.

[1513] 4. Emotion Engine

[1514] Device: The device uses a smartphone or in-car camera to recognize emotions from the user's facial expressions, tone of voice, heart rate, etc. This emotional data is collected and evaluated in real time.

[1515] Server: The server analyzes the emotion data sent from the emotion engine and evaluates the user's driving state, for example, to detect if the driver is nervous or tired.

[1516] Device: The device will adjust navigation advice based on the emotion engine's analysis. For example, if the user is nervous, it will provide calming advice or relaxing music.

[1517] Server: The server reconstructs a navigation plan according to the user's emotional state and sends it to the device, thereby providing the optimal route and advice for the user's mental state.

[1518] Device: The device provides users with navigation advice and music tailored by the emotion engine, thereby reducing the user's driving stress and supporting safe driving.

[1519] 5. Specific Examples

[1520] Example 1: When a user sets a destination, the device sends current location and destination information to the server and receives the optimal route. The emotion engine analyzes the user's facial expressions and tone of voice to evaluate their state of mind to drive safely. If the user is nervous, the device plays relaxing music and provides gentle voice guidance.

[1521] Example 2: When a user is driving for a long time, the emotion engine detects fatigue and notifies the server. The server recalculates the route and suggests places where users can take a break. The device notifies the user by voice, saying, "There is a rest area ahead. Would you like to take a short break?"

[1522] Example 3: For users who are nervous, the device provides information on recognized traffic signs and pedestrians in real time. For example, if the camera recognizes a "Stop" sign, the device will give a voice command saying, "There is a stop sign. Please stop," giving the user a sense of security.

[1523] As described above, this system combines real-time traffic information, multilingual support, image recognition technology, and an emotion engine to provide users with a safe and comfortable driving experience. By appropriately assessing the user's emotional state and adjusting navigation advice based on that assessment, the system achieves a less stressful driving experience.

[1524] The processing flow will be explained below.

[1525] Program processing flow and specific operations (including emotion engine)

[1526] 1. Providing real-time traffic information

[1527] Step 1: Collecting traffic information

[1528] Server: "The server collects traffic data in real time via API from road traffic management centers and cloud services."

[1529] Specific operation: Data such as traffic congestion, accidents, and road construction information is obtained using an API and stored in a database.

[1530] Step 2: Traffic information integration and analysis

[1531] Server: "The server integrates the collected traffic data into a database and uses machine learning algorithms to calculate the optimal route."

[1532] What it does: Analyzes traffic information data and calculates the optimal route to the user's destination.

[1533] Step 3: Distributing traffic information

[1534] Server: "The server sends the analyzed route information to the user terminal in real time."

[1535] Specific operation: The calculated optimal route is sent to the user's device via an HTTP request.

[1536] Step 4: Viewing information

[1537] Terminal: "The terminal displays real-time traffic information received from the server on a map."

[1538] Specific behavior: Uses OpenStreetMap and other map APIs to visually display received route information on a map.

[1539] 2. Multilingual support

[1540] Step 1: Get the language settings

[1541] Device: "The device acquires the language information set by the user and sends it to the server."

[1542] Specific behavior: Get the user's language preference from the app's settings menu and send it to the server.

[1543] Step 2: Performing the translation

[1544] Server: "The server translates traffic information and navigation advice into the user's language."

[1545] What it does: Uses a translation API to translate collected traffic information and navigation advice into the user's language of choice.

[1546] Step 3: Providing multilingual guidance

[1547] Device: "The device provides multilingual, translated information to the user via voice and text."

[1548] Specific operation: Using speech synthesis technology, translated navigation information is output as voice and simultaneously displayed as text on the screen.

[1549] 3. Image Recognition Technology

[1550] Step 1: Acquire camera footage

[1551] Device: "The device captures video from the smartphone camera and processes it for image recognition."

[1552] Specific operation: Uses the camera API to capture real-time video and store it in memory.

[1553] Step 2: Perform image recognition

[1554] Device: "The device applies image recognition algorithms to the captured video to detect traffic signs and pedestrians."

[1555] Specific operation: Using OpenCV and TensorFlow Lite, it analyzes signs and pedestrians in video and detects specific features.

[1556] Step 3: Sending the recognition results

[1557] Device: "The device sends the recognition results to the server."

[1558] Specific operation: Organizes the recognized information and sends it to the server via API.

[1559] Step 4: Generate feedback

[1560] Server: "The server analyzes the recognition results and generates advice and warnings for safe driving."

[1561] What it does: Evaluates the received recognition data and generates the necessary feedback.

[1562] Step 5: Deliver feedback to users

[1563] Terminal: "The terminal notifies the user of advice and warnings from the server."

[1564] Specific behavior: Displays a warning message on the screen and issues an audio alert.

[1565] 4. Emotion Engine

[1566] Step 1: Collecting emotion data

[1567] Device: "The device uses a smartphone or in-car camera to capture emotional data such as the user's facial expressions, tone of voice, and heart rate."

[1568] What it does: It uses facial recognition APIs, voice analysis APIs, and heart rate sensors to capture emotion-related data and evaluate it in real time.

[1569] Step 2: Sending emotion data

[1570] Device: "The device sends the collected emotion data to the server."

[1571] Specific operation: Organize emotion data and send it to the server via API.

[1572] Step 3: Analyze the sentiment data

[1573] Server: "The server uses the emotion engine to analyze the received emotion data and evaluate the user's driving state."

[1574] What it does: It uses analytics algorithms to identify the user's emotional state, such as whether they are nervous or tired.

[1575] Step 4: Adjust navigation based on emotions

[1576] Device: "The device will adjust its navigation advice based on the emotion engine's analysis."

[1577] Specific actions: For example, if the user is nervous, give gentle advice and play relaxing music.

[1578] Step 5: Emotionally recalculated route

[1579] Server: "The server reconstructs the optimal navigation plan based on the user's emotional state and sends it to the device."

[1580] What it does: Calculates a new route and provides the best route for the user's current mental state.

[1581] Step 6: Provide emotion-based feedback

[1582] Device: "The device provides users with navigation advice and music tailored by the emotion engine."

[1583] Specific behavior: Provides optimal feedback to the user through voice notifications and on-screen displays, supporting stress-free driving.

[1584] In this way, this system combines real-time traffic information, multilingual support, image recognition technology, and an emotion engine to provide users with a safe and comfortable driving experience. By properly recognizing the user's emotional state and adjusting navigation advice accordingly, it further reduces driving stress and supports safe driving.

[1585] Example 2

[1586] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1587] In today's traffic environment, drivers are required to select the optimal route based on traffic information updated in real time. However, systems that do not properly reflect traffic information, lack multilingual support, or do not take the driver's emotional state into account make it difficult to support safe and comfortable driving. This raises concerns about increased traffic accidents and driving stress. Therefore, the present invention aims to solve these issues and provide drivers with a safe and comfortable driving experience.

[1588] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1589] In this invention, the server includes means for collecting traffic information in real time from road traffic management agencies and cloud services, means for analyzing the collected traffic information using a machine learning algorithm to calculate an optimal route, and means for transmitting the analysis results to a user terminal. This allows the user to receive optimal navigation based on the latest traffic conditions. It also includes means for recognizing road signs and pedestrians using a camera and means for notifying the driver of warnings and advice based on the recognition results. This allows the user to receive appropriate warnings and advice in real time while driving. Furthermore, the system also includes means for collecting the user's facial expressions, tone of voice, and heart rate to analyze the user's emotional state, and means for adjusting navigation advice based on the emotional state, thereby supporting safe and comfortable driving according to the driver's emotional state.

[1590] "Real-time traffic information" refers to information that is obtained immediately after a traffic situation occurs, and includes current road conditions, congestion, accident information, road construction information, and the like.

[1591] A "machine learning algorithm" is a computational method that learns patterns from data and makes future decisions and predictions based on the results of that learning.

[1592] The "optimal route" is the route determined to be the most efficient from the user's starting point to the destination, taking into consideration factors such as time, distance, and traffic conditions.

[1593] A "user terminal" is an information terminal used by a driver, and includes a smartphone, tablet, in-vehicle display, etc.

[1594] "Visually displaying" means presenting information in a graphical format on a display or screen, including displays using maps, icons, text, etc.

[1595] A "camera" is a device used to capture images, such as a camera built into a smartphone or a camera attached to a vehicle.

[1596] "Road signs" are signs that indicate traffic rules and road conditions, and include stop signs, speed limit signs, no entry signs, and the like.

[1597] "Pedestrian" refers to a person crossing a road or walking on the side of the road.

[1598] "Emotional state" refers to the driver's psychological and emotional state, including states such as tension, fatigue, and relief.

[1599] "Navigation advice" is route guidance information provided to a user, including instructions in audio or text format.

[1600] "Warnings and advice" refers to alerts and instructions for actions to the user, including audible and visual alerts to warn of danger, and advice to encourage improved driving.

[1601] The present invention provides a system that collects traffic information in real time, provides the information in multiple languages, and recognizes the emotional state of the driver to support safe and comfortable driving. Detailed embodiments of the system are described below.

[1602] 1. Providing real-time traffic information

[1603] Server: The server collects real-time traffic data from road traffic management agencies and cloud services via API. The collected data includes information on traffic congestion, accidents, road construction, etc., and consolidates this information and stores it in a database.

[1604] Example: A server sends an API request every minute to get new traffic data and automatically stores the data in a database.

[1605] Server: The server analyzes the collected traffic information using machine learning algorithms to calculate the optimal route, using machine learning libraries such as TensorFlow and Scikit-learn.

[1606] Example: The server takes the user's current location and destination as input, runs an algorithm to calculate the optimal route, and stores the results in a database.

[1607] Server: The server sends the calculation results to the user's device in real time.

[1608] Example: The server sends a push notification to the user's device, providing a data packet containing optimal route information.

[1609] Terminal: The terminal visually displays the traffic information received from the server on a map and suggests the optimal route to the user. The map is displayed using OpenStreetMap (OSM) or Google Maps API.

[1610] Example: The device uses the Google Maps API to display a map and highlight the best route.

[1611] 2. Multilingual support

[1612] Terminal: The terminal acquires the language information set by the user and sends it to the server.

[1613] Example: A device reads the language information selected in the language setting menu and sends that information to a server.

[1614] Server: The server translates traffic information and navigation advice into the user's language using Google Cloud Translation API and DeepL.

[1615] Example: The server sends the acquired traffic information to a translation service via API and receives the translation results.

[1616] Terminal: The terminal provides the translated information to the user in voice and text format. It uses speech synthesis technology to convert the translated information into voice and display it on the display.

[1617] Example: The device starts a speech synthesis engine, converts the translated text into speech, and outputs it through the speaker, while displaying the text on the display.

[1618] 3. Image Recognition Technology

[1619] Device: The device uses the smartphone camera to capture real-time images of the surroundings, which are then analyzed using image recognition algorithms (e.g., OpenCV, TensorFlow Lite).

[1620] Example: The device activates the camera and starts the video stream.

[1621] Device: The device uses image recognition algorithms to recognize traffic signs and pedestrians from the captured video.

[1622] Example: The device runs image recognition algorithms on each video frame to detect signs and pedestrians.

[1623] Server: The server analyzes the recognition data sent from the device and generates warnings and advice for the driver.

[1624] Example: The server analyzes the received data and, if necessary, generates warning messages or driving advice and sends them to the terminal.

[1625] Terminal: The terminal notifies the user of advice from the server, for example providing audio and visual alerts to the driver if the speed limit is exceeded or if there is a pedestrian ahead.

[1626] Example: A device uses a speech synthesis engine to generate a warning message and output it over the speaker, and also displays the warning text on the screen.

[1627] 4. Emotion Engine

[1628] Device: The device uses a smartphone or in-car camera and microphone to capture the user's facial expressions, tone of voice, heart rate, and other data in real time.

[1629] Example: The device activates the camera and microphone to collect facial expressions, voice data, and data from a heart rate monitor.

[1630] Server: The server analyzes the emotion data sent from the emotion engine and evaluates the user's driving state, for example, detecting whether the user is nervous or tired.

[1631] Example: The server uses a machine learning model to analyze emotion data and evaluate the driver's state.

[1632] Device: The device will adjust its navigation advice based on the emotion engine's analysis.

[1633] Example: Based on the analysis received from the server, the device adjusts the settings of the navigation system, plays relaxing music, or provides gentle voice guidance.

[1634] Server: The server reconstructs the navigation plan according to the user's emotional state and sends it to the device.

[1635] Example: The server creates a recalculated navigation plan based on the results of an evaluation of the user's emotional state and sends it to the device.

[1636] Device: The device provides users with navigation advice and music tailored based on the emotion engine, thereby supporting safe driving.

[1637] Example: The device plays relaxing music selected from a playlist and outputs gentle voice prompts through the speaker while displaying instructions on the screen.

[1638] Prompt Sentence Examples

[1639] 1. Example 1: When a user sets a destination, the device sends current location and destination information to the server and receives the optimal route. The emotion engine analyzes the user's facial expressions and tone of voice to evaluate their level of confidence in driving. If the user is nervous, the device plays relaxing music and provides gentle voice guidance.

[1640] Example prompt: "Describe a scenario in which a user receives the best route to a destination and the emotion engine suggests relaxing music."

[1641] 2. Example 2: When a user is driving for a long time, the emotion engine detects fatigue and notifies the server. The server recalculates the route and suggests places where you can take a break. The device notifies the user by voice, "There is a rest area up ahead. Would you like to take a short break?"

[1642] Example prompt: "Describe a scenario where a user is driving for a long period of time, the emotion engine detects their fatigue, and the server suggests places where they can take a break."

[1643] 3. Example 3: For users who are nervous, the device provides information on recognized traffic signs and pedestrians in real time. For example, if the camera recognizes a "Stop" sign, the device will give a voice command saying, "There is a stop sign. Please stop," giving the user a sense of security.

[1644] Example prompt: "Describe a scenario in which a user feels anxious and is provided with real-time information about recognized traffic signs and pedestrians to promote safety."

[1645] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1646] Step 1: Collecting traffic data

[1647] Server: The server collects traffic data from road traffic management agencies and cloud services via API.

[1648] Input: API request

[1649] Data processing: The acquired data is analyzed in JSON format to extract information such as traffic congestion, accidents, and road construction information.

[1650] Output: Traffic information stored in a database

[1651] What it does: The server sends an API request every minute to get new traffic data, parses the data, and stores it in a database.

[1652] Step 2: Analyze the collected data

[1653] Server: The server uses machine learning algorithms to analyze the collected traffic information and calculate the optimal route.

[1654] Input: Traffic information data and user's current location and destination

[1655] Data processing: Calculate the optimal route using machine learning models (e.g., TensorFlow, Scikit-learn).

[1656] Output: Optimal route information

[1657] Specific operation: The server receives the user's current location and destination as input, calculates the optimal route through an algorithm, and stores the results in a database.

[1658] Step 3: Sending route information

[1659] Server: The server sends the calculation results to the user's device in real time.

[1660] Input: Optimal route information

[1661] Data processing: None

[1662] Output: Route information sent to the user terminal

[1663] Specific operation: The server sends a push notification to the user's device and provides a data packet containing optimal route information.

[1664] Step 4: View traffic information

[1665] Terminal: The terminal visually displays the traffic information received from the server on a map and presents the optimal route to the user.

[1666] Input: Route information sent

[1667] Data processing: Analyze the received data and display it on a map using a map API (e.g., OpenStreetMap, Google Maps API).

[1668] Output: Visual representation of the route

[1669] Specific operation: The device uses the map API to display a map and highlight the optimal route.

[1670] Step 5: Get and send language settings

[1671] Terminal: The terminal obtains the language information set by the user and sends it to the server.

[1672] Input: User's language preference

[1673] Data processing: None

[1674] Output: Language preference information sent to the server

[1675] Specific operation: The device reads the language information selected in the language setting menu and sends that information to the server.

[1676] Step 6: Translating traffic information and advice

[1677] Server: The server translates traffic information and navigation advice into the user's language.

[1678] Input: Traffic and navigation advice, language preference information

[1679] Data processing: Translation is performed using a translation API (e.g., Google Cloud Translation API, DeepL).

[1680] Output: Translated traffic information and navigation advice

[1681] Specific operation: The server sends the acquired traffic information and navigation advice to the translation service via API and receives the translation results.

[1682] Step 7: Provide translation information

[1683] Terminal: The terminal provides the translated information to the user in voice and text.

[1684] Input: Translated traffic and navigation advice

[1685] Data processing: Convert the data into speech using speech synthesis technology (e.g., speech synthesis engine), and display the text on the screen.

[1686] Output: Spoken and textual translation information

[1687] What happens: The device will launch its speech synthesis engine, convert the translated text into speech and output it through the speaker, and display the text on the display.

[1688] Step 8: Acquire and analyze footage

[1689] Device: The device uses the smartphone camera to capture real-time images of the surrounding area and analyzes them using image recognition algorithms.

[1690] Input: Camera image

[1691] Data processing: Run an image recognition algorithm (e.g., OpenCV, TensorFlow Lite) on each video frame to detect signs and pedestrians.

[1692] Output: Recognized traffic signs and pedestrian information

[1693] What happens: The device turns on the camera and starts streaming video. It then performs image recognition on each video frame to detect signs and pedestrians.

[1694] Step 9: Sending Recognition Data

[1695] Device: The device sends the recognized traffic signs and pedestrian information to the server.

[1696] Input: Recognized signs and pedestrian information

[1697] Data processing: None

[1698] Output: Identification information sent to the server

[1699] Specific operation: The device includes the recognized information in a data packet and sends it to the server.

[1700] Step 10: Generate warnings and advice

[1701] Server: The server analyzes the recognition data sent from the device and generates real-time warnings and advice for the driver.

[1702] Input: Recognition data

[1703] Data processing: Generate warnings and advice based on the recognition data.

[1704] Output: any warnings or advice generated

[1705] Specific operation: The server analyzes the received data and, if necessary, generates warning messages or driving advice and sends them to the terminal.

[1706] Step 11: Notification of warnings and advice

[1707] Terminal: The terminal notifies the user of warnings and advice from the server.

[1708] Input: Warnings and advice

[1709] Data processing: Converts data into speech using speech synthesis technology and displays the text on the screen.

[1710] Output: Audio and visual warnings and advice

[1711] What happens: The device uses a speech synthesis engine to generate a warning message and outputs it over the speaker, and also displays the warning text on the screen.

[1712] Step 12: Obtaining Emotion Data

[1713] Device: The device uses a smartphone, in-car camera, or microphone to capture the user's facial expressions, tone of voice, heart rate, and other information in real time.

[1714] Input: Camera video and audio data, heart rate monitor data

[1715] Data processing: Analyze the acquired data and identify the emotional state.

[1716] Output: Emotional state data

[1717] Specific operation: The device activates the camera and microphone to collect facial expressions, voice data, and heart rate monitor data.

[1718] Step 13: Analyze the sentiment data

[1719] Server: The server analyzes the emotion data sent from the emotion engine and evaluates the user's driving state.

[1720] Input: Emotional state data

[1721] Data processing: Using machine learning models to analyze emotion data and assess the driver's state.

[1722] Output: Driving condition evaluation data

[1723] Specific operation: The server uses a machine learning model to analyze emotion data and evaluate the driver's state.

[1724] Step 14: Adjusting Navigation Advice

[1725] Device: The device will adjust its navigation advice based on the emotion engine's analysis.

[1726] Input: Driving condition evaluation data

[1727] Data manipulation: Adjust navigation settings, change tips and music.

[1728] Output: Adjusted navigation advice

[1729] What it does: Based on the analysis received from the server, the device adjusts navigation settings, plays relaxing music, and provides gentle voice advice.

[1730] Step 15: Reconstructing the navigation plan

[1731] Server: The server reconstructs the navigation plan according to the user's emotional state and sends it to the device.

[1732] Input: Emotional state assessment data

[1733] Data transformation: Generate a new navigation plan.

[1734] Output: Reconstructed navigation plan

[1735] Specific operation: The server generates a recalculated navigation plan based on the results of the evaluation of the user's emotional state and sends it to the device.

[1736] Step 16: Offering advice and music

[1737] Device: The device provides users with navigation advice and music tailored to their needs based on the emotion engine, thereby supporting safe driving.

[1738] Input: Reconstructed navigation plan

[1739] Data processing: Provides voice advice and music based on the navigation plan.

[1740] Output: Audio advice and music provided

[1741] What it does: The device plays relaxing music selected from a playlist, emits gentle voice prompts through the speaker, and displays instructions on the screen.

[1742] (Application example 2)

[1743] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1744] In modern society, traffic congestion and accidents occur on a daily basis, and drivers need real-time traffic information. However, there have been no navigation systems that support multiple languages ​​or take into account the driver's emotional state. In addition, driver stress and fatigue can affect driving safety, and an effective system to mitigate these effects has been sought. Therefore, an object of the present invention is to provide a system that provides real-time traffic information and evaluates the driver's emotional state to improve driving safety and comfort.

[1745] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1746] In this invention, the server includes a means for collecting traffic information in real time, a means for analyzing the collected traffic information and calculating the optimal route, and a means for transmitting the analysis results to a user terminal. This allows the driver to receive optimal navigation based on the latest traffic conditions. Furthermore, by adding a means for collecting and recognizing the driver's emotional data, such as facial expressions, tone of voice, and heart rate, it is possible to provide relaxing music or gentle voice guidance when the driver is feeling tense or tired. Furthermore, by combining a means for recognizing road signs and pedestrians using a camera and notifying the user terminal of the recognition results, safety can be further improved.

[1747] "Real-time" refers to a method of processing data and providing results in accordance with the actual progress of time.

[1748] "Traffic information" refers to information relating to public transportation and vehicle traffic, such as road congestion, accidents, and construction work.

[1749] The term "route" refers to the route or path from a starting point to a destination.

[1750] A "user terminal" is an electronic device carried by a user, such as a smartphone, tablet, or in-vehicle terminal.

[1751] "Visually displaying" means showing data or information on a screen in the form of images or text.

[1752] A "camera" is a device for taking images or videos.

[1753] "Road signs" are signs installed to inform drivers and pedestrians of road safety and rules.

[1754] "Pedestrians" are people who walk on the road.

[1755] "Facial expressions" are facial muscle movements and expressions that indicate a person's emotional state.

[1756] "Tone of voice" refers to the timbre or tone of spoken language.

[1757] "Heart rate" is the number of times the heart beats within a certain period of time.

[1758] "Emotional data" means data collected to indicate the emotional state of a driver.

[1759] "Emotion recognition" means analyzing and evaluating a person's emotional state.

[1760] "Navigation advice" is advice to guide the driver to appropriate routes and important points to note.

[1761] "Music" refers to an art form that combines rhythm and melody of sounds.

[1762] "Voice guidance" refers to providing instructions and information audibly through a speaker.

[1763] "Means" refers to the tools, methods, and equipment used to achieve a specific purpose.

[1764] The present invention is a system that collects traffic information in real time and provides that information in multiple languages, and by combining it with an emotion engine, recognizes the driver's emotional state and supports safe and comfortable driving. An embodiment of the system will be described in detail below.

[1765] 1. Providing real-time traffic information

[1766] The server collects traffic data in real time via APIs from road traffic management centers, cloud services, etc. The collected data includes information on traffic congestion, accidents, road construction, etc., and consolidates this information and stores it in a database. The server analyzes the collected traffic information and calculates the optimal route. A machine learning algorithm is used for the calculation, generating the optimal route based on the user's current location and destination. The server sends the analysis results to the user's device in real time, allowing the user to receive navigation based on the latest traffic conditions. The user's device visually displays the traffic information received from the server on a map and provides the user with the optimal route. The map is displayed using OpenStreetMap or other map APIs.

[1767] 2. Multilingual support

[1768] The user device obtains the language information set by the user. This information is sent to the server, which applies different language settings to each user. The server translates traffic information and navigation advice into the user's language using a global API or other translation service. The user device then provides the translated information to the user in voice and text. Using speech synthesis technology, the navigation advice is output as voice in the user's language. At the same time, the translated information is displayed in text on the screen.

[1769] 3. Image Recognition Technology

[1770] The user device uses the smartphone camera to capture images of the surroundings in real time. This image is analyzed using an image recognition algorithm (e.g., OpenCV or TensorFlow Lite). The user device recognizes traffic signs and pedestrians from the captured image. The recognized information is used to assist the user's driving. The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time. The user device notifies the user of the advice from the server. For example, if the speed limit is exceeded or if a pedestrian is ahead, the user is provided with audio and visual alerts.

[1771] 4. Emotion Engine

[1772] The user device recognizes emotions from the user's facial expressions, tone of voice, heart rate, etc. using a smartphone or in-vehicle camera. This emotional data is collected and evaluated in real time. The server analyzes the emotional data sent from the emotion engine and evaluates the user's driving state. For example, it detects if the driver is nervous or tired. The user device adjusts navigation advice based on the emotion engine's analysis results. For example, if the user is nervous, it provides calming advice or relaxing music. The server reconstructs a navigation plan according to the user's emotional state and sends it to the device. This makes it possible to provide routes and advice optimal for the user's mental state. The user device provides the user with navigation advice and music adjusted by the emotion engine. This reduces the user's driving stress and supports safe driving.

[1773] 5. Specific Examples

[1774] Example 1: When a user sets a destination, the device sends current location and destination information to the server and receives the optimal route. The emotion engine analyzes the user's facial expressions and tone of voice to evaluate their state of mind to drive safely. If the user is nervous, the device plays relaxing music and provides gentle voice guidance.

[1775] Example 2: When a user is driving for a long time, the emotion engine detects fatigue and notifies the server. The server recalculates the route and suggests places where users can take a break. The device notifies the user by voice, saying, "There is a rest area ahead. Would you like to take a short break?"

[1776] Example 3: For users who are nervous, the device provides information on recognized traffic signs and pedestrians in real time. For example, if the camera recognizes a "Stop" sign, the device will give a voice command saying, "There is a stop sign. Please stop," to reassure the user.

[1777] The overall processing flow of this system involves the server and user terminal working together to provide traffic information and advice based on the driver's emotional state, thereby achieving safe and comfortable driving.

[1778] Example prompt: "Describe an application that recognizes a driver's emotions based on their facial expression, tone of voice, and heart rate to help them drive safely."

[1779] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1780] Step 1:

[1781] The server collects traffic data in real time via API from road traffic management centers, cloud services, etc. The input is traffic information (traffic congestion, accidents, road construction information, etc.) obtained from road traffic management centers and cloud services. This information is stored in a database and processed into a format that can be matched with the user's current location and destination information. The output is integrated traffic information data.

[1782] Step 2:

[1783] The server analyzes the collected traffic information using machine learning algorithms and calculates the optimal route. The inputs are the collected traffic information, the user's current location, and destination information. The server performs data calculations to generate the optimal route. The output is the optimized route information.

[1784] Step 3:

[1785] The server transmits the analysis result, the optimal route information, to the user terminal in real time. The input is the optimized route information, and the output is the navigation data transmitted to the user terminal.

[1786] Step 4:

[1787] The user device visually displays the optimal route information received from the server on a map. It uses OpenStreetMap and other map APIs to obtain map data and display the optimal route to the user. The input is the route information sent from the server and the map data obtained from the map API, and the output is a visual map display.

[1788] Step 5:

[1789] The user device uses the smartphone camera to capture images of the surroundings in real time and uses image recognition algorithms (OpenCV or TensorFlow Lite) to recognize traffic signs and pedestrians. The input is image data from the camera, which is analyzed using image recognition. The output is recognized traffic signs and pedestrian information.

[1790] Step 6:

[1791] The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. The input is traffic signs and pedestrian information, and by processing the data, it generates information that the driver should pay attention to. The output is a warning or advice sent to the user's device.

[1792] Step 7:

[1793] The user terminal notifies the user of warnings and advice from the server in real time. The input is the warnings and advice sent from the server and provided to the user as audio and visual alerts. The output is the notification of the warnings and advice given to the user.

[1794] Step 8:

[1795] The user device collects emotion data from the user's facial expressions, tone of voice, heart rate, etc. using a smartphone or in-car camera, and analyzes it with an emotion engine. The input is the user's facial expressions, tone of voice, and heart rate data, which are analyzed by the emotion engine. The output is the recognized emotional state.

[1796] Step 9:

[1797] The server analyzes the emotion data sent from the emotion engine and evaluates the user's driving state. The input is the emotional state data, and the server performs data calculations to evaluate the user's driving state. The output is navigation advice and music suggestions based on the emotional state to the user's device.

[1798] Step 10:

[1799] The user device provides navigation advice and relaxing music based on the analysis results of the emotion engine. The input is navigation advice and music data based on the emotional state, which are provided to the user visually and audibly. The output is support for safe and comfortable driving for the user.

[1800] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1801] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1802] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1803] [Fourth embodiment]

[1804] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1805] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1806] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1807] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1808] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1809] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1810] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1811] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1812] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1813] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1814] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1815] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1816] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1817] The present invention provides a system that supports safe and intuitive driving by collecting traffic information in real time, providing that information in multiple languages, and detecting traffic signs and pedestrians using image recognition technology. An embodiment of the system will be described in detail below.

[1818] 1. Providing real-time traffic information

[1819] Server: The server collects real-time traffic information from road traffic management centers, cloud services, etc. The collected data includes information on traffic congestion, accidents, road construction, etc., and integrates this information and stores it in a database.

[1820] Server: The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms to generate the optimal route based on the user's current location and destination.

[1821] Server: The server transmits the analysis results to the user's device in real time, allowing the user to receive navigation based on the latest traffic conditions.

[1822] Terminal: The terminal visually displays the traffic information received from the server on a map and provides the user with the optimal route. The map is displayed using OpenStreetMap or other map APIs.

[1823] 2. Multilingual support

[1824] Device: The device retrieves the user's language preference, which is then sent to the server, allowing different language settings to be applied for each user.

[1825] Server: The server translates traffic information and navigation advice into the user's language using a global API or other translation services.

[1826] Device: The device provides the translated information to the user in both voice and text. It uses speech synthesis technology to speak navigation advice in the user's language and simultaneously displays the translated text on the screen.

[1827] 3. Image Recognition Technology

[1828] Device: The device uses the smartphone camera to capture real-time images of the surroundings, which are then analyzed using image recognition algorithms (e.g., OpenCV or TensorFlow Lite).

[1829] Device: The device recognizes traffic signs and pedestrians from the captured video. The recognized information is used to assist the user in driving.

[1830] Server: The server analyzes the recognition data sent from the device and generates warnings and advice for the driver. This advice is sent to the device in real time.

[1831] Terminal: The terminal notifies the user of advice from the server, for example providing audio and visual alerts to the user if they exceed the speed limit or if there are pedestrians ahead.

[1832] 4. Specific Examples

[1833] Example 1: The user sets a destination. The device sends current location and destination information to the server and receives the optimal route. The device displays the route on a map and provides route guidance based on traffic information updated in real time. If a traffic jam occurs along the way, the server immediately calculates a new route and sends updated information to the device. As a result, the user can reach the destination smoothly.

[1834] Example 2: When a user is driving in a foreign country, the device provides navigation advice in the user's language. The server translates collected traffic information and recognition results into the user's native language and provides them via voice and text through the device. This allows the user to drive with confidence even in a different language environment.

[1835] Example 3: When the device's camera recognizes a road sign while driving, the information is sent to the server and the necessary warning is generated. For example, if the device recognizes a "Stop" sign, the server analyzes it and sends a warning to the device. The device will notify the user by voice, "There is a stop sign ahead," to help the user take appropriate action.

[1836] As described above, this system provides users with the latest traffic information in real time, supports the driving environment with multilingual support, and uses image recognition technology to support safe driving, thereby reducing driving anxiety and providing a comfortable driving experience.

[1837] The processing flow will be explained below.

[1838] Program processing flow and specific operations

[1839] 1. Providing real-time traffic information

[1840] Step 1: Collect traffic information

[1841] Server: "The server collects traffic data in real time via API from road traffic management centers and cloud services."

[1842] Specific operation: Data such as traffic congestion, accidents, and road construction information is obtained using an API and stored in a database.

[1843] Step 2: Traffic information integration and analysis

[1844] Server: "The server integrates the collected traffic data into a database and uses machine learning algorithms to calculate the optimal route."

[1845] What it does: Analyzes traffic information data and calculates the optimal route to the user's destination.

[1846] Step 3: Distributing traffic information

[1847] Server: "The server sends the analyzed route information to the user terminal in real time."

[1848] Specific operation: The calculated optimal route is sent to the user's device via an HTTP request.

[1849] Step 4: View information

[1850] Terminal: "The terminal displays real-time traffic information received from the server on a map."

[1851] Specific behavior: Uses OpenStreetMap and other map APIs to visually display received route information on a map.

[1852] 2. Multilingual support

[1853] Step 1: Get the language settings

[1854] Device: "The device acquires the language information set by the user and sends it to the server."

[1855] Specific behavior: Get the user's language preference from the app's settings menu and send it to the server.

[1856] Step 2: Perform the translation

[1857] Server: "The server translates traffic information and navigation advice into the user's language."

[1858] What it does: Uses a translation API to translate collected traffic information and navigation advice into the user's language of choice.

[1859] Step 3: Providing multilingual guidance

[1860] Device: "The device provides multilingual, translated information to the user via voice and text."

[1861] Specific operation: Using speech synthesis technology, translated navigation information is output as voice and simultaneously displayed as text on the screen.

[1862] 3. Image Recognition Technology

[1863] Step 1: Acquire camera footage

[1864] Device: "The device captures video from the smartphone camera and processes it for image recognition."

[1865] Specific operation: Uses the camera API to capture real-time video and store it in memory.

[1866] Step 2: Perform image recognition

[1867] Device: "The device applies image recognition algorithms to the captured video to detect traffic signs and pedestrians."

[1868] Specific operation: Using OpenCV and TensorFlow Lite, it analyzes signs and pedestrians in video and detects specific features.

[1869] Step 3: Sending the recognition results

[1870] Device: "The device sends the recognition results to the server."

[1871] Specific operation: Organizes the recognized information and sends it to the server via API.

[1872] Step 4: Generate feedback

[1873] Server: "The server analyzes the recognition results and generates advice and warnings for safe driving."

[1874] What it does: Evaluates the received recognition data and generates the necessary feedback.

[1875] Step 5: Deliver feedback to users

[1876] Terminal: "The terminal notifies the user of advice and warnings from the server."

[1877] Specific behavior: Displays a warning message on the screen and issues an audio alert.

[1878] 4. User Interface Operation and Feedback

[1879] Step 1: Set your destination

[1880] User: "The user enters and sets a destination in the application."

[1881] What it does: Select a destination on the map or enter text in the search bar to specify a destination.

[1882] Step 2: Start route guidance

[1883] Terminal: "The terminal starts navigation based on the route information received from the server."

[1884] What it does: Highlights the route on the map and starts voice guidance.

[1885] Step 3: Receive real-time feedback

[1886] Server: "The server monitors the user's real-time movements and updates the route as needed."

[1887] Specific operation: Constantly monitors the user's current location based on GPS information and recalculates the route according to changes in traffic conditions.

[1888] Step 4: Real-time notifications

[1889] Terminal: "The terminal notifies the user of updates from the server in real time."

[1890] Specific behavior: When a new route is provided, the user is immediately notified by map display and voice.

[1891] This enables the entire system to provide users with real-time, up-to-date and reliable traffic information and safe driving support.

[1892] Example 1

[1893] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1894] This invention relates to a system that efficiently collects and analyzes traffic information updated in real time and provides users with optimal routes. In particular, it aims to support safe and intuitive driving by recognizing the surrounding environment, such as traffic signs and pedestrians, from camera images and notifying the user of this information in multiple languages. Conventional systems have had issues with driver convenience and safety due to slow traffic information updates and insufficient multilingual support and real-time image recognition.

[1895] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1896] In this invention, the server includes a means for collecting traffic information in real time, a means for analyzing the collected traffic information and calculating an optimal route using a machine learning algorithm, and a means for transmitting the analysis results to a user terminal. This enables efficient and prompt provision of an optimal route based on the latest traffic information. The user terminal also includes a means for visually displaying the received traffic information on a map, a means for recognizing road signs and pedestrians using a camera, a means for transmitting the recognition results to the server and notifying the user terminal of analyzed warnings and advice, and a means for providing navigation advice using speech synthesis technology that supports multiple languages. This allows users to receive safe and intuitive driving support in multiple languages ​​in real time.

[1897] "Real-time" refers to processing and providing information instantly the moment it is generated.

[1898] "Traffic information" refers to information relating to traffic conditions on roads, including data on congestion, accidents, road construction, and the like.

[1899] A "machine learning algorithm" is a computational method that automatically learns patterns and rules by analyzing large amounts of data, and makes predictions and classifications.

[1900] "Optimal route" refers to the shortest and most efficient route for a user to reach a destination.

[1901] A "user terminal" is a device operated by a user, including a smartphone, tablet, car navigation system, etc.

[1902] "Visually displaying on a map" refers to visually displaying traffic and route information using a map application or navigation system.

[1903] "Camera" means a device for capturing video or images.

[1904] "Road sign" means a sign placed on a road to provide direction or information to motorists.

[1905] "Pedestrian" refers to a person who moves on foot on a road.

[1906] "Recognizing" refers to detecting and identifying specific objects (such as road signs or pedestrians) from the image captured by the camera.

[1907] "Analyze" refers to the detailed analysis of collected data and recognition results to derive meaningful information and predictions.

[1908] "Warning" means providing information to draw attention to a particular situation.

[1909] "Advice" means providing instructions or suggestions for specific action.

[1910] "Speech synthesis technology" is a technology that generates and plays back voices that resemble human voices from text data.

[1911] "Multilingual" means supporting multiple different languages.

[1912] "Translate" refers to converting text written in one language into another language.

[1913] The present invention provides a system that supports safe and intuitive driving by collecting traffic information in real time, providing that information in multiple languages, and detecting traffic signs and pedestrians using image recognition technology. An embodiment of the system will be described in detail below.

[1914] 1. Providing real-time traffic information

[1915] server:

[1916] The server continuously obtains real-time traffic information from the road traffic management center API or cloud services (e.g., general traffic information API), including information on traffic congestion, accidents, road construction, etc. The server stores this information in a database such as MongoDB and generates an integrated dataset.

[1917] server:

[1918] The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms (e.g., Scikit-learn or TensorFlow), and sends the results of this calculation to the user's device in real time.

[1919] Device:

[1920] The device visually displays the traffic information received from the server on a map and provides the user with the optimal route, using external services such as OpenStreetMap and Google Maps API.

[1921] 2. Multilingual navigation available

[1922] Device:

[1923] The terminal obtains the user's language setting and sends that information to the server.

[1924] server:

[1925] The server uses the Global Translate API (e.g., Google Translate API) to translate traffic information and navigation instructions into the user's preferred language.

[1926] Device:

[1927] The device provides the translated information to the user in real time using speech synthesis technology (e.g., Amazon Polly), and also displays the corresponding text information on a map.

[1928] 3. Traffic Sign and Pedestrian Detection Using Image Recognition Technology

[1929] Device:

[1930] The device uses its built-in camera to capture real-time images of its surroundings, which are then analyzed by image recognition algorithms (e.g., OpenCV or TensorFlow Lite) running on the device.

[1931] Device:

[1932] The device recognizes traffic signs and pedestrians in the video and sends the recognition results to the server in real time.

[1933] server:

[1934] The server analyzes the received recognition data and notifies the user's device of warnings and advice necessary for driving.

[1935] Device:

[1936] The device provides the user with audio and visual warnings and advice from the server, such as "There is a pedestrian ahead" or "You are exceeding the speed limit."

[1937] Specific examples

[1938] Example 1:

[1939] When a user inputs their current location and destination, the device sends the information to the server, which calculates the optimal route and sends it back to the device. The device then displays the route on a map and provides route guidance based on traffic information updated in real time. If traffic congestion occurs, the server calculates a new route and sends updated information to the device, allowing the user to reach their destination smoothly.

[1940] Example 2:

[1941] When a user is driving in a foreign country, the device will provide navigation advice in the user's language of choice. The server will translate traffic information and recognition results into the user's native language and provide them via voice and text via the device, allowing users to drive with confidence in different language environments.

[1942] Example 3:

[1943] When the device's camera recognizes a road sign while driving, the information is sent to the server, which generates the necessary warning. For example, if a "Stop" sign is recognized, the server analyzes it and sends a warning to the device. The device then notifies the user by voice, saying "There is a stop sign ahead," helping the user to take appropriate action.

[1944] Prompt Sentence Examples

[1945] 1. Give an example of a program that calculates the optimal route based on traffic information when a user inputs their current location and destination.

[1946] 2. Describe the process flow of a system that provides multilingual voice and text navigation based on real-time traffic data.

[1947] 3. Explain the process of a system that uses a smartphone camera and image recognition algorithms to detect traffic signs and pedestrians and alert the driver.

[1948] This invention allows users to utilize the latest traffic information in real time and easily implements a system that supports the driving environment in multiple languages, thereby enabling safe and efficient driving, reducing driving anxiety, and realizing a comfortable driving experience.

[1949] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1950] Step 1:

[1951] Real-time traffic information collection

[1952] Server: The server obtains real-time traffic information from the Road Traffic Management Center API and cloud services.

[1953] Input: Traffic information data received from the Road Traffic Management Center API and cloud services.

[1954] Data processing: The acquired traffic information is converted into JSON format and stored in a database.

[1955] Output: Updated traffic information dataset.

[1956] What it does: The server periodically sends API requests to retrieve new traffic information and stores it in MongoDB.

[1957] Step 2:

[1958] Traffic information analysis and optimal route calculation

[1959] Server: The server analyzes the collected traffic information and calculates the optimal route using machine learning algorithms (e.g., Scikit-learn, TensorFlow).

[1960] Input: Traffic information data stored in a database.

[1961] Data calculations: Using machine learning algorithms to calculate the optimal route based on the user's current location and destination.

[1962] Output: Optimal route information provided to the user.

[1963] Specific operation: The server applies Dijkstra's algorithm based on map data and traffic information to calculate the shortest route.

[1964] Step 3:

[1965] Sending optimal route information

[1966] Server: The server transmits the calculated optimal route information to the user's device in real time.

[1967] Input: Calculated optimal route information.

[1968] Data calculation: If necessary, convert the route information into JSON format so that it can be sent.

[1969] Output: Optimal route information sent to the user terminal.

[1970] Specific operation: The server sends JSON format data to the user terminal via the HTTP protocol.

[1971] Step 4:

[1972] Displaying traffic and route information

[1973] Terminal: The terminal visually displays the traffic and route information received from the server on a map.

[1974] Input: Traffic and route information received from the server.

[1975] Data processing: Data conversion to overlay information on a map.

[1976] Output: Map and route information visually displayed to the user.

[1977] Specific operation: The device displays a map using the Google Maps API and overlays the received route information on it.

[1978] Step 5:

[1979] Multilingual navigation provided

[1980] Device: The device obtains the user's language preference and sends it to the server.

[1981] Input: User's language preference information.

[1982] Output: Language preference data sent to the server.

[1983] Specific behavior: The device obtains the user's selected language from the language settings screen.

[1984] Server: The server translates traffic information and navigation instructions into the user's preferred language.

[1985] Input: Traffic information, navigation instructions, and user language preference data.

[1986] Data Calculation: Translate data into a specified language using the Global Translate API.

[1987] Output: Translated traffic and navigation instructions.

[1988] Specific operation: The server calls the Google Translate API to translate the data and sends the results to the device.

[1989] Terminal: The terminal synthesizes the translated information and provides it to the user.

[1990] Input: Translated traffic and navigation instructions.

[1991] Data processing: Generate voice data using a speech synthesis application (e.g., Amazon Polly).

[1992] Output: Audio and text information provided to the user.

[1993] Specific operation: The device uses Amazon Polly to play a voice message saying "Go straight to your destination" and also displays the text on the map.

[1994] Step 6:

[1995] Traffic sign and pedestrian detection using image recognition

[1996] Device: The device uses its built-in camera to capture real-time images of its surroundings.

[1997] Input: Video data captured by the camera.

[1998] Output: Video data that is passed to the image recognition algorithm.

[1999] Specific operation: The device starts the camera and acquires streaming video data.

[2000] Terminal: The terminal analyzes the video data using image recognition algorithms (e.g., OpenCV or TensorFlow Lite).

[2001] Input: Video data captured by the camera.

[2002] Data calculations: Image recognition algorithms identify traffic signs and pedestrians.

[2003] Output: Information on recognized traffic signs and pedestrians.

[2004] Specific operation: The device uses OpenCV to recognize stop signs and pedestrians.

[2005] Device: Sends the recognition results to the server.

[2006] Input: The recognized information.

[2007] Output: Recognition result data sent to the server.

[2008] Specific operation: The device converts the recognized data into JSON format and sends it to the server.

[2009] Server: The server analyzes the received recognition data and generates any necessary warnings or advice.

[2010] Input: Recognition data sent from the device.

[2011] Data calculations: Analyze perception data to generate warnings and advice.

[2012] Output: Warnings and advice sent to the user's terminal.

[2013] Specific operation: Based on the analysis results, the server generates a warning such as "There is a pedestrian ahead" and sends it to the device.

[2014] Terminal: The terminal notifies the user of warnings and advice from the server by voice and display.

[2015] Input: Any warnings or advice received from the server.

[2016] Output: The audio and visual information provided to the user.

[2017] Specific operation: The device will announce "There is a pedestrian ahead" and display an icon on the screen.

[2018] (Application example 1)

[2019] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2020] Conventional navigation systems only provide traffic information and are limited in their ability to respond to changes in traffic conditions in real time. Furthermore, they lack driving assistance in multilingual environments and the ability to recognize road signs and pedestrians in real time, making them inadequate for supporting safe driving. There is a need to solve these issues and provide safer and more efficient driving assistance.

[2021] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2022] In this invention, the server includes means for collecting traffic information in real time, means for analyzing the collected traffic information and calculating the optimal route, and means for transmitting the analysis results to the user terminal, thereby enabling the server to respond to changes in traffic conditions in real time and provide the optimal route.

[2023] Furthermore, this invention provides safe and intuitive driving assistance by incorp...

Claims

1. A means of collecting traffic information in real time; A means of analyzing the collected traffic information and calculating the optimal route; means for transmitting the analysis results to a user terminal; a means for visually displaying the received traffic information in the user terminal; A means of recognizing road signs and pedestrians using cameras, means for notifying a user terminal of the recognition result; A system including:

2. 10. The system of claim 1, further comprising means for translating and providing traffic information and navigation advice in multiple languages.

3. 10. The system of claim 1, further comprising means for providing translated information in audio and text based on a user-preferred language.

Images