system

Abstract

Provide a system. [Solution] a means for the in-vehicle terminal to collect data from a plurality of sensors in the vehicle and periodically transmit the data to a central server; a central server for analyzing the data in real time and generating an alert when an anomaly is detected; means for notifying a user of the alert to a mobile terminal and providing a countermeasure for the abnormality; a means of collecting user feedback and storing that data for analysis; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current vehicle trouble response systems often result in a delay between the time a driver recognizes a problem and the time it takes to address it. Another problem is that the information provided is insufficient, and specific troubleshooting methods and information on the nearest support service are not provided. Furthermore, there is a problem that the system is difficult to improve because a feedback loop after a problem occurs is not established. The present invention aims to solve these problems and provide a vehicle driving assistance system that can respond to problems more quickly and effectively. [Means for solving the problem]

[0005] The present invention relates to a system that includes an on-board device that collects data from multiple sensors in a vehicle, a central server that analyzes the collected data in real time and detects abnormalities, and a means for notifying the user's mobile device of countermeasures to the abnormality. Specifically, the on-board device collects a wide range of data, including GPS data, speed data, acceleration data, engine data, and tire pressure data, and periodically transmits it to the central server. The central server analyzes the collected data and generates an alert if an abnormality is detected. The generated alert is sent to the user's mobile device, providing specific details of the problem, how to deal with it, and information about the nearest support service. In addition, by collecting user feedback, storing this data, and using it for analysis, the system can be continuously improved. This allows users to respond to problems quickly and accurately.

[0006] An "on-board terminal" is a device installed in a vehicle that has the function of collecting data from multiple sensors and transmitting it to a central server.

[0007] A "sensor" is a device that detects specific physical or environmental conditions (e.g., GPS data, speed data, acceleration data, engine data, tire pressure data) and provides that information to an in-vehicle terminal.

[0008] "Data collection" refers to the act of an in-vehicle terminal collecting information obtained from sensors installed in the vehicle.

[0009] The "central server" is a computer system that receives data sent from the vehicle-mounted terminal, analyzes it in real time, and detects abnormalities.

[0010] "Real-time analysis" is a processing method in which data is analyzed and results are derived almost immediately after it is transmitted.

[0011] "Anomaly detection" is the act of a central server identifying data that deviates from normal ranges through real-time analysis.

[0012] An "alert" is a notification message generated by a central server when an abnormality is detected, and is information that warns the user and provides instructions on how to deal with the problem.

[0013] A "mobile terminal" is a portable communication device such as a smartphone or tablet that a user has.

[0014] "Countermeasures" is information indicating specific actions and procedures that a user should take in response to a detected abnormality.

[0015] "Feedback" refers to information such as opinions, impressions, and areas for improvement provided by system users after dealing with a problem.

[0016] "Support service information" is information about the nearest service station or repair shop that can be used when the user encounters a problem. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to a system for collecting data from a plurality of sensors in a vehicle, analyzing the data to detect abnormalities, and providing a user with a prompt and optimal countermeasure. Specific embodiments for carrying out the present invention will be described below.

[0039] An in-vehicle terminal is a device installed in a vehicle that collects data from GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, tire pressure sensors, etc. This data is sent to a central server at regular intervals. For example, every 5 seconds, the data from each sensor is collected into packets and sent to the server.

[0040] The central server receives data sent from the in-vehicle devices, analyzes the data in real time, and generates an alert if anomalies or irregular patterns are detected. For example, if the engine temperature significantly exceeds normal values, an alert will be generated indicating a possible overheating.

[0041] The generated alert is sent to the user's mobile device (such as a smartphone or tablet). By opening the application, the user can check the details of the alert. For example, specific countermeasures such as "The engine temperature is abnormally high. Immediately stop the engine and allow it to cool" are displayed.

[0042] The system also provides users with emergency assistance service information. Based on the user's current location, information about the nearest gas station or repair shop and directions to the nearest shop are displayed within the application. For example, specific directions such as "The nearest repair shop is 1.5 kilometers away. Please follow the route below" are provided.

[0043] After a user resolves a problem through the system, they will receive a notification requesting feedback. They can use the feedback form to express their thoughts and the effectiveness of the resolution. This feedback data will be stored on a central server and used to improve future prediction models and services.

[0044] As a concrete example, consider the case where a user experiences a drop in tire pressure while driving. The in-vehicle device detects abnormal data from the tire pressure sensor and sends it to a central server. The server immediately analyzes the abnormality, generates an alert stating "Tire pressure is below the specified value," and notifies the user's mobile device. The user confirms the alert and follows the directions to the nearest gas station. They then report their experience resolving the problem in a feedback form. This allows the system to improve the accuracy of data analysis and alert generation in future cases.

[0045] As described above, the system of the present invention can efficiently and effectively resolve problems that occur while driving a vehicle, and provide a safe and secure driving environment.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The device collects data from the vehicle's GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor, and other sensors.

[0049] Step 2:

[0050] The data collected by the device is aggregated into packets every 5 seconds. For example, the latest data values ​​obtained from each sensor are organized into a single data set.

[0051] Step 3:

[0052] The device sends the assembled data packets to a central server, which then sends the data over the internet or a communications network.

[0053] Step 4:

[0054] The server receives the data packets sent from the terminal. The server has a receiving function to process large amounts of data quickly.

[0055] Step 5:

[0056] The server analyzes the received data and uses specific algorithms to detect abnormal values ​​and patterns in the data from each sensor. For example, if the engine temperature exceeds the normal range, it will determine that this is an abnormality.

[0057] Step 6:

[0058] If the server detects an abnormality through analysis, it generates an alert, which includes information about the problem, recommended actions to take, and emergency instructions.

[0059] Step 7:

[0060] The server notifies the user of the generated alert via push notification, SMS message, or other methods.

[0061] Step 8:

[0062] The user's mobile device will receive an alert and display a notification, and the user can view more information through the application.

[0063] Step 9:

[0064] The user can review the details of the alert and follow the remedial action provided by the system, for example, "Engine temperature is too high. Stop engine and allow it to cool."

[0065] Step 10:

[0066] After the user has finished troubleshooting, the mobile device displays a feedback form, in which the user can enter their level of satisfaction, the results of the troubleshooting, and other information.

[0067] Step 11:

[0068] The mobile devices collect and send the feedback data to a central server, allowing the system to use the feedback information to improve future operations and the accuracy of its prediction models.

[0069] The above is the specific program processing flow of this system, which allows the user to quickly detect problems that occur during operation and take appropriate countermeasures.

[0070] Example 1

[0071] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0072] Modern automobiles are equipped with numerous sensors that can monitor the vehicle's condition. However, there is a lack of systems that can properly analyze sensor data and provide prompt and accurate countermeasures when an abnormality is detected. They also lack the ability to provide emergency assistance information based on the user's current location. Furthermore, there is often no way to collect user feedback to help improve the system. A system that can solve these problems and improve safety and convenience while driving is needed.

[0073] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0074] In this invention, the server includes: a means for an in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server; a means for the central server to analyze the data in real time and generate an alert when an abnormality is detected; a means for notifying the user of the alert to the user's mobile device and providing countermeasures for the abnormality; a means for providing information on the nearest support service based on the user's current location information; and a means for collecting feedback from the user, saving the data, and using it for analysis. This allows the system to quickly detect vehicle abnormalities, provide the user with appropriate countermeasures, and provide support service information, enabling the user to take prompt and appropriate action. Furthermore, collecting feedback data and using it for analysis enables continuous improvement of the system.

[0075] An "on-board terminal" is a device that is installed in a vehicle and collects data from various sensors and transmits it to a central server.

[0076] A "sensor" is a device that monitors the state of a vehicle and measures specific physical quantities, such as position information, speed information, movement data, mechanical information, and tire pressure information.

[0077] The "central server" is a computer system that receives and analyzes data sent from the vehicle-mounted terminal, and if it detects an abnormality, it generates an alert and notifies the user.

[0078] "Data" means a series of measurements or numbers collected from sensors, including location information, speed information, movement data, mechanical information, and tire pressure information.

[0079] An "alert" is a notification message generated when the central server detects an abnormality, informing the user of the nature of the abnormality and how to deal with it.

[0080] A "user's mobile device" is a portable electronic device, such as a smartphone or tablet, that is carried by a user and is used to receive alert notifications and check detailed information.

[0081] "Feedback" refers to opinions and impressions submitted by users after a problem has been resolved, as well as data for system improvement.

[0082] "Support service information" is information about locations and services that are useful for solving vehicle problems, such as the nearest gas station or repair shop, provided based on the user's current location.

[0083] "Location information" is data indicating the current location of the vehicle, and is obtained by a GPS sensor or the like.

[0084] "Speed ​​information" is data indicating the speed of the vehicle, and is acquired by a speed sensor.

[0085] "Movement data" is a series of information related to the movement of a vehicle, and is acquired by an acceleration sensor or the like.

[0086] "Mechanical information" is data indicating the mechanical state of the vehicle, such as the engine condition and temperature, and is acquired by an engine temperature sensor or the like.

[0087] "Tire pressure information" is data indicating the air pressure of the vehicle's tires, and is acquired by a tire pressure sensor.

[0088] These definitions clarify the meaning of the words contained in the claims and facilitate understanding of the scope and nature of the invention.

[0089] This invention is a system that collects data from multiple sensors in a vehicle, analyzes the data to detect abnormalities, and provides the user with a prompt and optimal response. How this invention is implemented will be specifically described below.

[0090] In-vehicle terminal

[0091] An in-vehicle terminal is a device that collects data from multiple sensors installed in a vehicle. The sensors used include a GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, and tire pressure sensor. The terminal obtains location information, speed information, movement data, mechanical information, and tire pressure information from these sensors.

[0092] The onboard device compiles the collected data into packets every five seconds and transmits them to a central server using a cellular communication module. For example, every five seconds, a data packet might be generated that reports an engine temperature of 90 degrees and tire pressure of 32 PSI, and the data is then transmitted to the central server.

[0093] Central Server

[0094] The central server receives data sent from the in-vehicle terminals and analyzes it in real time. The server uses a database management system (e.g., MySQL (registered trademark) or PostgreSQL) to store the received data. It also runs programs that use data analysis libraries (e.g., Python's pandas or scikit-learn) to detect outliers and irregular patterns. If an abnormality is detected, the server generates an alert according to the content. For example, if the engine temperature exceeds 100 degrees, it generates an alert stating, "The engine temperature is abnormally high."

[0095] The generated alert is immediately sent to the user's mobile device. Specifically, the alert message is sent using the push notification function.

[0096] User's mobile device

[0097] The user's mobile device is the device that receives the alert notification. This includes smartphones and tablets. The user can check the notification and open an application to view detailed information about the alert. For example, specific countermeasures such as "The engine temperature is abnormally high. Immediately stop the engine and allow it to cool down" are displayed.

[0098] Additionally, the system will provide information about the nearest assistance service based on the user's current location, for example, by displaying a message within the application saying, "The nearest gas station is 1.5 kilometers away. Please follow the route below."

[0099] feedback

[0100] After solving a problem, users report their impressions and effectiveness of the solution through a feedback form. The feedback data is stored on a central server and used for future analysis of prediction models and system improvements.

[0101] Prompt Sentence Examples

[0102] For example, by entering a prompt such as "Please explain what to do if the vehicle's engine temperature becomes abnormally high" into the generative AI model, it is possible to generate an explanation of specific countermeasures.

[0103] The present invention is a system that quickly detects abnormalities in a vehicle and provides the user with appropriate countermeasures, thereby improving safety and convenience for the user.

[0104] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0105] Step 1:

[0106] Terminal: The in-vehicle terminal collects data from various sensors installed in the vehicle (GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor, etc.).

[0107] Input: Data from each sensor (location information, speed information, movement data, machine information, tire pressure information).

[0108] How it works: Every 5 seconds, the onboard device aggregates this data and generates a time-stamped data packet.

[0109] Output: Consolidated data packet (e.g., engine temperature = 90 degrees, tire pressure = 32 PSI, time = 2023-10-10 10:00:05).

[0110] Step 2:

[0111] Terminal: The in-vehicle terminal generates and transmits the data packets to the central server.

[0112] Input: Consolidated data packet.

[0113] How it works: It uses a cellular communication module to send data packets over the internet to a central server.

[0114] Output: Data packets sent to the central server.

[0115] Step 3:

[0116] Server: The central server receives data packets sent from the in-vehicle terminals.

[0117] Input: Data packet sent from the in-vehicle terminal.

[0118] What it does: Uses the data reception module to receive packets and store them in a database (e.g. MySQL or PostgreSQL).

[0119] Output: Vehicle status data stored in a database.

[0120] Step 4:

[0121] Server: The central server analyzes the received data in real time and detects anomalies.

[0122] Input: Vehicle condition data stored in a database.

[0123] How it works: Using data analysis libraries (e.g., Python's pandas, scikit-learn), it uses rule-based or machine learning models to detect outliers and irregular patterns. For example, an engine temperature above 100 degrees is considered abnormal.

[0124] Output: Alert information when an abnormality is detected (e.g. engine temperature abnormality).

[0125] Step 5:

[0126] Server: When an anomaly is detected, a corresponding alert is generated and sent to the user's mobile device.

[0127] Input: Alert information if an anomaly is detected.

[0128] What it does: Uses push notifications to send an alert to the user's mobile device, for example, a message saying "Engine temperature is too high."

[0129] Output: An alert notification that appears on the user's mobile device.

[0130] Step 6:

[0131] User: The user receives an alert notification on their mobile device and opens the application to view more information.

[0132] Input: Alert notification.

[0133] Action: Open the application and check the details of the error and the countermeasures. For example, the countermeasures displayed will be "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool."

[0134] Output: A screen showing detailed information about the anomaly and specific countermeasures.

[0135] Step 7:

[0136] Server: Provides information about the nearest support service based on the user's current location.

[0137] Input: User's current location.

[0138] How it works: Using GPS location information, it searches a support service database and obtains information on the nearest gas station or repair shop.

[0139] Output: Information about the nearest assistance service and directions displayed on the user's mobile device (e.g., "The nearest gas station is 1.5 kilometers away. Please follow the route below.").

[0140] Step 8:

[0141] Server: Collect user feedback after the problem is resolved.

[0142] Input: Feedback request after troubleshooting.

[0143] What it does: Sends a feedback form to the user's mobile device.

[0144] Output: User-entered feedback (e.g., "I successfully resolved my engine overheating issue. Your guidance was very clear and helpful.").

[0145] Step 9:

[0146] User: Fill in the feedback form with your thoughts and opinions on how the problem was resolved and submit it.

[0147] Input: User feedback.

[0148] Action: Enter the required information into the feedback form and click the submit button.

[0149] Output: The feedback data sent.

[0150] Step 10:

[0151] Server: Stores the feedback data and analyzes it for future prediction models and system improvements.

[0152] Input: The submitted feedback data.

[0153] How it works: Feedback data is stored in a database and analyzed using a data analysis library.

[0154] Output: Data analysis results for improved predictive models and system improvements.

[0155] (Application example 1)

[0156] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0157] Detecting abnormalities and problems that occur during the operation of autonomous vehicles in real time and providing users with prompt and appropriate countermeasures is important for improving operational safety and efficiency. However, with conventional systems, there is a time lag between the detection of an abnormality and instructions to respond, as well as limitations in notification methods, making it difficult for users to take appropriate action immediately. In addition, feedback data is not properly collected and analyzed, and there are cases where it cannot be used to improve future countermeasures.

[0158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0159] In this invention, the server includes: means for the in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server; means for the central server to analyze the data in real time and generate an alert when an abnormality is detected; means for notifying the user of the alert to the user's mobile terminal and wearable device and providing countermeasures for the abnormality; and means for collecting feedback from the user, saving the data, and using it for analysis. This minimizes the time lag from abnormality detection to response instructions, allowing the user to immediately confirm and respond via a wearable device such as smart glasses, thereby significantly improving the safety and efficiency of driving.

[0160] An "on-board terminal" is a device installed in a vehicle that collects data from multiple sensors and transmits it to a central server.

[0161] The "multiple sensors" are various sensor devices that collect data such as vehicle position, speed, acceleration, engine temperature, and tire pressure.

[0162] The "central server" is a computer system that analyzes data sent from the vehicle-mounted terminal in real time and generates an alert when an abnormality is detected.

[0163] An "alert" is a warning or notification generated when the central server detects an abnormality, and is information intended to prompt the user to take prompt action.

[0164] A "mobile terminal" is a computing device that a user can carry with them, such as a smartphone or tablet.

[0165] A "wearable device" is a device that can be worn by a user, such as smart glasses or a wristwatch-type terminal.

[0166] "Countermeasures" refer to specific actions or procedures that a user should take when an abnormality occurs.

[0167] "Feedback" refers to information provided by users about the effectiveness of the problem resolution, their impressions, and their experiences, and is data that will be used for future data analysis and service improvement.

[0168] "Location data" refers to data relating to the current location of a vehicle obtained using a GPS or the like.

[0169] "Speed ​​data" is data relating to the vehicle's traveling speed.

[0170] "Acceleration data" is data relating to the acceleration of the vehicle.

[0171] "Engine data" refers to data relating to the temperature and operating conditions of the engine.

[0172] "Tire pressure data" refers to data relating to tire air pressure.

[0173] "Support services" refer to service facilities such as gas stations and repair shops that users can use when there is a problem with their vehicle.

[0174] This invention is a system that detects abnormalities and problems that occur during the operation of an autonomous vehicle in real time and provides users with prompt and appropriate countermeasures. This system includes an in-vehicle terminal, a central server, a mobile terminal, and a wearable device.

[0175] First, the in-vehicle terminal collects data from multiple sensors installed in the vehicle. The collected data includes location data, speed data, acceleration data, engine data, and tire pressure data. This data is sent to a central server at regular intervals (for example, every 5 seconds).

[0176] The central server then analyzes the data sent from the in-vehicle device in real time. If an abnormality is detected during the analysis, an alert is immediately generated. This alert includes the specific details of the problem, how to deal with it, and information about the nearest support service.

[0177] The generated alerts are sent to the user's mobile devices (e.g., smartphones, tablets) and wearable devices (e.g., smart glasses), allowing the user to immediately check the alerts and take appropriate action.

[0178] As a specific example, if the engine temperature exceeds the normal range, an alert such as the following is generated: "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool." This alert is sent to the mobile device and smart glasses, and can be immediately confirmed by the user.

[0179] The system also includes a means to collect user feedback. After resolving a problem, users can submit their impressions and feedback through a feedback form. This feedback data is stored on a central server and used for future data analysis and service improvement.

[0180] The hardware includes various vehicle sensors (e.g., GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, tire pressure sensors), smart glasses (e.g., Google® Glass®, Vuzix Blade), mobile devices (e.g., smartphones, tablets), and a central server.

[0181] The software includes APIs (e.g., RESTful APIs) for collecting data from sensors and sending it to a central server, Python scripts for data analysis, SDKs for smart glasses (e.g., Google Glass SDK, Vuzix SDK), and applications for sending notifications.

[0182] Using generative AI models, it is also possible to automatically generate specific and practical countermeasures when an abnormality is detected. For example, the following is an example of a prompt when an abnormality in engine temperature is detected: "The vehicle's engine temperature is above the normal range. Immediately shut down the engine and allow it to cool. Check the sensor data again to see if the engine temperature has returned to normal."

[0183] This minimizes the time lag between detecting an abnormality and issuing instructions to respond, significantly improving safety and efficiency during operation.

[0184] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0185] Step 1:

[0186] The in-vehicle terminal collects data from multiple sensors installed in the vehicle (position sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor). The input is data from each sensor, and the output is a packet of collected data.

[0187] Step 2:

[0188] The in-vehicle terminal transmits the collected sensor data as packets to the central server at regular intervals (e.g., every 5 seconds). The input is the data packets collected in step 1, and the output is the data packets transmitted to the central server.

[0189] Step 3:

[0190] The central server receives data packets sent from the in-vehicle terminal and analyzes them in real time. During the analysis, it detects outliers and irregular patterns. The input is the data packets sent from the in-vehicle terminal, and the output is the analysis results.

[0191] Step 4:

[0192] If the central server detects an abnormality, it generates an alert. This alert contains the specific details of the problem, how to deal with it, and information about the nearest support service. The input is the analysis result obtained in step 3, and the output is the generated alert.

[0193] Step 5:

[0194] The central server notifies the generated alerts to the user's mobile terminal and wearable device. The input is the generated alert information, and the output is the alert displayed on the user's device.

[0195] Step 6:

[0196] The user checks the alert through a mobile terminal or wearable device and takes appropriate action according to the countermeasures presented. The input is the alert information displayed on the device, and the output is the user's specific action.

[0197] Step 7:

[0198] After resolving a problem, the user submits the results and impressions through a feedback form. The input is the user's feedback information, and the output is the submitted feedback data.

[0199] Step 8:

[0200] The central server stores the collected feedback data and uses it for future data analysis and service improvement. The input is the feedback data submitted by users, and the output is the stored data and analysis results.

[0201] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0202] The present invention combines a system that collects data from multiple sensors in a vehicle, analyzes the data to detect abnormalities, and provides the user with a prompt and optimal response, with an emotion engine that recognizes the user's emotional state. Specific embodiments for implementing the present invention will be described below.

[0203] An in-vehicle terminal is a device installed in a vehicle that collects data from GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, tire pressure sensors, etc. This data is sent to a central server at regular intervals. For example, every 5 seconds, the data from each sensor is collected into packets and sent to the server.

[0204] The central server receives data sent from the in-vehicle device. The received data is analyzed in real time, and if an abnormal value or irregular pattern is detected, an alert is generated. For example, if the engine temperature exceeds the normal range, an abnormality is detected based on that data.

[0205] The generated alert is sent to the user's mobile device (such as a smartphone or tablet). By opening the application, the user can check the details of the alert. For example, specific countermeasures such as "The engine temperature is abnormally high. Immediately stop the engine and allow it to cool" are displayed.

[0206] The present invention also incorporates an emotion engine for recognizing the user's emotional state, which uses the camera and / or microphone on the user's mobile device to analyze the user's facial expressions and voice to detect emotions such as stress, anger, fatigue, etc. in real time.

[0207] Based on the emotional state detected by the emotion engine, the system can customize responses to the user. For example, if the user is stressed, the system can provide relaxing music and voice guidance along with an alert notification. If the user shows high levels of anger, the system can provide gentle advice on safe driving.

[0208] As a concrete example, consider the case where a user's tire pressure drops while driving. The in-vehicle device detects abnormal data from the tire pressure sensor and sends it to a central server. The server immediately analyzes the abnormality, generates an alert stating "Tire pressure is below the specified value," and notifies the user's mobile device. The user's mobile device then uses a camera and microphone to analyze the user's facial expressions and voice, and detects that the user is feeling stressed at this point. In response, the system notifies the user of the alert, plays relaxing music, and provides route guidance to the nearest gas station. After the problem is resolved, a feedback form is displayed and the user's experience is sent to the server.

[0209] This allows the system to not only respond to problems but also provide support that takes into account the user's emotional state, allowing the user to continue driving with peace of mind while the system continues to improve.

[0210] The processing flow will be explained below.

[0211] Step 1:

[0212] The device collects data from the vehicle's GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor, and other sensors.

[0213] Step 2:

[0214] The terminal collects data and compiles it into packets every 5 seconds. The terminal then organizes the latest data values ​​obtained from each sensor into a single data set.

[0215] Step 3:

[0216] The device sends the assembled data packets to a central server, which then sends the data over the internet or a communications network.

[0217] Step 4:

[0218] The server receives the data packets sent from the terminal. The server has a receiving function to process large amounts of data quickly.

[0219] Step 5:

[0220] The server analyzes the data it receives in real time. Specific algorithms are used to detect abnormal values ​​and patterns in the data from each sensor. For example, if the engine temperature exceeds the normal range, it will determine that this is an abnormality.

[0221] Step 6:

[0222] If the server detects an abnormality through analysis, it generates an alert, which includes information about the problem, recommended actions to take, and emergency instructions.

[0223] Step 7:

[0224] The server notifies the user of the generated alert via push notification, SMS message, or other methods.

[0225] Step 8:

[0226] The user's mobile device will receive an alert and display a notification, and the user can view more information through the application.

[0227] Step 9:

[0228] The emotion engine analyzes the user's facial expressions and voice using the camera and microphone installed in the user's mobile device, and detects emotional states such as stress, anger, and fatigue in real time.

[0229] Step 10:

[0230] The emotion engine sends the user's emotional state to the server, for example, if the user is showing a high level of stress, it provides that information to the server.

[0231] Step 11:

[0232] The server takes into account the user's emotional state and sends a customized response to the alert to the user's mobile device, such as providing relaxing music or a voice guide.

[0233] Step 12:

[0234] The user reviews the alert details and follows the customized remedial action provided by the system, for example, "Engine temperature is too high. Stop engine and allow it to cool down," while listening to relaxing music.

[0235] Step 13:

[0236] After the user has finished troubleshooting, the mobile device displays a feedback form, in which the user can enter information such as their level of satisfaction, the outcome of the troubleshooting, and their emotional state.

[0237] Step 14:

[0238] The mobile devices collect and send the feedback data to a central server, allowing the system to use the feedback information to improve future operations and the accuracy of its prediction models.

[0239] The above is the specific program processing flow of this system, which allows the user to quickly detect problems that occur while driving, take appropriate countermeasures, and receive support that takes into consideration the user's emotional state.

[0240] Example 2

[0241] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0242] Modern vehicle management systems detect vehicle anomalies and notify the user, but do not take into account the user's emotional state. As a result, users may feel excessive stress or anger when an abnormality occurs, making it difficult to respond optimally. The present invention aims to provide a system that not only analyzes data collected from multiple sensors in the vehicle to detect anomalies, but also recognizes the user's emotional state and provides optimal countermeasures based on that information.

[0243] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0244] In this invention, the server includes: means for an in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server; means for the central server to analyze the data in real time and generate an alert when an abnormality is detected; means for notifying the user of the alert to a mobile device and providing countermeasures for the abnormality; means for the mobile device to analyze the user's facial expressions and voice using a camera and microphone to detect the user's emotional state in real time; means for customizing the content of the alert and countermeasures based on the emotional state detected by the emotion engine; and means for collecting feedback from the user, saving the data, and using it for analysis. This enables a quick and optimal response to vehicle abnormalities while also taking the user's emotional state into consideration.

[0245] An "on-board terminal" is a device installed in a vehicle that collects data from multiple sensors and transmits it to a central server.

[0246] A "sensor" is a device that detects and collects information about the vehicle's status and the external environment, including location data, speed data, acceleration data, engine temperature data, and tire pressure data.

[0247] The "central server" is a server device that receives data sent from the vehicle-mounted terminal, analyzes it in real time, and detects abnormalities.

[0248] An "alert" is a warning message generated by the central server when an abnormality is detected as a result of data analysis.

[0249] A "mobile terminal" is a mobile device such as a smartphone or tablet owned by a user, which receives alert notifications from a central server.

[0250] The "emotion engine" is a system that uses the camera and microphone of a mobile device to analyze the user's facial expressions and voice, and detects the user's emotional state in real time.

[0251] "Feedback" refers to opinions and ratings collected from users, and is data used to improve the system and evaluate alert responses.

[0252] "Relaxing music" is music intended to relieve the user's stress and tension, and can be played by the emotion engine based on the user's emotional state.

[0253] A "trouble" is an event that prevents the vehicle from operating normally, including vehicle abnormalities and breakdowns.

[0254] The present invention provides a system that collects data from multiple sensors in a vehicle, analyzes the data to detect abnormalities, and also recognizes the emotional state of the user, and provides optimal countermeasures based on the collected data. Specific embodiments for carrying out the present invention will be described below.

[0255] System configuration

[0256] The system consists of the following main parts:

[0257] In-vehicle terminal

[0258] Central Server

[0259] Mobile devices

[0260] Emotion Engine

[0261] In-vehicle terminal

[0262] An onboard device is a device installed in a vehicle that collects data from the following sensors:

[0263] GPS sensor

[0264] Speed ​​sensor

[0265] Accelerometer

[0266] Engine temperature sensor

[0267] tire pressure sensor

[0268] The data collected from these sensors is aggregated into packets at regular intervals (e.g., every 5 seconds) and sent to a central server.

[0269] Specific working example:

[0270] When a vehicle turns a sharp corner, the acceleration sensor detects high lateral acceleration (G-force), and the onboard terminal compiles it into packets.

[0271] Central Server

[0272] The central server receives data sent from the in-vehicle device and analyzes it in real time. If an abnormal value or irregular pattern is detected in the received data, an alert is generated. This alert is sent to the user's mobile device.

[0273] Specific working example:

[0274] If the engine temperature sensor detects a temperature outside the normal range, an alert will be generated stating, "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool." and sent to the user's mobile device.

[0275] Mobile devices

[0276] The mobile device is a user's smartphone or tablet, which receives an alert notification from the central server. By opening the application, the user can check the details of the alert and receive suggestions for countermeasures.

[0277] Specific working example:

[0278] If the tire pressure drops below the specified value, the system will display an alert stating "Tire pressure is below the specified value" along with directions to the nearest gas station.

[0279] Emotion engine and customized responses

[0280] The emotion engine uses the mobile device's camera and microphone to analyze the user's facial expressions and voice to detect emotional states such as stress, anger, fatigue, etc. in real time. Based on the results, the system further customizes the immediate response measures for the user.

[0281] Specific working example:

[0282] If the user is recognized as being in a stressful state, relaxing music is played on the mobile device and a guide voice prompts the user to "relax."

[0283] Gathering feedback

[0284] After the problem is resolved, the user's mobile device will display a feedback form to collect the user's experience, which will be stored on a central server and used to improve the system in the future.

[0285] Specific working example:

[0286] After the user has resolved the issue, a feedback form is displayed asking "How was your experience with our service?", and the user's input is received and sent to a central server.

[0287] Prompt Sentence Examples

[0288] "Lateral acceleration data detected by the accelerometer is collected and sent to a central server."

[0289] "Generates an alert if engine temperatures exceed normal range."

[0290] "When tire pressure drops, the user is given directions to the nearest gas station."

[0291] "If the user is feeling stressed, it will play relaxing music and provide a guide voice."

[0292] This allows the system to provide integrated support that takes into account both troubleshooting and the user's emotional state, allowing the user to continue driving with peace of mind.

[0293] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0294] Step 1:

[0295] The onboard terminal collects data from each sensor.

[0296] Inputs: GPS data, speed data, acceleration data, engine temperature data, tire pressure data.

[0297] Data processing / calculation: The raw data obtained from each sensor is compiled into packets by time.

[0298] Output: Sensor data packets.

[0299] Specific operation: When a vehicle turns a sharp corner, the acceleration sensor detects high lateral acceleration (G-force), and the onboard terminal assembles it into packets.

[0300] Step 2:

[0301] The in-vehicle terminal transmits the collected data packets to a central server.

[0302] Input: Sensor data packets.

[0303] Data processing / calculation: Encryption of packet data and securing of communication lines.

[0304] Output: Sending encrypted sensor data packets.

[0305] How it works: The in-vehicle device sends encrypted data packets to a central server via Wi-Fi or a cellular network.

[0306] Step 3:

[0307] A central server receives the data packets and analyzes them in real time.

[0308] Input: Encrypted sensor data packet.

[0309] Data processing / computation: Decrypting encrypted data and applying algorithms to detect outliers and irregular patterns.

[0310] Output: Anomaly detection results and alert generation.

[0311] Specific operation: The central server analyzes the received engine temperature data and compares it with the reference range to detect abnormalities. For example, if the engine temperature exceeds the normal range, an alert is generated.

[0312] Step 4:

[0313] A central server generates alerts and sends them to the user's mobile device.

[0314] Input: Anomaly detection results.

[0315] Data processing / calculation: Generating alert messages and creating notification data packets.

[0316] Output: Alert notification to mobile device.

[0317] Specific operation: If the engine temperature is abnormally high, an alert will be generated stating "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool down" and a notification will be sent to the mobile device.

[0318] Step 5:

[0319] The mobile device uses a camera and microphone to analyze the user's facial expressions and voice to detect their emotional state in real time.

[0320] Input: User's facial expression data, voice data.

[0321] Data processing / computation: Apply image and audio analysis algorithms to identify emotional states.

[0322] Output: User's emotional state data.

[0323] Specific operation: The user's voice is captured and recognized as "stress" by the emotion engine. "Stress" is also detected from facial expressions.

[0324] Step 6:

[0325] Based on the emotional state detected by the emotion engine, a central server customizes alert notification content and responses.

[0326] Input: User emotional state data.

[0327] Data processing / computation: Applying customized algorithms based on emotional state data.

[0328] Output: Customized alert notifications and remedial actions.

[0329] Specific operation: If the user is recognized as being in a stressful state, relaxing music will be played on the mobile device and a guide voice will prompt the user to "relax."

[0330] Step 7:

[0331] After the problem is solved, the mobile terminal displays a feedback form to collect feedback from the user.

[0332] Input: User's post-resolution feedback answer.

[0333] Data processing / calculation: Collection of feedback data and storage in a database for analysis.

[0334] Output: Save feedback data.

[0335] Specific behavior: After the problem is resolved, a feedback form asking "How was your service?" is displayed, and user input is collected and sent to a central server.

[0336] (Application example 2)

[0337] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0338] Conventional vehicle management systems have been effective to a certain extent in detecting abnormalities and notifying drivers using sensors, but they have limitations in providing optimal countermeasures that take into account the user's emotional state. Furthermore, when users feel stressed or fatigued, they do not provide sufficient support to alleviate these conditions. Therefore, there is a need for a system that is ergonomically friendly to users while improving driving safety.

[0339] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0340] In this invention, the server includes means for an in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server, means for the central server to analyze the data in real time and generate an alert when an abnormality is detected, means for notifying the user of the alert to the user's mobile terminal and providing countermeasures for the abnormality, means for recognizing the user's emotional state and customizing the countermeasures based on the emotional state, and means for collecting feedback from the user and saving the data for use in analysis, thereby making it possible to provide a system that not only detects abnormalities but also responds to the user's emotional state.

[0341] An "on-board terminal" is a device installed in a vehicle that collects data from multiple sensors and transmits the data to a central server.

[0342] A "sensor" is a device used to detect the condition and movement of a vehicle, and is used to collect data such as GPS, speed, acceleration, engine, and tire pressure.

[0343] A "central server" is a device or system that analyzes data sent from vehicle-mounted terminals in real time, detects abnormalities, and generates alerts.

[0344] An "alert" is a warning notification that is generated when an abnormality is detected based on data collected from a sensor, and is intended to inform the user of information about the abnormality and how to deal with it.

[0345] A "mobile terminal" is a device that can be carried by a user, including a smartphone or tablet, for receiving alert notifications and abnormality response measures.

[0346] "Emotional state" refers to the mental state detected from the user's facial expression and voice, and includes, for example, stress, anger, fatigue, etc.

[0347] An "emotion engine" is a system or software that analyzes a user's facial expressions and voice data to recognize their emotional state.

[0348] "Feedback" refers to opinions and information provided by users about their experience using the system, which is collected for the purpose of improving and optimizing the system.

[0349] The system of the present invention is composed of an in-vehicle terminal, a central server, an emotion engine, and a user's mobile terminal. The specific configuration and operation will be explained below.

[0350] In-vehicle terminal

[0351] The in-vehicle terminal is equipped with multiple sensors, such as a GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, and tire pressure sensor. These sensors constantly monitor the vehicle's condition and movement and collect data. The collected data is sent to a central server at regular intervals, for example, every five seconds.

[0352] Central Server

[0353] The central server receives and analyzes data sent from the in-vehicle device in real time. If an abnormal value is detected as a result of the analysis, an alert is generated. For example, if the engine temperature exceeds the normal range, the data is judged to be abnormal and an alert is generated based on that. The generated alert is sent to the user's mobile device.

[0354] Mobile Devices and Emotion Engines

[0355] The user's mobile device is equipped with an emotion engine that uses a camera and microphone to analyze the user's facial expressions and voice. The emotion engine recognizes the user's mental state (stress, anger, fatigue, etc.) in real time. For example, if the user is in a stressful state when an alert is generated, the system will play relaxing music and suggest optimal countermeasures, such as providing route guidance to the nearest gas station.

[0356] Feedback and Improvements

[0357] The system collects user feedback and is continuously improved based on experience. Feedback is collected via mobile devices and analyzed and stored on a central server. This improves the accuracy of the system and the user experience.

[0358] Specific examples

[0359] For example, consider the case where a user's tire pressure drops while driving. The in-vehicle device detects abnormal data from the tire pressure sensor and sends it to a central server. The server immediately analyzes the abnormality, generates an alert stating "Tire pressure is below the specified value," and notifies the user's mobile device. The user's mobile device then uses its camera and microphone to analyze the user's facial expressions and voice, and detects that the user is feeling stressed at this point. In response, the system plays relaxing music along with the alert notification and provides route guidance to the nearest gas station.

[0360] Prompt Sentence Examples

[0361] Anomaly Detection Systems:

[0362] Sensor data such as GPS, speed, engine temperature, and tire pressure collected by the in-vehicle device is sent to a central server every five seconds for real-time analysis. If an abnormality is detected, an alert is generated and a notification of countermeasures is sent to the user's smartphone. In addition, the system analyzes the user's emotional state using a camera and microphone, and plays relaxing music if the user is under stress.

[0363] As described above, the embodiments of the present invention can provide a safe and comfortable driving environment by detecting abnormalities in a vehicle and responding to the emotional state of the user.

[0364] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0365] Step 1:

[0366] The in-vehicle terminal collects data from multiple sensors, specifically GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, and tire pressure sensors. The input is data from each sensor, and the output is a collection of that data.

[0367] Step 2:

[0368] The data collected by the in-vehicle device is sent to a central server every five seconds. Specifically, the data is packaged into packets and sent over the Internet. The input is a collection of data collected from the sensors, and the output is the data packet sent to the central server.

[0369] Step 3:

[0370] The central server analyzes the received data in real time, stores it in a database, and runs algorithms to detect abnormal values ​​and patterns. The input is the data packets sent from the in-vehicle device, and the output is the result of the anomaly detection.

[0371] Step 4:

[0372] When the central server detects an anomaly, it generates an alert. Specifically, it creates a warning message depending on the type of anomaly. The input is the anomaly detection result, and the output is the generated alert message.

[0373] Step 5:

[0374] The generated alert is sent to the user's mobile device. Specifically, the alert message is pushed to the mobile device. The input is the alert message, and the output is the alert displayed on the mobile device.

[0375] Step 6:

[0376] The user's mobile device receives the alert and uses its camera and microphone to analyze the user's emotional state. Specifically, the camera video and audio data are input into an emotion engine to recognize emotional states such as stress, anger, and fatigue. The input is data obtained from the camera and microphone, and the output is the analyzed emotional state.

[0377] Step 7:

[0378] The emotion engine customizes responses based on the user's emotional state, such as playing relaxing music or running an algorithm to suggest optimal responses. The input is the analyzed emotional state, and the output is a customized response provided to the user.

[0379] Step 8:

[0380] User feedback is collected and used to improve the system. Specifically, feedback is collected from users via an application and stored in a database. The input is feedback data from users, and the output is the feedback stored in the database.

[0381] Step 9:

[0382] The feedback data is analyzed and reflected in system improvements. Specifically, the feedback data is analyzed, system improvements are extracted, and software updates and algorithm optimization are carried out. The input is the feedback data, and the output is improved system functions.

[0383] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0384] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0385] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0386] [Second embodiment]

[0387] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0388] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0389] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0390] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0391] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0392] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0393] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0394] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0395] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0396] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0397] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0398] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0399] The present invention relates to a system for collecting data from a plurality of sensors in a vehicle, analyzing the data to detect abnormalities, and providing a user with a prompt and optimal countermeasure. Specific embodiments for carrying out the present invention will be described below.

[0400] An in-vehicle terminal is a device installed in a vehicle that collects data from GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, tire pressure sensors, etc. This data is sent to a central server at regular intervals. For example, every 5 seconds, the data from each sensor is collected into packets and sent to the server.

[0401] The central server receives data sent from the in-vehicle devices, analyzes the data in real time, and generates an alert if anomalies or irregular patterns are detected. For example, if the engine temperature significantly exceeds normal values, an alert will be generated indicating a possible overheating.

[0402] The generated alert is sent to the user's mobile device (such as a smartphone or tablet). By opening the application, the user can check the details of the alert. For example, specific countermeasures such as "The engine temperature is abnormally high. Immediately stop the engine and allow it to cool" are displayed.

[0403] The system also provides users with emergency assistance service information. Based on the user's current location, information about the nearest gas station or repair shop and directions to the nearest shop are displayed within the application. For example, specific directions such as "The nearest repair shop is 1.5 kilometers away. Please follow the route below" are provided.

[0404] After a user resolves a problem through the system, they will receive a notification requesting feedback. They can use the feedback form to express their thoughts and the effectiveness of the resolution. This feedback data will be stored on a central server and used to improve future prediction models and services.

[0405] As a concrete example, consider the case where a user experiences a drop in tire pressure while driving. The in-vehicle device detects abnormal data from the tire pressure sensor and sends it to a central server. The server immediately analyzes the abnormality, generates an alert stating "Tire pressure is below the specified value," and notifies the user's mobile device. The user confirms the alert and follows the directions to the nearest gas station. They then report their experience resolving the problem in a feedback form. This allows the system to improve the accuracy of data analysis and alert generation in future cases.

[0406] As described above, the system of the present invention can efficiently and effectively resolve problems that occur while driving a vehicle, and provide a safe and secure driving environment.

[0407] The processing flow will be explained below.

[0408] Step 1:

[0409] The device collects data from the vehicle's GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor, and other sensors.

[0410] Step 2:

[0411] The data collected by the device is aggregated into packets every 5 seconds. For example, the latest data values ​​obtained from each sensor are organized into a single data set.

[0412] Step 3:

[0413] The device sends the assembled data packets to a central server, which then sends the data over the internet or a communications network.

[0414] Step 4:

[0415] The server receives the data packets sent from the terminal. The server has a receiving function to process large amounts of data quickly.

[0416] Step 5:

[0417] The server analyzes the received data and uses specific algorithms to detect abnormal values ​​and patterns in the data from each sensor. For example, if the engine temperature exceeds the normal range, it will determine that this is an abnormality.

[0418] Step 6:

[0419] If the server detects an abnormality through analysis, it generates an alert, which includes information about the problem, recommended actions to take, and emergency instructions.

[0420] Step 7:

[0421] The server notifies the user of the generated alert via push notification, SMS message, or other methods.

[0422] Step 8:

[0423] The user's mobile device will receive an alert and display a notification, and the user can view more information through the application.

[0424] Step 9:

[0425] The user can review the details of the alert and follow the remedial action provided by the system, for example, "Engine temperature is too high. Stop engine and allow it to cool."

[0426] Step 10:

[0427] After the user has finished troubleshooting, the mobile device displays a feedback form, in which the user can enter their level of satisfaction, the results of the troubleshooting, and other information.

[0428] Step 11:

[0429] The mobile devices collect and send the feedback data to a central server, allowing the system to use the feedback information to improve future operations and the accuracy of its prediction models.

[0430] The above is the specific program processing flow of this system, which allows the user to quickly detect problems that occur during operation and take appropriate countermeasures.

[0431] Example 1

[0432] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0433] Modern automobiles are equipped with numerous sensors that can monitor the vehicle's condition. However, there is a lack of systems that can properly analyze sensor data and provide prompt and accurate countermeasures when an abnormality is detected. They also lack the ability to provide emergency assistance information based on the user's current location. Furthermore, there is often no way to collect user feedback to help improve the system. A system that can solve these problems and improve safety and convenience while driving is needed.

[0434] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0435] In this invention, the server includes: a means for an in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server; a means for the central server to analyze the data in real time and generate an alert when an abnormality is detected; a means for notifying the user of the alert to the user's mobile device and providing countermeasures for the abnormality; a means for providing information on the nearest support service based on the user's current location information; and a means for collecting feedback from the user, saving the data, and using it for analysis. This allows the system to quickly detect vehicle abnormalities, provide the user with appropriate countermeasures, and provide support service information, enabling the user to take prompt and appropriate action. Furthermore, collecting feedback data and using it for analysis enables continuous improvement of the system.

[0436] An "on-board terminal" is a device that is installed in a vehicle and collects data from various sensors and transmits it to a central server.

[0437] A "sensor" is a device that monitors the state of a vehicle and measures specific physical quantities, such as position information, speed information, movement data, mechanical information, and tire pressure information.

[0438] The "central server" is a computer system that receives and analyzes data sent from the vehicle-mounted terminal, and if it detects an abnormality, it generates an alert and notifies the user.

[0439] "Data" means a series of measurements or numbers collected from sensors, including location information, speed information, movement data, mechanical information, and tire pressure information.

[0440] An "alert" is a notification message generated when the central server detects an abnormality, informing the user of the nature of the abnormality and how to deal with it.

[0441] A "user's mobile device" is a portable electronic device, such as a smartphone or tablet, that is carried by a user and is used to receive alert notifications and check detailed information.

[0442] "Feedback" refers to opinions and impressions submitted by users after a problem has been resolved, as well as data for system improvement.

[0443] "Support service information" is information about locations and services that are useful for solving vehicle problems, such as the nearest gas station or repair shop, provided based on the user's current location.

[0444] "Location information" is data indicating the current location of the vehicle, and is obtained by a GPS sensor or the like.

[0445] "Speed ​​information" is data indicating the speed of the vehicle, and is acquired by a speed sensor.

[0446] "Movement data" is a series of information related to the movement of a vehicle, and is acquired by an acceleration sensor or the like.

[0447] "Mechanical information" is data indicating the mechanical state of the vehicle, such as the engine condition and temperature, and is acquired by an engine temperature sensor or the like.

[0448] "Tire pressure information" is data indicating the air pressure of the vehicle's tires, and is acquired by a tire pressure sensor.

[0449] These definitions clarify the meaning of the words contained in the claims and facilitate understanding of the scope and nature of the invention.

[0450] This invention is a system that collects data from multiple sensors in a vehicle, analyzes the data to detect abnormalities, and provides the user with a prompt and optimal response. How this invention is implemented will be specifically described below.

[0451] In-vehicle terminal

[0452] An in-vehicle terminal is a device that collects data from multiple sensors installed in a vehicle. The sensors used include a GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, and tire pressure sensor. The terminal obtains location information, speed information, movement data, mechanical information, and tire pressure information from these sensors.

[0453] The onboard device compiles the collected data into packets every five seconds and transmits them to a central server using a cellular communication module. For example, every five seconds, a data packet might be generated that reports an engine temperature of 90 degrees and tire pressure of 32 PSI, and the data is then transmitted to the central server.

[0454] Central Server

[0455] The central server receives data sent from the in-vehicle devices and analyzes it in real time. The server uses a database management system (e.g., MySQL or PostgreSQL) to store the received data. It also uses a data analysis library (e.g., Python's pandas or scikit-learn) to run programs that detect outliers and irregular patterns. If an abnormality is detected, the server generates an alert according to the content. For example, if the engine temperature exceeds 100 degrees, it generates an alert stating, "The engine temperature is abnormally high."

[0456] The generated alert is immediately sent to the user's mobile device. Specifically, the alert message is sent using the push notification function.

[0457] User's mobile device

[0458] The user's mobile device is the device that receives the alert notification. This includes smartphones and tablets. The user can check the notification and open an application to view detailed information about the alert. For example, specific countermeasures such as "The engine temperature is abnormally high. Immediately stop the engine and allow it to cool down" are displayed.

[0459] Additionally, the system will provide information about the nearest assistance service based on the user's current location, for example, by displaying a message within the application saying, "The nearest gas station is 1.5 kilometers away. Please follow the route below."

[0460] feedback

[0461] After solving a problem, users report their impressions and effectiveness of the solution through a feedback form. The feedback data is stored on a central server and used for future analysis of prediction models and system improvements.

[0462] Prompt Sentence Examples

[0463] For example, by entering a prompt such as "Please explain what to do if the vehicle's engine temperature becomes abnormally high" into the generative AI model, it is possible to generate an explanation of specific countermeasures.

[0464] The present invention is a system that quickly detects abnormalities in a vehicle and provides the user with appropriate countermeasures, thereby improving safety and convenience for the user.

[0465] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0466] Step 1:

[0467] Terminal: The in-vehicle terminal collects data from various sensors installed in the vehicle (GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor, etc.).

[0468] Input: Data from each sensor (location information, speed information, movement data, machine information, tire pressure information).

[0469] How it works: Every 5 seconds, the onboard device aggregates this data and generates a time-stamped data packet.

[0470] Output: Consolidated data packet (e.g., engine temperature = 90 degrees, tire pressure = 32 PSI, time = 2023-10-10 10:00:05).

[0471] Step 2:

[0472] Terminal: The in-vehicle terminal generates and transmits the data packets to the central server.

[0473] Input: Consolidated data packet.

[0474] How it works: It uses a cellular communication module to send data packets over the internet to a central server.

[0475] Output: Data packets sent to the central server.

[0476] Step 3:

[0477] Server: The central server receives data packets sent from the in-vehicle terminals.

[0478] Input: Data packet sent from the in-vehicle terminal.

[0479] What it does: Uses the data reception module to receive packets and store them in a database (e.g. MySQL or PostgreSQL).

[0480] Output: Vehicle status data stored in a database.

[0481] Step 4:

[0482] Server: The central server analyzes the received data in real time and detects anomalies.

[0483] Input: Vehicle condition data stored in a database.

[0484] How it works: Using data analysis libraries (e.g., Python's pandas, scikit-learn), it uses rule-based or machine learning models to detect outliers and irregular patterns. For example, an engine temperature above 100 degrees is considered abnormal.

[0485] Output: Alert information when an abnormality is detected (e.g. engine temperature abnormality).

[0486] Step 5:

[0487] Server: When an anomaly is detected, a corresponding alert is generated and sent to the user's mobile device.

[0488] Input: Alert information if an anomaly is detected.

[0489] What it does: Uses push notifications to send an alert to the user's mobile device, for example, a message saying "Engine temperature is too high."

[0490] Output: An alert notification that appears on the user's mobile device.

[0491] Step 6:

[0492] User: The user receives an alert notification on their mobile device and opens the application to view more information.

[0493] Input: Alert notification.

[0494] Action: Open the application and check the details of the error and the countermeasures. For example, the countermeasures displayed will be "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool."

[0495] Output: A screen showing detailed information about the anomaly and specific countermeasures.

[0496] Step 7:

[0497] Server: Provides information about the nearest support service based on the user's current location.

[0498] Input: User's current location.

[0499] How it works: Using GPS location information, it searches a support service database and obtains information on the nearest gas station or repair shop.

[0500] Output: Information about the nearest assistance service and directions displayed on the user's mobile device (e.g., "The nearest gas station is 1.5 kilometers away. Please follow the route below.").

[0501] Step 8:

[0502] Server: Collect user feedback after the problem is resolved.

[0503] Input: Feedback request after troubleshooting.

[0504] What it does: Sends a feedback form to the user's mobile device.

[0505] Output: User-entered feedback (e.g., "I successfully resolved my engine overheating issue. Your guidance was very clear and helpful.").

[0506] Step 9:

[0507] User: Fill in the feedback form with your thoughts and opinions on how the problem was resolved and submit it.

[0508] Input: User feedback.

[0509] Action: Enter the required information into the feedback form and click the submit button.

[0510] Output: The feedback data sent.

[0511] Step 10:

[0512] Server: Stores the feedback data and analyzes it for future prediction models and system improvements.

[0513] Input: The submitted feedback data.

[0514] How it works: Feedback data is stored in a database and analyzed using a data analysis library.

[0515] Output: Data analysis results for improved predictive models and system improvements.

[0516] (Application example 1)

[0517] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0518] Detecting abnormalities and problems that occur during the operation of autonomous vehicles in real time and providing users with prompt and appropriate countermeasures is important for improving operational safety and efficiency. However, with conventional systems, there is a time lag between the detection of an abnormality and instructions to respond, as well as limitations in notification methods, making it difficult for users to take appropriate action immediately. In addition, feedback data is not properly collected and analyzed, and there are cases where it cannot be used to improve future countermeasures.

[0519] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0520] In this invention, the server includes: means for the in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server; means for the central server to analyze the data in real time and generate an alert when an abnormality is detected; means for notifying the user of the alert to the user's mobile terminal and wearable device and providing countermeasures for the abnormality; and means for collecting feedback from the user, saving the data, and using it for analysis. This minimizes the time lag from abnormality detection to response instructions, allowing the user to immediately confirm and respond via a wearable device such as smart glasses, thereby significantly improving the safety and efficiency of driving.

[0521] An "on-board terminal" is a device installed in a vehicle that collects data from multiple sensors and transmits it to a central server.

[0522] The "multiple sensors" are various sensor devices that collect data such as vehicle position, speed, acceleration, engine temperature, and tire pressure.

[0523] The "central server" is a computer system that analyzes data sent from the vehicle-mounted terminal in real time and generates an alert when an abnormality is detected.

[0524] An "alert" is a warning or notification generated when the central server detects an abnormality, and is information intended to prompt the user to take prompt action.

[0525] A "mobile terminal" is a computing device that a user can carry with them, such as a smartphone or tablet.

[0526] A "wearable device" is a device that can be worn by a user, such as smart glasses or a wristwatch-type terminal.

[0527] "Countermeasures" refer to specific actions or procedures that a user should take when an abnormality occurs.

[0528] "Feedback" refers to information provided by users about the effectiveness of the problem resolution, their impressions, and their experiences, and is data that will be used for future data analysis and service improvement.

[0529] "Location data" refers to data relating to the current location of a vehicle obtained using a GPS or the like.

[0530] "Speed ​​data" is data relating to the vehicle's traveling speed.

[0531] "Acceleration data" is data relating to the acceleration of the vehicle.

[0532] "Engine data" refers to data relating to the temperature and operating conditions of the engine.

[0533] "Tire pressure data" refers to data relating to tire air pressure.

[0534] "Support services" refer to service facilities such as gas stations and repair shops that users can use when there is a problem with their vehicle.

[0535] This invention is a system that detects abnormalities and problems that occur during the operation of an autonomous vehicle in real time and provides users with prompt and appropriate countermeasures. This system includes an in-vehicle terminal, a central server, a mobile terminal, and a wearable device.

[0536] First, the in-vehicle terminal collects data from multiple sensors installed in the vehicle. The collected data includes location data, speed data, acceleration data, engine data, and tire pressure data. This data is sent to a central server at regular intervals (for example, every 5 seconds).

[0537] The central server then analyzes the data sent from the in-vehicle device in real time. If an abnormality is detected during the analysis, an alert is immediately generated. This alert includes the specific details of the problem, how to deal with it, and information about the nearest support service.

[0538] The generated alerts are sent to the user's mobile devices (e.g., smartphones, tablets) and wearable devices (e.g., smart glasses), allowing the user to immediately check the alerts and take appropriate action.

[0539] As a specific example, if the engine temperature exceeds the normal range, an alert such as the following is generated: "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool." This alert is sent to the mobile device and smart glasses, and can be immediately confirmed by the user.

[0540] The system also includes a means to collect user feedback. After resolving a problem, users can submit their impressions and feedback through a feedback form. This feedback data is stored on a central server and used for future data analysis and service improvement.

[0541] The hardware includes various vehicle sensors (e.g., GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, tire pressure sensors), smart glasses (e.g., Google Glass, Vuzix Blade), mobile devices (e.g., smartphones, tablets), and a central server.

[0542] The software includes APIs (e.g., RESTful APIs) for collecting data from sensors and sending it to a central server, Python scripts for data analysis, SDKs for smart glasses (e.g., Google Glass SDK, Vuzix SDK), and applications for sending notifications.

[0543] Using generative AI models, it is also possible to automatically generate specific and practical countermeasures when an abnormality is detected. For example, the following is an example of a prompt when an abnormality in engine temperature is detected: "The vehicle's engine temperature is above the normal range. Immediately shut down the engine and allow it to cool. Check the sensor data again to see if the engine temperature has returned to normal."

[0544] This minimizes the time lag between detecting an abnormality and issuing instructions to respond, significantly improving safety and efficiency during operation.

[0545] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0546] Step 1:

[0547] The in-vehicle terminal collects data from multiple sensors installed in the vehicle (position sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor). The input is data from each sensor, and the output is a packet of collected data.

[0548] Step 2:

[0549] The in-vehicle terminal transmits the collected sensor data as packets to the central server at regular intervals (e.g., every 5 seconds). The input is the data packets collected in step 1, and the output is the data packets transmitted to the central server.

[0550] Step 3:

[0551] The central server receives data packets sent from the in-vehicle terminal and analyzes them in real time. During the analysis, it detects outliers and irregular patterns. The input is the data packets sent from the in-vehicle terminal, and the output is the analysis results.

[0552] Step 4:

[0553] If the central server detects an abnormality, it generates an alert. This alert contains the specific details of the problem, how to deal with it, and information about the nearest support service. The input is the analysis result obtained in step 3, and the output is the generated alert.

[0554] Step 5:

[0555] The central server notifies the generated alerts to the user's mobile terminal and wearable device. The input is the generated alert information, and the output is the alert displayed on the user's device.

[0556] Step 6:

[0557] The user checks the alert through a mobile terminal or wearable device and takes appropriate action according to the countermeasures presented. The input is the alert information displayed on the device, and the output is the user's specific action.

[0558] Step 7:

[0559] After resolving a problem, the user submits the results and impressions through a feedback form. The input is the user's feedback information, and the output is the submitted feedback data.

[0560] Step 8:

[0561] The central server stores the collected feedback data and uses it for future data analysis and service improvement. The input is the feedback data submitted by users, and the output is the stored data and analysis results.

[0562] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0563] The present invention combines a system that collects data from multiple sensors in a vehicle, analyzes the data to detect abnormalities, and provides the user with a prompt and optimal response, with an emotion engine that recognizes the user's emotional state. Specific embodiments for implementing the present invention will be described below.

[0564] An in-vehicle terminal is a device installed in a vehicle that collects data from GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, tire pressure sensors, etc. This data is sent to a central server at regular intervals. For example, every 5 seconds, the data from each sensor is collected into packets and sent to the server.

[0565] The central server receives data sent from the in-vehicle device. The received data is analyzed in real time, and if an abnormal value or irregular pattern is detected, an alert is generated. For example, if the engine temperature exceeds the normal range, an abnormality is detected based on that data.

[0566] The generated alert is sent to the user's mobile device (such as a smartphone or tablet). By opening the application, the user can check the details of the alert. For example, specific countermeasures such as "The engine temperature is abnormally high. Immediately stop the engine and allow it to cool" are displayed.

[0567] The present invention also incorporates an emotion engine for recognizing the user's emotional state, which uses the camera and / or microphone on the user's mobile device to analyze the user's facial expressions and voice to detect emotions such as stress, anger, fatigue, etc. in real time.

[0568] Based on the emotional state detected by the emotion engine, the system can customize responses to the user. For example, if the user is stressed, the system can provide relaxing music and voice guidance along with an alert notification. If the user shows high levels of anger, the system can provide gentle advice on safe driving.

[0569] As a concrete example, consider the case where a user's tire pressure drops while driving. The in-vehicle device detects abnormal data from the tire pressure sensor and sends it to a central server. The server immediately analyzes the abnormality, generates an alert stating "Tire pressure is below the specified value," and notifies the user's mobile device. The user's mobile device then uses a camera and microphone to analyze the user's facial expressions and voice, and detects that the user is feeling stressed at this point. In response, the system notifies the user of the alert, plays relaxing music, and provides route guidance to the nearest gas station. After the problem is resolved, a feedback form is displayed and the user's experience is sent to the server.

[0570] This allows the system to not only respond to problems but also provide support that takes into account the user's emotional state, allowing the user to continue driving with peace of mind while the system continues to improve.

[0571] The processing flow will be explained below.

[0572] Step 1:

[0573] The device collects data from the vehicle's GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor, and other sensors.

[0574] Step 2:

[0575] The terminal collects data and compiles it into packets every 5 seconds. The terminal then organizes the latest data values ​​obtained from each sensor into a single data set.

[0576] Step 3:

[0577] The device sends the assembled data packets to a central server, which then sends the data over the internet or a communications network.

[0578] Step 4:

[0579] The server receives the data packets sent from the terminal. The server has a receiving function to process large amounts of data quickly.

[0580] Step 5:

[0581] The server analyzes the data it receives in real time. Specific algorithms are used to detect abnormal values ​​and patterns in the data from each sensor. For example, if the engine temperature exceeds the normal range, it will determine that this is an abnormality.

[0582] Step 6:

[0583] If the server detects an abnormality through analysis, it generates an alert, which includes information about the problem, recommended actions to take, and emergency instructions.

[0584] Step 7:

[0585] The server notifies the user of the generated alert via push notification, SMS message, or other methods.

[0586] Step 8:

[0587] The user's mobile device will receive an alert and display a notification, and the user can view more information through the application.

[0588] Step 9:

[0589] The emotion engine analyzes the user's facial expressions and voice using the camera and microphone installed in the user's mobile device, and detects emotional states such as stress, anger, and fatigue in real time.

[0590] Step 10:

[0591] The emotion engine sends the user's emotional state to the server, for example, if the user is showing a high level of stress, it provides that information to the server.

[0592] Step 11:

[0593] The server takes into account the user's emotional state and sends a customized response to the alert to the user's mobile device, such as providing relaxing music or a voice guide.

[0594] Step 12:

[0595] The user reviews the alert details and follows the customized remedial action provided by the system, for example, "Engine temperature is too high. Stop engine and allow it to cool down," while listening to relaxing music.

[0596] Step 13:

[0597] After the user has finished troubleshooting, the mobile device displays a feedback form, in which the user can enter information such as their level of satisfaction, the outcome of the troubleshooting, and their emotional state.

[0598] Step 14:

[0599] The mobile devices collect and send the feedback data to a central server, allowing the system to use the feedback information to improve future operations and the accuracy of its prediction models.

[0600] The above is the specific program processing flow of this system, which allows the user to quickly detect problems that occur while driving, take appropriate countermeasures, and receive support that takes into consideration the user's emotional state.

[0601] Example 2

[0602] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0603] Modern vehicle management systems detect vehicle anomalies and notify the user, but do not take into account the user's emotional state. As a result, users may feel excessive stress or anger when an abnormality occurs, making it difficult to respond optimally. The present invention aims to provide a system that not only analyzes data collected from multiple sensors in the vehicle to detect anomalies, but also recognizes the user's emotional state and provides optimal countermeasures based on that information.

[0604] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0605] In this invention, the server includes: means for an in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server; means for the central server to analyze the data in real time and generate an alert when an abnormality is detected; means for notifying the user of the alert to a mobile device and providing countermeasures for the abnormality; means for the mobile device to analyze the user's facial expressions and voice using a camera and microphone to detect the user's emotional state in real time; means for customizing the content of the alert and countermeasures based on the emotional state detected by the emotion engine; and means for collecting feedback from the user, saving the data, and using it for analysis. This enables a quick and optimal response to vehicle abnormalities while also taking the user's emotional state into consideration.

[0606] An "on-board terminal" is a device installed in a vehicle that collects data from multiple sensors and transmits it to a central server.

[0607] A "sensor" is a device that detects and collects information about the vehicle's status and the external environment, including location data, speed data, acceleration data, engine temperature data, and tire pressure data.

[0608] The "central server" is a server device that receives data sent from the vehicle-mounted terminal, analyzes it in real time, and detects abnormalities.

[0609] An "alert" is a warning message generated by the central server when an abnormality is detected as a result of data analysis.

[0610] A "mobile terminal" is a mobile device such as a smartphone or tablet owned by a user, which receives alert notifications from a central server.

[0611] The "emotion engine" is a system that uses the camera and microphone of a mobile device to analyze the user's facial expressions and voice, and detects the user's emotional state in real time.

[0612] "Feedback" refers to opinions and ratings collected from users, and is data used to improve the system and evaluate alert responses.

[0613] "Relaxing music" is music intended to relieve the user's stress and tension, and can be played by the emotion engine based on the user's emotional state.

[0614] A "trouble" is an event that prevents the vehicle from operating normally, including vehicle abnormalities and breakdowns.

[0615] The present invention provides a system that collects data from multiple sensors in a vehicle, analyzes the data to detect abnormalities, and also recognizes the emotional state of the user, and provides optimal countermeasures based on the collected data. Specific embodiments for carrying out the present invention will be described below.

[0616] System configuration

[0617] The system consists of the following main parts:

[0618] In-vehicle terminal

[0619] Central Server

[0620] Mobile devices

[0621] Emotion Engine

[0622] In-vehicle terminal

[0623] An onboard device is a device installed in a vehicle that collects data from the following sensors:

[0624] GPS sensor

[0625] Speed ​​sensor

[0626] Accelerometer

[0627] Engine temperature sensor

[0628] tire pressure sensor

[0629] The data collected from these sensors is aggregated into packets at regular intervals (e.g., every 5 seconds) and sent to a central server.

[0630] Specific working example:

[0631] When a vehicle turns a sharp corner, the acceleration sensor detects high lateral acceleration (G-force), and the onboard terminal compiles it into packets.

[0632] Central Server

[0633] The central server receives data sent from the in-vehicle device and analyzes it in real time. If an abnormal value or irregular pattern is detected in the received data, an alert is generated. This alert is sent to the user's mobile device.

[0634] Specific working example:

[0635] If the engine temperature sensor detects a temperature outside the normal range, an alert will be generated stating, "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool." and sent to the user's mobile device.

[0636] Mobile devices

[0637] The mobile device is a user's smartphone or tablet, which receives an alert notification from the central server. By opening the application, the user can check the details of the alert and receive suggestions for countermeasures.

[0638] Specific working example:

[0639] If the tire pressure drops below the specified value, the system will display an alert stating "Tire pressure is below the specified value" along with directions to the nearest gas station.

[0640] Emotion engine and customized responses

[0641] The emotion engine uses the mobile device's camera and microphone to analyze the user's facial expressions and voice to detect emotional states such as stress, anger, fatigue, etc. in real time. Based on the results, the system further customizes the immediate response measures for the user.

[0642] Specific working example:

[0643] If the user is recognized as being in a stressful state, relaxing music is played on the mobile device and a guide voice prompts the user to "relax."

[0644] Gathering feedback

[0645] After the problem is resolved, the user's mobile device will display a feedback form to collect the user's experience, which will be stored on a central server and used to improve the system in the future.

[0646] Specific working example:

[0647] After the user has resolved the issue, a feedback form is displayed asking "How was your experience with our service?", and the user's input is received and sent to a central server.

[0648] Prompt Sentence Examples

[0649] "Lateral acceleration data detected by the accelerometer is collected and sent to a central server."

[0650] "Generates an alert if engine temperatures exceed normal range."

[0651] "When tire pressure drops, the user is given directions to the nearest gas station."

[0652] "If the user is feeling stressed, it will play relaxing music and provide a guide voice."

[0653] This allows the system to provide integrated support that takes into account both troubleshooting and the user's emotional state, allowing the user to continue driving with peace of mind.

[0654] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0655] Step 1:

[0656] The onboard terminal collects data from each sensor.

[0657] Inputs: GPS data, speed data, acceleration data, engine temperature data, tire pressure data.

[0658] Data processing / calculation: The raw data obtained from each sensor is compiled into packets by time.

[0659] Output: Sensor data packets.

[0660] Specific operation: When a vehicle turns a sharp corner, the acceleration sensor detects high lateral acceleration (G-force), and the onboard terminal assembles it into packets.

[0661] Step 2:

[0662] The in-vehicle terminal transmits the collected data packets to a central server.

[0663] Input: Sensor data packets.

[0664] Data processing / calculation: Encryption of packet data and securing of communication lines.

[0665] Output: Sending encrypted sensor data packets.

[0666] How it works: The in-vehicle device sends encrypted data packets to a central server via Wi-Fi or a cellular network.

[0667] Step 3:

[0668] A central server receives the data packets and analyzes them in real time.

[0669] Input: Encrypted sensor data packet.

[0670] Data processing / computation: Decrypting encrypted data and applying algorithms to detect outliers and irregular patterns.

[0671] Output: Anomaly detection results and alert generation.

[0672] Specific operation: The central server analyzes the received engine temperature data and compares it with the reference range to detect abnormalities. For example, if the engine temperature exceeds the normal range, an alert is generated.

[0673] Step 4:

[0674] A central server generates alerts and sends them to the user's mobile device.

[0675] Input: Anomaly detection results.

[0676] Data processing / calculation: Generating alert messages and creating notification data packets.

[0677] Output: Alert notification to mobile device.

[0678] Specific operation: If the engine temperature is abnormally high, an alert will be generated stating "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool down" and a notification will be sent to the mobile device.

[0679] Step 5:

[0680] The mobile device uses a camera and microphone to analyze the user's facial expressions and voice to detect their emotional state in real time.

[0681] Input: User's facial expression data, voice data.

[0682] Data processing / computation: Apply image and audio analysis algorithms to identify emotional states.

[0683] Output: User's emotional state data.

[0684] Specific operation: The user's voice is captured and recognized as "stress" by the emotion engine. "Stress" is also detected from facial expressions.

[0685] Step 6:

[0686] Based on the emotional state detected by the emotion engine, a central server customizes alert notification content and responses.

[0687] Input: User emotional state data.

[0688] Data processing / computation: Applying customized algorithms based on emotional state data.

[0689] Output: Customized alert notifications and remedial actions.

[0690] Specific operation: If the user is recognized as being in a stressful state, relaxing music will be played on the mobile device and a guide voice will prompt the user to "relax."

[0691] Step 7:

[0692] After the problem is solved, the mobile terminal displays a feedback form to collect feedback from the user.

[0693] Input: User's post-resolution feedback answer.

[0694] Data processing / calculation: Collection of feedback data and storage in a database for analysis.

[0695] Output: Save feedback data.

[0696] Specific behavior: After the problem is resolved, a feedback form asking "How was your service?" is displayed, and user input is collected and sent to a central server.

[0697] (Application example 2)

[0698] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0699] Conventional vehicle management systems have been effective to a certain extent in detecting abnormalities and notifying drivers using sensors, but they have limitations in providing optimal countermeasures that take into account the user's emotional state. Furthermore, when users feel stressed or fatigued, they do not provide sufficient support to alleviate these conditions. Therefore, there is a need for a system that is ergonomically friendly to users while improving driving safety.

[0700] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0701] In this invention, the server includes means for an in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server, means for the central server to analyze the data in real time and generate an alert when an abnormality is detected, means for notifying the user of the alert to the user's mobile terminal and providing countermeasures for the abnormality, means for recognizing the user's emotional state and customizing the countermeasures based on the emotional state, and means for collecting feedback from the user and saving the data for use in analysis, thereby making it possible to provide a system that not only detects abnormalities but also responds to the user's emotional state.

[0702] An "on-board terminal" is a device installed in a vehicle that collects data from multiple sensors and transmits the data to a central server.

[0703] A "sensor" is a device used to detect the condition and movement of a vehicle, and is used to collect data such as GPS, speed, acceleration, engine, and tire pressure.

[0704] A "central server" is a device or system that analyzes data sent from vehicle-mounted terminals in real time, detects abnormalities, and generates alerts.

[0705] An "alert" is a warning notification that is generated when an abnormality is detected based on data collected from a sensor, and is intended to inform the user of information about the abnormality and how to deal with it.

[0706] A "mobile terminal" is a device that can be carried by a user, including a smartphone or tablet, for receiving alert notifications and abnormality response measures.

[0707] "Emotional state" refers to the mental state detected from the user's facial expression and voice, and includes, for example, stress, anger, fatigue, etc.

[0708] An "emotion engine" is a system or software that analyzes a user's facial expressions and voice data to recognize their emotional state.

[0709] "Feedback" refers to opinions and information provided by users about their experience using the system, which is collected for the purpose of improving and optimizing the system.

[0710] The system of the present invention is composed of an in-vehicle terminal, a central server, an emotion engine, and a user's mobile terminal. The specific configuration and operation will be explained below.

[0711] In-vehicle terminal

[0712] The in-vehicle terminal is equipped with multiple sensors, such as a GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, and tire pressure sensor. These sensors constantly monitor the vehicle's condition and movement and collect data. The collected data is sent to a central server at regular intervals, for example, every five seconds.

[0713] Central Server

[0714] The central server receives and analyzes data sent from the in-vehicle device in real time. If an abnormal value is detected as a result of the analysis, an alert is generated. For example, if the engine temperature exceeds the normal range, the data is judged to be abnormal and an alert is generated based on that. The generated alert is sent to the user's mobile device.

[0715] Mobile Devices and Emotion Engines

[0716] The user's mobile device is equipped with an emotion engine that uses a camera and microphone to analyze the user's facial expressions and voice. The emotion engine recognizes the user's mental state (stress, anger, fatigue, etc.) in real time. For example, if the user is in a stressful state when an alert is generated, the system will play relaxing music and suggest optimal countermeasures, such as providing route guidance to the nearest gas station.

[0717] Feedback and Improvements

[0718] The system collects user feedback and is continuously improved based on experience. Feedback is collected via mobile devices and analyzed and stored on a central server. This improves the accuracy of the system and the user experience.

[0719] Specific examples

[0720] For example, consider the case where a user's tire pressure drops while driving. The in-vehicle device detects abnormal data from the tire pressure sensor and sends it to a central server. The server immediately analyzes the abnormality, generates an alert stating "Tire pressure is below the specified value," and notifies the user's mobile device. The user's mobile device then uses its camera and microphone to analyze the user's facial expressions and voice, and detects that the user is feeling stressed at this point. In response, the system plays relaxing music along with the alert notification and provides route guidance to the nearest gas station.

[0721] Prompt Sentence Examples

[0722] Anomaly Detection Systems:

[0723] Sensor data such as GPS, speed, engine temperature, and tire pressure collected by the in-vehicle device is sent to a central server every five seconds for real-time analysis. If an abnormality is detected, an alert is generated and a notification of countermeasures is sent to the user's smartphone. In addition, the system analyzes the user's emotional state using a camera and microphone, and plays relaxing music if the user is under stress.

[0724] As described above, the embodiments of the present invention can provide a safe and comfortable driving environment by detecting abnormalities in a vehicle and responding to the emotional state of the user.

[0725] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0726] Step 1:

[0727] The in-vehicle terminal collects data from multiple sensors, specifically GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, and tire pressure sensors. The input is data from each sensor, and the output is a collection of that data.

[0728] Step 2:

[0729] The data collected by the in-vehicle device is sent to a central server every five seconds. Specifically, the data is packaged into packets and sent over the Internet. The input is a collection of data collected from the sensors, and the output is the data packet sent to the central server.

[0730] Step 3:

[0731] The central server analyzes the received data in real time, stores it in a database, and runs algorithms to detect abnormal values ​​and patterns. The input is the data packets sent from the in-vehicle device, and the output is the result of the anomaly detection.

[0732] Step 4:

[0733] When the central server detects an anomaly, it generates an alert. Specifically, it creates a warning message depending on the type of anomaly. The input is the anomaly detection result, and the output is the generated alert message.

[0734] Step 5:

[0735] The generated alert is sent to the user's mobile device. Specifically, the alert message is pushed to the mobile device. The input is the alert message, and the output is the alert displayed on the mobile device.

[0736] Step 6:

[0737] The user's mobile device receives the alert and uses its camera and microphone to analyze the user's emotional state. Specifically, the camera video and audio data are input into an emotion engine to recognize emotional states such as stress, anger, and fatigue. The input is data obtained from the camera and microphone, and the output is the analyzed emotional state.

[0738] Step 7:

[0739] The emotion engine customizes responses based on the user's emotional state, such as playing relaxing music or running an algorithm to suggest optimal responses. The input is the analyzed emotional state, and the output is a customized response provided to the user.

[0740] Step 8:

[0741] User feedback is collected and used to improve the system. Specifically, feedback is collected from users via an application and stored in a database. The input is feedback data from users, and the output is the feedback stored in the database.

[0742] Step 9:

[0743] The feedback data is analyzed and reflected in system improvements. Specifically, the feedback data is analyzed, system improvements are extracted, and software updates and algorithm optimization are carried out. The input is the feedback data, and the output is improved system functions.

[0744] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0745] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0746] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0747] [Third embodiment]

[0748] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0749] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0750] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0751] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0752] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0753] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0754] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0755] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0756] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0757] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0758] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0759] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0760] The present invention relates to a system for collecting data from a plurality of sensors in a vehicle, analyzing the data to detect abnormalities, and providing a user with a prompt and optimal countermeasure. Specific embodiments for carrying out the present invention will be described below.

[0761] An in-vehicle terminal is a device installed in a vehicle that collects data from GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, tire pressure sensors, etc. This data is sent to a central server at regular intervals. For example, every 5 seconds, the data from each sensor is collected into packets and sent to the server.

[0762] The central server receives data sent from the in-vehicle devices, analyzes the data in real time, and generates an alert if anomalies or irregular patterns are detected. For example, if the engine temperature significantly exceeds normal values, an alert will be generated indicating a possible overheating.

[0763] The generated alert is sent to the user's mobile device (such as a smartphone or tablet). By opening the application, the user can check the details of the alert. For example, specific countermeasures such as "The engine temperature is abnormally high. Immediately stop the engine and allow it to cool" are displayed.

[0764] The system also provides users with emergency assistance service information. Based on the user's current location, information about the nearest gas station or repair shop and directions to the nearest shop are displayed within the application. For example, specific directions such as "The nearest repair shop is 1.5 kilometers away. Please follow the route below" are provided.

[0765] After a user resolves a problem through the system, they will receive a notification requesting feedback. They can use the feedback form to express their thoughts and the effectiveness of the resolution. This feedback data will be stored on a central server and used to improve future prediction models and services.

[0766] As a concrete example, consider the case where a user experiences a drop in tire pressure while driving. The in-vehicle device detects abnormal data from the tire pressure sensor and sends it to a central server. The server immediately analyzes the abnormality, generates an alert stating "Tire pressure is below the specified value," and notifies the user's mobile device. The user confirms the alert and follows the directions to the nearest gas station. They then report their experience resolving the problem in a feedback form. This allows the system to improve the accuracy of data analysis and alert generation in future cases.

[0767] As described above, the system of the present invention can efficiently and effectively resolve problems that occur while driving a vehicle, and provide a safe and secure driving environment.

[0768] The processing flow will be explained below.

[0769] Step 1:

[0770] The device collects data from the vehicle's GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor, and other sensors.

[0771] Step 2:

[0772] The data collected by the device is aggregated into packets every 5 seconds. For example, the latest data values ​​obtained from each sensor are organized into a single data set.

[0773] Step 3:

[0774] The device sends the assembled data packets to a central server, which then sends the data over the internet or a communications network.

[0775] Step 4:

[0776] The server receives the data packets sent from the terminal. The server has a receiving function to process large amounts of data quickly.

[0777] Step 5:

[0778] The server analyzes the received data and uses specific algorithms to detect abnormal values ​​and patterns in the data from each sensor. For example, if the engine temperature exceeds the normal range, it will determine that this is an abnormality.

[0779] Step 6:

[0780] If the server detects an abnormality through analysis, it generates an alert, which includes information about the problem, recommended actions to take, and emergency instructions.

[0781] Step 7:

[0782] The server notifies the user of the generated alert via push notification, SMS message, or other methods.

[0783] Step 8:

[0784] The user's mobile device will receive an alert and display a notification, and the user can view more information through the application.

[0785] Step 9:

[0786] The user can review the details of the alert and follow the remedial action provided by the system, for example, "Engine temperature is too high. Stop engine and allow it to cool."

[0787] Step 10:

[0788] After the user has finished troubleshooting, the mobile device displays a feedback form, in which the user can enter their level of satisfaction, the results of the troubleshooting, and other information.

[0789] Step 11:

[0790] The mobile devices collect and send the feedback data to a central server, allowing the system to use the feedback information to improve future operations and the accuracy of its prediction models.

[0791] The above is the specific program processing flow of this system, which allows the user to quickly detect problems that occur during operation and take appropriate countermeasures.

[0792] Example 1

[0793] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0794] Modern automobiles are equipped with numerous sensors that can monitor the vehicle's condition. However, there is a lack of systems that can properly analyze sensor data and provide prompt and accurate countermeasures when an abnormality is detected. They also lack the ability to provide emergency assistance information based on the user's current location. Furthermore, there is often no way to collect user feedback to help improve the system. A system that can solve these problems and improve safety and convenience while driving is needed.

[0795] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0796] In this invention, the server includes: a means for an in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server; a means for the central server to analyze the data in real time and generate an alert when an abnormality is detected; a means for notifying the user of the alert to the user's mobile device and providing countermeasures for the abnormality; a means for providing information on the nearest support service based on the user's current location information; and a means for collecting feedback from the user, saving the data, and using it for analysis. This allows the system to quickly detect vehicle abnormalities, provide the user with appropriate countermeasures, and provide support service information, enabling the user to take prompt and appropriate action. Furthermore, collecting feedback data and using it for analysis enables continuous improvement of the system.

[0797] An "on-board terminal" is a device that is installed in a vehicle and collects data from various sensors and transmits it to a central server.

[0798] A "sensor" is a device that monitors the state of a vehicle and measures specific physical quantities, such as position information, speed information, movement data, mechanical information, and tire pressure information.

[0799] The "central server" is a computer system that receives and analyzes data sent from the vehicle-mounted terminal, and if it detects an abnormality, it generates an alert and notifies the user.

[0800] "Data" means a series of measurements or numbers collected from sensors, including location information, speed information, movement data, mechanical information, and tire pressure information.

[0801] An "alert" is a notification message generated when the central server detects an abnormality, informing the user of the nature of the abnormality and how to deal with it.

[0802] A "user's mobile device" is a portable electronic device, such as a smartphone or tablet, that is carried by a user and is used to receive alert notifications and check detailed information.

[0803] "Feedback" refers to opinions and impressions submitted by users after a problem has been resolved, as well as data for system improvement.

[0804] "Support service information" is information about locations and services that are useful for solving vehicle problems, such as the nearest gas station or repair shop, provided based on the user's current location.

[0805] "Location information" is data indicating the current location of the vehicle, and is obtained by a GPS sensor or the like.

[0806] "Speed ​​information" is data indicating the speed of the vehicle, and is acquired by a speed sensor.

[0807] "Movement data" is a series of information related to the movement of a vehicle, and is acquired by an acceleration sensor or the like.

[0808] "Mechanical information" is data indicating the mechanical state of the vehicle, such as the engine condition and temperature, and is acquired by an engine temperature sensor or the like.

[0809] "Tire pressure information" is data indicating the air pressure of the vehicle's tires, and is acquired by a tire pressure sensor.

[0810] These definitions clarify the meaning of the words contained in the claims and facilitate understanding of the scope and nature of the invention.

[0811] This invention is a system that collects data from multiple sensors in a vehicle, analyzes the data to detect abnormalities, and provides the user with a prompt and optimal response. How this invention is implemented will be specifically described below.

[0812] In-vehicle terminal

[0813] An in-vehicle terminal is a device that collects data from multiple sensors installed in a vehicle. The sensors used include a GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, and tire pressure sensor. The terminal obtains location information, speed information, movement data, mechanical information, and tire pressure information from these sensors.

[0814] The onboard device compiles the collected data into packets every five seconds and transmits them to a central server using a cellular communication module. For example, every five seconds, a data packet might be generated that reports an engine temperature of 90 degrees and tire pressure of 32 PSI, and the data is then transmitted to the central server.

[0815] Central Server

[0816] The central server receives data sent from the in-vehicle devices and analyzes it in real time. The server uses a database management system (e.g., MySQL or PostgreSQL) to store the received data. It also uses a data analysis library (e.g., Python's pandas or scikit-learn) to run programs that detect outliers and irregular patterns. If an abnormality is detected, the server generates an alert according to the content. For example, if the engine temperature exceeds 100 degrees, it generates an alert stating, "The engine temperature is abnormally high."

[0817] The generated alert is immediately sent to the user's mobile device. Specifically, the alert message is sent using the push notification function.

[0818] User's mobile device

[0819] The user's mobile device is the device that receives the alert notification. This includes smartphones and tablets. The user can check the notification and open an application to view detailed information about the alert. For example, specific countermeasures such as "The engine temperature is abnormally high. Immediately stop the engine and allow it to cool down" are displayed.

[0820] Additionally, the system will provide information about the nearest assistance service based on the user's current location, for example, by displaying a message within the application saying, "The nearest gas station is 1.5 kilometers away. Please follow the route below."

[0821] feedback

[0822] After solving a problem, users report their impressions and effectiveness of the solution through a feedback form. The feedback data is stored on a central server and used for future analysis of prediction models and system improvements.

[0823] Prompt Sentence Examples

[0824] For example, by entering a prompt such as "Please explain what to do if the vehicle's engine temperature becomes abnormally high" into the generative AI model, it is possible to generate an explanation of specific countermeasures.

[0825] The present invention is a system that quickly detects abnormalities in a vehicle and provides the user with appropriate countermeasures, thereby improving safety and convenience for the user.

[0826] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0827] Step 1:

[0828] Terminal: The in-vehicle terminal collects data from various sensors installed in the vehicle (GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor, etc.).

[0829] Input: Data from each sensor (location information, speed information, movement data, machine information, tire pressure information).

[0830] How it works: Every 5 seconds, the onboard device aggregates this data and generates a time-stamped data packet.

[0831] Output: Consolidated data packet (e.g., engine temperature = 90 degrees, tire pressure = 32 PSI, time = 2023-10-10 10:00:05).

[0832] Step 2:

[0833] Terminal: The in-vehicle terminal generates and transmits the data packets to the central server.

[0834] Input: Consolidated data packet.

[0835] How it works: It uses a cellular communication module to send data packets over the internet to a central server.

[0836] Output: Data packets sent to the central server.

[0837] Step 3:

[0838] Server: The central server receives data packets sent from the in-vehicle terminals.

[0839] Input: Data packet sent from the in-vehicle terminal.

[0840] What it does: Uses the data reception module to receive packets and store them in a database (e.g. MySQL or PostgreSQL).

[0841] Output: Vehicle status data stored in a database.

[0842] Step 4:

[0843] Server: The central server analyzes the received data in real time and detects anomalies.

[0844] Input: Vehicle condition data stored in a database.

[0845] How it works: Using data analysis libraries (e.g., Python's pandas, scikit-learn), it uses rule-based or machine learning models to detect outliers and irregular patterns. For example, an engine temperature above 100 degrees is considered abnormal.

[0846] Output: Alert information when an abnormality is detected (e.g. engine temperature abnormality).

[0847] Step 5:

[0848] Server: When an anomaly is detected, a corresponding alert is generated and sent to the user's mobile device.

[0849] Input: Alert information if an anomaly is detected.

[0850] What it does: Uses push notifications to send an alert to the user's mobile device, for example, a message saying "Engine temperature is too high."

[0851] Output: An alert notification that appears on the user's mobile device.

[0852] Step 6:

[0853] User: The user receives an alert notification on their mobile device and opens the application to view more information.

[0854] Input: Alert notification.

[0855] Action: Open the application and check the details of the error and the countermeasures. For example, the countermeasures displayed will be "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool."

[0856] Output: A screen showing detailed information about the anomaly and specific countermeasures.

[0857] Step 7:

[0858] Server: Provides information about the nearest support service based on the user's current location.

[0859] Input: User's current location.

[0860] How it works: Using GPS location information, it searches a support service database and obtains information on the nearest gas station or repair shop.

[0861] Output: Information about the nearest assistance service and directions displayed on the user's mobile device (e.g., "The nearest gas station is 1.5 kilometers away. Please follow the route below.").

[0862] Step 8:

[0863] Server: Collect user feedback after the problem is resolved.

[0864] Input: Feedback request after troubleshooting.

[0865] What it does: Sends a feedback form to the user's mobile device.

[0866] Output: User-entered feedback (e.g., "I successfully resolved my engine overheating issue. Your guidance was very clear and helpful.").

[0867] Step 9:

[0868] User: Fill in the feedback form with your thoughts and opinions on how the problem was resolved and submit it.

[0869] Input: User feedback.

[0870] Action: Enter the required information into the feedback form and click the submit button.

[0871] Output: The feedback data sent.

[0872] Step 10:

[0873] Server: Stores the feedback data and analyzes it for future prediction models and system improvements.

[0874] Input: The submitted feedback data.

[0875] How it works: Feedback data is stored in a database and analyzed using a data analysis library.

[0876] Output: Data analysis results for improved predictive models and system improvements.

[0877] (Application example 1)

[0878] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0879] Detecting abnormalities and problems that occur during the operation of autonomous vehicles in real time and providing users with prompt and appropriate countermeasures is important for improving operational safety and efficiency. However, with conventional systems, there is a time lag between the detection of an abnormality and instructions to respond, as well as limitations in notification methods, making it difficult for users to take appropriate action immediately. In addition, feedback data is not properly collected and analyzed, and there are cases where it cannot be used to improve future countermeasures.

[0880] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0881] In this invention, the server includes: means for the in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server; means for the central server to analyze the data in real time and generate an alert when an abnormality is detected; means for notifying the user of the alert to the user's mobile terminal and wearable device and providing countermeasures for the abnormality; and means for collecting feedback from the user, saving the data, and using it for analysis. This minimizes the time lag from abnormality detection to response instructions, allowing the user to immediately confirm and respond via a wearable device such as smart glasses, thereby significantly improving the safety and efficiency of driving.

[0882] An "on-board terminal" is a device installed in a vehicle that collects data from multiple sensors and transmits it to a central server.

[0883] The "multiple sensors" are various sensor devices that collect data such as vehicle position, speed, acceleration, engine temperature, and tire pressure.

[0884] The "central server" is a computer system that analyzes data sent from the vehicle-mounted terminal in real time and generates an alert when an abnormality is detected.

[0885] An "alert" is a warning or notification generated when the central server detects an abnormality, and is information intended to prompt the user to take prompt action.

[0886] A "mobile terminal" is a computing device that a user can carry with them, such as a smartphone or tablet.

[0887] A "wearable device" is a device that can be worn by a user, such as smart glasses or a wristwatch-type terminal.

[0888] "Countermeasures" refer to specific actions or procedures that a user should take when an abnormality occurs.

[0889] "Feedback" refers to information provided by users about the effectiveness of the problem resolution, their impressions, and their experiences, and is data that will be used for future data analysis and service improvement.

[0890] "Location data" refers to data relating to the current location of a vehicle obtained using a GPS or the like.

[0891] "Speed ​​data" is data relating to the vehicle's traveling speed.

[0892] "Acceleration data" is data relating to the acceleration of the vehicle.

[0893] "Engine data" refers to data relating to the temperature and operating conditions of the engine.

[0894] "Tire pressure data" refers to data relating to tire air pressure.

[0895] "Support services" refer to service facilities such as gas stations and repair shops that users can use when there is a problem with their vehicle.

[0896] This invention is a system that detects abnormalities and problems that occur during the operation of an autonomous vehicle in real time and provides users with prompt and appropriate countermeasures. This system includes an in-vehicle terminal, a central server, a mobile terminal, and a wearable device.

[0897] First, the in-vehicle terminal collects data from multiple sensors installed in the vehicle. The collected data includes location data, speed data, acceleration data, engine data, and tire pressure data. This data is sent to a central server at regular intervals (for example, every 5 seconds).

[0898] The central server then analyzes the data sent from the in-vehicle device in real time. If an abnormality is detected during the analysis, an alert is immediately generated. This alert includes the specific details of the problem, how to deal with it, and information about the nearest support service.

[0899] The generated alerts are sent to the user's mobile devices (e.g., smartphones, tablets) and wearable devices (e.g., smart glasses), allowing the user to immediately check the alerts and take appropriate action.

[0900] As a specific example, if the engine temperature exceeds the normal range, an alert such as the following is generated: "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool." This alert is sent to the mobile device and smart glasses, and can be immediately confirmed by the user.

[0901] The system also includes a means to collect user feedback. After resolving a problem, users can submit their impressions and feedback through a feedback form. This feedback data is stored on a central server and used for future data analysis and service improvement.

[0902] The hardware includes various vehicle sensors (e.g., GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, tire pressure sensors), smart glasses (e.g., Google Glass, Vuzix Blade), mobile devices (e.g., smartphones, tablets), and a central server.

[0903] The software includes APIs (e.g., RESTful APIs) for collecting data from sensors and sending it to a central server, Python scripts for data analysis, SDKs for smart glasses (e.g., Google Glass SDK, Vuzix SDK), and applications for sending notifications.

[0904] Using generative AI models, it is also possible to automatically generate specific and practical countermeasures when an abnormality is detected. For example, the following is an example of a prompt when an abnormality in engine temperature is detected: "The vehicle's engine temperature is above the normal range. Immediately shut down the engine and allow it to cool. Check the sensor data again to see if the engine temperature has returned to normal."

[0905] This minimizes the time lag between detecting an abnormality and issuing instructions to respond, significantly improving safety and efficiency during operation.

[0906] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0907] Step 1:

[0908] The in-vehicle terminal collects data from multiple sensors installed in the vehicle (position sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor). The input is data from each sensor, and the output is a packet of collected data.

[0909] Step 2:

[0910] The in-vehicle terminal transmits the collected sensor data as packets to the central server at regular intervals (e.g., every 5 seconds). The input is the data packets collected in step 1, and the output is the data packets transmitted to the central server.

[0911] Step 3:

[0912] The central server receives data packets sent from the in-vehicle terminal and analyzes them in real time. During the analysis, it detects outliers and irregular patterns. The input is the data packets sent from the in-vehicle terminal, and the output is the analysis results.

[0913] Step 4:

[0914] If the central server detects an abnormality, it generates an alert. This alert contains the specific details of the problem, how to deal with it, and information about the nearest support service. The input is the analysis result obtained in step 3, and the output is the generated alert.

[0915] Step 5:

[0916] The central server notifies the generated alerts to the user's mobile terminal and wearable device. The input is the generated alert information, and the output is the alert displayed on the user's device.

[0917] Step 6:

[0918] The user checks the alert through a mobile terminal or wearable device and takes appropriate action according to the countermeasures presented. The input is the alert information displayed on the device, and the output is the user's specific action.

[0919] Step 7:

[0920] After resolving a problem, the user submits the results and impressions through a feedback form. The input is the user's feedback information, and the output is the submitted feedback data.

[0921] Step 8:

[0922] The central server stores the collected feedback data and uses it for future data analysis and service improvement. The input is the feedback data submitted by users, and the output is the stored data and analysis results.

[0923] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0924] The present invention combines a system that collects data from multiple sensors in a vehicle, analyzes the data to detect abnormalities, and provides the user with a prompt and optimal response, with an emotion engine that recognizes the user's emotional state. Specific embodiments for implementing the present invention will be described below.

[0925] An in-vehicle terminal is a device installed in a vehicle that collects data from GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, tire pressure sensors, etc. This data is sent to a central server at regular intervals. For example, every 5 seconds, the data from each sensor is collected into packets and sent to the server.

[0926] The central server receives data sent from the in-vehicle device. The received data is analyzed in real time, and if an abnormal value or irregular pattern is detected, an alert is generated. For example, if the engine temperature exceeds the normal range, an abnormality is detected based on that data.

[0927] The generated alert is sent to the user's mobile device (such as a smartphone or tablet). By opening the application, the user can check the details of the alert. For example, specific countermeasures such as "The engine temperature is abnormally high. Immediately stop the engine and allow it to cool" are displayed.

[0928] The present invention also incorporates an emotion engine for recognizing the user's emotional state, which uses the camera and / or microphone on the user's mobile device to analyze the user's facial expressions and voice to detect emotions such as stress, anger, fatigue, etc. in real time.

[0929] Based on the emotional state detected by the emotion engine, the system can customize responses to the user. For example, if the user is stressed, the system can provide relaxing music and voice guidance along with an alert notification. If the user shows high levels of anger, the system can provide gentle advice on safe driving.

[0930] As a concrete example, consider the case where a user's tire pressure drops while driving. The in-vehicle device detects abnormal data from the tire pressure sensor and sends it to a central server. The server immediately analyzes the abnormality, generates an alert stating "Tire pressure is below the specified value," and notifies the user's mobile device. The user's mobile device then uses a camera and microphone to analyze the user's facial expressions and voice, and detects that the user is feeling stressed at this point. In response, the system notifies the user of the alert, plays relaxing music, and provides route guidance to the nearest gas station. After the problem is resolved, a feedback form is displayed and the user's experience is sent to the server.

[0931] This allows the system to not only respond to problems but also provide support that takes into account the user's emotional state, allowing the user to continue driving with peace of mind while the system continues to improve.

[0932] The processing flow will be explained below.

[0933] Step 1:

[0934] The device collects data from the vehicle's GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor, and other sensors.

[0935] Step 2:

[0936] The terminal collects data and compiles it into packets every 5 seconds. The terminal then organizes the latest data values ​​obtained from each sensor into a single data set.

[0937] Step 3:

[0938] The device sends the assembled data packets to a central server, which then sends the data over the internet or a communications network.

[0939] Step 4:

[0940] The server receives the data packets sent from the terminal. The server has a receiving function to process large amounts of data quickly.

[0941] Step 5:

[0942] The server analyzes the data it receives in real time. Specific algorithms are used to detect abnormal values ​​and patterns in the data from each sensor. For example, if the engine temperature exceeds the normal range, it will determine that this is an abnormality.

[0943] Step 6:

[0944] If the server detects an abnormality through analysis, it generates an alert, which includes information about the problem, recommended actions to take, and emergency instructions.

[0945] Step 7:

[0946] The server notifies the user of the generated alert via push notification, SMS message, or other methods.

[0947] Step 8:

[0948] The user's mobile device will receive an alert and display a notification, and the user can view more information through the application.

[0949] Step 9:

[0950] The emotion engine analyzes the user's facial expressions and voice using the camera and microphone installed in the user's mobile device, and detects emotional states such as stress, anger, and fatigue in real time.

[0951] Step 10:

[0952] The emotion engine sends the user's emotional state to the server, for example, if the user is showing a high level of stress, it provides that information to the server.

[0953] Step 11:

[0954] The server takes into account the user's emotional state and sends a customized response to the alert to the user's mobile device, such as providing relaxing music or a voice guide.

[0955] Step 12:

[0956] The user reviews the alert details and follows the customized remedial action provided by the system, for example, "Engine temperature is too high. Stop engine and allow it to cool down," while listening to relaxing music.

[0957] Step 13:

[0958] After the user has finished troubleshooting, the mobile device displays a feedback form, in which the user can enter information such as their level of satisfaction, the outcome of the troubleshooting, and their emotional state.

[0959] Step 14:

[0960] The mobile devices collect and send the feedback data to a central server, allowing the system to use the feedback information to improve future operations and the accuracy of its prediction models.

[0961] The above is the specific program processing flow of this system, which allows the user to quickly detect problems that occur while driving, take appropriate countermeasures, and receive support that takes into consideration the user's emotional state.

[0962] Example 2

[0963] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0964] Modern vehicle management systems detect vehicle anomalies and notify the user, but do not take into account the user's emotional state. As a result, users may feel excessive stress or anger when an abnormality occurs, making it difficult to respond optimally. The present invention aims to provide a system that not only analyzes data collected from multiple sensors in the vehicle to detect anomalies, but also recognizes the user's emotional state and provides optimal countermeasures based on that information.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0966] In this invention, the server includes: means for an in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server; means for the central server to analyze the data in real time and generate an alert when an abnormality is detected; means for notifying the user of the alert to a mobile device and providing countermeasures for the abnormality; means for the mobile device to analyze the user's facial expressions and voice using a camera and microphone to detect the user's emotional state in real time; means for customizing the content of the alert and countermeasures based on the emotional state detected by the emotion engine; and means for collecting feedback from the user, saving the data, and using it for analysis. This enables a quick and optimal response to vehicle abnormalities while also taking the user's emotional state into consideration.

[0967] An "on-board terminal" is a device installed in a vehicle that collects data from multiple sensors and transmits it to a central server.

[0968] A "sensor" is a device that detects and collects information about the vehicle's status and the external environment, including location data, speed data, acceleration data, engine temperature data, and tire pressure data.

[0969] The "central server" is a server device that receives data sent from the vehicle-mounted terminal, analyzes it in real time, and detects abnormalities.

[0970] An "alert" is a warning message generated by the central server when an abnormality is detected as a result of data analysis.

[0971] A "mobile terminal" is a mobile device such as a smartphone or tablet owned by a user, which receives alert notifications from a central server.

[0972] The "emotion engine" is a system that uses the camera and microphone of a mobile device to analyze the user's facial expressions and voice, and detects the user's emotional state in real time.

[0973] "Feedback" refers to opinions and ratings collected from users, and is data used to improve the system and evaluate alert responses.

[0974] "Relaxing music" is music intended to relieve the user's stress and tension, and can be played by the emotion engine based on the user's emotional state.

[0975] A "trouble" is an event that prevents the vehicle from operating normally, including vehicle abnormalities and breakdowns.

[0976] The present invention provides a system that collects data from multiple sensors in a vehicle, analyzes the data to detect abnormalities, and also recognizes the emotional state of the user, and provides optimal countermeasures based on the collected data. Specific embodiments for carrying out the present invention will be described below.

[0977] System configuration

[0978] The system consists of the following main parts:

[0979] In-vehicle terminal

[0980] Central Server

[0981] Mobile devices

[0982] Emotion Engine

[0983] In-vehicle terminal

[0984] An onboard device is a device installed in a vehicle that collects data from the following sensors:

[0985] GPS sensor

[0986] Speed ​​sensor

[0987] Accelerometer

[0988] Engine temperature sensor

[0989] tire pressure sensor

[0990] The data collected from these sensors is aggregated into packets at regular intervals (e.g., every 5 seconds) and sent to a central server.

[0991] Specific working example:

[0992] When a vehicle turns a sharp corner, the acceleration sensor detects high lateral acceleration (G-force), and the onboard terminal compiles it into packets.

[0993] Central Server

[0994] The central server receives data sent from the in-vehicle device and analyzes it in real time. If an abnormal value or irregular pattern is detected in the received data, an alert is generated. This alert is sent to the user's mobile device.

[0995] Specific working example:

[0996] If the engine temperature sensor detects a temperature outside the normal range, an alert will be generated stating, "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool." and sent to the user's mobile device.

[0997] Mobile devices

[0998] The mobile device is a user's smartphone or tablet, which receives an alert notification from the central server. By opening the application, the user can check the details of the alert and receive suggestions for countermeasures.

[0999] Specific working example:

[1000] If the tire pressure drops below the specified value, the system will display an alert stating "Tire pressure is below the specified value" along with directions to the nearest gas station.

[1001] Emotion engine and customized responses

[1002] The emotion engine uses the mobile device's camera and microphone to analyze the user's facial expressions and voice to detect emotional states such as stress, anger, fatigue, etc. in real time. Based on the results, the system further customizes the immediate response measures for the user.

[1003] Specific working example:

[1004] If the user is recognized as being in a stressful state, relaxing music is played on the mobile device and a guide voice prompts the user to "relax."

[1005] Gathering feedback

[1006] After the problem is resolved, the user's mobile device will display a feedback form to collect the user's experience, which will be stored on a central server and used to improve the system in the future.

[1007] Specific working example:

[1008] After the user has resolved the issue, a feedback form is displayed asking "How was your experience with our service?", and the user's input is received and sent to a central server.

[1009] Prompt Sentence Examples

[1010] "Lateral acceleration data detected by the accelerometer is collected and sent to a central server."

[1011] "Generates an alert if engine temperatures exceed normal range."

[1012] "When tire pressure drops, the user is given directions to the nearest gas station."

[1013] "If the user is feeling stressed, it will play relaxing music and provide a guide voice."

[1014] This allows the system to provide integrated support that takes into account both troubleshooting and the user's emotional state, allowing the user to continue driving with peace of mind.

[1015] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1016] Step 1:

[1017] The onboard terminal collects data from each sensor.

[1018] Inputs: GPS data, speed data, acceleration data, engine temperature data, tire pressure data.

[1019] Data processing / calculation: The raw data obtained from each sensor is compiled into packets by time.

[1020] Output: Sensor data packets.

[1021] Specific operation: When a vehicle turns a sharp corner, the acceleration sensor detects high lateral acceleration (G-force), and the onboard terminal assembles it into packets.

[1022] Step 2:

[1023] The in-vehicle terminal transmits the collected data packets to a central server.

[1024] Input: Sensor data packets.

[1025] Data processing / calculation: Encryption of packet data and securing of communication lines.

[1026] Output: Sending encrypted sensor data packets.

[1027] How it works: The in-vehicle device sends encrypted data packets to a central server via Wi-Fi or a cellular network.

[1028] Step 3:

[1029] A central server receives the data packets and analyzes them in real time.

[1030] Input: Encrypted sensor data packet.

[1031] Data processing / computation: Decrypting encrypted data and applying algorithms to detect outliers and irregular patterns.

[1032] Output: Anomaly detection results and alert generation.

[1033] Specific operation: The central server analyzes the received engine temperature data and compares it with the reference range to detect abnormalities. For example, if the engine temperature exceeds the normal range, an alert is generated.

[1034] Step 4:

[1035] A central server generates alerts and sends them to the user's mobile device.

[1036] Input: Anomaly detection results.

[1037] Data processing / calculation: Generating alert messages and creating notification data packets.

[1038] Output: Alert notification to mobile device.

[1039] Specific operation: If the engine temperature is abnormally high, an alert will be generated stating "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool down" and a notification will be sent to the mobile device.

[1040] Step 5:

[1041] The mobile device uses a camera and microphone to analyze the user's facial expressions and voice to detect their emotional state in real time.

[1042] Input: User's facial expression data, voice data.

[1043] Data processing / computation: Apply image and audio analysis algorithms to identify emotional states.

[1044] Output: User's emotional state data.

[1045] Specific operation: The user's voice is captured and recognized as "stress" by the emotion engine. "Stress" is also detected from facial expressions.

[1046] Step 6:

[1047] Based on the emotional state detected by the emotion engine, a central server customizes alert notification content and responses.

[1048] Input: User emotional state data.

[1049] Data processing / computation: Applying customized algorithms based on emotional state data.

[1050] Output: Customized alert notifications and remedial actions.

[1051] Specific operation: If the user is recognized as being in a stressful state, relaxing music will be played on the mobile device and a guide voice will prompt the user to "relax."

[1052] Step 7:

[1053] After the problem is solved, the mobile terminal displays a feedback form to collect feedback from the user.

[1054] Input: User's post-resolution feedback answer.

[1055] Data processing / calculation: Collection of feedback data and storage in a database for analysis.

[1056] Output: Save feedback data.

[1057] Specific behavior: After the problem is resolved, a feedback form asking "How was your service?" is displayed, and user input is collected and sent to a central server.

[1058] (Application example 2)

[1059] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1060] Conventional vehicle management systems have been effective to a certain extent in detecting abnormalities and notifying drivers using sensors, but they have limitations in providing optimal countermeasures that take into account the user's emotional state. Furthermore, when users feel stressed or fatigued, they do not provide sufficient support to alleviate these conditions. Therefore, there is a need for a system that is ergonomically friendly to users while improving driving safety.

[1061] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1062] In this invention, the server includes means for an in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server, means for the central server to analyze the data in real time and generate an alert when an abnormality is detected, means for notifying the user of the alert to the user's mobile terminal and providing countermeasures for the abnormality, means for recognizing the user's emotional state and customizing the countermeasures based on the emotional state, and means for collecting feedback from the user and saving the data for use in analysis, thereby making it possible to provide a system that not only detects abnormalities but also responds to the user's emotional state.

[1063] An "on-board terminal" is a device installed in a vehicle that collects data from multiple sensors and transmits the data to a central server.

[1064] A "sensor" is a device used to detect the condition and movement of a vehicle, and is used to collect data such as GPS, speed, acceleration, engine, and tire pressure.

[1065] A "central server" is a device or system that analyzes data sent from vehicle-mounted terminals in real time, detects abnormalities, and generates alerts.

[1066] An "alert" is a warning notification that is generated when an abnormality is detected based on data collected from a sensor, and is intended to inform the user of information about the abnormality and how to deal with it.

[1067] A "mobile terminal" is a device that can be carried by a user, including a smartphone or tablet, for receiving alert notifications and abnormality response measures.

[1068] "Emotional state" refers to the mental state detected from the user's facial expression and voice, and includes, for example, stress, anger, fatigue, etc.

[1069] An "emotion engine" is a system or software that analyzes a user's facial expressions and voice data to recognize their emotional state.

[1070] "Feedback" refers to opinions and information provided by users about their experience using the system, which is collected for the purpose of improving and optimizing the system.

[1071] The system of the present invention is composed of an in-vehicle terminal, a central server, an emotion engine, and a user's mobile terminal. The specific configuration and operation will be explained below.

[1072] In-vehicle terminal

[1073] The in-vehicle terminal is equipped with multiple sensors, such as a GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, and tire pressure sensor. These sensors constantly monitor the vehicle's condition and movement and collect data. The collected data is sent to a central server at regular intervals, for example, every five seconds.

[1074] Central Server

[1075] The central server receives and analyzes data sent from the in-vehicle device in real time. If an abnormal value is detected as a result of the analysis, an alert is generated. For example, if the engine temperature exceeds the normal range, the data is judged to be abnormal and an alert is generated based on that. The generated alert is sent to the user's mobile device.

[1076] Mobile Devices and Emotion Engines

[1077] The user's mobile device is equipped with an emotion engine that uses a camera and microphone to analyze the user's facial expressions and voice. The emotion engine recognizes the user's mental state (stress, anger, fatigue, etc.) in real time. For example, if the user is in a stressful state when an alert is generated, the system will play relaxing music and suggest optimal countermeasures, such as providing route guidance to the nearest gas station.

[1078] Feedback and Improvements

[1079] The system collects user feedback and is continuously improved based on experience. Feedback is collected via mobile devices and analyzed and stored on a central server. This improves the accuracy of the system and the user experience.

[1080] Specific examples

[1081] For example, consider the case where a user's tire pressure drops while driving. The in-vehicle device detects abnormal data from the tire pressure sensor and sends it to a central server. The server immediately analyzes the abnormality, generates an alert stating "Tire pressure is below the specified value," and notifies the user's mobile device. The user's mobile device then uses its camera and microphone to analyze the user's facial expressions and voice, and detects that the user is feeling stressed at this point. In response, the system plays relaxing music along with the alert notification and provides route guidance to the nearest gas station.

[1082] Prompt Sentence Examples

[1083] Anomaly Detection Systems:

[1084] Sensor data such as GPS, speed, engine temperature, and tire pressure collected by the in-vehicle device is sent to a central server every five seconds for real-time analysis. If an abnormality is detected, an alert is generated and a notification of countermeasures is sent to the user's smartphone. In addition, the system analyzes the user's emotional state using a camera and microphone, and plays relaxing music if the user is under stress.

[1085] As described above, the embodiments of the present invention can provide a safe and comfortable driving environment by detecting abnormalities in a vehicle and responding to the emotional state of the user.

[1086] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1087] Step 1:

[1088] The in-vehicle terminal collects data from multiple sensors, specifically GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, and tire pressure sensors. The input is data from each sensor, and the output is a collection of that data.

[1089] Step 2:

[1090] The data collected by the in-vehicle device is sent to a central server every five seconds. Specifically, the data is packaged into packets and sent over the Internet. The input is a collection of data collected from the sensors, and the output is the data packet sent to the central server.

[1091] Step 3:

[1092] The central server analyzes the received data in real time, stores it in a database, and runs algorithms to detect abnormal values ​​and patterns. The input is the data packets sent from the in-vehicle device, and the output is the result of the anomaly detection.

[1093] Step 4:

[1094] When the central server detects an anomaly, it generates an alert. Specifically, it creates a warning message depending on the type of anomaly. The input is the anomaly detection result, and the output is the generated alert message.

[1095] Step 5:

[1096] The generated alert is sent to the user's mobile device. Specifically, the alert message is pushed to the mobile device. The input is the alert message, and the output is the alert displayed on the mobile device.

[1097] Step 6:

[1098] The user's mobile device receives the alert and uses its camera and microphone to analyze the user's emotional state. Specifically, the camera video and audio data are input into an emotion engine to recognize emotional states such as stress, anger, and fatigue. The input is data obtained from the camera and microphone, and the output is the analyzed emotional state.

[1099] Step 7:

[1100] The emotion engine customizes responses based on the user's emotional state, such as playing relaxing music or running an algorithm to suggest optimal responses. The input is the analyzed emotional state, and the output is a customized response provided to the user.

[1101] Step 8:

[1102] User feedback is collected and used to improve the system. Specifically, feedback is collected from users via an application and stored in a database. The input is feedback data from users, and the output is the feedback stored in the database.

[1103] Step 9:

[1104] The feedback data is analyzed and reflected in system improvements. Specifically, the feedback data is analyzed, system improvements are extracted, and software updates and algorithm optimization are carried out. The input is the feedback data, and the output is improved system functions.

[1105] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1106] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1107] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1108] [Fourth embodiment]

[1109] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1110] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1111] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1112] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1113] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1114] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1115] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1116] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1117] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1118] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1119] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1120] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1121] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1122] The present invention relates to a system for collecting data from a plurality of sensors in a vehicle, analyzing the data to detect abnormalities, and providing a user with a prompt and optimal countermeasure. Specific embodiments for carrying out the present invention will be described below.

[1123] An in-vehicle terminal is a device installed in a vehicle that collects data from GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, tire pressure sensors, etc. This data is sent to a central server at regular intervals. For example, every 5 seconds, the data from each sensor is collected into packets and sent to the server.

[1124] The central server receives data sent from the in-vehicle devices, analyzes the data in real time, and generates an alert if anomalies or irregular patterns are detected. For example, if the engine temperature significantly exceeds normal values, an alert will be generated indicating a possible overheating.

[1125] The generated alert is sent to the user's mobile device (such as a smartphone or tablet). By opening the application, the user can check the details of the alert. For example, specific countermeasures such as "The engine temperature is abnormally high. Immediately stop the engine and allow it to cool" are displayed.

[1126] The system also provides users with emergency assistance service information. Based on the user's current location, information about the nearest gas station or repair shop and directions to the nearest shop are displayed within the application. For example, specific directions such as "The nearest repair shop is 1.5 kilometers away. Please follow the route below" are provided.

[1127] After a user resolves a problem through the system, they will receive a notification requesting feedback. They can use the feedback form to express their thoughts and the effectiveness of the resolution. This feedback data will be stored on a central server and used to improve future prediction models and services.

[1128] As a concrete example, consider the case where a user experiences a drop in tire pressure while driving. The in-vehicle device detects abnormal data from the tire pressure sensor and sends it to a central server. The server immediately analyzes the abnormality, generates an alert stating "Tire pressure is below the specified value," and notifies the user's mobile device. The user confirms the alert and follows the directions to the nearest gas station. They then report their experience resolving the problem in a feedback form. This allows the system to improve the accuracy of data analysis and alert generation in future cases.

[1129] As described above, the system of the present invention can efficiently and effectively resolve problems that occur while driving a vehicle, and provide a safe and secure driving environment.

[1130] The processing flow will be explained below.

[1131] Step 1:

[1132] The device collects data from the vehicle's GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor, and other sensors.

[1133] Step 2:

[1134] The data collected by the device is aggregated into packets every 5 seconds. For example, the latest data values ​​obtained from each sensor are organized into a single data set.

[1135] Step 3:

[1136] The device sends the assembled data packets to a central server, which then sends the data over the internet or a communications network.

[1137] Step 4:

[1138] The server receives the data packets sent from the terminal. The server has a receiving function to process large amounts of data quickly.

[1139] Step 5:

[1140] The server analyzes the received data and uses specific algorithms to detect abnormal values ​​and patterns in the data from each sensor. For example, if the engine temperature exceeds the normal range, it will determine that this is an abnormality.

[1141] Step 6:

[1142] If the server detects an abnormality through analysis, it generates an alert, which includes information about the problem, recommended actions to take, and emergency instructions.

[1143] Step 7:

[1144] The server notifies the user of the generated alert via push notification, SMS message, or other methods.

[1145] Step 8:

[1146] The user's mobile device will receive an alert and display a notification, and the user can view more information through the application.

[1147] Step 9:

[1148] The user can review the details of the alert and follow the remedial action provided by the system, for example, "Engine temperature is too high. Stop engine and allow it to cool."

[1149] Step 10:

[1150] After the user has finished troubleshooting, the mobile device displays a feedback form, in which the user can enter their level of satisfaction, the results of the troubleshooting, and other information.

[1151] Step 11:

[1152] The mobile devices collect and send the feedback data to a central server, allowing the system to use the feedback information to improve future operations and the accuracy of its prediction models.

[1153] The above is the specific program processing flow of this system, which allows the user to quickly detect problems that occur during operation and take appropriate countermeasures.

[1154] Example 1

[1155] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1156] Modern automobiles are equipped with numerous sensors that can monitor the vehicle's condition. However, there is a lack of systems that can properly analyze sensor data and provide prompt and accurate countermeasures when an abnormality is detected. They also lack the ability to provide emergency assistance information based on the user's current location. Furthermore, there is often no way to collect user feedback to help improve the system. A system that can solve these problems and improve safety and convenience while driving is needed.

[1157] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1158] In this invention, the server includes: a means for an in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server; a means for the central server to analyze the data in real time and generate an alert when an abnormality is detected; a means for notifying the user of the alert to the user's mobile device and providing countermeasures for the abnormality; a means for providing information on the nearest support service based on the user's current location information; and a means for collecting feedback from the user, saving the data, and using it for analysis. This allows the system to quickly detect vehicle abnormalities, provide the user with appropriate countermeasures, and provide support service information, enabling the user to take prompt and appropriate action. Furthermore, collecting feedback data and using it for analysis enables continuous improvement of the system.

[1159] An "on-board terminal" is a device that is installed in a vehicle and collects data from various sensors and transmits it to a central server.

[1160] A "sensor" is a device that monitors the state of a vehicle and measures specific physical quantities, such as position information, speed information, movement data, mechanical information, and tire pressure information.

[1161] The "central server" is a computer system that receives and analyzes data sent from the vehicle-mounted terminal, and if it detects an abnormality, it generates an alert and notifies the user.

[1162] "Data" means a series of measurements or numbers collected from sensors, including location information, speed information, movement data, mechanical information, and tire pressure information.

[1163] An "alert" is a notification message generated when the central server detects an abnormality, informing the user of the nature of the abnormality and how to deal with it.

[1164] A "user's mobile device" is a portable electronic device, such as a smartphone or tablet, that is carried by a user and is used to receive alert notifications and check detailed information.

[1165] "Feedback" refers to opinions and impressions submitted by users after a problem has been resolved, as well as data for system improvement.

[1166] "Support service information" is information about locations and services that are useful for solving vehicle problems, such as the nearest gas station or repair shop, provided based on the user's current location.

[1167] "Location information" is data indicating the current location of the vehicle, and is obtained by a GPS sensor or the like.

[1168] "Speed ​​information" is data indicating the speed of the vehicle, and is acquired by a speed sensor.

[1169] "Movement data" is a series of information related to the movement of a vehicle, and is acquired by an acceleration sensor or the like.

[1170] "Mechanical information" is data indicating the mechanical state of the vehicle, such as the engine condition and temperature, and is acquired by an engine temperature sensor or the like.

[1171] "Tire pressure information" is data indicating the air pressure of the vehicle's tires, and is acquired by a tire pressure sensor.

[1172] These definitions clarify the meaning of the words contained in the claims and facilitate understanding of the scope and nature of the invention.

[1173] This invention is a system that collects data from multiple sensors in a vehicle, analyzes the data to detect abnormalities, and provides the user with a prompt and optimal response. How this invention is implemented will be specifically described below.

[1174] In-vehicle terminal

[1175] An in-vehicle terminal is a device that collects data from multiple sensors installed in a vehicle. The sensors used include a GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, and tire pressure sensor. The terminal obtains location information, speed information, movement data, mechanical information, and tire pressure information from these sensors.

[1176] The onboard device compiles the collected data into packets every five seconds and transmits them to a central server using a cellular communication module. For example, every five seconds, a data packet might be generated that reports an engine temperature of 90 degrees and tire pressure of 32 PSI, and the data is then transmitted to the central server.

[1177] Central Server

[1178] The central server receives data sent from the in-vehicle devices and analyzes it in real time. The server uses a database management system (e.g., MySQL or PostgreSQL) to store the received data. It also uses a data analysis library (e.g., Python's pandas or scikit-learn) to run programs that detect outliers and irregular patterns. If an abnormality is detected, the server generates an alert according to the content. For example, if the engine temperature exceeds 100 degrees, it generates an alert stating, "The engine temperature is abnormally high."

[1179] The generated alert is immediately sent to the user's mobile device. Specifically, the alert message is sent using the push notification function.

[1180] User's mobile device

[1181] The user's mobile device is the device that receives the alert notification. This includes smartphones and tablets. The user can check the notification and open an application to view detailed information about the alert. For example, specific countermeasures such as "The engine temperature is abnormally high. Immediately stop the engine and allow it to cool down" are displayed.

[1182] Additionally, the system will provide information about the nearest assistance service based on the user's current location, for example, by displaying a message within the application saying, "The nearest gas station is 1.5 kilometers away. Please follow the route below."

[1183] feedback

[1184] After solving a problem, users report their impressions and effectiveness of the solution through a feedback form. The feedback data is stored on a central server and used for future analysis of prediction models and system improvements.

[1185] Prompt Sentence Examples

[1186] For example, by entering a prompt such as "Please explain what to do if the vehicle's engine temperature becomes abnormally high" into the generative AI model, it is possible to generate an explanation of specific countermeasures.

[1187] The present invention is a system that quickly detects abnormalities in a vehicle and provides the user with appropriate countermeasures, thereby improving safety and convenience for the user.

[1188] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1189] Step 1:

[1190] Terminal: The in-vehicle terminal collects data from various sensors installed in the vehicle (GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor, etc.).

[1191] Input: Data from each sensor (location information, speed information, movement data, machine information, tire pressure information).

[1192] How it works: Every 5 seconds, the onboard device aggregates this data and generates a time-stamped data packet.

[1193] Output: Consolidated data packet (e.g., engine temperature = 90 degrees, tire pressure = 32 PSI, time = 2023-10-10 10:00:05).

[1194] Step 2:

[1195] Terminal: The in-vehicle terminal generates and transmits the data packets to the central server.

[1196] Input: Consolidated data packet.

[1197] How it works: It uses a cellular communication module to send data packets over the internet to a central server.

[1198] Output: Data packets sent to the central server.

[1199] Step 3:

[1200] Server: The central server receives data packets sent from the in-vehicle terminals.

[1201] Input: Data packet sent from the in-vehicle terminal.

[1202] What it does: Uses the data reception module to receive packets and store them in a database (e.g. MySQL or PostgreSQL).

[1203] Output: Vehicle status data stored in a database.

[1204] Step 4:

[1205] Server: The central server analyzes the received data in real time and detects anomalies.

[1206] Input: Vehicle condition data stored in a database.

[1207] How it works: Using data analysis libraries (e.g., Python's pandas, scikit-learn), it uses rule-based or machine learning models to detect outliers and irregular patterns. For example, an engine temperature above 100 degrees is considered abnormal.

[1208] Output: Alert information when an abnormality is detected (e.g. engine temperature abnormality).

[1209] Step 5:

[1210] Server: When an anomaly is detected, a corresponding alert is generated and sent to the user's mobile device.

[1211] Input: Alert information if an anomaly is detected.

[1212] What it does: Uses push notifications to send an alert to the user's mobile device, for example, a message saying "Engine temperature is too high."

[1213] Output: An alert notification that appears on the user's mobile device.

[1214] Step 6:

[1215] User: The user receives an alert notification on their mobile device and opens the application to view more information.

[1216] Input: Alert notification.

[1217] Action: Open the application and check the details of the error and the countermeasures. For example, the countermeasures displayed will be "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool."

[1218] Output: A screen showing detailed information about the anomaly and specific countermeasures.

[1219] Step 7:

[1220] Server: Provides information about the nearest support service based on the user's current location.

[1221] Input: User's current location.

[1222] How it works: Using GPS location information, it searches a support service database and obtains information on the nearest gas station or repair shop.

[1223] Output: Information about the nearest assistance service and directions displayed on the user's mobile device (e.g., "The nearest gas station is 1.5 kilometers away. Please follow the route below.").

[1224] Step 8:

[1225] Server: Collect user feedback after the problem is resolved.

[1226] Input: Feedback request after troubleshooting.

[1227] What it does: Sends a feedback form to the user's mobile device.

[1228] Output: User-entered feedback (e.g., "I successfully resolved my engine overheating issue. Your guidance was very clear and helpful.").

[1229] Step 9:

[1230] User: Fill in the feedback form with your thoughts and opinions on how the problem was resolved and submit it.

[1231] Input: User feedback.

[1232] Action: Enter the required information into the feedback form and click the submit button.

[1233] Output: The feedback data sent.

[1234] Step 10:

[1235] Server: Stores the feedback data and analyzes it for future prediction models and system improvements.

[1236] Input: The submitted feedback data.

[1237] How it works: Feedback data is stored in a database and analyzed using a data analysis library.

[1238] Output: Data analysis results for improved predictive models and system improvements.

[1239] (Application example 1)

[1240] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1241] Detecting abnormalities and problems that occur during the operation of autonomous vehicles in real time and providing users with prompt and appropriate countermeasures is important for improving operational safety and efficiency. However, with conventional systems, there is a time lag between the detection of an abnormality and instructions to respond, as well as limitations in notification methods, making it difficult for users to take appropriate action immediately. In addition, feedback data is not properly collected and analyzed, and there are cases where it cannot be used to improve future countermeasures.

[1242] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1243] In this invention, the server includes: means for the in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server; means for the central server to analyze the data in real time and generate an alert when an abnormality is detected; means for notifying the user of the alert to the user's mobile terminal and wearable device and providing countermeasures for the abnormality; and means for collecting feedback from the user, saving the data, and using it for analysis. This minimizes the time lag from abnormality detection to response instructions, allowing the user to immediately confirm and respond via a wearable device such as smart glasses, thereby significantly improving the safety and efficiency of driving.

[1244] An "on-board terminal" is a device installed in a vehicle that collects data from multiple sensors and transmits it to a central server.

[1245] The "multiple sensors" are various sensor devices that collect data such as vehicle position, speed, acceleration, engine temperature, and tire pressure.

[1246] The "central server" is a computer system that analyzes data sent from the vehicle-mounted terminal in real time and generates an alert when an abnormality is detected.

[1247] An "alert" is a warning or notification generated when the central server detects an abnormality, and is information intended to prompt the user to take prompt action.

[1248] A "mobile terminal" is a computing device that a user can carry with them, such as a smartphone or tablet.

[1249] A "wearable device" is a device that can be worn by a user, such as smart glasses or a wristwatch-type terminal.

[1250] "Countermeasures" refer to specific actions or procedures that a user should take when an abnormality occurs.

[1251] "Feedback" refers to information provided by users about the effectiveness of the problem resolution, their impressions, and their experiences, and is data that will be used for future data analysis and service improvement.

[1252] "Location data" refers to data relating to the current location of a vehicle obtained using a GPS or the like.

[1253] "Speed ​​data" is data relating to the vehicle's traveling speed.

[1254] "Acceleration data" is data relating to the acceleration of the vehicle.

[1255] "Engine data" refers to data relating to the temperature and operating conditions of the engine.

[1256] "Tire pressure data" refers to data relating to tire air pressure.

[1257] "Support services" refer to service facilities such as gas stations and repair shops that users can use when there is a problem with their vehicle.

[1258] This invention is a system that detects abnormalities and problems that occur during the operation of an autonomous vehicle in real time and provides users with prompt and appropriate countermeasures. This system includes an in-vehicle terminal, a central server, a mobile terminal, and a wearable device.

[1259] First, the in-vehicle terminal collects data from multiple sensors installed in the vehicle. The collected data includes location data, speed data, acceleration data, engine data, and tire pressure data. This data is sent to a central server at regular intervals (for example, every 5 seconds).

[1260] The central server then analyzes the data sent from the in-vehicle device in real time. If an abnormality is detected during the analysis, an alert is immediately generated. This alert includes the specific details of the problem, how to deal with it, and information about the nearest support service.

[1261] The generated alerts are sent to the user's mobile devices (e.g., smartphones, tablets) and wearable devices (e.g., smart glasses), allowing the user to immediately check the alerts and take appropriate action.

[1262] As a specific example, if the engine temperature exceeds the normal range, an alert such as the following is generated: "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool." This alert is sent to the mobile device and smart glasses, and can be immediately confirmed by the user.

[1263] The system also includes a means to collect user feedback. After resolving a problem, users can submit their impressions and feedback through a feedback form. This feedback data is stored on a central server and used for future data analysis and service improvement.

[1264] The hardware includes various vehicle sensors (e.g., GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, tire pressure sensors), smart glasses (e.g., Google Glass, Vuzix Blade), mobile devices (e.g., smartphones, tablets), and a central server.

[1265] The software includes APIs (e.g., RESTful APIs) for collecting data from sensors and sending it to a central server, Python scripts for data analysis, SDKs for smart glasses (e.g., Google Glass SDK, Vuzix SDK), and applications for sending notifications.

[1266] Using generative AI models, it is also possible to automatically generate specific and practical countermeasures when an abnormality is detected. For example, the following is an example of a prompt when an abnormality in engine temperature is detected: "The vehicle's engine temperature is above the normal range. Immediately shut down the engine and allow it to cool. Check the sensor data again to see if the engine temperature has returned to normal."

[1267] This minimizes the time lag between detecting an abnormality and issuing instructions to respond, significantly improving safety and efficiency during operation.

[1268] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1269] Step 1:

[1270] The in-vehicle terminal collects data from multiple sensors installed in the vehicle (position sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor). The input is data from each sensor, and the output is a packet of collected data.

[1271] Step 2:

[1272] The in-vehicle terminal transmits the collected sensor data as packets to the central server at regular intervals (e.g., every 5 seconds). The input is the data packets collected in step 1, and the output is the data packets transmitted to the central server.

[1273] Step 3:

[1274] The central server receives data packets sent from the in-vehicle terminal and analyzes them in real time. During the analysis, it detects outliers and irregular patterns. The input is the data packets sent from the in-vehicle terminal, and the output is the analysis results.

[1275] Step 4:

[1276] If the central server detects an abnormality, it generates an alert. This alert contains the specific details of the problem, how to deal with it, and information about the nearest support service. The input is the analysis result obtained in step 3, and the output is the generated alert.

[1277] Step 5:

[1278] The central server notifies the generated alerts to the user's mobile terminal and wearable device. The input is the generated alert information, and the output is the alert displayed on the user's device.

[1279] Step 6:

[1280] The user checks the alert through a mobile terminal or wearable device and takes appropriate action according to the countermeasures presented. The input is the alert information displayed on the device, and the output is the user's specific action.

[1281] Step 7:

[1282] After resolving a problem, the user submits the results and impressions through a feedback form. The input is the user's feedback information, and the output is the submitted feedback data.

[1283] Step 8:

[1284] The central server stores the collected feedback data and uses it for future data analysis and service improvement. The input is the feedback data submitted by users, and the output is the stored data and analysis results.

[1285] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1286] The present invention combines a system that collects data from multiple sensors in a vehicle, analyzes the data to detect abnormalities, and provides the user with a prompt and optimal response, with an emotion engine that recognizes the user's emotional state. Specific embodiments for implementing the present invention will be described below.

[1287] An in-vehicle terminal is a device installed in a vehicle that collects data from GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, tire pressure sensors, etc. This data is sent to a central server at regular intervals. For example, every 5 seconds, the data from each sensor is collected into packets and sent to the server.

[1288] The central server receives data sent from the in-vehicle device. The received data is analyzed in real time, and if an abnormal value or irregular pattern is detected, an alert is generated. For example, if the engine temperature exceeds the normal range, an abnormality is detected based on that data.

[1289] The generated alert is sent to the user's mobile device (such as a smartphone or tablet). By opening the application, the user can check the details of the alert. For example, specific countermeasures such as "The engine temperature is abnormally high. Immediately stop the engine and allow it to cool" are displayed.

[1290] The present invention also incorporates an emotion engine for recognizing the user's emotional state, which uses the camera and / or microphone on the user's mobile device to analyze the user's facial expressions and voice to detect emotions such as stress, anger, fatigue, etc. in real time.

[1291] Based on the emotional state detected by the emotion engine, the system can customize responses to the user. For example, if the user is stressed, the system can provide relaxing music and voice guidance along with an alert notification. If the user shows high levels of anger, the system can provide gentle advice on safe driving.

[1292] As a concrete example, consider the case where a user's tire pressure drops while driving. The in-vehicle device detects abnormal data from the tire pressure sensor and sends it to a central server. The server immediately analyzes the abnormality, generates an alert stating "Tire pressure is below the specified value," and notifies the user's mobile device. The user's mobile device then uses a camera and microphone to analyze the user's facial expressions and voice, and detects that the user is feeling stressed at this point. In response, the system notifies the user of the alert, plays relaxing music, and provides route guidance to the nearest gas station. After the problem is resolved, a feedback form is displayed and the user's experience is sent to the server.

[1293] This allows the system to not only respond to problems but also provide support that takes into account the user's emotional state, allowing the user to continue driving with peace of mind while the system continues to improve.

[1294] The processing flow will be explained below.

[1295] Step 1:

[1296] The device collects data from the vehicle's GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, tire pressure sensor, and other sensors.

[1297] Step 2:

[1298] The terminal collects data and compiles it into packets every 5 seconds. The terminal then organizes the latest data values ​​obtained from each sensor into a single data set.

[1299] Step 3:

[1300] The device sends the assembled data packets to a central server, which then sends the data over the internet or a communications network.

[1301] Step 4:

[1302] The server receives the data packets sent from the terminal. The server has a receiving function to process large amounts of data quickly.

[1303] Step 5:

[1304] The server analyzes the data it receives in real time. Specific algorithms are used to detect abnormal values ​​and patterns in the data from each sensor. For example, if the engine temperature exceeds the normal range, it will determine that this is an abnormality.

[1305] Step 6:

[1306] If the server detects an abnormality through analysis, it generates an alert, which includes information about the problem, recommended actions to take, and emergency instructions.

[1307] Step 7:

[1308] The server notifies the user of the generated alert via push notification, SMS message, or other methods.

[1309] Step 8:

[1310] The user's mobile device will receive an alert and display a notification, and the user can view more information through the application.

[1311] Step 9:

[1312] The emotion engine analyzes the user's facial expressions and voice using the camera and microphone installed in the user's mobile device, and detects emotional states such as stress, anger, and fatigue in real time.

[1313] Step 10:

[1314] The emotion engine sends the user's emotional state to the server, for example, if the user is showing a high level of stress, it provides that information to the server.

[1315] Step 11:

[1316] The server takes into account the user's emotional state and sends a customized response to the alert to the user's mobile device, such as providing relaxing music or a voice guide.

[1317] Step 12:

[1318] The user reviews the alert details and follows the customized remedial action provided by the system, for example, "Engine temperature is too high. Stop engine and allow it to cool down," while listening to relaxing music.

[1319] Step 13:

[1320] After the user has finished troubleshooting, the mobile device displays a feedback form, in which the user can enter information such as their level of satisfaction, the outcome of the troubleshooting, and their emotional state.

[1321] Step 14:

[1322] The mobile devices collect and send the feedback data to a central server, allowing the system to use the feedback information to improve future operations and the accuracy of its prediction models.

[1323] The above is the specific program processing flow of this system, which allows the user to quickly detect problems that occur while driving, take appropriate countermeasures, and receive support that takes into consideration the user's emotional state.

[1324] Example 2

[1325] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1326] Modern vehicle management systems detect vehicle anomalies and notify the user, but do not take into account the user's emotional state. As a result, users may feel excessive stress or anger when an abnormality occurs, making it difficult to respond optimally. The present invention aims to provide a system that not only analyzes data collected from multiple sensors in the vehicle to detect anomalies, but also recognizes the user's emotional state and provides optimal countermeasures based on that information.

[1327] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1328] In this invention, the server includes: means for an in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server; means for the central server to analyze the data in real time and generate an alert when an abnormality is detected; means for notifying the user of the alert to a mobile device and providing countermeasures for the abnormality; means for the mobile device to analyze the user's facial expressions and voice using a camera and microphone to detect the user's emotional state in real time; means for customizing the content of the alert and countermeasures based on the emotional state detected by the emotion engine; and means for collecting feedback from the user, saving the data, and using it for analysis. This enables a quick and optimal response to vehicle abnormalities while also taking the user's emotional state into consideration.

[1329] An "on-board terminal" is a device installed in a vehicle that collects data from multiple sensors and transmits it to a central server.

[1330] A "sensor" is a device that detects and collects information about the vehicle's status and the external environment, including location data, speed data, acceleration data, engine temperature data, and tire pressure data.

[1331] The "central server" is a server device that receives data sent from the vehicle-mounted terminal, analyzes it in real time, and detects abnormalities.

[1332] An "alert" is a warning message generated by the central server when an abnormality is detected as a result of data analysis.

[1333] A "mobile terminal" is a mobile device such as a smartphone or tablet owned by a user, which receives alert notifications from a central server.

[1334] The "emotion engine" is a system that uses the camera and microphone of a mobile device to analyze the user's facial expressions and voice, and detects the user's emotional state in real time.

[1335] "Feedback" refers to opinions and ratings collected from users, and is data used to improve the system and evaluate alert responses.

[1336] "Relaxing music" is music intended to relieve the user's stress and tension, and can be played by the emotion engine based on the user's emotional state.

[1337] A "trouble" is an event that prevents the vehicle from operating normally, including vehicle abnormalities and breakdowns.

[1338] The present invention provides a system that collects data from multiple sensors in a vehicle, analyzes the data to detect abnormalities, and also recognizes the emotional state of the user, and provides optimal countermeasures based on the collected data. Specific embodiments for carrying out the present invention will be described below.

[1339] System configuration

[1340] The system consists of the following main parts:

[1341] In-vehicle terminal

[1342] Central Server

[1343] Mobile devices

[1344] Emotion Engine

[1345] In-vehicle terminal

[1346] An onboard device is a device installed in a vehicle that collects data from the following sensors:

[1347] GPS sensor

[1348] Speed ​​sensor

[1349] Accelerometer

[1350] Engine temperature sensor

[1351] tire pressure sensor

[1352] The data collected from these sensors is aggregated into packets at regular intervals (e.g., every 5 seconds) and sent to a central server.

[1353] Specific working example:

[1354] When a vehicle turns a sharp corner, the acceleration sensor detects high lateral acceleration (G-force), and the onboard terminal compiles it into packets.

[1355] Central Server

[1356] The central server receives data sent from the in-vehicle device and analyzes it in real time. If an abnormal value or irregular pattern is detected in the received data, an alert is generated. This alert is sent to the user's mobile device.

[1357] Specific working example:

[1358] If the engine temperature sensor detects a temperature outside the normal range, an alert will be generated stating, "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool." and sent to the user's mobile device.

[1359] Mobile devices

[1360] The mobile device is a user's smartphone or tablet, which receives an alert notification from the central server. By opening the application, the user can check the details of the alert and receive suggestions for countermeasures.

[1361] Specific working example:

[1362] If the tire pressure drops below the specified value, the system will display an alert stating "Tire pressure is below the specified value" along with directions to the nearest gas station.

[1363] Emotion engine and customized responses

[1364] The emotion engine uses the mobile device's camera and microphone to analyze the user's facial expressions and voice to detect emotional states such as stress, anger, fatigue, etc. in real time. Based on the results, the system further customizes the immediate response measures for the user.

[1365] Specific working example:

[1366] If the user is recognized as being in a stressful state, relaxing music is played on the mobile device and a guide voice prompts the user to "relax."

[1367] Gathering feedback

[1368] After the problem is resolved, the user's mobile device will display a feedback form to collect the user's experience, which will be stored on a central server and used to improve the system in the future.

[1369] Specific working example:

[1370] After the user has resolved the issue, a feedback form is displayed asking "How was your experience with our service?", and the user's input is received and sent to a central server.

[1371] Prompt Sentence Examples

[1372] "Lateral acceleration data detected by the accelerometer is collected and sent to a central server."

[1373] "Generates an alert if engine temperatures exceed normal range."

[1374] "When tire pressure drops, the user is given directions to the nearest gas station."

[1375] "If the user is feeling stressed, it will play relaxing music and provide a guide voice."

[1376] This allows the system to provide integrated support that takes into account both troubleshooting and the user's emotional state, allowing the user to continue driving with peace of mind.

[1377] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1378] Step 1:

[1379] The onboard terminal collects data from each sensor.

[1380] Inputs: GPS data, speed data, acceleration data, engine temperature data, tire pressure data.

[1381] Data processing / calculation: The raw data obtained from each sensor is compiled into packets by time.

[1382] Output: Sensor data packets.

[1383] Specific operation: When a vehicle turns a sharp corner, the acceleration sensor detects high lateral acceleration (G-force), and the onboard terminal assembles it into packets.

[1384] Step 2:

[1385] The in-vehicle terminal transmits the collected data packets to a central server.

[1386] Input: Sensor data packets.

[1387] Data processing / calculation: Encryption of packet data and securing of communication lines.

[1388] Output: Sending encrypted sensor data packets.

[1389] How it works: The in-vehicle device sends encrypted data packets to a central server via Wi-Fi or a cellular network.

[1390] Step 3:

[1391] A central server receives the data packets and analyzes them in real time.

[1392] Input: Encrypted sensor data packet.

[1393] Data processing / computation: Decrypting encrypted data and applying algorithms to detect outliers and irregular patterns.

[1394] Output: Anomaly detection results and alert generation.

[1395] Specific operation: The central server analyzes the received engine temperature data and compares it with the reference range to detect abnormalities. For example, if the engine temperature exceeds the normal range, an alert is generated.

[1396] Step 4:

[1397] A central server generates alerts and sends them to the user's mobile device.

[1398] Input: Anomaly detection results.

[1399] Data processing / calculation: Generating alert messages and creating notification data packets.

[1400] Output: Alert notification to mobile device.

[1401] Specific operation: If the engine temperature is abnormally high, an alert will be generated stating "The engine temperature is abnormally high. Stop the engine immediately and allow it to cool down" and a notification will be sent to the mobile device.

[1402] Step 5:

[1403] The mobile device uses a camera and microphone to analyze the user's facial expressions and voice to detect their emotional state in real time.

[1404] Input: User's facial expression data, voice data.

[1405] Data processing / computation: Apply image and audio analysis algorithms to identify emotional states.

[1406] Output: User's emotional state data.

[1407] Specific operation: The user's voice is captured and recognized as "stress" by the emotion engine. "Stress" is also detected from facial expressions.

[1408] Step 6:

[1409] Based on the emotional state detected by the emotion engine, a central server customizes alert notification content and responses.

[1410] Input: User emotional state data.

[1411] Data processing / computation: Applying customized algorithms based on emotional state data.

[1412] Output: Customized alert notifications and remedial actions.

[1413] Specific operation: If the user is recognized as being in a stressful state, relaxing music will be played on the mobile device and a guide voice will prompt the user to "relax."

[1414] Step 7:

[1415] After the problem is solved, the mobile terminal displays a feedback form to collect feedback from the user.

[1416] Input: User's post-resolution feedback answer.

[1417] Data processing / calculation: Collection of feedback data and storage in a database for analysis.

[1418] Output: Save feedback data.

[1419] Specific behavior: After the problem is resolved, a feedback form asking "How was your service?" is displayed, and user input is collected and sent to a central server.

[1420] (Application example 2)

[1421] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1422] Conventional vehicle management systems have been effective to a certain extent in detecting abnormalities and notifying drivers using sensors, but they have limitations in providing optimal countermeasures that take into account the user's emotional state. Furthermore, when users feel stressed or fatigued, they do not provide sufficient support to alleviate these conditions. Therefore, there is a need for a system that is ergonomically friendly to users while improving driving safety.

[1423] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1424] In this invention, the server includes means for an in-vehicle terminal to collect data from multiple sensors in the vehicle and periodically transmit the data to a central server, means for the central server to analyze the data in real time and generate an alert when an abnormality is detected, means for notifying the user of the alert to the user's mobile terminal and providing countermeasures for the abnormality, means for recognizing the user's emotional state and customizing the countermeasures based on the emotional state, and means for collecting feedback from the user and saving the data for use in analysis, thereby making it possible to provide a system that not only detects abnormalities but also responds to the user's emotional state.

[1425] An "on-board terminal" is a device installed in a vehicle that collects data from multiple sensors and transmits the data to a central server.

[1426] A "sensor" is a device used to detect the condition and movement of a vehicle, and is used to collect data such as GPS, speed, acceleration, engine, and tire pressure.

[1427] A "central server" is a device or system that analyzes data sent from vehicle-mounted terminals in real time, detects abnormalities, and generates alerts.

[1428] An "alert" is a warning notification that is generated when an abnormality is detected based on data collected from a sensor, and is intended to inform the user of information about the abnormality and how to deal with it.

[1429] A "mobile terminal" is a device that can be carried by a user, including a smartphone or tablet, for receiving alert notifications and abnormality response measures.

[1430] "Emotional state" refers to the mental state detected from the user's facial expression and voice, and includes, for example, stress, anger, fatigue, etc.

[1431] An "emotion engine" is a system or software that analyzes a user's facial expressions and voice data to recognize their emotional state.

[1432] "Feedback" refers to opinions and information provided by users about their experience using the system, which is collected for the purpose of improving and optimizing the system.

[1433] The system of the present invention is composed of an in-vehicle terminal, a central server, an emotion engine, and a user's mobile terminal. The specific configuration and operation will be explained below.

[1434] In-vehicle terminal

[1435] The in-vehicle terminal is equipped with multiple sensors, such as a GPS sensor, speed sensor, acceleration sensor, engine temperature sensor, and tire pressure sensor. These sensors constantly monitor the vehicle's condition and movement and collect data. The collected data is sent to a central server at regular intervals, for example, every five seconds.

[1436] Central Server

[1437] The central server receives and analyzes data sent from the in-vehicle device in real time. If an abnormal value is detected as a result of the analysis, an alert is generated. For example, if the engine temperature exceeds the normal range, the data is judged to be abnormal and an alert is generated based on that. The generated alert is sent to the user's mobile device.

[1438] Mobile Devices and Emotion Engines

[1439] The user's mobile device is equipped with an emotion engine that uses a camera and microphone to analyze the user's facial expressions and voice. The emotion engine recognizes the user's mental state (stress, anger, fatigue, etc.) in real time. For example, if the user is in a stressful state when an alert is generated, the system will play relaxing music and suggest optimal countermeasures, such as providing route guidance to the nearest gas station.

[1440] Feedback and Improvements

[1441] The system collects user feedback and is continuously improved based on experience. Feedback is collected via mobile devices and analyzed and stored on a central server. This improves the accuracy of the system and the user experience.

[1442] Specific examples

[1443] For example, consider the case where a user's tire pressure drops while driving. The in-vehicle device detects abnormal data from the tire pressure sensor and sends it to a central server. The server immediately analyzes the abnormality, generates an alert stating "Tire pressure is below the specified value," and notifies the user's mobile device. The user's mobile device then uses its camera and microphone to analyze the user's facial expressions and voice, and detects that the user is feeling stressed at this point. In response, the system plays relaxing music along with the alert notification and provides route guidance to the nearest gas station.

[1444] Prompt Sentence Examples

[1445] Anomaly Detection Systems:

[1446] Sensor data such as GPS, speed, engine temperature, and tire pressure collected by the in-vehicle device is sent to a central server every five seconds for real-time analysis. If an abnormality is detected, an alert is generated and a notification of countermeasures is sent to the user's smartphone. In addition, the system analyzes the user's emotional state using a camera and microphone, and plays relaxing music if the user is under stress.

[1447] As described above, the embodiments of the present invention can provide a safe and comfortable driving environment by detecting abnormalities in a vehicle and responding to the emotional state of the user.

[1448] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1449] Step 1:

[1450] The in-vehicle terminal collects data from multiple sensors, specifically GPS sensors, speed sensors, acceleration sensors, engine temperature sensors, and tire pressure sensors. The input is data from each sensor, and the output is a collection of that data.

[1451] Step 2:

[1452] The data collected by the in-vehicle device is sent to a central server every five seconds. Specifically, the data is packaged into packets and sent over the Internet. The input is a collection of data collected from the sensors, and the output is the data packet sent to the central server.

[1453] Step 3:

[1454] The central server analyzes the received data in real time, stores it in a database, and runs algorithms to detect abnormal values ​​and patterns. The input is the data packets sent from the in-vehicle device, and the output is the result of the anomaly detection.

[1455] Step 4:

[1456] When the central server detects an anomaly, it generates an alert. Specifically, it creates a warning message depending on the type of anomaly. The input is the anomaly detection result, and the output is the generated alert message.

[1457] Step 5:

[1458] The generated alert is sent to the user's mobile device. Specifically, the alert message is pushed to the mobile device. The input is the alert message, and the output is the alert displayed on the mobile device.

[1459] Step 6:

[1460] The user's mobile device receives the alert and uses its camera and microphone to analyze the user's emotional state. Specifically, the camera video and audio data are input into an emotion engine to recognize emotional states such as stress, anger, and fatigue. The input is data obtained from the camera and microphone, and the output is the analyzed emotional state.

[1461] Step 7:

[1462] The emotion engine customizes responses based on the user's emotional state, such as playing relaxing music or running an algorithm to suggest optimal responses. The input is the analyzed emotional state, and the output is a customized response provided to the user.

[1463] Step 8:

[1464] User feedback is collected and used to improve the system. Specifically, feedback is collected from users via an application and stored in a database. The input is feedback data from users, and the output is the feedback stored in the database.

[1465] Step 9:

[1466] The feedback data is analyzed and reflected in system improvements. Specifically, the feedback data is analyzed, system improvements are extracted, and software updates and algorithm optimization are carried out. The input is the feedback data, and the output is improved system functions.

[1467] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1468] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1469] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1470] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1471] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1472] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1473] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1474] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1475] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1476] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1477] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1478] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1479] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1480] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1481] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1482] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1483] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1484] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1485] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1486] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1487] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1488] The following is further disclosed regarding the above embodiment.

[1489] (Claim 1)

[1490] a means for the in-vehicle terminal to collect data from a plurality of sensors in the vehicle and periodically transmit the data to a central server;

[1491] a central server for analyzing the data in real time and generating an alert when an anomaly is detected;

[1492] means for notifying a user of the alert to a mobile terminal and providing a countermeasure for the abnormality;

[1493] a means of collecting user feedback and storing that data for analysis;

[1494] A system including:

[1495] (Claim 2)

[1496] 2. The system of claim 1, wherein the data collected by the in-vehicle terminal includes GPS data, speed data, acceleration data, engine data, and tire pressure data.

[1497] (Claim 3)

[1498] 2. The system according to claim 1, wherein the alert generated by the central server includes a specific description of the problem, how to deal with it, and information on the nearest support service.

[1499] "Example 1"

[1500] (Claim 1)

[1501] a means for the in-vehicle terminal to collect data from a plurality of sensors in the vehicle and periodically transmit the data to a central server;

[1502] a central server for analyzing the data in real time and generating an alert when an anomaly is detected;

[1503] means for notifying a user of the alert to a mobile terminal and providing a countermeasure for the abnormality;

[1504] A means for providing information on the nearest support service based on the user's current location information;

[1505] a means of collecting user feedback and storing that data for analysis;

[1506] A system including:

[1507] (Claim 2)

[1508] 2. The system of claim 1, wherein the data collected by the in-vehicle terminal includes location information, speed information, movement data, mechanical information, and tire pressure information.

[1509] (Claim 3)

[1510] 2. The system according to claim 1, wherein the alert generated by the central server includes the specific details of the problem, how to deal with it, and information on the nearest support service.

[1511] "Application Example 1"

[1512] (Claim 1)

[1513] a means for the in-vehicle terminal to collect data from a plurality of sensors in the vehicle and periodically transmit the data to a central server;

[1514] a central server for analyzing the data in real time and generating an alert when an anomaly is detected;

[1515] means for notifying the user of the alert to the user's mobile terminal and wearable device and providing a countermeasure for the abnormality;

[1516] a means of collecting user feedback and storing that data for analysis;

[1517] A system including:

[1518] (Claim 2)

[1519] 2. The system of claim 1, wherein the data collected by the vehicle-mounted terminal includes location data, speed data, acceleration data, engine data, and tire pressure data.

[1520] (Claim 3)

[1521] 2. The system according to claim 1, wherein the alert generated by the central server includes a specific description of the problem, how to deal with it, and information on the nearest support service.

[1522] "Example 2: Combining Emotion Engines"

[1523] (Claim 1)

[1524] a means for the in-vehicle terminal to collect data from a plurality of sensors in the vehicle and periodically transmit the data to a central server;

[1525] a central server for analyzing the data in real time and generating an alert when an anomaly is detected;

[1526] means for notifying a user of the alert to a mobile terminal and providing a countermeasure for the abnormality;

[1527] A means for the mobile device to analyze the user's facial expressions and voice using a camera and a microphone to detect the user's emotional state in real time;

[1528] a means for customizing alert notification content and responses based on the emotional state detected by the emotion engine;

[1529] a means of collecting user feedback and storing that data for analysis;

[1530] A system including:

[1531] (Claim 2)

[1532] 2. The system of claim 1, wherein the data collected by the vehicle-mounted terminal includes location data, speed data, acceleration data, engine temperature data, and tire pressure data.

[1533] (Claim 3)

[1534] 2. The system according to claim 1, wherein the alert generated by the central server includes a specific description of the problem, how to deal with it, and information on the nearest support service.

[1535] "Application example 2 when combining emotion engines"

[1536] (Claim 1)

[1537] a means for the in-vehicle terminal to collect data from a plurality of sensors in the vehicle and periodically transmit the data to a central server;

[1538] a central server for analyzing the data in real time and generating an alert when an anomaly is detected;

[1539] means for notifying a user of the alert to a mobile terminal and providing a countermeasure for the abnormality;

[1540] means for recognizing the emotional state of a user and customizing a response based on the emotional state;

[1541] a means of collecting user feedback and storing that data for analysis;

[1542] A system including:

[1543] (Claim 2)

[1544] 2. The system of claim 1, wherein the data collected by the in-vehicle terminal includes GPS data, speed data, acceleration data, engine data, and tire pressure data.

[1545] (Claim 3)

[1546] 2. The system according to claim 1, wherein the alert generated by the central server includes a specific description of the problem, how to deal with it, and information on the nearest support service. [Explanation of symbols]

[1547] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for the in-vehicle terminal to collect data from a plurality of sensors in the vehicle and periodically transmit the data to a central server; a central server for analyzing the data in real time and generating an alert when an anomaly is detected; means for notifying a user of the alert to a mobile terminal and providing a countermeasure for the abnormality; a means of collecting user feedback and storing that data for analysis; A system including:

2. 2. The system of claim 1, wherein the data collected by the vehicle-mounted terminal includes GPS data, speed data, acceleration data, engine data, and tire pressure data.

3. 2. The system according to claim 1, wherein the alert generated by the central server includes a specific problem description, a method for dealing with the problem, and information on the nearest support service.

Images