System

Abstract

A system is provided.SOLUTION: A system includes means for receiving sensor data collected from a vehicle, data analysis means including a generation model for analyzing the received sensor data in real time and detecting a failure sign, notification means for notifying a driver of the detected failure sign, repair shop search means for searching for a nearest repair shop when a failure is expected, and reservation procedure means for performing a reservation procedure to the searched repair shop.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional vehicle failure sign detection systems do not perform real-time data analysis, and often miss signs of failure. As a result, unexpected vehicle breakdowns occur, often resulting in emergency repairs and expensive maintenance costs. Furthermore, when repairs are needed, it is difficult to quickly find the nearest repair shop, and the time and effort required to complete repair procedures is a challenge. [Means for solving the problem]

[0005] The present invention provides a system that receives sensor data collected from a vehicle, analyzes it in real time, detects signs of a malfunction, and notifies the driver. The system of the present invention includes a means for receiving sensor data from the vehicle in real time, a data analysis means including a generative model that analyzes the received sensor data and detects signs of a malfunction, and a notification means for notifying the driver of the detected signs of a malfunction. The system also includes a repair shop search means and a reservation processing means that, when a malfunction is predicted, search for the nearest repair shop and make a reservation. This system enables early detection of unexpected vehicle malfunctions and enables prompt and appropriate repairs.

[0006] "Sensor data" refers to data collected from various sensors installed inside a vehicle, and includes information on the vehicle's engine condition, braking system, tire pressure, temperature, etc.

[0007] "Real-time" refers to data collection and processing occurring almost instantaneously, with minimal actual time lag.

[0008] A "generative model" is a mathematical model that uses machine learning algorithms to detect patterns and anomalies from input data.

[0009] "Data analysis means" refers to hardware or software components that analyze received data and make specific decisions based on its contents.

[0010] "Notification means" refers to a means for notifying the driver of detected results or information, and includes voice messages and display screens.

[0011] The "repair shop search means" is a system component that has the function of searching for the most suitable repair shop based on the current location of the vehicle and other conditions.

[0012] "Reservation procedure means" refers to hardware and software components for automatically making a reservation for repairs at a selected repair shop. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The embodiments of the present invention will be specifically described below.

[0035] Sensor data collection and transmission

[0036] The device collects data from multiple sensors installed in the vehicle. This sensor data includes various vehicle information such as engine status, braking system, tire pressure, and temperature. The collected data is compiled into a single data packet within the device. The device then transmits this data packet to a server in real time.

[0037] Data analysis and failure prediction

[0038] The server receives sensor data sent from the device. The received data is analyzed in real time using a generative model. This generative model can detect abnormal patterns and signs of faults using machine learning algorithms. The server uses the analysis results to determine whether there are signs of a fault in the vehicle.

[0039] Driver Notification

[0040] If the server detects any signs of a malfunction, it sends the information to the terminal in real time. The terminal then notifies the driver of the received signs of a malfunction. The notification method can take the form of a voice message or a display to warn the driver of the presence of signs of a malfunction.

[0041] Find and book a repair shop

[0042] If a malfunction is detected, the server searches for the nearest repair shop based on the vehicle's current location. The search results generate a list of multiple repair shops. This list is provided to the driver via their terminal. The driver can then select the repair shop of their choice from the provided list.

[0043] Once the selection is complete, the server automatically schedules a repair with the selected repair shop, including details such as the vehicle's condition and the driver's preferred date and time, ensuring a fast and timely repair.

[0044] Specific examples

[0045] For example, suppose an onboard sensor detects an abnormal engine temperature while a vehicle is traveling on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative model and detects signs of engine failure. As a result, the server sends information about the signs of failure to the device, and the device notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[0046] The server then searches for the nearest repair shop based on the vehicle's current location and sends a list of several repair shops to the terminal. Once the driver selects the desired repair shop, the server proceeds with the reservation procedure at that repair shop. This series of steps allows the driver to quickly discover the problem and take appropriate action.

[0047] This invention is a system that significantly contributes to extending the lifespan of vehicles and reducing maintenance costs by analyzing various sensor data on vehicles in real time and quickly detecting and notifying signs of malfunction.

[0048] The processing flow will be explained below.

[0049] Step 1:

[0050] The device collects various data from sensors installed inside the vehicle, including information on engine status, braking system, tire pressure, temperature, etc.

[0051] Step 2:

[0052] The device collects sensor data and assembles it into a single data packet, which contains information from various sensors.

[0053] Step 3:

[0054] The device sends the bundled data packets to the server in real time using HTTP POST requests.

[0055] Step 4:

[0056] The server receives the sensor data sent from the device and temporarily stores the data on the server side.

[0057] Step 5:

[0058] The server analyzes the received sensor data using a generative model, which uses machine learning algorithms to detect abnormal patterns and signs of failure.

[0059] Step 6:

[0060] The server determines whether any signs of failure have been detected from the analysis results, and if so, sends that information to the terminal.

[0061] Step 7:

[0062] The terminal receives the notification of the malfunction sign sent from the server and notifies the driver of this information using means such as a voice message or a display.

[0063] Step 8:

[0064] When a malfunction symptom is detected, the server searches for the nearest repair shop based on the current location of the vehicle, and generates a list of repair shops.

[0065] Step 9:

[0066] The server sends the generated list of repair shops to the terminal, which displays the list to the driver and prompts them to make a selection.

[0067] Step 10:

[0068] The user (driver) selects the desired repair shop from a list of repair shops via the terminal, and the selected information is sent to the server.

[0069] Step 11:

[0070] The server automatically processes reservations with the selected repair shops, including details such as the vehicle's condition and the driver's desired date and time.

[0071] Step 12:

[0072] The server notifies the terminal that the reservation procedure is complete, and the terminal notifies the driver that the reservation is complete, providing support to ensure the vehicle repair is carried out smoothly.

[0073] Example 1

[0074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0075] Conventional vehicle management systems lacked an integrated means for collecting and analyzing sensor data, making it difficult to quickly identify and notify signs of malfunction. Furthermore, they lacked the functionality to automatically search for and reserve the most suitable repair shop when signs of malfunction were detected, making it difficult to perform fast and efficient maintenance. This created the challenge of preventing vehicle breakdowns and reducing maintenance costs.

[0076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0077] In this invention, the server includes means for receiving sensor data collected from the vehicle, data analysis means for analyzing the received sensor data in real time and including a generative model for detecting signs of failure, notification means for notifying the user of the detected signs of failure, repair facility search means for searching for the nearest repair facility when a failure is predicted, and reservation procedure means for making a reservation at the searched repair facility. This makes it possible to analyze the sensor data in real time and quickly detect and notify signs of failure, thereby significantly improving the efficiency and effectiveness of vehicle maintenance.

[0078] "Vehicle" refers to a moving body such as an automobile or motorcycle that travels on the ground.

[0079] "Sensor data" refers to information collected from various sensors installed in a vehicle, and includes data on the power plant status, braking system, tire pressure, temperature, etc.

[0080] "Means for receiving" refers to communication equipment or programs for obtaining sensor data from other devices or systems.

[0081] "Data analysis means" refers to an analysis device or program that processes received sensor data and includes a generative model for detecting abnormalities or signs of failure.

[0082] A "generative model" refers to a data model that has been trained using machine learning algorithms to make predictions or analyses.

[0083] "Notification means" refers to devices or programs such as voice messages or display screens that notify the user of detected signs of a malfunction.

[0084] "Repair facility search means" refers to a device or program for searching for the nearest repair facility based on the current location of the vehicle when a malfunction symptom is detected.

[0085] "Reservation processing means" refers to a device or program for automatically making a reservation at a searched repair facility.

[0086] "User" means the person who drives or owns a Vehicle.

[0087] "Real-time" refers to data collection and processing occurring immediately, without delay.

[0088] The present invention is a system that uses sensor data collected from a vehicle to detect signs of a malfunction, notifies the user, and searches for the nearest repair facility and completes the reservation procedure. This system is configured and operates as follows.

[0089] Sensor data collection and transmission

[0090] The terminal collects data from multiple sensors installed in the vehicle. These sensors include sensors for measuring the status of the powertrain, braking system, tire pressure, and temperature. The collected sensor data is compiled into a single data packet within the terminal. For example, data such as 90 degrees from the engine temperature sensor and 2.5 bar from the tire pressure sensor is collected.

[0091] The device then transmits the data packets to the server in real time over Wi-Fi or 4G / 5G networks, minimizing data latency.

[0092] Data analysis and failure prediction

[0093] The server receives sensor data sent from the device. The received data is analyzed in real time using a generative AI model using data analysis software such as TensorFlow or PyTorch. During the analysis, machine learning algorithms are used to detect abnormal patterns and signs of malfunction. For example, if the engine temperature is outside the normal range (above 90 degrees), it is recognized as a sign of malfunction.

[0094] Driver Notification

[0095] If a malfunction sign is detected, the server sends that information to the terminal. The terminal then notifies the driver of the received malfunction sign notification. This notification method can include a voice message or a display. Specifically, the message displayed will say, "An abnormality has been detected in the engine. Immediate repair is required."

[0096] Find and book a repair shop

[0097] If a malfunction is detected, the server searches for the nearest repair facility based on the vehicle's current location. The server references an online database and generates a list of multiple repair facilities. This list is provided to the user via their terminal. The user can select the desired repair facility from the provided list. Once the selection is complete, the server automatically completes the reservation procedure at the selected repair facility. The reservation procedure includes detailed information such as the vehicle's condition and the user's desired date and time.

[0098] Specific examples

[0099] For example, suppose a sensor detects an abnormal engine temperature while a vehicle is driving on a highway. This data is sent by the device to a server, which then analyzes it using a generative AI model. As a result of the analysis, signs of engine failure are detected. The server sends this information to the device, which then notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[0100] The server then searches for the nearest repair facility based on the vehicle's current location and sends a list of several repair facilities to the terminal. Once the driver selects the desired repair facility, the server proceeds with the reservation procedure at that repair facility. This series of steps allows the driver to quickly discover the malfunction and take appropriate action.

[0101] Prompt Sentence Examples

[0102] "We have acquired abnormal engine temperature data. Please analyze this data to detect signs of failure."

[0103] "Find the nearest repair shops based on your vehicle's current location and generate a list."

[0104] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0105] System program processing flow

[0106] Step 1: Collect sensor data

[0107] Specific details:

[0108] The device collects data from multiple sensors installed in the vehicle, including sensors to measure the status of the powerplant, braking system, tire pressure, and temperature.

[0109] Input and Output:

[0110] The input is raw data from each sensor, and the output is a data packet that aggregates this data.

[0111] Specific behavior:

[0112] For example, it collects data such as 90 degrees from the engine temperature sensor and 2.5 bar from the tire pressure sensor.

[0113] Step 2: Generate data packets

[0114] Specific details:

[0115] Collected sensor data is compiled into a single data packet.

[0116] Input and Output:

[0117] The input is the data collected from each sensor in step 1. The output is a data packet that integrates this data.

[0118] Specific behavior:

[0119] It generates a data packet containing information such as engine temperature at 90 degrees and tire pressure at 2.5 bar.

[0120] Step 3: Sending data packets

[0121] Specific details:

[0122] The device transmits the generated data packets to the server in real time using Wi-Fi or 4G / 5G networks.

[0123] Input and Output:

[0124] The input is the data packet generated in step 2. The output is the data packet sent to the server.

[0125] Specific behavior:

[0126] The data packets are sent to a server over a communication network (e.g., 4G).

[0127] Step 4: Receiving the data

[0128] Specific details:

[0129] The server receives the data packets sent from the terminal.

[0130] Input and Output:

[0131] The input is the data packets sent from the device, and the output is the sensor data stored on the server.

[0132] Specific behavior:

[0133] The server checks and stores the received data.

[0134] Step 5: Data analysis

[0135] Specific details:

[0136] Based on the received data, real-time analysis is performed using a generative AI model.

[0137] Input and Output:

[0138] The input is sensor data stored on a server, and the output is the analysis results, which include signs of failure.

[0139] Specific behavior:

[0140] For example, TensorFlow analyzes that the engine temperature is abnormal (above 90 degrees).

[0141] Step 6: Pre-fault detection

[0142] Specific details:

[0143] The server detects abnormal patterns and signs of failure based on the analysis results.

[0144] Input and Output:

[0145] The input is the analysis result from the generative AI model, and the output is sensor data that indicates a potential failure.

[0146] Specific behavior:

[0147] Since the engine temperature is 90 degrees, this is detected as a sign of a malfunction.

[0148] Step 7: Sending notifications

[0149] Specific details:

[0150] If a fault sign is detected, the server transmits the information to the terminal.

[0151] Input and Output:

[0152] The input is the sensor data that indicates a failure. The output is the notification data sent to the terminal.

[0153] Specific behavior:

[0154] A message is sent to the terminal saying, "An abnormality has been detected in the engine. Immediate repair is required."

[0155] Step 8: Notify users

[0156] Specific details:

[0157] The terminal notifies the user of the received failure warning by means of a voice message or a display.

[0158] Input and Output:

[0159] The input is the notification data sent by the server. The output is the notification the user receives.

[0160] Specific behavior:

[0161] For example, a message such as "An abnormality has been detected in the engine. Immediate repair is required" may be displayed on the screen.

[0162] Step 9: Find a repair facility

[0163] Specific details:

[0164] The server searches for the nearest repair facility based on the vehicle's current location.

[0165] Input and Output:

[0166] The input is the vehicle's current location data. The output is a list of repair facilities.

[0167] Specific behavior:

[0168] For example, GPS data can be used to search an internet database for the nearest repair facility from your current location.

[0169] Step 10: Submit a list of repair facilities

[0170] Specific details:

[0171] The server sends the generated list of repair facilities to the terminal.

[0172] Input and Output:

[0173] The input is the list of repair facilities found. The output is the list sent to the terminal.

[0174] Specific behavior:

[0175] A number of potential repair facilities are sent to the device.

[0176] Step 11: User Selects Repair Facility

[0177] Specific details:

[0178] The terminal displays a list of repair facilities to the user, and the user selects the desired repair facility.

[0179] Input and Output:

[0180] The input is a list of repair facilities. The output is the user's selected repair facility.

[0181] Specific behavior:

[0182] The user selects the desired repair facility from the device screen.

[0183] Step 12: Complete the booking process

[0184] Specific details:

[0185] The server automatically processes reservations with the selected repair facility, including detailed information such as the vehicle's status and the user's desired date and time.

[0186] Input and Output:

[0187] The input is the repair facility selected by the user and the fault information. The output is a reservation completion notification to the repair facility.

[0188] Specific behavior:

[0189] A reservation message is sent to the repair facility stating, "Please repair the engine abnormality on XX date at XX time."

[0190] (Application example 1)

[0191] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0192] Conventional systems lacked the means to effectively collect various vehicle sensor data and analyze and notify abnormalities and signs of failure in real time. Furthermore, when a failure was predicted, the process of searching for and booking a repair shop was cumbersome for drivers, making it difficult to respond quickly. This meant that the system was unable to fully extend the lifespan of vehicles or reduce maintenance costs.

[0193] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0194] In this invention, the server includes: means for receiving sensor data collected from the vehicle; data analysis means including a generative model that analyzes the received sensor data in real time and detects signs of failure; means for notifying the driver of the detected signs of failure; repair shop search means for searching for the nearest repair shop when a failure is predicted; reservation processing means for making a reservation at the searched repair shop; data transmission means for collecting various sensor data from the vehicle via a smartphone and transmitting it to a cloud server; and means for analyzing the sensor data on the cloud server, detecting signs of failure or abnormalities, and notifying the driver in real time. This makes it possible to quickly collect and analyze various sensor data from the vehicle and notify the driver of signs of failure in real time. Furthermore, when a failure is predicted, the server searches for the nearest repair shop and automatically processes the reservation, thereby speeding up maintenance response and reducing the driver's hassle.

[0195] A "vehicle" is a moving body, including an automobile, and is a means of transportation intended to travel on roads.

[0196] "Sensor data" refers to data collected by various sensors installed in a vehicle, including engine status, braking system, tire pressure, temperature, etc.

[0197] "Means for receiving" refers to the means for inputting sensor data from the vehicle to a server or terminal.

[0198] A "generative model" is a model trained using machine learning algorithms to make predictions or classifications based on input data.

[0199] "Data analysis means" refers to a means for analyzing received sensor data in real time and detecting abnormalities or signs of failure.

[0200] The "notification means" is a means for notifying the driver of signs of malfunction or abnormalities based on the analysis results.

[0201] The "repair shop search means" is a means for searching for the nearest repair shop based on the vehicle's location information when a breakdown is expected.

[0202] The "reservation procedure means" is a means for automatically making a reservation to visit the searched repair shop.

[0203] A "smartphone" is a multi-functional mobile phone terminal that has mobile communication functions and Internet connection functions.

[0204] A "cloud server" is a server accessed via the Internet that provides computing resources for storing and analyzing large amounts of data.

[0205] The "data transmission means" is a means for transmitting collected sensor data from the smartphone in the vehicle to the cloud server.

[0206] "Real-time" refers to processing occurring immediately without delay.

[0207] "Abnormal" refers to a condition of a vehicle or its parts that is different from its normal operating condition.

[0208] This invention is a system that collects various sensor data from a vehicle, analyzes it in real time, and detects signs of failure, and is particularly applicable to autonomous vehicles.

[0209] First, the vehicle is equipped with multiple sensors that measure the engine, braking system, tire pressure, temperature, etc. These sensor data are collected via a smartphone installed in the vehicle, which then compiles the data into a single data packet and sends it to a cloud server.

[0210] A generative model trained using machine learning algorithms is installed on the cloud server. This generative model analyzes the received sensor data in real time and detects abnormal patterns and signs of malfunction, such as when the engine temperature is higher than normal or when tire pressure drops suddenly.

[0211] Depending on the analysis results, the server will notify the smartphone of any detected signs of malfunction. Notification methods include voice messages and display on the screen. For example, the server will warn the driver by saying, "An abnormality has been detected in the engine. Immediate repair is required."

[0212] The server then searches for the nearest repair shop based on the vehicle's current location. The search results are sent to the smartphone as a list of multiple repair shops. When the driver selects the desired repair shop, the server automatically makes a reservation with that shop. The reservation procedure includes detailed information such as the vehicle's status and the driver's desired date and time.

[0213] The hardware used includes various sensors in the vehicle, smartphones, and cloud servers. The software includes a data collection application running on the smartphone and a data analysis program on the cloud server. In particular, the machine learning models used on the cloud server use libraries such as scikit-learn and TensorFlow.

[0214] As a concrete example, consider a case where an abnormal engine temperature is detected while driving on a highway. This data is sent to a cloud server via a smartphone and analyzed by a generative model. If a sign of engine failure is detected as a result, a notification is displayed on the smartphone stating, "There is an abnormality in the engine. Repairs are required immediately." The server then searches for the nearest repair shop and sends a list of several shops to the smartphone. When the driver selects the desired repair shop, the server automatically processes the reservation procedure at that shop.

[0215] An example of a prompt is:

[0216] "GPS location: {lat}, {lng}. Engine temperature: {engine_temp} degrees. Brake status: {brake_status}. Tire pressure: {tire_pressure} psi. If an abnormality is detected, we will notify you of the nearest repair shop and start the appointment process."

[0217] Based on this prompt, the system analyzes data collected in real time and automates fault prediction and repair shop searches.

[0218] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0219] Step 1:

[0220] The smartphone collects data from various vehicle sensors (engine, brake system, tire pressure, temperature, etc.). The sensor data is input, and the smartphone compiles the collected data into a single data packet. This data packet is output. Specifically, the smartphone converts the voltage and signals from the sensors into digital data and packets it in the appropriate format.

[0221] Step 2:

[0222] The smartphone sends a data packet to the cloud server. The input is the data packet generated in step 1, and the output is the data packet sent over the Internet. Specifically, the smartphone sends an HTTP request to the specified server URL using Wi-Fi or the mobile network.

[0223] Step 3:

[0224] The cloud server analyzes the received data packets. The input is the data packets and the output is the analysis results. A generative AI model is used for this analysis to detect anomalies and signs of failure. Specifically, a Python program installed on the cloud server analyzes the data packets and uses machine learning algorithms to perform pattern matching and predictions.

[0225] Step 4:

[0226] If the cloud server detects a sign of a failure based on the analysis results, it sends a notification to the smartphone. The input is the analysis results and the output is a notification message. Specifically, the cloud server generates a message based on the analysis results and sends a push notification to the smartphone using a service such as Firebase Cloud Messaging.

[0227] Step 5:

[0228] The smartphone receives the notification message and displays a warning to the driver. The input is the notification message, and the output is a voice message or a warning displayed on the screen. Specifically, the smartphone application is triggered by the notification and displays a message on the user interface.

[0229] Step 6:

[0230] The cloud server searches for the nearest repair shop based on the vehicle's current location. The input is the vehicle's GPS data, and the output is a list of repair shops. Specifically, the cloud server uses a map service such as Google Maps API to obtain the location information of repair shops based on the current location.

[0231] Step 7:

[0232] A list of repair shops is sent to the smartphone, and the driver selects one. The input is the list of repair shops, and the output is the repair shop selected by the driver. Specifically, the smartphone application displays a repair shop selection interface and provides operations for the driver to select.

[0233] Step 8:

[0234] The cloud server then completes the reservation procedure at the selected repair shop. The input is the repair shop selected by the driver and the vehicle status data, and the output is a reservation confirmation message. Specifically, the cloud server sends an HTTP request to the repair shop's reservation system to complete the reservation procedure.

[0235] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0236] The embodiments of the present invention will be specifically described below.

[0237] Sensor data collection and transmission

[0238] The terminal collects data from multiple sensors installed in the vehicle, including information on engine status, braking system, tire pressure, temperature, etc. The collected data is compiled into a single data packet within the terminal and then transmitted to a server in real time.

[0239] Data analysis and failure prediction

[0240] The server receives sensor data sent from the device. The received data is analyzed in real time using a generative model. This generative model can detect abnormal patterns and signs of faults using machine learning algorithms. The server uses the analysis results to determine whether there are signs of a fault in the vehicle.

[0241] Recognizing user emotions with an emotion engine

[0242] The device is equipped with an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's tone of voice, facial expressions, body movements, etc., and can recognize the user's emotional state in real time. The emotion engine sends the recognized emotion data to the server, which then incorporates this data into the analysis results.

[0243] Driver notification and adaptation

[0244] The server selects an appropriate notification method based on the signs of a malfunction and the emotional data obtained from the emotion engine. For example, if the server determines that the driver is under stress, it instructs the server to provide gentle voice notifications and relaxation support. The notification is sent via the device, gently informing the driver of the presence of signs of a malfunction.

[0245] Find and book a repair shop

[0246] If a malfunction is detected, the server searches for the nearest repair shop based on the vehicle's current location. As a result of the search, a list of multiple repair shops is generated and provided to the driver via the terminal. The driver can then select the desired repair shop from the provided list.

[0247] Reservation procedure

[0248] The server automatically schedules a reservation with the selected repair shop, including details such as the vehicle's condition and the driver's preferred date and time, ensuring fast and timely repairs.

[0249] Specific examples

[0250] For example, suppose an onboard sensor detects an abnormal engine temperature while a vehicle is traveling on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative model and detects signs of engine failure. As a result, the server sends information about the signs of failure to the device, and the device notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[0251] Meanwhile, the emotion engine analyzes the driver's facial expressions and voice tone to determine that the driver is under stress. Based on this information, the server instructs the device to issue a gentle voice notification to the driver. The server then searches for the nearest repair shop based on the vehicle's current location and sends a list of multiple repair shops to the device. Once the driver selects their preferred repair shop, the server automatically makes a reservation with that shop. This series of steps allows the driver to quickly discover the problem and take appropriate action.

[0252] This system significantly contributes to extending the life of a vehicle and reducing maintenance costs by analyzing various vehicle sensor data in real time and quickly detecting and notifying signs of malfunction. In addition, by combining it with an emotion engine, it provides appropriate notifications according to the driver's emotional state, realizing a more user-friendly system.

[0253] The processing flow will be explained below.

[0254] Step 1:

[0255] The device collects various data from multiple sensors installed in the vehicle, including information on engine status, braking system, tire pressure, temperature, etc.

[0256] Step 2:

[0257] The device aggregates the collected sensor data into a single data packet, which is then sent to the server in real time.

[0258] Step 3:

[0259] The server receives the sensor data sent from the device and temporarily stores the received data within the server.

[0260] Step 4:

[0261] The server uses the generative model to analyze the incoming sensor data in real time, using machine learning algorithms to detect abnormal patterns and signs of failure.

[0262] Step 5:

[0263] The server determines whether any signs of failure have been detected from the analysis results, and if so, sends that information to the terminal.

[0264] Step 6:

[0265] The terminal receives the notification of the malfunction signs sent from the server. The terminal notifies the driver of this information. Specifically, the terminal warns the driver of the existence of the malfunction signs using a voice message or a display.

[0266] Step 7:

[0267] The device's emotion engine analyzes the user's (driver's) voice tone, facial expressions, body movements, etc. to recognize the user's emotional state in real time. The recognized emotion data is sent from the device to the server.

[0268] Step 8:

[0269] The server analyzes the emotional data sent from the emotion engine and determines whether the driver is in a stressed or relaxed state.

[0270] Step 9:

[0271] The server selects the appropriate notification method based on the malfunction signs and emotional data. For example, if it determines that the driver is under stress, it will provide gentle voice notifications or relaxation support.

[0272] Step 10:

[0273] The device notifies the driver based on instructions from the server, including voice messages, displaying information on the screen, and even playing relaxation music.

[0274] Step 11:

[0275] When a malfunction symptom is detected, the server searches for the nearest repair shop based on the vehicle's current location, and generates a list of multiple repair shops as a search result.

[0276] Step 12:

[0277] The server sends the generated repair shop list to the terminal, which displays the list to the driver, allowing the driver to select the desired repair shop.

[0278] Step 13:

[0279] The driver (user) selects the desired repair shop from a list of repair shops via the terminal, and the selected information is sent to the server.

[0280] Step 14:

[0281] The server automatically processes reservations with the selected repair shop, and the reservation includes detailed information such as the vehicle's condition and the driver's desired date and time.

[0282] Step 15:

[0283] The server notifies the terminal that the reservation procedure is complete, and the terminal notifies the driver that the reservation is complete, providing support to ensure the vehicle repair is carried out smoothly.

[0284] Example 2

[0285] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0286] Conventional vehicle operation assistance systems have the ability to detect and notify drivers of signs of malfunction based on sensor data, but they lack the ability to provide appropriate responses and notifications that take into account the user's emotional state. This can result in inadequate timing for notifications sent to stressed drivers. Furthermore, functions for searching for the nearest repair facility based on signs of malfunction and making reservations are not fully integrated, placing a significant burden on the driver. These issues need to be resolved.

[0287] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0288] In this invention, the server includes means for receiving sensor data collected from the vehicle, data analysis means including a generative model that analyzes the received sensor data in real time and detects signs of failure, emotion recognition means for recognizing the user's emotional state and collecting emotion data, notification means for notifying the driver based on the detected signs of failure and emotion data, repair facility search means for searching for the nearest repair facility when a failure is predicted, and reservation procedure means for making a reservation at the searched repair facility. This makes it possible to quickly detect and notify signs of failure and quickly take appropriate repair measures while taking the driver's emotional state into consideration.

[0289] A "vehicle" is a means of transportation used as a means of transportation, and is a general term for a vehicle that mainly includes an engine, a braking system, etc.

[0290] "Sensor data" refers to data collected from various sensors attached to a vehicle, and includes information on engine status, braking system, tire pressure, temperature, etc.

[0291] "Means for receiving" refers to a function or device that acquires data sent from outside.

[0292] A "generative model" is a model that uses machine learning algorithms to analyze data and detect abnormal patterns and signs of failure.

[0293] "Data analysis means" refers to the functions and technologies for analyzing received data and extracting meaningful information.

[0294] "Emotion recognition means" refers to devices or technologies that analyze and recognize a user's emotional state in real time from their tone of voice, facial expressions, body movements, etc.

[0295] "Notification means" refers to a function or device for notifying the user of analysis results and important information.

[0296] A "repair facility" is a location that provides equipment and services for repairing vehicle breakdowns and maintenance.

[0297] "Repair facility search means" refers to a function or technology for searching for the nearest repair facility based on information such as the vehicle's current location.

[0298] "Reservation procedure means" refers to a function or system for automatically making reservations for repairs or maintenance at a repair facility.

[0299] This invention is a system that detects signs of malfunction and notifies the driver based on sensor data collected from the vehicle and user emotional data. The system aims to improve driver safety and comfort by analyzing sensor data in real time and providing appropriate notifications and responses.

[0300] Hardware Configuration

[0301] The device is installed inside the vehicle and collects data from multiple sensors (engine condition sensors, brake system sensors, tire pressure sensors, temperature sensors, etc.) It is also equipped with a camera and microphone for emotion recognition, which capture the user's voice tone, facial expressions, and body movements.

[0302] The server is a central system for receiving and analyzing collected sensor data and emotion data in real time. It is built on a computer equipped with a high-performance processor and large memory capacity. To perform data analysis, a generative AI model using a machine learning algorithm is implemented. For example, machine learning frameworks such as TensorFlow and PyTorch are used.

[0303] Software Configuration

[0304] The device contains software to collect sensor data and compile it into a single data packet. It also contains an emotion engine to analyze the user's emotional data. The emotion engine has algorithms that determine the user's emotional state, such as stress or relief, based on the user's tone of voice, facial expressions, and body movements.

[0305] The server is equipped with a generative AI model for analyzing the received data. This AI model has the ability to detect signs of malfunction from the received sensor data. It also includes an algorithm for selecting the optimal notification method for the driver based on emotion recognition results.

[0306] Specific examples

[0307] Suppose an onboard sensor detects an abnormal engine temperature while a vehicle is traveling on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative AI model and detects signs of engine failure. As a result, the server sends information about the signs of failure to the device, and the device notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[0308] Meanwhile, the emotion engine analyzes the driver's facial expressions and voice tone to determine if the driver is under stress. Based on this information, the server instructs the device to issue a gentle voice notification to the driver. The server then searches for the nearest repair shop based on the vehicle's current location and sends a list of multiple repair shops to the device. Once the driver selects the desired repair shop, the server automatically completes the reservation procedure with that repair shop.

[0309] Prompt Sentence Examples

[0310] For example, the following prompt statements can be used to specify system behavior:

[0311] "What is the procedure if the onboard sensors detect an abnormal engine temperature while driving on the highway? Also, what kind of notification or support will be provided if the driver feels stressed at that time?"

[0312] In this way, the system of the present invention significantly contributes to extending the life of a vehicle and reducing maintenance costs by analyzing various sensor data of the vehicle in real time and quickly detecting and notifying signs of malfunction. In addition, by providing notifications that take the user's emotional state into consideration, it realizes a more user-friendly system.

[0313] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0314] Step 1: Collect sensor data

[0315] The terminal collects data from sensors installed inside the vehicle, such as those for the engine condition, braking system, tire pressure, and temperature. The collected data is temporarily stored in the terminal's memory. The input is data from each sensor, and the output is compiled into a single data packet. Specifically, when the tire pressure sensor detects pressure below a specified value, it records the data in memory.

[0316] Step 2: Sending sensor data

[0317] The terminal aggregates the collected sensor data into a single packet and transmits it in encrypted form to the server in real time. The input is the aggregated data packet, and the output is a transmission success message to the server. Specifically, the terminal aggregates engine status data, brake system data, tire pressure data, and temperature data, encrypts it, and transmits it to the server.

[0318] Step 3: Receiving the data

[0319] The server receives sensor data sent from the device. The input is an encrypted data packet, and the output is decrypted sensor data. Specifically, the server receives the data packet, decrypts it, and obtains the data of each sensor.

[0320] Step 4: Data analysis and failure detection

[0321] The server analyzes the received data in real time using a generative AI model to detect abnormal patterns and signs of failure. The input is decoded sensor data, and the output is the presence or absence of signs of failure and detailed data. Specifically, if the engine temperature exceeds the normal range, it is determined to be a sign of engine failure.

[0322] Step 5: Collecting sentiment data

[0323] The device analyzes the user's voice tone, facial expressions, and body movements to recognize their emotional state in real time. The input is the user's voice and video data, and the output is analyzed emotional data. Specifically, the camera and microphone capture the user's facial expressions and voice, which are then analyzed by the emotion engine.

[0324] Step 6: Sending Emotion Data

[0325] The device sends the recognized emotion data to the server. The input is the analyzed emotion data, and the output is a transmission success message to the server. Specifically, the device sends data indicating that the user is in a stressed state to the server.

[0326] Step 7: Decide on notifications

[0327] The server determines the optimal notification method based on the received sensor data and emotional data. The input is failure prediction data and emotional data, and the output is the notification content and method. Specifically, if an engine abnormality is detected and the user is in a stressful state, a gentle voice notification will be selected.

[0328] Step 8: Notify drivers

[0329] The device follows instructions from the server and notifies the driver of signs of malfunction and their emotions. The input is the content and method of the notification, and the output is the notification to the driver. Specifically, it notifies the driver by voice, saying, "An abnormality has been detected in the engine. Immediate repair is required."

[0330] Step 9: Find a repair facility

[0331] The server searches for the nearest repair facility based on the vehicle's current location. The input is the vehicle's location data, and the output is a list of repair facilities. Specifically, it uses GPS data to search for nearby repair facilities and generates the list.

[0332] Step 10: Select and book a repair facility

[0333] The user selects the desired repair facility from a list of repair facilities provided by the terminal. The information of the selected repair facility is sent to the server, and the reservation procedure is carried out automatically. The input is the user's selection information, and the output is a reservation confirmation message. Specifically, the user taps the desired repair facility, and the server contacts that facility to complete the reservation procedure.

[0334] (Application example 2)

[0335] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0336] Modern cars are equipped with many sensors, but there are only a limited number of systems that can analyze this sensor data in real time and predict potential breakdowns in advance. Furthermore, there are no systems that recognize the driver's emotional state and immediately take appropriate action based on that. This creates a problem: drivers tend to feel stressed when they encounter a breakdown, making it difficult to quickly repair the vehicle.

[0337] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving sensor data collected from the vehicle, data analysis means including a generative model that analyzes the received sensor data in real time and detects signs of failure, emotion recognition means that recognizes the user's emotional state and incorporates that data into the analysis results, notification means that selects an appropriate notification method based on the detected signs of failure and the user's emotional state and notifies the driver, repair facility search means that, when a failure is predicted, searches for the nearest repair facility based on the vehicle location, and reservation procedure means that makes a reservation for the searched repair facility. This enables rapid detection of signs of failure, appropriate notification according to the driver's emotions, and immediate repair arrangements.

[0338] "Sensor data" refers to data collected from multiple sensors installed in a vehicle, and includes information on engine status, braking system, tire pressure, temperature, etc.

[0339] "Real-time analysis" refers to analyzing sensor data immediately as it is collected.

[0340] "Failure signs" are data based on collected sensor data that indicate signs of impending failure in various parts of the vehicle.

[0341] A "generative model" is a model that uses machine learning algorithms to analyze sensor data and detect abnormal patterns and signs of failure.

[0342] The "data analysis means" is a device or software for analyzing received sensor data and detecting signs of failure.

[0343] The "emotion recognition means" is a device or software that analyzes the user's tone of voice, facial expressions, body movements, etc., and recognizes the user's emotional state in real time.

[0344] The "notification means" is a device or software that selects an appropriate notification method based on the detected signs of failure and the user's emotional state, and notifies the driver.

[0345] The "repair facility search means" is a device or software for searching for the nearest repair facility based on the current location of the vehicle when a failure symptom is detected.

[0346] The "reservation procedure means" is a device or software for automatically carrying out reservation procedures with the searched repair facility.

[0347] The embodiments of the present invention will be specifically described below.

[0348] Sensor data collection and transmission

[0349] The server collects sensor data from multiple sensors installed in the vehicle, including information on engine status, braking system, tire pressure, temperature, etc. The collected data is compiled into a single data packet on the device and then transmitted to the server in real time.

[0350] Data analysis and failure prediction

[0351] The server receives sensor data sent from the device and analyzes it in real time using a generative model. This generative model uses machine learning algorithms to detect abnormal patterns and signs of malfunction. As a result of the analysis, it determines whether there are signs of a malfunction in the vehicle.

[0352] Recognizing user emotions with an emotion engine

[0353] The server uses the smartphone's camera and microphone to analyze the driver's facial expressions and tone of voice to recognize their emotional state in real time. This emotional data is then added to the analysis results and sent to the server.

[0354] Driver notification and adaptation

[0355] The server selects the appropriate notification method based on the malfunction signs and emotion recognition data. For example, if the server determines that the driver is under stress, it instructs the device to provide gentle voice notifications or relaxation support. Based on this, the device gently notifies the driver of the presence of malfunction signs.

[0356] Find and book a repair shop

[0357] If a malfunction is detected, the server searches for the nearest repair facility based on the vehicle's current location. A list of multiple repair facilities is generated as a result of the search and provided to the driver via the terminal. The driver can then select the desired repair facility from the provided list.

[0358] Reservation procedure

[0359] The server automatically schedules a reservation with the selected repair facility, including details such as the vehicle's condition and the driver's preferred date and time, ensuring fast and timely repairs.

[0360] Hardware and software used

[0361] Hardware: Sensors in the vehicle (engine sensor, tire pressure sensor, temperature sensor), smartphone (camera, microphone)

[0362] Software: Server-based machine learning algorithms, generative models, emotion recognition engines, cloud servers, notification systems

[0363] Specific examples

[0364] For example, suppose an onboard sensor detects an abnormal engine temperature while a vehicle is driving on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative model to detect signs of engine failure. Based on the analysis results and the driver's emotion recognition data, the server instructs the device to send an appropriate notification. The device notifies the driver, "An abnormality has been detected in the engine. Please remain calm and take action." The server then searches for the nearest repair facility based on the vehicle's current location and sends a list of multiple repair facilities to the device. When the driver selects the desired repair facility, the server automatically processes the reservation at that facility.

[0365] Prompt Sentence Examples

[0366] Failure prediction prompt

[0367] Prompt: Detect pre-failure engine conditions based on data sent from sensors. Example: Engine temperature is outside normal range.

[0368] Data: {"Engine Status": "Abnormal", "Brake System": "Normal", "Tire Pressure": 32, "Temperature": 110}

[0369] Result: Premature failure. Engine requires immediate repair.

[0370] Driver emotion recognition prompts

[0371] Prompt: Based on camera and microphone data, determine if the driver is stressed. Example: Talking with a stern face.

[0372] Data: {"Facial Expression": "Stern", "Voice Tone": "High-pitched"}

[0373] Result: The driver is stressed. Use a gentle tone to notify them.

[0374] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0375] Step 1:

[0376] The terminal collects sensor data from multiple sensors installed in the vehicle, such as engine status, braking system, tire pressure, and temperature, and compiles it into a data packet. The input is data from the sensors, and the output is a data packet.

[0377] Step 2:

[0378] The terminal sends the collected data packets to a server in real time. The server inputs the received sensor data into an analysis platform. Specifically, a communication protocol such as an HTTP request is used. The input is the data packets, and the output is the data sent to the server.

[0379] Step 3:

[0380] The server analyzes the received sensor data in real time using a generative model. This uses machine learning algorithms to detect abnormal patterns and signs of failure. The input is the sensor data, and the output is the analysis results.

[0381] Step 4:

[0382] The server recognizes the driver's emotional state using data provided by the smartphone's camera and microphone. Specifically, it uses image processing and voice analysis technology (e.g., OpenCV, voice analysis software). The input is data from the camera and microphone, and the output is emotional state data.

[0383] Step 5:

[0384] The server selects an appropriate notification method based on the failure signs and emotion recognition data and sends instructions to the device. For example, if the driver is in a stressful state, it will instruct a gentle voice notification. The input is the analysis results and emotional state data, and the output is notification instruction data.

[0385] Step 6:

[0386] The terminal notifies the driver based on instructions from the server. For example, it may notify the driver that "An abnormality has been detected in the engine. Please remain calm and take appropriate action." The input is notification instruction data, and the output is a notification to the user.

[0387] Step 7:

[0388] When a malfunction symptom is detected, the server searches for the nearest repair facility based on the vehicle's current location information. Here, GPS information and a database of repair facilities are used. The input is location information and malfunction symptom data, and the output is a list of repair facilities.

[0389] Step 8:

[0390] The terminal displays the list of repair facilities sent from the server to the driver and lets the driver select the desired repair facility. The input is the list of repair facilities and the output is the user's selection.

[0391] Step 9:

[0392] The server automatically processes the reservation with the selected repair facility. Specifically, it accesses the repair facility's reservation system and communicates the vehicle's status and the driver's desired date and time. The input is the user's selection and vehicle information, and the output is reservation confirmation.

[0393] Step 10:

[0394] The server sends the reservation confirmation information to the terminal, and the terminal notifies the driver. The input is the reservation confirmation data, and the output is the reservation confirmation notification to the user.

[0395] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0396] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0397] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0398] [Second embodiment]

[0399] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0400] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0401] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0402] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0403] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0404] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0405] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0406] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0407] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0408] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0409] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0410] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0411] The embodiments of the present invention will be specifically described below.

[0412] Sensor data collection and transmission

[0413] The device collects data from multiple sensors installed in the vehicle. This sensor data includes various vehicle information such as engine status, braking system, tire pressure, and temperature. The collected data is compiled into a single data packet within the device. The device then transmits this data packet to a server in real time.

[0414] Data analysis and failure prediction

[0415] The server receives sensor data sent from the device. The received data is analyzed in real time using a generative model. This generative model can detect abnormal patterns and signs of faults using machine learning algorithms. The server uses the analysis results to determine whether there are signs of a fault in the vehicle.

[0416] Driver Notification

[0417] When a malfunction sign is detected, the server sends the information to the terminal in real time. The terminal then notifies the driver of the malfunction sign. The notification method can take the form of a voice message or a display to warn the driver of the presence of a malfunction sign.

[0418] Find and book a repair shop

[0419] If a malfunction is detected, the server searches for the nearest repair shop based on the vehicle's current location. The search results generate a list of multiple repair shops. This list is provided to the driver via their terminal. The driver can then select the repair shop of their choice from the provided list.

[0420] Once the selection is complete, the server automatically schedules a reservation with the selected repair shop, including details such as the vehicle's condition and the driver's preferred date and time, ensuring a fast and timely repair.

[0421] Specific examples

[0422] For example, suppose an onboard sensor detects an abnormal engine temperature while a vehicle is traveling on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative model and detects signs of engine failure. As a result, the server sends information about the signs of failure to the device, and the device notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[0423] The server then searches for the nearest repair shop based on the vehicle's current location and sends a list of several repair shops to the terminal. Once the driver selects the desired repair shop, the server proceeds with the reservation procedure at that repair shop. This series of steps allows the driver to quickly discover the problem and take appropriate action.

[0424] This system analyzes various vehicle sensor data in real time and quickly detects and notifies of signs of malfunction, thereby significantly contributing to extending the lifespan of vehicles and reducing maintenance costs.

[0425] The processing flow will be explained below.

[0426] Step 1:

[0427] The device collects various data from sensors installed inside the vehicle, including information on engine status, braking system, tire pressure, temperature, etc.

[0428] Step 2:

[0429] The device collects sensor data and assembles it into a single data packet, which contains information from various sensors.

[0430] Step 3:

[0431] The device sends the bundled data packets to the server in real time using HTTP POST requests.

[0432] Step 4:

[0433] The server receives the sensor data sent from the device and temporarily stores the data on the server side.

[0434] Step 5:

[0435] The server analyzes the received sensor data using a generative model, which uses machine learning algorithms to detect abnormal patterns and signs of failure.

[0436] Step 6:

[0437] The server determines whether any signs of failure have been detected from the analysis results, and if so, sends that information to the terminal.

[0438] Step 7:

[0439] The terminal receives the notification of the malfunction sign sent from the server and notifies the driver of this information using means such as a voice message or a display.

[0440] Step 8:

[0441] When a malfunction symptom is detected, the server searches for the nearest repair shop based on the current location of the vehicle, and generates a list of repair shops.

[0442] Step 9:

[0443] The server sends the generated list of repair shops to the terminal, which displays the list to the driver and prompts them to make a selection.

[0444] Step 10:

[0445] The user (driver) selects the desired repair shop from a list of repair shops via the terminal, and the selected information is sent to the server.

[0446] Step 11:

[0447] The server automatically processes reservations with the selected repair shops, including details such as the vehicle's condition and the driver's desired date and time.

[0448] Step 12:

[0449] The server notifies the terminal that the reservation procedure is complete, and the terminal notifies the driver that the reservation is complete, providing support to ensure the vehicle repair is carried out smoothly.

[0450] Example 1

[0451] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0452] Conventional vehicle management systems lacked an integrated means for collecting and analyzing sensor data, making it difficult to quickly identify and notify signs of malfunction. Furthermore, they lacked the functionality to automatically search for and reserve the most suitable repair shop when signs of malfunction were detected, making it difficult to perform fast and efficient maintenance. This created the challenge of preventing vehicle breakdowns and reducing maintenance costs.

[0453] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0454] In this invention, the server includes means for receiving sensor data collected from the vehicle, data analysis means for analyzing the received sensor data in real time and including a generative model for detecting signs of failure, notification means for notifying the user of the detected signs of failure, repair facility search means for searching for the nearest repair facility when a failure is predicted, and reservation procedure means for making a reservation at the searched repair facility. This makes it possible to analyze the sensor data in real time and quickly detect and notify signs of failure, thereby significantly improving the efficiency and effectiveness of vehicle maintenance.

[0455] "Vehicle" refers to a moving body such as an automobile or motorcycle that travels on the ground.

[0456] "Sensor data" refers to information collected from various sensors installed in a vehicle, and includes data on the power plant status, braking system, tire pressure, temperature, etc.

[0457] "Means for receiving" refers to communication equipment or programs for obtaining sensor data from other devices or systems.

[0458] "Data analysis means" refers to an analysis device or program that processes received sensor data and includes a generative model for detecting abnormalities or signs of failure.

[0459] A "generative model" refers to a data model that has been trained using machine learning algorithms to make predictions or analyses.

[0460] "Notification means" refers to devices or programs such as voice messages or display screens that notify the user of detected signs of a malfunction.

[0461] "Repair facility search means" refers to a device or program for searching for the nearest repair facility based on the current location of the vehicle when a malfunction symptom is detected.

[0462] "Reservation processing means" refers to a device or program for automatically making a reservation at a searched repair facility.

[0463] "User" means the person who drives or owns a Vehicle.

[0464] "Real-time" refers to data collection and processing occurring immediately, without delay.

[0465] The present invention is a system that uses sensor data collected from a vehicle to detect signs of a malfunction, notifies the user, and searches for the nearest repair facility and completes the reservation procedure. This system is configured and operates as follows.

[0466] Sensor data collection and transmission

[0467] The terminal collects data from multiple sensors installed in the vehicle. These sensors include sensors for measuring the status of the powertrain, braking system, tire pressure, and temperature. The collected sensor data is compiled into a single data packet within the terminal. For example, data such as 90 degrees from the engine temperature sensor and 2.5 bar from the tire pressure sensor is collected.

[0468] The device then transmits the data packets to the server in real time over Wi-Fi or 4G / 5G networks, minimizing data latency.

[0469] Data analysis and failure prediction

[0470] The server receives sensor data sent from the device. The received data is analyzed in real time using a generative AI model using data analysis software such as TensorFlow or PyTorch. During the analysis, machine learning algorithms are used to detect abnormal patterns and signs of malfunction. For example, if the engine temperature is outside the normal range (above 90 degrees), it is recognized as a sign of malfunction.

[0471] Driver Notification

[0472] If a malfunction sign is detected, the server sends that information to the terminal. The terminal then notifies the driver of the received malfunction sign notification. This notification method can include a voice message or a display. Specifically, the message displayed will say, "An abnormality has been detected in the engine. Immediate repair is required."

[0473] Find and book a repair shop

[0474] If a malfunction is detected, the server searches for the nearest repair facility based on the vehicle's current location. The server references an online database and generates a list of multiple repair facilities. This list is provided to the user via their terminal. The user can select the desired repair facility from the provided list. Once the selection is complete, the server automatically completes the reservation procedure at the selected repair facility. The reservation procedure includes detailed information such as the vehicle's condition and the user's desired date and time.

[0475] Specific examples

[0476] For example, suppose a sensor detects an abnormal engine temperature while a vehicle is driving on a highway. This data is sent by the device to a server, which then analyzes it using a generative AI model. As a result of the analysis, signs of engine failure are detected. The server sends this information to the device, which then notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[0477] The server then searches for the nearest repair facility based on the vehicle's current location and sends a list of several repair facilities to the terminal. Once the driver selects the desired repair facility, the server proceeds with the reservation procedure at that repair facility. This series of steps allows the driver to quickly discover the malfunction and take appropriate action.

[0478] Prompt Sentence Examples

[0479] "We have acquired abnormal engine temperature data. Please analyze this data to detect signs of failure."

[0480] "Find the nearest repair shops based on your vehicle's current location and generate a list."

[0481] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0482] System program processing flow

[0483] Step 1: Collect sensor data

[0484] Specific details:

[0485] The device collects data from multiple sensors installed in the vehicle, including sensors to measure the status of the powerplant, braking system, tire pressure, and temperature.

[0486] Input and Output:

[0487] The input is raw data from each sensor, and the output is a data packet that aggregates this data.

[0488] Specific behavior:

[0489] For example, it collects data such as 90 degrees from the engine temperature sensor and 2.5 bar from the tire pressure sensor.

[0490] Step 2: Generate data packets

[0491] Specific details:

[0492] Collected sensor data is compiled into a single data packet.

[0493] Input and Output:

[0494] The input is the data collected from each sensor in step 1. The output is a data packet that integrates this data.

[0495] Specific behavior:

[0496] It generates a data packet containing information such as engine temperature at 90 degrees and tire pressure at 2.5 bar.

[0497] Step 3: Sending data packets

[0498] Specific details:

[0499] The device transmits the generated data packets to the server in real time using Wi-Fi or 4G / 5G networks.

[0500] Input and Output:

[0501] The input is the data packet generated in step 2. The output is the data packet sent to the server.

[0502] Specific behavior:

[0503] The data packets are sent to a server over a communication network (e.g., 4G).

[0504] Step 4: Receiving the data

[0505] Specific details:

[0506] The server receives the data packets sent from the terminal.

[0507] Input and Output:

[0508] The input is the data packets sent from the device, and the output is the sensor data stored on the server.

[0509] Specific behavior:

[0510] The server checks and stores the received data.

[0511] Step 5: Data analysis

[0512] Specific details:

[0513] Based on the received data, real-time analysis is performed using a generative AI model.

[0514] Input and Output:

[0515] The input is sensor data stored on a server, and the output is the analysis results, which include signs of failure.

[0516] Specific behavior:

[0517] For example, TensorFlow analyzes that the engine temperature is abnormal (above 90 degrees).

[0518] Step 6: Pre-fault detection

[0519] Specific details:

[0520] The server detects abnormal patterns and signs of failure based on the analysis results.

[0521] Input and Output:

[0522] The input is the analysis result from the generative AI model, and the output is sensor data that indicates a potential failure.

[0523] Specific behavior:

[0524] Since the engine temperature is 90 degrees, this is detected as a sign of a malfunction.

[0525] Step 7: Sending notifications

[0526] Specific details:

[0527] If a fault sign is detected, the server transmits the information to the terminal.

[0528] Input and Output:

[0529] The input is the sensor data that indicates a failure. The output is the notification data sent to the terminal.

[0530] Specific behavior:

[0531] A message is sent to the terminal saying, "An abnormality has been detected in the engine. Immediate repair is required."

[0532] Step 8: Notify users

[0533] Specific details:

[0534] The terminal notifies the user of the received failure warning by means of a voice message or a display.

[0535] Input and Output:

[0536] The input is the notification data sent by the server. The output is the notification the user receives.

[0537] Specific behavior:

[0538] For example, a message such as "An abnormality has been detected in the engine. Immediate repair is required" may be displayed on the screen.

[0539] Step 9: Find a repair facility

[0540] Specific details:

[0541] The server searches for the nearest repair facility based on the vehicle's current location.

[0542] Input and Output:

[0543] The input is the vehicle's current location data. The output is a list of repair facilities.

[0544] Specific behavior:

[0545] For example, GPS data can be used to search an internet database for the nearest repair facility from your current location.

[0546] Step 10: Submit a list of repair facilities

[0547] Specific details:

[0548] The server sends the generated list of repair facilities to the terminal.

[0549] Input and Output:

[0550] The input is the list of repair facilities found. The output is the list sent to the terminal.

[0551] Specific behavior:

[0552] A number of potential repair facilities are sent to the device.

[0553] Step 11: User Selects Repair Facility

[0554] Specific details:

[0555] The terminal displays a list of repair facilities to the user, and the user selects the desired repair facility.

[0556] Input and Output:

[0557] The input is a list of repair facilities. The output is the user's selected repair facility.

[0558] Specific behavior:

[0559] The user selects the desired repair facility from the device screen.

[0560] Step 12: Complete the booking process

[0561] Specific details:

[0562] The server automatically processes reservations with the selected repair facility, including detailed information such as the vehicle's status and the user's desired date and time.

[0563] Input and Output:

[0564] The input is the repair facility selected by the user and the fault information. The output is a reservation completion notification to the repair facility.

[0565] Specific behavior:

[0566] A reservation message is sent to the repair facility stating, "Please repair the engine abnormality on XX date at XX time."

[0567] (Application example 1)

[0568] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0569] Conventional systems lacked the means to effectively collect various vehicle sensor data and analyze and notify abnormalities and signs of failure in real time. Furthermore, when a failure was predicted, the process of searching for and booking a repair shop was cumbersome for drivers, making it difficult to respond quickly. This meant that the system was unable to fully extend the lifespan of vehicles or reduce maintenance costs.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0571] In this invention, the server includes: means for receiving sensor data collected from the vehicle; data analysis means including a generative model that analyzes the received sensor data in real time and detects signs of failure; means for notifying the driver of the detected signs of failure; repair shop search means for searching for the nearest repair shop when a failure is predicted; reservation processing means for making a reservation at the searched repair shop; data transmission means for collecting various sensor data from the vehicle via a smartphone and transmitting it to a cloud server; and means for analyzing the sensor data on the cloud server, detecting signs of failure or abnormalities, and notifying the driver in real time. This makes it possible to quickly collect and analyze various sensor data from the vehicle and notify the driver of signs of failure in real time. Furthermore, when a failure is predicted, the server searches for the nearest repair shop and automatically processes the reservation, thereby speeding up maintenance response and reducing the driver's hassle.

[0572] A "vehicle" is a moving body, including an automobile, and is a means of transportation intended to travel on roads.

[0573] "Sensor data" refers to data collected by various sensors installed in a vehicle, including engine status, braking system, tire pressure, temperature, etc.

[0574] "Means for receiving" refers to the means for inputting sensor data from the vehicle to a server or terminal.

[0575] A "generative model" is a model trained using machine learning algorithms to make predictions or classifications based on input data.

[0576] "Data analysis means" refers to a means for analyzing received sensor data in real time and detecting abnormalities or signs of failure.

[0577] The "notification means" is a means for notifying the driver of signs of malfunction or abnormalities based on the analysis results.

[0578] The "repair shop search means" is a means for searching for the nearest repair shop based on the vehicle's location information when a breakdown is expected.

[0579] The "reservation procedure means" is a means for automatically making a reservation to visit the searched repair shop.

[0580] A "smartphone" is a multi-functional mobile phone terminal that has mobile communication functions and Internet connection functions.

[0581] A "cloud server" is a server accessed via the Internet that provides computing resources for storing and analyzing large amounts of data.

[0582] The "data transmission means" is a means for transmitting collected sensor data from the smartphone in the vehicle to the cloud server.

[0583] "Real-time" refers to processing occurring immediately without delay.

[0584] "Abnormal" refers to a condition of a vehicle or its parts that is different from its normal operating condition.

[0585] This invention is a system that collects various sensor data from a vehicle, analyzes it in real time, and detects signs of failure, and is particularly applicable to autonomous vehicles.

[0586] First, the vehicle is equipped with multiple sensors that measure the engine, braking system, tire pressure, temperature, etc. These sensor data are collected via a smartphone installed in the vehicle, which then compiles the data into a single data packet and sends it to a cloud server.

[0587] A generative model trained using machine learning algorithms is installed on the cloud server. This generative model analyzes the received sensor data in real time and detects abnormal patterns and signs of malfunction, such as when the engine temperature is higher than normal or when tire pressure drops suddenly.

[0588] Depending on the analysis results, the server will notify the smartphone of any detected signs of malfunction. Notification methods include voice messages and display on the screen. For example, the server will warn the driver by saying, "An abnormality has been detected in the engine. Immediate repair is required."

[0589] The server then searches for the nearest repair shop based on the vehicle's current location. The search results are sent to the smartphone as a list of multiple repair shops. When the driver selects the desired repair shop, the server automatically makes a reservation with that shop. The reservation procedure includes detailed information such as the vehicle's status and the driver's desired date and time.

[0590] The hardware used includes various sensors in the vehicle, smartphones, and cloud servers. The software includes a data collection application running on the smartphone and a data analysis program on the cloud server. In particular, the machine learning models used on the cloud server use libraries such as scikit-learn and TensorFlow.

[0591] As a concrete example, consider a case where an abnormal engine temperature is detected while driving on a highway. This data is sent to a cloud server via a smartphone and analyzed by a generative model. If a sign of engine failure is detected as a result, a notification is displayed on the smartphone stating, "There is an abnormality in the engine. Repairs are required immediately." The server then searches for the nearest repair shop and sends a list of several shops to the smartphone. When the driver selects the desired repair shop, the server automatically processes the reservation procedure at that shop.

[0592] An example of a prompt is:

[0593] "GPS location: {lat}, {lng}. Engine temperature: {engine_temp} degrees. Brake status: {brake_status}. Tire pressure: {tire_pressure} psi. If an abnormality is detected, we will notify you of the nearest repair shop and start the appointment process."

[0594] Based on this prompt, the system analyzes data collected in real time and automates fault prediction and repair shop searches.

[0595] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0596] Step 1:

[0597] The smartphone collects data from various vehicle sensors (engine, brake system, tire pressure, temperature, etc.). The sensor data is input, and the smartphone compiles the collected data into a single data packet. This data packet is output. Specifically, the smartphone converts the voltage and signals from the sensors into digital data and packets it in the appropriate format.

[0598] Step 2:

[0599] The smartphone sends a data packet to the cloud server. The input is the data packet generated in step 1, and the output is the data packet sent over the Internet. Specifically, the smartphone sends an HTTP request to the specified server URL using Wi-Fi or the mobile network.

[0600] Step 3:

[0601] The cloud server analyzes the received data packets. The input is the data packets and the output is the analysis results. A generative AI model is used for this analysis to detect anomalies and signs of failure. Specifically, a Python program installed on the cloud server analyzes the data packets and uses machine learning algorithms to perform pattern matching and predictions.

[0602] Step 4:

[0603] If the cloud server detects a sign of a failure based on the analysis results, it sends a notification to the smartphone. The input is the analysis results and the output is a notification message. Specifically, the cloud server generates a message based on the analysis results and sends a push notification to the smartphone using a service such as Firebase Cloud Messaging.

[0604] Step 5:

[0605] The smartphone receives the notification message and displays a warning to the driver. The input is the notification message, and the output is a voice message or a warning displayed on the screen. Specifically, the smartphone application is triggered by the notification and displays a message on the user interface.

[0606] Step 6:

[0607] The cloud server searches for the nearest repair shop based on the vehicle's current location. The input is the vehicle's GPS data, and the output is a list of repair shops. Specifically, the cloud server uses a map service such as Google Maps API to obtain the location information of repair shops based on the current location.

[0608] Step 7:

[0609] A list of repair shops is sent to the smartphone, and the driver selects one. The input is the list of repair shops, and the output is the repair shop selected by the driver. Specifically, the smartphone application displays a repair shop selection interface and provides operations for the driver to select.

[0610] Step 8:

[0611] The cloud server then completes the reservation procedure with the selected repair shop. The input is the repair shop selected by the driver and the vehicle status data, and the output is a reservation confirmation message. Specifically, the cloud server sends an HTTP request to the repair shop's reservation system to complete the reservation procedure.

[0612] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0613] The embodiments of the present invention will be specifically described below.

[0614] Sensor data collection and transmission

[0615] The device collects data from multiple sensors installed in the vehicle, including information on engine status, braking system, tire pressure, temperature, etc. The collected data is compiled into a single data packet within the device and then transmitted to a server in real time.

[0616] Data analysis and failure prediction

[0617] The server receives sensor data sent from the device. The received data is analyzed in real time using a generative model. This generative model can detect abnormal patterns and signs of faults using machine learning algorithms. The server uses the analysis results to determine whether there are signs of a fault in the vehicle.

[0618] Recognizing user emotions with an emotion engine

[0619] The device is equipped with an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's tone of voice, facial expressions, body movements, etc., and can recognize the user's emotional state in real time. The emotion engine sends the recognized emotion data to the server, which then incorporates this data into the analysis results.

[0620] Driver notification and adaptation

[0621] The server selects an appropriate notification method based on the signs of a malfunction and the emotional data obtained from the emotion engine. For example, if the server determines that the driver is under stress, it instructs the server to provide gentle voice notifications and relaxation support. The notification is sent via the device, gently informing the driver of the presence of signs of a malfunction.

[0622] Find and book a repair shop

[0623] If a malfunction is detected, the server searches for the nearest repair shop based on the vehicle's current location. As a result of the search, a list of multiple repair shops is generated and provided to the driver via the terminal. The driver can then select the desired repair shop from the provided list.

[0624] Reservation procedure

[0625] The server automatically schedules a reservation with the selected repair shop, including details such as the vehicle's condition and the driver's preferred date and time, ensuring fast and timely repairs.

[0626] Specific examples

[0627] For example, suppose an onboard sensor detects an abnormal engine temperature while a vehicle is traveling on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative model and detects signs of engine failure. As a result, the server sends information about the signs of failure to the device, and the device notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[0628] Meanwhile, the emotion engine analyzes the driver's facial expressions and voice tone to determine that the driver is under stress. Based on this information, the server instructs the device to issue a gentle voice notification to the driver. The server then searches for the nearest repair shop based on the vehicle's current location and sends a list of multiple repair shops to the device. Once the driver selects their preferred repair shop, the server automatically makes a reservation with that shop. This series of steps allows the driver to quickly discover the problem and take appropriate action.

[0629] This system significantly contributes to extending the life of a vehicle and reducing maintenance costs by analyzing various vehicle sensor data in real time and quickly detecting and notifying signs of malfunction. In addition, by combining it with an emotion engine, it provides appropriate notifications according to the driver's emotional state, realizing a more user-friendly system.

[0630] The processing flow will be explained below.

[0631] Step 1:

[0632] The device collects various data from multiple sensors installed inside the vehicle, including information on engine status, braking system, tire pressure, temperature, etc.

[0633] Step 2:

[0634] The device aggregates the collected sensor data into a single data packet, which is then sent to the server in real time.

[0635] Step 3:

[0636] The server receives the sensor data sent from the device and temporarily stores the received data within the server.

[0637] Step 4:

[0638] The server uses the generative model to analyze the incoming sensor data in real time, using machine learning algorithms to detect abnormal patterns and signs of failure.

[0639] Step 5:

[0640] The server determines whether any signs of failure have been detected from the analysis results, and if so, sends that information to the terminal.

[0641] Step 6:

[0642] The terminal receives the notification of the malfunction signs sent from the server. The terminal notifies the driver of this information. Specifically, the terminal warns the driver of the existence of the malfunction signs using a voice message or a display.

[0643] Step 7:

[0644] The device's emotion engine analyzes the user's (driver's) voice tone, facial expressions, body movements, etc. to recognize the user's emotional state in real time. The recognized emotion data is sent from the device to the server.

[0645] Step 8:

[0646] The server analyzes the emotional data sent from the emotion engine and determines whether the driver is in a stressed or relaxed state.

[0647] Step 9:

[0648] The server selects the appropriate notification method based on the malfunction signs and emotional data. For example, if it determines that the driver is under stress, it will provide gentle voice notifications or relaxation support.

[0649] Step 10:

[0650] The device notifies the driver based on instructions from the server, including voice messages, displaying information on the screen, and even playing relaxation music.

[0651] Step 11:

[0652] When a malfunction symptom is detected, the server searches for the nearest repair shop based on the vehicle's current location, and generates a list of multiple repair shops as a search result.

[0653] Step 12:

[0654] The server sends the generated repair shop list to the terminal, which displays the list to the driver, allowing the driver to select the desired repair shop.

[0655] Step 13:

[0656] The driver (user) selects the desired repair shop from a list of repair shops via the terminal, and the selected information is sent to the server.

[0657] Step 14:

[0658] The server automatically processes reservations with the selected repair shop, and the reservation includes detailed information such as the vehicle's condition and the driver's desired date and time.

[0659] Step 15:

[0660] The server notifies the terminal that the reservation procedure is complete, and the terminal notifies the driver that the reservation is complete, providing support to ensure the vehicle repair is carried out smoothly.

[0661] Example 2

[0662] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0663] Conventional vehicle operation assistance systems have the ability to detect and notify drivers of signs of malfunction based on sensor data, but they lack the ability to provide appropriate responses and notifications that take into account the user's emotional state. This can result in inadequate timing for notifications sent to stressed drivers. Furthermore, functions for searching for the nearest repair facility based on signs of malfunction and making reservations are not fully integrated, placing a significant burden on the driver. These issues need to be resolved.

[0664] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0665] In this invention, the server includes means for receiving sensor data collected from the vehicle, data analysis means including a generative model that analyzes the received sensor data in real time and detects signs of failure, emotion recognition means for recognizing the user's emotional state and collecting emotion data, notification means for notifying the driver based on the detected signs of failure and emotion data, repair facility search means for searching for the nearest repair facility when a failure is predicted, and reservation procedure means for making a reservation at the searched repair facility. This makes it possible to quickly detect and notify signs of failure and quickly take appropriate repair measures while taking the driver's emotional state into consideration.

[0666] A "vehicle" is a means of transportation used as a means of transportation, and is a general term for a vehicle that mainly includes an engine, a braking system, etc.

[0667] "Sensor data" refers to data collected from various sensors attached to a vehicle, and includes information on engine status, braking system, tire pressure, temperature, etc.

[0668] "Means for receiving" refers to a function or device that acquires data sent from outside.

[0669] A "generative model" is a model that uses machine learning algorithms to analyze data and detect abnormal patterns and signs of failure.

[0670] "Data analysis means" refers to the functions and technologies for analyzing received data and extracting meaningful information.

[0671] "Emotion recognition means" refers to devices or technologies that analyze and recognize a user's emotional state in real time from their voice tone, facial expressions, body movements, etc.

[0672] "Notification means" refers to a function or device for notifying the user of analysis results and important information.

[0673] A "repair facility" is a location that provides equipment and services for repairing vehicle breakdowns and maintenance.

[0674] "Repair facility search means" refers to a function or technology for searching for the nearest repair facility based on information such as the vehicle's current location.

[0675] "Reservation procedure means" refers to a function or system for automatically making reservations for repairs or maintenance at a repair facility.

[0676] This invention is a system that detects signs of malfunction and notifies the driver based on sensor data collected from the vehicle and user emotional data. The system aims to improve driver safety and comfort by analyzing sensor data in real time and providing appropriate notifications and responses.

[0677] Hardware Configuration

[0678] The device is installed inside the vehicle and collects data from multiple sensors (engine condition sensors, brake system sensors, tire pressure sensors, temperature sensors, etc.) It is also equipped with a camera and microphone for emotion recognition, which capture the user's voice tone, facial expressions, and body movements.

[0679] The server is a central system for receiving and analyzing collected sensor data and emotion data in real time. It is built on a computer equipped with a high-performance processor and large memory capacity. To perform data analysis, a generative AI model using a machine learning algorithm is implemented. For example, machine learning frameworks such as TensorFlow and PyTorch are used.

[0680] Software Configuration

[0681] The device contains software to collect sensor data and compile it into a single data packet. It also contains an emotion engine to analyze the user's emotional data. The emotion engine has algorithms that determine the user's emotional state, such as stress or relief, based on the user's tone of voice, facial expressions, and body movements.

[0682] The server is equipped with a generative AI model for analyzing the received data. This AI model has the ability to detect signs of malfunction from the received sensor data. It also includes an algorithm for selecting the optimal notification method for the driver based on the emotion recognition results.

[0683] Specific examples

[0684] Suppose an onboard sensor detects an abnormal engine temperature while a vehicle is traveling on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative AI model and detects signs of engine failure. As a result, the server sends information about the signs of failure to the device, and the device notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[0685] Meanwhile, the emotion engine analyzes the driver's facial expressions and voice tone to determine if the driver is under stress. Based on this information, the server instructs the device to issue a gentle voice notification to the driver. The server then searches for the nearest repair shop based on the vehicle's current location and sends a list of multiple repair shops to the device. Once the driver selects the desired repair shop, the server automatically completes the reservation procedure with that repair shop.

[0686] Prompt Sentence Examples

[0687] For example, the following prompt statements can be used to specify system behavior:

[0688] "What is the procedure if the onboard sensors detect an abnormal engine temperature while driving on the highway? Also, what kind of notification or support will be provided if the driver feels stressed at that time?"

[0689] In this way, the system of the present invention significantly contributes to extending the life of a vehicle and reducing maintenance costs by analyzing various sensor data of the vehicle in real time and quickly detecting and notifying signs of malfunction. In addition, by providing notifications that take the user's emotional state into consideration, it realizes a more user-friendly system.

[0690] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0691] Step 1: Collect sensor data

[0692] The terminal collects data from sensors installed inside the vehicle, such as those for the engine condition, braking system, tire pressure, and temperature. The collected data is temporarily stored in the terminal's memory. The input is data from each sensor, and the output is compiled into a single data packet. Specifically, when the tire pressure sensor detects pressure below a specified value, it records the data in memory.

[0693] Step 2: Sending sensor data

[0694] The terminal aggregates the collected sensor data into a single packet and transmits it in encrypted form to the server in real time. The input is the aggregated data packet, and the output is a transmission success message to the server. Specifically, the terminal aggregates engine status data, brake system data, tire pressure data, and temperature data, encrypts it, and transmits it to the server.

[0695] Step 3: Receiving the data

[0696] The server receives sensor data sent from the device. The input is an encrypted data packet, and the output is decrypted sensor data. Specifically, the server receives the data packet, decrypts it, and obtains the data of each sensor.

[0697] Step 4: Data analysis and failure detection

[0698] The server analyzes the received data in real time using a generative AI model to detect abnormal patterns and signs of failure. The input is decoded sensor data, and the output is the presence or absence of signs of failure and detailed data. Specifically, if the engine temperature exceeds the normal range, it is determined to be a sign of engine failure.

[0699] Step 5: Collecting emotion data

[0700] The device analyzes the user's voice tone, facial expressions, and body movements to recognize their emotional state in real time. The input is the user's voice and video data, and the output is analyzed emotional data. Specifically, the camera and microphone capture the user's facial expressions and voice, which are then analyzed by the emotion engine.

[0701] Step 6: Sending Emotion Data

[0702] The device sends the recognized emotion data to the server. The input is the analyzed emotion data, and the output is a transmission success message to the server. Specifically, the device sends data indicating that the user is in a stressed state to the server.

[0703] Step 7: Decide on notifications

[0704] The server determines the optimal notification method based on the received sensor data and emotional data. The input is failure prediction data and emotional data, and the output is the notification content and method. Specifically, if an engine abnormality is detected and the user is in a stressful state, a gentle voice notification will be selected.

[0705] Step 8: Notify drivers

[0706] The device follows instructions from the server and notifies the driver of signs of malfunction and their emotions. The input is the content and method of the notification, and the output is the notification to the driver. Specifically, it notifies the driver by voice, saying, "An abnormality has been detected in the engine. Immediate repair is required."

[0707] Step 9: Find a repair facility

[0708] The server searches for the nearest repair facility based on the vehicle's current location. The input is the vehicle's location data, and the output is a list of repair facilities. Specifically, it uses GPS data to search for nearby repair facilities and generates the list.

[0709] Step 10: Select and book a repair facility

[0710] The user selects the desired repair facility from a list of repair facilities provided by the terminal. The information of the selected repair facility is sent to the server, and the reservation procedure is carried out automatically. The input is the user's selection information, and the output is a reservation confirmation message. Specifically, the user taps the desired repair facility, and the server contacts that facility to complete the reservation procedure.

[0711] (Application example 2)

[0712] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0713] Modern cars are equipped with many sensors, but there are only a limited number of systems that can analyze this sensor data in real time and predict potential breakdowns in advance. Furthermore, there are no systems that recognize the driver's emotional state and immediately take appropriate action based on that. This creates a problem: drivers tend to feel stressed when they encounter a breakdown, making it difficult to quickly repair the vehicle.

[0714] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving sensor data collected from the vehicle, data analysis means including a generative model that analyzes the received sensor data in real time and detects signs of failure, emotion recognition means that recognizes the user's emotional state and incorporates that data into the analysis results, notification means that selects an appropriate notification method based on the detected signs of failure and the user's emotional state and notifies the driver, repair facility search means that, when a failure is predicted, searches for the nearest repair facility based on the vehicle location, and reservation procedure means that makes a reservation for the searched repair facility. This enables rapid detection of signs of failure, appropriate notification according to the driver's emotions, and immediate repair arrangements.

[0715] "Sensor data" refers to data collected from multiple sensors installed in a vehicle, and includes information on engine status, braking system, tire pressure, temperature, etc.

[0716] "Real-time analysis" refers to analyzing sensor data immediately as it is collected.

[0717] "Failure signs" are data based on collected sensor data that indicate signs of impending failure in various parts of the vehicle.

[0718] A "generative model" is a model that uses machine learning algorithms to analyze sensor data and detect abnormal patterns and signs of failure.

[0719] The "data analysis means" is a device or software for analyzing received sensor data and detecting signs of failure.

[0720] The "emotion recognition means" is a device or software that analyzes the user's tone of voice, facial expressions, body movements, etc., and recognizes the user's emotional state in real time.

[0721] The "notification means" is a device or software that selects an appropriate notification method based on the detected signs of failure and the user's emotional state, and notifies the driver.

[0722] The "repair facility search means" is a device or software for searching for the nearest repair facility based on the current location of the vehicle when a failure symptom is detected.

[0723] The "reservation procedure means" is a device or software for automatically carrying out reservation procedures with the searched repair facility.

[0724] The embodiments of the present invention will be specifically described below.

[0725] Sensor data collection and transmission

[0726] The server collects sensor data from multiple sensors installed in the vehicle, including information on engine status, braking system, tire pressure, temperature, etc. The collected data is compiled into a single data packet on the device and then transmitted to the server in real time.

[0727] Data analysis and failure prediction

[0728] The server receives sensor data sent from the device and analyzes it in real time using a generative model. This generative model uses machine learning algorithms to detect abnormal patterns and signs of malfunction. As a result of the analysis, it determines whether there are signs of a malfunction in the vehicle.

[0729] Recognizing user emotions with an emotion engine

[0730] The server uses the smartphone's camera and microphone to analyze the driver's facial expressions and tone of voice to recognize their emotional state in real time. This emotional data is then added to the analysis results and sent to the server.

[0731] Driver notification and adaptation

[0732] The server selects the appropriate notification method based on the malfunction signs and emotion recognition data. For example, if the server determines that the driver is under stress, it instructs the device to provide gentle voice notifications or relaxation support. Based on this, the device gently notifies the driver of the presence of malfunction signs.

[0733] Find and book a repair shop

[0734] If a malfunction is detected, the server searches for the nearest repair facility based on the vehicle's current location. A list of multiple repair facilities is generated as a result of the search and provided to the driver via the terminal. The driver can then select the desired repair facility from the provided list.

[0735] Reservation procedure

[0736] The server automatically schedules a reservation with the selected repair facility, including details such as the vehicle's condition and the driver's preferred date and time, ensuring fast and timely repairs.

[0737] Hardware and software used

[0738] Hardware: Sensors in the vehicle (engine sensor, tire pressure sensor, temperature sensor), smartphone (camera, microphone)

[0739] Software: Server-based machine learning algorithms, generative models, emotion recognition engines, cloud servers, notification systems

[0740] Specific examples

[0741] For example, suppose an onboard sensor detects an abnormal engine temperature while a vehicle is driving on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative model to detect signs of engine failure. Based on the analysis results and the driver's emotion recognition data, the server instructs the device to send an appropriate notification. The device notifies the driver, "An abnormality has been detected in the engine. Please remain calm and take action." The server then searches for the nearest repair facility based on the vehicle's current location and sends a list of multiple repair facilities to the device. When the driver selects the desired repair facility, the server automatically processes the reservation at that facility.

[0742] Prompt Sentence Examples

[0743] Failure prediction prompt

[0744] Prompt: Detect engine pre-failure based on data sent from sensors. Example: Engine temperature is outside normal range.

[0745] Data: {"Engine Status": "Abnormal", "Brake System": "Normal", "Tire Pressure": 32, "Temperature": 110}

[0746] Result: Premature failure. Engine requires immediate repair.

[0747] Driver emotion recognition prompts

[0748] Prompt: Based on camera and microphone data, determine if the driver is stressed. Example: Talking with a stern face.

[0749] Data: {"Facial Expression": "Stern", "Voice Tone": "High-pitched"}

[0750] Result: The driver is stressed. Use a gentle tone to notify them.

[0751] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0752] Step 1:

[0753] The terminal collects sensor data from multiple sensors installed in the vehicle, such as engine status, braking system, tire pressure, and temperature, and compiles it into a data packet. The input is data from the sensors, and the output is a data packet.

[0754] Step 2:

[0755] The terminal sends the collected data packets to a server in real time. The server inputs the received sensor data into an analysis platform. Specifically, a communication protocol such as an HTTP request is used. The input is the data packets, and the output is the data sent to the server.

[0756] Step 3:

[0757] The server analyzes the received sensor data in real time using a generative model. This uses machine learning algorithms to detect abnormal patterns and signs of failure. The input is the sensor data, and the output is the analysis results.

[0758] Step 4:

[0759] The server recognizes the driver's emotional state using data provided by the smartphone's camera and microphone. Specifically, it uses image processing and voice analysis technology (e.g., OpenCV, voice analysis software). The input is data from the camera and microphone, and the output is emotional state data.

[0760] Step 5:

[0761] The server selects an appropriate notification method based on the failure signs and emotion recognition data and sends instructions to the device. For example, if the driver is in a stressful state, it will instruct a gentle voice notification. The input is the analysis results and emotional state data, and the output is notification instruction data.

[0762] Step 6:

[0763] The terminal notifies the driver based on instructions from the server. For example, it may notify the driver that "An abnormality has been detected in the engine. Please remain calm and take appropriate action." The input is notification instruction data, and the output is a notification to the user.

[0764] Step 7:

[0765] When a malfunction symptom is detected, the server searches for the nearest repair facility based on the vehicle's current location information. Here, GPS information and a database of repair facilities are used. The input is location information and malfunction symptom data, and the output is a list of repair facilities.

[0766] Step 8:

[0767] The terminal displays the list of repair facilities sent from the server to the driver and lets the driver select the desired repair facility. The input is the list of repair facilities and the output is the user's selection.

[0768] Step 9:

[0769] The server automatically processes the reservation with the selected repair facility. Specifically, it accesses the repair facility's reservation system and communicates the vehicle's status and the driver's desired date and time. The input is the user's selection and vehicle information, and the output is reservation confirmation.

[0770] Step 10:

[0771] The server sends the reservation confirmation information to the terminal, and the terminal notifies the driver. The input is the reservation confirmation data, and the output is the reservation confirmation notification to the user.

[0772] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0773] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0774] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0775] [Third embodiment]

[0776] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0777] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0778] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0779] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0780] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0781] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0782] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0783] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0784] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0785] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0786] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0787] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0788] The embodiments of the present invention will be specifically described below.

[0789] Sensor data collection and transmission

[0790] The device collects data from multiple sensors installed in the vehicle. This sensor data includes various vehicle information such as engine status, braking system, tire pressure, and temperature. The collected data is compiled into a single data packet within the device. The device then transmits this data packet to a server in real time.

[0791] Data analysis and failure prediction

[0792] The server receives sensor data sent from the device. The received data is analyzed in real time using a generative model. This generative model can detect abnormal patterns and signs of faults using machine learning algorithms. The server uses the analysis results to determine whether there are signs of a fault in the vehicle.

[0793] Driver Notification

[0794] When a malfunction sign is detected, the server sends the information to the terminal in real time. The terminal then notifies the driver of the malfunction sign. The notification method can take the form of a voice message or a display to warn the driver of the presence of a malfunction sign.

[0795] Find and book a repair shop

[0796] If a malfunction is detected, the server searches for the nearest repair shop based on the vehicle's current location. The search results generate a list of multiple repair shops. This list is provided to the driver via their terminal. The driver can then select the repair shop of their choice from the provided list.

[0797] Once the selection is complete, the server automatically schedules a reservation with the selected repair shop, including details such as the vehicle's condition and the driver's preferred date and time, ensuring a fast and timely repair.

[0798] Specific examples

[0799] For example, suppose an onboard sensor detects an abnormal engine temperature while a vehicle is traveling on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative model and detects signs of engine failure. As a result, the server sends information about the signs of failure to the device, and the device notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[0800] The server then searches for the nearest repair shop based on the vehicle's current location and sends a list of several repair shops to the terminal. Once the driver selects the desired repair shop, the server proceeds with the reservation procedure at that repair shop. This series of steps allows the driver to quickly discover the problem and take appropriate action.

[0801] This system analyzes various vehicle sensor data in real time and quickly detects and notifies of signs of malfunction, thereby significantly contributing to extending the lifespan of vehicles and reducing maintenance costs.

[0802] The processing flow will be explained below.

[0803] Step 1:

[0804] The device collects various data from sensors installed inside the vehicle, including information on engine status, braking system, tire pressure, temperature, etc.

[0805] Step 2:

[0806] The device collects sensor data and assembles it into a single data packet, which contains information from various sensors.

[0807] Step 3:

[0808] The device sends the bundled data packets to the server in real time using HTTP POST requests.

[0809] Step 4:

[0810] The server receives the sensor data sent from the device and temporarily stores the data on the server side.

[0811] Step 5:

[0812] The server analyzes the received sensor data using a generative model, which uses machine learning algorithms to detect abnormal patterns and signs of failure.

[0813] Step 6:

[0814] The server determines whether any signs of failure have been detected from the analysis results, and if so, sends that information to the terminal.

[0815] Step 7:

[0816] The terminal receives the notification of the malfunction sign sent from the server and notifies the driver of this information using means such as a voice message or a display.

[0817] Step 8:

[0818] When a malfunction symptom is detected, the server searches for the nearest repair shop based on the current location of the vehicle, and generates a list of repair shops.

[0819] Step 9:

[0820] The server sends the generated list of repair shops to the terminal, which displays the list to the driver and prompts them to make a selection.

[0821] Step 10:

[0822] The user (driver) selects the desired repair shop from a list of repair shops via the terminal, and the selected information is sent to the server.

[0823] Step 11:

[0824] The server automatically processes reservations with the selected repair shops, including details such as the vehicle's condition and the driver's desired date and time.

[0825] Step 12:

[0826] The server notifies the terminal that the reservation procedure is complete, and the terminal notifies the driver that the reservation is complete, providing support to ensure the vehicle repair is carried out smoothly.

[0827] Example 1

[0828] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0829] Conventional vehicle management systems lacked an integrated means for collecting and analyzing sensor data, making it difficult to quickly identify and notify signs of malfunction. Furthermore, they lacked the functionality to automatically search for and reserve the most suitable repair shop when signs of malfunction were detected, making it difficult to perform fast and efficient maintenance. This created the challenge of preventing vehicle breakdowns and reducing maintenance costs.

[0830] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0831] In this invention, the server includes means for receiving sensor data collected from the vehicle, data analysis means for analyzing the received sensor data in real time and including a generative model for detecting signs of failure, notification means for notifying the user of the detected signs of failure, repair facility search means for searching for the nearest repair facility when a failure is predicted, and reservation procedure means for making a reservation at the searched repair facility. This makes it possible to analyze the sensor data in real time and quickly detect and notify signs of failure, thereby significantly improving the efficiency and effectiveness of vehicle maintenance.

[0832] "Vehicle" refers to a moving body such as an automobile or motorcycle that travels on the ground.

[0833] "Sensor data" refers to information collected from various sensors installed in a vehicle, and includes data on the power plant status, braking system, tire pressure, temperature, etc.

[0834] "Means for receiving" refers to communication equipment or programs for obtaining sensor data from other devices or systems.

[0835] "Data analysis means" refers to an analysis device or program that processes received sensor data and includes a generative model for detecting abnormalities or signs of failure.

[0836] A "generative model" refers to a data model that has been trained using machine learning algorithms to make predictions or analyses.

[0837] "Notification means" refers to devices or programs such as voice messages or display screens that notify the user of detected signs of a malfunction.

[0838] "Repair facility search means" refers to a device or program for searching for the nearest repair facility based on the current location of the vehicle when a malfunction symptom is detected.

[0839] "Reservation processing means" refers to a device or program for automatically making a reservation at a searched repair facility.

[0840] "User" means the person who drives or owns a Vehicle.

[0841] "Real-time" refers to data collection and processing occurring immediately, without delay.

[0842] The present invention is a system that uses sensor data collected from a vehicle to detect signs of a malfunction, notifies the user, and searches for the nearest repair facility and completes the reservation procedure. This system is configured and operates as follows.

[0843] Sensor data collection and transmission

[0844] The terminal collects data from multiple sensors installed in the vehicle. These sensors include sensors for measuring the status of the powertrain, braking system, tire pressure, and temperature. The collected sensor data is compiled into a single data packet within the terminal. For example, data such as 90 degrees from the engine temperature sensor and 2.5 bar from the tire pressure sensor is collected.

[0845] The device then transmits the data packets to the server in real time over Wi-Fi or 4G / 5G networks, minimizing data latency.

[0846] Data analysis and failure prediction

[0847] The server receives sensor data sent from the device. The received data is analyzed in real time using a generative AI model using data analysis software such as TensorFlow or PyTorch. During the analysis, machine learning algorithms are used to detect abnormal patterns and signs of malfunction. For example, if the engine temperature is outside the normal range (above 90 degrees), it is recognized as a sign of malfunction.

[0848] Driver Notification

[0849] If a malfunction sign is detected, the server sends that information to the terminal. The terminal then notifies the driver of the received malfunction sign notification. This notification method can include a voice message or a display. Specifically, the message displayed will say, "An abnormality has been detected in the engine. Immediate repair is required."

[0850] Find and book a repair shop

[0851] If a malfunction is detected, the server searches for the nearest repair facility based on the vehicle's current location. The server references an online database and generates a list of multiple repair facilities. This list is provided to the user via their terminal. The user can select the desired repair facility from the provided list. Once the selection is complete, the server automatically completes the reservation procedure at the selected repair facility. The reservation procedure includes detailed information such as the vehicle's condition and the user's desired date and time.

[0852] Specific examples

[0853] For example, suppose a sensor detects an abnormal engine temperature while a vehicle is driving on a highway. This data is sent by the device to a server, which then analyzes it using a generative AI model. As a result of the analysis, signs of engine failure are detected. The server sends this information to the device, which then notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[0854] The server then searches for the nearest repair facility based on the vehicle's current location and sends a list of several repair facilities to the terminal. Once the driver selects the desired repair facility, the server proceeds with the reservation procedure at that repair facility. This series of steps allows the driver to quickly discover the malfunction and take appropriate action.

[0855] Prompt Sentence Examples

[0856] "We have acquired abnormal engine temperature data. Please analyze this data to detect signs of failure."

[0857] "Find the nearest repair shops based on your vehicle's current location and generate a list."

[0858] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0859] System program processing flow

[0860] Step 1: Collect sensor data

[0861] Specific details:

[0862] The device collects data from multiple sensors installed in the vehicle, including sensors to measure the status of the powerplant, braking system, tire pressure, and temperature.

[0863] Input and Output:

[0864] The input is raw data from each sensor, and the output is a data packet that aggregates this data.

[0865] Specific behavior:

[0866] For example, it collects data such as 90 degrees from the engine temperature sensor and 2.5 bar from the tire pressure sensor.

[0867] Step 2: Generate data packets

[0868] Specific details:

[0869] Collected sensor data is compiled into a single data packet.

[0870] Input and Output:

[0871] The input is the data collected from each sensor in step 1. The output is a data packet that integrates this data.

[0872] Specific behavior:

[0873] It generates a data packet containing information such as engine temperature at 90 degrees and tire pressure at 2.5 bar.

[0874] Step 3: Sending data packets

[0875] Specific details:

[0876] The device transmits the generated data packets to the server in real time using Wi-Fi or 4G / 5G networks.

[0877] Input and Output:

[0878] The input is the data packet generated in step 2. The output is the data packet sent to the server.

[0879] Specific behavior:

[0880] The data packets are sent to a server over a communication network (e.g., 4G).

[0881] Step 4: Receiving the data

[0882] Specific details:

[0883] The server receives the data packets sent from the terminal.

[0884] Input and Output:

[0885] The input is the data packets sent from the device, and the output is the sensor data stored on the server.

[0886] Specific behavior:

[0887] The server checks and stores the received data.

[0888] Step 5: Data analysis

[0889] Specific details:

[0890] Based on the received data, real-time analysis is performed using a generative AI model.

[0891] Input and Output:

[0892] The input is sensor data stored on a server, and the output is the analysis results, which include signs of failure.

[0893] Specific behavior:

[0894] For example, TensorFlow analyzes that the engine temperature is abnormal (above 90 degrees).

[0895] Step 6: Pre-fault detection

[0896] Specific details:

[0897] The server detects abnormal patterns and signs of failure based on the analysis results.

[0898] Input and Output:

[0899] The input is the analysis result from the generative AI model, and the output is sensor data that indicates a potential failure.

[0900] Specific behavior:

[0901] Since the engine temperature is 90 degrees, this is detected as a sign of a malfunction.

[0902] Step 7: Sending notifications

[0903] Specific details:

[0904] If a fault sign is detected, the server transmits the information to the terminal.

[0905] Input and Output:

[0906] The input is the sensor data that indicates a failure. The output is the notification data sent to the terminal.

[0907] Specific behavior:

[0908] A message is sent to the terminal saying, "An abnormality has been detected in the engine. Immediate repair is required."

[0909] Step 8: Notify users

[0910] Specific details:

[0911] The terminal notifies the user of the received failure warning by means of a voice message or a display.

[0912] Input and Output:

[0913] The input is the notification data sent by the server. The output is the notification the user receives.

[0914] Specific behavior:

[0915] For example, a message such as "An abnormality has been detected in the engine. Immediate repair is required" may be displayed on the screen.

[0916] Step 9: Find a repair facility

[0917] Specific details:

[0918] The server searches for the nearest repair facility based on the vehicle's current location.

[0919] Input and Output:

[0920] The input is the vehicle's current location data. The output is a list of repair facilities.

[0921] Specific behavior:

[0922] For example, GPS data can be used to search an internet database for the nearest repair facility from your current location.

[0923] Step 10: Submit a list of repair facilities

[0924] Specific details:

[0925] The server sends the generated list of repair facilities to the terminal.

[0926] Input and Output:

[0927] The input is the list of repair facilities found. The output is the list sent to the terminal.

[0928] Specific behavior:

[0929] A number of potential repair facilities are sent to the device.

[0930] Step 11: User Selects Repair Facility

[0931] Specific details:

[0932] The terminal displays a list of repair facilities to the user, and the user selects the desired repair facility.

[0933] Input and Output:

[0934] The input is a list of repair facilities. The output is the user's selected repair facility.

[0935] Specific behavior:

[0936] The user selects the desired repair facility from the device screen.

[0937] Step 12: Complete the booking process

[0938] Specific details:

[0939] The server automatically processes reservations with the selected repair facility, including detailed information such as the vehicle's status and the user's desired date and time.

[0940] Input and Output:

[0941] The input is the repair facility selected by the user and the fault information. The output is a reservation completion notification to the repair facility.

[0942] Specific behavior:

[0943] A reservation message is sent to the repair facility stating, "Please repair the engine abnormality on XX date at XX time."

[0944] (Application example 1)

[0945] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0946] Conventional systems lacked the means to effectively collect various vehicle sensor data and analyze and notify abnormalities and signs of failure in real time. Furthermore, when a failure was predicted, the process of searching for and booking a repair shop was cumbersome for drivers, making it difficult to respond quickly. This meant that the system was unable to fully extend the lifespan of vehicles or reduce maintenance costs.

[0947] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0948] In this invention, the server includes: means for receiving sensor data collected from the vehicle; data analysis means including a generative model that analyzes the received sensor data in real time and detects signs of failure; means for notifying the driver of the detected signs of failure; repair shop search means for searching for the nearest repair shop when a failure is predicted; reservation processing means for making a reservation at the searched repair shop; data transmission means for collecting various sensor data from the vehicle via a smartphone and transmitting it to a cloud server; and means for analyzing the sensor data on the cloud server, detecting signs of failure or abnormalities, and notifying the driver in real time. This makes it possible to quickly collect and analyze various sensor data from the vehicle and notify the driver of signs of failure in real time. Furthermore, when a failure is predicted, the server searches for the nearest repair shop and automatically processes the reservation, thereby speeding up maintenance response and reducing the driver's hassle.

[0949] A "vehicle" is a moving body, including an automobile, and is a means of transportation intended to travel on roads.

[0950] "Sensor data" refers to data collected by various sensors installed in a vehicle, including engine status, braking system, tire pressure, temperature, etc.

[0951] "Means for receiving" refers to the means for inputting sensor data from the vehicle to a server or terminal.

[0952] A "generative model" is a model trained using machine learning algorithms to make predictions or classifications based on input data.

[0953] "Data analysis means" refers to a means for analyzing received sensor data in real time and detecting abnormalities or signs of failure.

[0954] The "notification means" is a means for notifying the driver of signs of malfunction or abnormalities based on the analysis results.

[0955] The "repair shop search means" is a means for searching for the nearest repair shop based on the vehicle's location information when a breakdown is expected.

[0956] The "reservation procedure means" is a means for automatically making a reservation to visit the searched repair shop.

[0957] A "smartphone" is a multi-functional mobile phone terminal that has mobile communication functions and Internet connection functions.

[0958] A "cloud server" is a server accessed via the Internet that provides computing resources for storing and analyzing large amounts of data.

[0959] The "data transmission means" is a means for transmitting collected sensor data from the smartphone in the vehicle to the cloud server.

[0960] "Real-time" refers to processing occurring immediately without delay.

[0961] "Abnormal" refers to a condition of a vehicle or its parts that is different from its normal operating condition.

[0962] This invention is a system that collects various sensor data from a vehicle, analyzes it in real time, and detects signs of failure, and is particularly applicable to autonomous vehicles.

[0963] First, the vehicle is equipped with multiple sensors that measure the engine, braking system, tire pressure, temperature, etc. These sensor data are collected via a smartphone installed in the vehicle, which then compiles the data into a single data packet and sends it to a cloud server.

[0964] A generative model trained using machine learning algorithms is installed on the cloud server. This generative model analyzes the received sensor data in real time and detects abnormal patterns and signs of malfunction, such as when the engine temperature is higher than normal or when tire pressure drops suddenly.

[0965] Depending on the analysis results, the server will notify the smartphone of any detected signs of malfunction. Notification methods include voice messages and display on the screen. For example, the server will warn the driver by saying, "An abnormality has been detected in the engine. Immediate repair is required."

[0966] The server then searches for the nearest repair shop based on the vehicle's current location. The search results are sent to the smartphone as a list of multiple repair shops. When the driver selects the desired repair shop, the server automatically makes a reservation with that shop. The reservation procedure includes detailed information such as the vehicle's status and the driver's desired date and time.

[0967] The hardware used includes various sensors in the vehicle, smartphones, and cloud servers. The software includes a data collection application running on the smartphone and a data analysis program on the cloud server. In particular, the machine learning models used on the cloud server use libraries such as scikit-learn and TensorFlow.

[0968] As a concrete example, consider a case where an abnormal engine temperature is detected while driving on a highway. This data is sent to a cloud server via a smartphone and analyzed by a generative model. If a sign of engine failure is detected as a result, a notification is displayed on the smartphone stating, "There is an abnormality in the engine. Repairs are required immediately." The server then searches for the nearest repair shop and sends a list of several shops to the smartphone. When the driver selects the desired repair shop, the server automatically processes the reservation procedure at that shop.

[0969] An example of a prompt is:

[0970] "GPS location: {lat}, {lng}. Engine temperature: {engine_temp} degrees. Brake status: {brake_status}. Tire pressure: {tire_pressure} psi. If an abnormality is detected, we will notify you of the nearest repair shop and start the appointment process."

[0971] Based on this prompt, the system analyzes data collected in real time and automates fault prediction and repair shop searches.

[0972] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0973] Step 1:

[0974] The smartphone collects data from various vehicle sensors (engine, brake system, tire pressure, temperature, etc.). The sensor data is input, and the smartphone compiles the collected data into a single data packet. This data packet is output. Specifically, the smartphone converts the voltage and signals from the sensors into digital data and packets it in the appropriate format.

[0975] Step 2:

[0976] The smartphone sends a data packet to the cloud server. The input is the data packet generated in step 1, and the output is the data packet sent over the Internet. Specifically, the smartphone sends an HTTP request to the specified server URL using Wi-Fi or the mobile network.

[0977] Step 3:

[0978] The cloud server analyzes the received data packets. The input is the data packets and the output is the analysis results. A generative AI model is used for this analysis to detect anomalies and signs of failure. Specifically, a Python program installed on the cloud server analyzes the data packets and uses machine learning algorithms to perform pattern matching and predictions.

[0979] Step 4:

[0980] If the cloud server detects a sign of a failure based on the analysis results, it sends a notification to the smartphone. The input is the analysis results and the output is a notification message. Specifically, the cloud server generates a message based on the analysis results and sends a push notification to the smartphone using a service such as Firebase Cloud Messaging.

[0981] Step 5:

[0982] The smartphone receives the notification message and displays a warning to the driver. The input is the notification message, and the output is a voice message or a warning displayed on the screen. Specifically, the smartphone application is triggered by the notification and displays a message on the user interface.

[0983] Step 6:

[0984] The cloud server searches for the nearest repair shop based on the vehicle's current location. The input is the vehicle's GPS data, and the output is a list of repair shops. Specifically, the cloud server uses a map service such as Google Maps API to obtain the location information of repair shops based on the current location.

[0985] Step 7:

[0986] A list of repair shops is sent to the smartphone, and the driver selects one. The input is the list of repair shops, and the output is the repair shop selected by the driver. Specifically, the smartphone application displays a repair shop selection interface and provides operations for the driver to select.

[0987] Step 8:

[0988] The cloud server then completes the reservation procedure with the selected repair shop. The input is the repair shop selected by the driver and the vehicle status data, and the output is a reservation confirmation message. Specifically, the cloud server sends an HTTP request to the repair shop's reservation system to complete the reservation procedure.

[0989] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0990] The embodiments of the present invention will be specifically described below.

[0991] Sensor data collection and transmission

[0992] The device collects data from multiple sensors installed in the vehicle, including information on engine status, braking system, tire pressure, temperature, etc. The collected data is compiled into a single data packet within the device and then transmitted to a server in real time.

[0993] Data analysis and failure prediction

[0994] The server receives sensor data sent from the device. The received data is analyzed in real time using a generative model. This generative model can detect abnormal patterns and signs of faults using machine learning algorithms. The server uses the analysis results to determine whether there are signs of a fault in the vehicle.

[0995] Recognizing user emotions with an emotion engine

[0996] The device is equipped with an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's tone of voice, facial expressions, body movements, etc., and can recognize the user's emotional state in real time. The emotion engine sends the recognized emotion data to the server, which then incorporates this data into the analysis results.

[0997] Driver notification and adaptation

[0998] The server selects an appropriate notification method based on the signs of a malfunction and the emotional data obtained from the emotion engine. For example, if the server determines that the driver is under stress, it instructs the server to provide gentle voice notifications and relaxation support. The notification is sent via the device, gently informing the driver of the presence of signs of a malfunction.

[0999] Find and book a repair shop

[1000] If a malfunction is detected, the server searches for the nearest repair shop based on the vehicle's current location. As a result of the search, a list of multiple repair shops is generated and provided to the driver via the terminal. The driver can then select the desired repair shop from the provided list.

[1001] Reservation procedure

[1002] The server automatically schedules a reservation with the selected repair shop, including details such as the vehicle's condition and the driver's preferred date and time, ensuring fast and timely repairs.

[1003] Specific examples

[1004] For example, suppose an onboard sensor detects an abnormal engine temperature while a vehicle is traveling on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative model and detects signs of engine failure. As a result, the server sends information about the signs of failure to the device, and the device notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[1005] Meanwhile, the emotion engine analyzes the driver's facial expressions and voice tone to determine that the driver is under stress. Based on this information, the server instructs the device to issue a gentle voice notification to the driver. The server then searches for the nearest repair shop based on the vehicle's current location and sends a list of multiple repair shops to the device. Once the driver selects their preferred repair shop, the server automatically makes a reservation with that shop. This series of steps allows the driver to quickly discover the problem and take appropriate action.

[1006] This system significantly contributes to extending the life of a vehicle and reducing maintenance costs by analyzing various vehicle sensor data in real time and quickly detecting and notifying signs of malfunction. In addition, by combining it with an emotion engine, it provides appropriate notifications according to the driver's emotional state, realizing a more user-friendly system.

[1007] The processing flow will be explained below.

[1008] Step 1:

[1009] The device collects various data from multiple sensors installed inside the vehicle, including information on engine status, braking system, tire pressure, temperature, etc.

[1010] Step 2:

[1011] The device aggregates the collected sensor data into a single data packet, which is then sent to the server in real time.

[1012] Step 3:

[1013] The server receives the sensor data sent from the device and temporarily stores the received data within the server.

[1014] Step 4:

[1015] The server uses the generative model to analyze the incoming sensor data in real time, using machine learning algorithms to detect abnormal patterns and signs of failure.

[1016] Step 5:

[1017] The server determines whether any signs of failure have been detected from the analysis results, and if so, sends that information to the terminal.

[1018] Step 6:

[1019] The terminal receives the notification of the malfunction signs sent from the server. The terminal notifies the driver of this information. Specifically, the terminal warns the driver of the existence of the malfunction signs using a voice message or a display.

[1020] Step 7:

[1021] The device's emotion engine analyzes the user's (driver's) voice tone, facial expressions, body movements, etc. to recognize the user's emotional state in real time. The recognized emotion data is sent from the device to the server.

[1022] Step 8:

[1023] The server analyzes the emotional data sent from the emotion engine and determines whether the driver is in a stressed or relaxed state.

[1024] Step 9:

[1025] The server selects the appropriate notification method based on the malfunction signs and emotional data. For example, if it determines that the driver is under stress, it will provide gentle voice notifications or relaxation support.

[1026] Step 10:

[1027] The device notifies the driver based on instructions from the server, including voice messages, displaying information on the screen, and even playing relaxation music.

[1028] Step 11:

[1029] When a malfunction symptom is detected, the server searches for the nearest repair shop based on the vehicle's current location, and generates a list of multiple repair shops as a search result.

[1030] Step 12:

[1031] The server sends the generated repair shop list to the terminal, which displays the list to the driver, allowing the driver to select the desired repair shop.

[1032] Step 13:

[1033] The driver (user) selects the desired repair shop from a list of repair shops via the terminal, and the selected information is sent to the server.

[1034] Step 14:

[1035] The server automatically processes reservations with the selected repair shop, and the reservation includes detailed information such as the vehicle's condition and the driver's desired date and time.

[1036] Step 15:

[1037] The server notifies the terminal that the reservation procedure is complete, and the terminal notifies the driver that the reservation is complete, providing support to ensure the vehicle repair is carried out smoothly.

[1038] Example 2

[1039] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1040] Conventional vehicle operation assistance systems have the ability to detect and notify drivers of signs of malfunction based on sensor data, but they lack the ability to provide appropriate responses and notifications that take into account the user's emotional state. This can result in inadequate timing for notifications sent to stressed drivers. Furthermore, functions for searching for the nearest repair facility based on signs of malfunction and making reservations are not fully integrated, placing a significant burden on the driver. These issues need to be resolved.

[1041] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1042] In this invention, the server includes means for receiving sensor data collected from the vehicle, data analysis means including a generative model that analyzes the received sensor data in real time and detects signs of failure, emotion recognition means for recognizing the user's emotional state and collecting emotion data, notification means for notifying the driver based on the detected signs of failure and emotion data, repair facility search means for searching for the nearest repair facility when a failure is predicted, and reservation procedure means for making a reservation at the searched repair facility. This makes it possible to quickly detect and notify signs of failure and quickly take appropriate repair measures while taking the driver's emotional state into consideration.

[1043] A "vehicle" is a means of transportation used as a means of transportation, and is a general term for a vehicle that mainly includes an engine, a braking system, etc.

[1044] "Sensor data" refers to data collected from various sensors attached to a vehicle, and includes information on engine status, braking system, tire pressure, temperature, etc.

[1045] "Means for receiving" refers to a function or device that acquires data sent from outside.

[1046] A "generative model" is a model that uses machine learning algorithms to analyze data and detect abnormal patterns and signs of failure.

[1047] "Data analysis means" refers to the functions and technologies for analyzing received data and extracting meaningful information.

[1048] "Emotion recognition means" refers to devices or technologies that analyze and recognize a user's emotional state in real time from their voice tone, facial expressions, body movements, etc.

[1049] "Notification means" refers to a function or device for notifying the user of analysis results and important information.

[1050] A "repair facility" is a location that provides equipment and services for repairing vehicle breakdowns and maintenance.

[1051] "Repair facility search means" refers to a function or technology for searching for the nearest repair facility based on information such as the vehicle's current location.

[1052] "Reservation procedure means" refers to a function or system for automatically making reservations for repairs or maintenance at a repair facility.

[1053] This invention is a system that detects signs of malfunction and notifies the driver based on sensor data collected from the vehicle and user emotional data. The system aims to improve driver safety and comfort by analyzing sensor data in real time and providing appropriate notifications and responses.

[1054] Hardware Configuration

[1055] The device is installed inside the vehicle and collects data from multiple sensors (engine condition sensors, brake system sensors, tire pressure sensors, temperature sensors, etc.) It is also equipped with a camera and microphone for emotion recognition, which capture the user's voice tone, facial expressions, and body movements.

[1056] The server is a central system for receiving and analyzing collected sensor data and emotion data in real time. It is built on a computer equipped with a high-performance processor and large memory capacity. To perform data analysis, a generative AI model using a machine learning algorithm is implemented. For example, machine learning frameworks such as TensorFlow and PyTorch are used.

[1057] Software Configuration

[1058] The device contains software to collect sensor data and compile it into a single data packet. It also contains an emotion engine to analyze the user's emotional data. The emotion engine has algorithms that determine the user's emotional state, such as stress or relief, based on the user's tone of voice, facial expressions, and body movements.

[1059] The server is equipped with a generative AI model for analyzing the received data. This AI model has the ability to detect signs of malfunction from the received sensor data. It also includes an algorithm for selecting the optimal notification method for the driver based on the emotion recognition results.

[1060] Specific examples

[1061] Suppose an onboard sensor detects an abnormal engine temperature while a vehicle is traveling on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative AI model and detects signs of engine failure. As a result, the server sends information about the signs of failure to the device, and the device notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[1062] Meanwhile, the emotion engine analyzes the driver's facial expressions and voice tone to determine if the driver is under stress. Based on this information, the server instructs the device to issue a gentle voice notification to the driver. The server then searches for the nearest repair shop based on the vehicle's current location and sends a list of multiple repair shops to the device. Once the driver selects the desired repair shop, the server automatically completes the reservation procedure with that repair shop.

[1063] Prompt Sentence Examples

[1064] For example, the following prompt statements can be used to specify system behavior:

[1065] "What is the procedure if the onboard sensors detect an abnormal engine temperature while driving on the highway? Also, what kind of notification or support will be provided if the driver feels stressed at that time?"

[1066] In this way, the system of the present invention significantly contributes to extending the life of a vehicle and reducing maintenance costs by analyzing various sensor data of the vehicle in real time and quickly detecting and notifying signs of malfunction. In addition, by providing notifications that take the user's emotional state into consideration, it realizes a more user-friendly system.

[1067] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1068] Step 1: Collect sensor data

[1069] The terminal collects data from sensors installed inside the vehicle, such as those for the engine condition, braking system, tire pressure, and temperature. The collected data is temporarily stored in the terminal's memory. The input is data from each sensor, and the output is compiled into a single data packet. Specifically, when the tire pressure sensor detects pressure below a specified value, it records the data in memory.

[1070] Step 2: Sending sensor data

[1071] The terminal aggregates the collected sensor data into a single packet and transmits it in encrypted form to the server in real time. The input is the aggregated data packet, and the output is a transmission success message to the server. Specifically, the terminal aggregates engine status data, brake system data, tire pressure data, and temperature data, encrypts it, and transmits it to the server.

[1072] Step 3: Receiving the data

[1073] The server receives sensor data sent from the device. The input is an encrypted data packet, and the output is decrypted sensor data. Specifically, the server receives the data packet, decrypts it, and obtains the data of each sensor.

[1074] Step 4: Data analysis and failure detection

[1075] The server analyzes the received data in real time using a generative AI model to detect abnormal patterns and signs of failure. The input is decoded sensor data, and the output is the presence or absence of signs of failure and detailed data. Specifically, if the engine temperature exceeds the normal range, it is determined to be a sign of engine failure.

[1076] Step 5: Collecting emotion data

[1077] The device analyzes the user's voice tone, facial expressions, and body movements to recognize their emotional state in real time. The input is the user's voice and video data, and the output is analyzed emotional data. Specifically, the camera and microphone capture the user's facial expressions and voice, which are then analyzed by the emotion engine.

[1078] Step 6: Sending Emotion Data

[1079] The device sends the recognized emotion data to the server. The input is the analyzed emotion data, and the output is a transmission success message to the server. Specifically, the device sends data indicating that the user is in a stressed state to the server.

[1080] Step 7: Decide on notifications

[1081] The server determines the optimal notification method based on the received sensor data and emotional data. The input is failure prediction data and emotional data, and the output is the notification content and method. Specifically, if an engine abnormality is detected and the user is in a stressful state, a gentle voice notification will be selected.

[1082] Step 8: Notify drivers

[1083] The device follows instructions from the server and notifies the driver of signs of malfunction and their emotions. The input is the content and method of the notification, and the output is the notification to the driver. Specifically, it notifies the driver by voice, saying, "An abnormality has been detected in the engine. Immediate repair is required."

[1084] Step 9: Find a repair facility

[1085] The server searches for the nearest repair facility based on the vehicle's current location. The input is the vehicle's location data, and the output is a list of repair facilities. Specifically, it uses GPS data to search for nearby repair facilities and generates the list.

[1086] Step 10: Select and book a repair facility

[1087] The user selects the desired repair facility from a list of repair facilities provided by the terminal. The information of the selected repair facility is sent to the server, and the reservation procedure is carried out automatically. The input is the user's selection information, and the output is a reservation confirmation message. Specifically, the user taps the desired repair facility, and the server contacts that facility to complete the reservation procedure.

[1088] (Application example 2)

[1089] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1090] Modern cars are equipped with many sensors, but there are only a limited number of systems that can analyze this sensor data in real time and predict potential breakdowns in advance. Furthermore, there are no systems that recognize the driver's emotional state and immediately take appropriate action based on that. This creates a problem: drivers tend to feel stressed when they encounter a breakdown, making it difficult to quickly repair the vehicle.

[1091] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving sensor data collected from the vehicle, data analysis means including a generative model that analyzes the received sensor data in real time and detects signs of failure, emotion recognition means that recognizes the user's emotional state and incorporates that data into the analysis results, notification means that selects an appropriate notification method based on the detected signs of failure and the user's emotional state and notifies the driver, repair facility search means that, when a failure is predicted, searches for the nearest repair facility based on the vehicle location, and reservation procedure means that makes a reservation for the searched repair facility. This enables rapid detection of signs of failure, appropriate notification according to the driver's emotions, and immediate repair arrangements.

[1092] "Sensor data" refers to data collected from multiple sensors installed in a vehicle, and includes information on engine status, braking system, tire pressure, temperature, etc.

[1093] "Real-time analysis" refers to analyzing sensor data immediately as it is collected.

[1094] "Failure signs" are data based on collected sensor data that indicate signs of impending failure in various parts of the vehicle.

[1095] A "generative model" is a model that uses machine learning algorithms to analyze sensor data and detect abnormal patterns and signs of failure.

[1096] The "data analysis means" is a device or software for analyzing received sensor data and detecting signs of failure.

[1097] The "emotion recognition means" is a device or software that analyzes the user's tone of voice, facial expressions, body movements, etc., and recognizes the user's emotional state in real time.

[1098] The "notification means" is a device or software that selects an appropriate notification method based on the detected signs of failure and the user's emotional state, and notifies the driver.

[1099] The "repair facility search means" is a device or software for searching for the nearest repair facility based on the current location of the vehicle when a failure symptom is detected.

[1100] The "reservation procedure means" is a device or software for automatically carrying out reservation procedures with the searched repair facility.

[1101] The embodiments of the present invention will be specifically described below.

[1102] Sensor data collection and transmission

[1103] The server collects sensor data from multiple sensors installed in the vehicle, including information on engine status, braking system, tire pressure, temperature, etc. The collected data is compiled into a single data packet on the device and then transmitted to the server in real time.

[1104] Data analysis and failure prediction

[1105] The server receives sensor data sent from the device and analyzes it in real time using a generative model. This generative model uses machine learning algorithms to detect abnormal patterns and signs of malfunction. As a result of the analysis, it determines whether there are signs of a malfunction in the vehicle.

[1106] Recognizing user emotions with an emotion engine

[1107] The server uses the smartphone's camera and microphone to analyze the driver's facial expressions and tone of voice to recognize their emotional state in real time. This emotional data is then added to the analysis results and sent to the server.

[1108] Driver notification and adaptation

[1109] The server selects the appropriate notification method based on the malfunction signs and emotion recognition data. For example, if the server determines that the driver is under stress, it instructs the device to provide gentle voice notifications or relaxation support. Based on this, the device gently notifies the driver of the presence of malfunction signs.

[1110] Find and book a repair shop

[1111] If a malfunction is detected, the server searches for the nearest repair facility based on the vehicle's current location. A list of multiple repair facilities is generated as a result of the search and provided to the driver via the terminal. The driver can then select the desired repair facility from the provided list.

[1112] Reservation procedure

[1113] The server automatically schedules a reservation with the selected repair facility, including details such as the vehicle's condition and the driver's preferred date and time, ensuring fast and timely repairs.

[1114] Hardware and software used

[1115] Hardware: Sensors in the vehicle (engine sensor, tire pressure sensor, temperature sensor), smartphone (camera, microphone)

[1116] Software: Server-based machine learning algorithms, generative models, emotion recognition engines, cloud servers, notification systems

[1117] Specific examples

[1118] For example, suppose an onboard sensor detects an abnormal engine temperature while a vehicle is driving on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative model to detect signs of engine failure. Based on the analysis results and the driver's emotion recognition data, the server instructs the device to send an appropriate notification. The device notifies the driver, "An abnormality has been detected in the engine. Please remain calm and take action." The server then searches for the nearest repair facility based on the vehicle's current location and sends a list of multiple repair facilities to the device. When the driver selects the desired repair facility, the server automatically processes the reservation at that facility.

[1119] Prompt Sentence Examples

[1120] Failure prediction prompt

[1121] Prompt: Detect engine pre-failure based on data sent from sensors. Example: Engine temperature is outside normal range.

[1122] Data: {"Engine Status": "Abnormal", "Brake System": "Normal", "Tire Pressure": 32, "Temperature": 110}

[1123] Result: Premature failure. Engine requires immediate repair.

[1124] Driver emotion recognition prompts

[1125] Prompt: Based on camera and microphone data, determine if the driver is stressed. Example: Talking with a stern face.

[1126] Data: {"Facial Expression": "Stern", "Voice Tone": "High-pitched"}

[1127] Result: The driver is stressed. Use a gentle tone to notify them.

[1128] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1129] Step 1:

[1130] The terminal collects sensor data from multiple sensors installed in the vehicle, such as engine status, braking system, tire pressure, and temperature, and compiles it into a data packet. The input is data from the sensors, and the output is a data packet.

[1131] Step 2:

[1132] The terminal sends the collected data packets to a server in real time. The server inputs the received sensor data into an analysis platform. Specifically, a communication protocol such as an HTTP request is used. The input is the data packets, and the output is the data sent to the server.

[1133] Step 3:

[1134] The server analyzes the received sensor data in real time using a generative model. This uses machine learning algorithms to detect abnormal patterns and signs of failure. The input is the sensor data, and the output is the analysis results.

[1135] Step 4:

[1136] The server recognizes the driver's emotional state using data provided by the smartphone's camera and microphone. Specifically, it uses image processing and voice analysis technology (e.g., OpenCV, voice analysis software). The input is data from the camera and microphone, and the output is emotional state data.

[1137] Step 5:

[1138] The server selects an appropriate notification method based on the failure signs and emotion recognition data and sends instructions to the device. For example, if the driver is in a stressful state, it will instruct a gentle voice notification. The input is the analysis results and emotional state data, and the output is notification instruction data.

[1139] Step 6:

[1140] The terminal notifies the driver based on instructions from the server. For example, it may notify the driver that "An abnormality has been detected in the engine. Please remain calm and take appropriate action." The input is notification instruction data, and the output is a notification to the user.

[1141] Step 7:

[1142] When a malfunction symptom is detected, the server searches for the nearest repair facility based on the vehicle's current location information. Here, GPS information and a database of repair facilities are used. The input is location information and malfunction symptom data, and the output is a list of repair facilities.

[1143] Step 8:

[1144] The terminal displays the list of repair facilities sent from the server to the driver and lets the driver select the desired repair facility. The input is the list of repair facilities and the output is the user's selection.

[1145] Step 9:

[1146] The server automatically processes the reservation with the selected repair facility. Specifically, it accesses the repair facility's reservation system and communicates the vehicle's status and the driver's desired date and time. The input is the user's selection and vehicle information, and the output is reservation confirmation.

[1147] Step 10:

[1148] The server sends the reservation confirmation information to the terminal, and the terminal notifies the driver. The input is the reservation confirmation data, and the output is the reservation confirmation notification to the user.

[1149] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1150] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1151] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1152] [Fourth embodiment]

[1153] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1154] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1155] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1156] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1157] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1158] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1159] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1160] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1161] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1162] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1163] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1164] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1165] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1166] The embodiments of the present invention will be specifically described below.

[1167] Sensor data collection and transmission

[1168] The device collects data from multiple sensors installed in the vehicle. This sensor data includes various vehicle information such as engine status, braking system, tire pressure, and temperature. The collected data is compiled into a single data packet within the device. The device then transmits this data packet to a server in real time.

[1169] Data analysis and failure prediction

[1170] The server receives sensor data sent from the device. The received data is analyzed in real time using a generative model. This generative model can detect abnormal patterns and signs of faults using machine learning algorithms. The server uses the analysis results to determine whether there are signs of a fault in the vehicle.

[1171] Driver Notification

[1172] When a malfunction sign is detected, the server sends the information to the terminal in real time. The terminal then notifies the driver of the malfunction sign. The notification method can take the form of a voice message or a display to warn the driver of the presence of a malfunction sign.

[1173] Find and book a repair shop

[1174] If a malfunction is detected, the server searches for the nearest repair shop based on the vehicle's current location. The search results generate a list of multiple repair shops. This list is provided to the driver via their terminal. The driver can then select the repair shop of their choice from the provided list.

[1175] Once the selection is complete, the server automatically schedules a reservation with the selected repair shop, including details such as the vehicle's condition and the driver's preferred date and time, ensuring a fast and timely repair.

[1176] Specific examples

[1177] For example, suppose an onboard sensor detects an abnormal engine temperature while a vehicle is traveling on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative model and detects signs of engine failure. As a result, the server sends information about the signs of failure to the device, and the device notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[1178] The server then searches for the nearest repair shop based on the vehicle's current location and sends a list of several repair shops to the terminal. Once the driver selects the desired repair shop, the server proceeds with the reservation procedure at that repair shop. This series of steps allows the driver to quickly discover the problem and take appropriate action.

[1179] This system analyzes various vehicle sensor data in real time and quickly detects and notifies of signs of malfunction, thereby significantly contributing to extending the lifespan of vehicles and reducing maintenance costs.

[1180] The processing flow will be explained below.

[1181] Step 1:

[1182] The device collects various data from sensors installed inside the vehicle, including information on engine status, braking system, tire pressure, temperature, etc.

[1183] Step 2:

[1184] The device collects sensor data and assembles it into a single data packet, which contains information from various sensors.

[1185] Step 3:

[1186] The device sends the bundled data packets to the server in real time using HTTP POST requests.

[1187] Step 4:

[1188] The server receives the sensor data sent from the device and temporarily stores the data on the server side.

[1189] Step 5:

[1190] The server analyzes the received sensor data using a generative model, which uses machine learning algorithms to detect abnormal patterns and signs of failure.

[1191] Step 6:

[1192] The server determines whether any signs of failure have been detected from the analysis results, and if so, sends that information to the terminal.

[1193] Step 7:

[1194] The terminal receives the notification of the malfunction sign sent from the server and notifies the driver of this information using means such as a voice message or a display.

[1195] Step 8:

[1196] When a malfunction symptom is detected, the server searches for the nearest repair shop based on the current location of the vehicle, and generates a list of repair shops.

[1197] Step 9:

[1198] The server sends the generated list of repair shops to the terminal, which displays the list to the driver and prompts them to make a selection.

[1199] Step 10:

[1200] The user (driver) selects the desired repair shop from a list of repair shops via the terminal, and the selected information is sent to the server.

[1201] Step 11:

[1202] The server automatically processes reservations with the selected repair shops, including details such as the vehicle's condition and the driver's desired date and time.

[1203] Step 12:

[1204] The server notifies the terminal that the reservation procedure is complete, and the terminal notifies the driver that the reservation is complete, providing support to ensure the vehicle repair is carried out smoothly.

[1205] Example 1

[1206] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1207] Conventional vehicle management systems lacked an integrated means for collecting and analyzing sensor data, making it difficult to quickly identify and notify signs of malfunction. Furthermore, they lacked the functionality to automatically search for and reserve the most suitable repair shop when signs of malfunction were detected, making it difficult to perform fast and efficient maintenance. This created the challenge of preventing vehicle breakdowns and reducing maintenance costs.

[1208] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1209] In this invention, the server includes means for receiving sensor data collected from the vehicle, data analysis means for analyzing the received sensor data in real time and including a generative model for detecting signs of failure, notification means for notifying the user of the detected signs of failure, repair facility search means for searching for the nearest repair facility when a failure is predicted, and reservation procedure means for making a reservation at the searched repair facility. This makes it possible to analyze the sensor data in real time and quickly detect and notify signs of failure, thereby significantly improving the efficiency and effectiveness of vehicle maintenance.

[1210] "Vehicle" refers to a moving body such as an automobile or motorcycle that travels on the ground.

[1211] "Sensor data" refers to information collected from various sensors installed in a vehicle, and includes data on the power plant status, braking system, tire pressure, temperature, etc.

[1212] "Means for receiving" refers to communication equipment or programs for obtaining sensor data from other devices or systems.

[1213] "Data analysis means" refers to an analysis device or program that processes received sensor data and includes a generative model for detecting abnormalities or signs of failure.

[1214] A "generative model" refers to a data model that has been trained using machine learning algorithms to make predictions or analyses.

[1215] "Notification means" refers to devices or programs such as voice messages or display screens that notify the user of detected signs of a malfunction.

[1216] "Repair facility search means" refers to a device or program for searching for the nearest repair facility based on the current location of the vehicle when a malfunction symptom is detected.

[1217] "Reservation processing means" refers to a device or program for automatically making a reservation at a searched repair facility.

[1218] "User" means the person who drives or owns a Vehicle.

[1219] "Real-time" refers to data collection and processing occurring immediately, without delay.

[1220] The present invention is a system that uses sensor data collected from a vehicle to detect signs of a malfunction, notifies the user, and searches for the nearest repair facility and completes the reservation procedure. This system is configured and operates as follows.

[1221] Sensor data collection and transmission

[1222] The terminal collects data from multiple sensors installed in the vehicle. These sensors include sensors for measuring the status of the powertrain, braking system, tire pressure, and temperature. The collected sensor data is compiled into a single data packet within the terminal. For example, data such as 90 degrees from the engine temperature sensor and 2.5 bar from the tire pressure sensor is collected.

[1223] The device then transmits the data packets to the server in real time over Wi-Fi or 4G / 5G networks, minimizing data latency.

[1224] Data analysis and failure prediction

[1225] The server receives sensor data sent from the device. The received data is analyzed in real time using a generative AI model using data analysis software such as TensorFlow or PyTorch. During the analysis, machine learning algorithms are used to detect abnormal patterns and signs of malfunction. For example, if the engine temperature is outside the normal range (above 90 degrees), it is recognized as a sign of malfunction.

[1226] Driver Notification

[1227] If a malfunction sign is detected, the server sends that information to the terminal. The terminal then notifies the driver of the received malfunction sign notification. This notification method can include a voice message or a display. Specifically, the message displayed will say, "An abnormality has been detected in the engine. Immediate repair is required."

[1228] Find and book a repair shop

[1229] If a malfunction is detected, the server searches for the nearest repair facility based on the vehicle's current location. The server references an online database and generates a list of multiple repair facilities. This list is provided to the user via their terminal. The user can select the desired repair facility from the provided list. Once the selection is complete, the server automatically completes the reservation procedure at the selected repair facility. The reservation procedure includes detailed information such as the vehicle's condition and the user's desired date and time.

[1230] Specific examples

[1231] For example, suppose a sensor detects an abnormal engine temperature while a vehicle is driving on a highway. This data is sent by the device to a server, which then analyzes it using a generative AI model. As a result of the analysis, signs of engine failure are detected. The server sends this information to the device, which then notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[1232] The server then searches for the nearest repair facility based on the vehicle's current location and sends a list of several repair facilities to the terminal. Once the driver selects the desired repair facility, the server proceeds with the reservation procedure at that repair facility. This series of steps allows the driver to quickly discover the malfunction and take appropriate action.

[1233] Prompt Sentence Examples

[1234] "We have acquired abnormal engine temperature data. Please analyze this data to detect signs of failure."

[1235] "Find the nearest repair shops based on your vehicle's current location and generate a list."

[1236] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1237] System program processing flow

[1238] Step 1: Collect sensor data

[1239] Specific details:

[1240] The device collects data from multiple sensors installed in the vehicle, including sensors to measure the status of the powerplant, braking system, tire pressure, and temperature.

[1241] Input and Output:

[1242] The input is raw data from each sensor, and the output is a data packet that aggregates this data.

[1243] Specific behavior:

[1244] For example, it collects data such as 90 degrees from the engine temperature sensor and 2.5 bar from the tire pressure sensor.

[1245] Step 2: Generate data packets

[1246] Specific details:

[1247] Collected sensor data is compiled into a single data packet.

[1248] Input and Output:

[1249] The input is the data collected from each sensor in step 1. The output is a data packet that integrates this data.

[1250] Specific behavior:

[1251] It generates a data packet containing information such as engine temperature at 90 degrees and tire pressure at 2.5 bar.

[1252] Step 3: Sending data packets

[1253] Specific details:

[1254] The device transmits the generated data packets to the server in real time using Wi-Fi or 4G / 5G networks.

[1255] Input and Output:

[1256] The input is the data packet generated in step 2. The output is the data packet sent to the server.

[1257] Specific behavior:

[1258] The data packets are sent to a server over a communication network (e.g., 4G).

[1259] Step 4: Receiving the data

[1260] Specific details:

[1261] The server receives the data packets sent from the terminal.

[1262] Input and Output:

[1263] The input is the data packets sent from the device, and the output is the sensor data stored on the server.

[1264] Specific behavior:

[1265] The server checks and stores the received data.

[1266] Step 5: Data analysis

[1267] Specific details:

[1268] Based on the received data, real-time analysis is performed using a generative AI model.

[1269] Input and Output:

[1270] The input is sensor data stored on a server, and the output is the analysis results, which include signs of failure.

[1271] Specific behavior:

[1272] For example, TensorFlow analyzes that the engine temperature is abnormal (above 90 degrees).

[1273] Step 6: Pre-fault detection

[1274] Specific details:

[1275] The server detects abnormal patterns and signs of failure based on the analysis results.

[1276] Input and Output:

[1277] The input is the analysis result from the generative AI model, and the output is sensor data that indicates a potential failure.

[1278] Specific behavior:

[1279] Since the engine temperature is 90 degrees, this is detected as a sign of a malfunction.

[1280] Step 7: Sending notifications

[1281] Specific details:

[1282] If a fault sign is detected, the server transmits the information to the terminal.

[1283] Input and Output:

[1284] The input is the sensor data that indicates a failure. The output is the notification data sent to the terminal.

[1285] Specific behavior:

[1286] A message is sent to the terminal saying, "An abnormality has been detected in the engine. Immediate repair is required."

[1287] Step 8: Notify users

[1288] Specific details:

[1289] The terminal notifies the user of the received failure warning by means of a voice message or a display.

[1290] Input and Output:

[1291] The input is the notification data sent by the server. The output is the notification the user receives.

[1292] Specific behavior:

[1293] For example, a message such as "An abnormality has been detected in the engine. Immediate repair is required" may be displayed on the screen.

[1294] Step 9: Find a repair facility

[1295] Specific details:

[1296] The server searches for the nearest repair facility based on the vehicle's current location.

[1297] Input and Output:

[1298] The input is the vehicle's current location data. The output is a list of repair facilities.

[1299] Specific behavior:

[1300] For example, GPS data can be used to search an internet database for the nearest repair facility from your current location.

[1301] Step 10: Submit a list of repair facilities

[1302] Specific details:

[1303] The server sends the generated list of repair facilities to the terminal.

[1304] Input and Output:

[1305] The input is the list of repair facilities found. The output is the list sent to the terminal.

[1306] Specific behavior:

[1307] A number of potential repair facilities are sent to the device.

[1308] Step 11: User Selects Repair Facility

[1309] Specific details:

[1310] The terminal displays a list of repair facilities to the user, and the user selects the desired repair facility.

[1311] Input and Output:

[1312] The input is a list of repair facilities. The output is the user's selected repair facility.

[1313] Specific behavior:

[1314] The user selects the desired repair facility from the device screen.

[1315] Step 12: Complete the booking process

[1316] Specific details:

[1317] The server automatically processes reservations with the selected repair facility, including detailed information such as the vehicle's status and the user's desired date and time.

[1318] Input and Output:

[1319] The input is the repair facility selected by the user and the fault information. The output is a reservation completion notification to the repair facility.

[1320] Specific behavior:

[1321] A reservation message is sent to the repair facility stating, "Please repair the engine abnormality on XX date at XX time."

[1322] (Application example 1)

[1323] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1324] Conventional systems lacked the means to effectively collect various vehicle sensor data and analyze and notify abnormalities and signs of failure in real time. Furthermore, when a failure was predicted, the process of searching for and booking a repair shop was cumbersome for drivers, making it difficult to respond quickly. This meant that the system was unable to fully extend the lifespan of vehicles or reduce maintenance costs.

[1325] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1326] In this invention, the server includes: means for receiving sensor data collected from the vehicle; data analysis means including a generative model that analyzes the received sensor data in real time and detects signs of failure; means for notifying the driver of the detected signs of failure; repair shop search means for searching for the nearest repair shop when a failure is predicted; reservation processing means for making a reservation at the searched repair shop; data transmission means for collecting various sensor data from the vehicle via a smartphone and transmitting it to a cloud server; and means for analyzing the sensor data on the cloud server, detecting signs of failure or abnormalities, and notifying the driver in real time. This makes it possible to quickly collect and analyze various sensor data from the vehicle and notify the driver of signs of failure in real time. Furthermore, when a failure is predicted, the server searches for the nearest repair shop and automatically processes the reservation, thereby speeding up maintenance response and reducing the driver's hassle.

[1327] A "vehicle" is a moving body, including an automobile, and is a means of transportation intended to travel on roads.

[1328] "Sensor data" refers to data collected by various sensors installed in a vehicle, including engine status, braking system, tire pressure, temperature, etc.

[1329] "Means for receiving" refers to the means for inputting sensor data from the vehicle to a server or terminal.

[1330] A "generative model" is a model trained using machine learning algorithms to make predictions or classifications based on input data.

[1331] "Data analysis means" refers to a means for analyzing received sensor data in real time and detecting abnormalities or signs of failure.

[1332] The "notification means" is a means for notifying the driver of signs of malfunction or abnormalities based on the analysis results.

[1333] The "repair shop search means" is a means for searching for the nearest repair shop based on the vehicle's location information when a breakdown is expected.

[1334] The "reservation procedure means" is a means for automatically making a reservation to visit the searched repair shop.

[1335] A "smartphone" is a multi-functional mobile phone terminal that has mobile communication functions and Internet connection functions.

[1336] A "cloud server" is a server accessed via the Internet that provides computing resources for storing and analyzing large amounts of data.

[1337] The "data transmission means" is a means for transmitting collected sensor data from the smartphone in the vehicle to the cloud server.

[1338] "Real-time" refers to processing occurring immediately without delay.

[1339] "Abnormal" refers to a condition of a vehicle or its parts that is different from its normal operating condition.

[1340] This invention is a system that collects various sensor data from a vehicle, analyzes it in real time, and detects signs of failure, and is particularly applicable to autonomous vehicles.

[1341] First, the vehicle is equipped with multiple sensors that measure the engine, braking system, tire pressure, temperature, etc. These sensor data are collected via a smartphone installed in the vehicle, which then compiles the data into a single data packet and sends it to a cloud server.

[1342] A generative model trained using machine learning algorithms is installed on the cloud server. This generative model analyzes the received sensor data in real time and detects abnormal patterns and signs of malfunction, such as when the engine temperature is higher than normal or when tire pressure drops suddenly.

[1343] Depending on the analysis results, the server will notify the smartphone of any detected signs of malfunction. Notification methods include voice messages and display on the screen. For example, the server will warn the driver by saying, "An abnormality has been detected in the engine. Immediate repair is required."

[1344] The server then searches for the nearest repair shop based on the vehicle's current location. The search results are sent to the smartphone as a list of multiple repair shops. When the driver selects the desired repair shop, the server automatically makes a reservation with that shop. The reservation procedure includes detailed information such as the vehicle's status and the driver's desired date and time.

[1345] The hardware used includes various sensors in the vehicle, smartphones, and cloud servers. The software includes a data collection application running on the smartphone and a data analysis program on the cloud server. In particular, the machine learning models used on the cloud server use libraries such as scikit-learn and TensorFlow.

[1346] As a concrete example, consider a case where an abnormal engine temperature is detected while driving on a highway. This data is sent to a cloud server via a smartphone and analyzed by a generative model. If a sign of engine failure is detected as a result, a notification is displayed on the smartphone stating, "There is an abnormality in the engine. Repairs are required immediately." The server then searches for the nearest repair shop and sends a list of several shops to the smartphone. When the driver selects the desired repair shop, the server automatically processes the reservation procedure at that shop.

[1347] An example of a prompt is:

[1348] "GPS location: {lat}, {lng}. Engine temperature: {engine_temp} degrees. Brake status: {brake_status}. Tire pressure: {tire_pressure} psi. If an abnormality is detected, we will notify you of the nearest repair shop and start the appointment process."

[1349] Based on this prompt, the system analyzes data collected in real time and automates fault prediction and repair shop searches.

[1350] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1351] Step 1:

[1352] The smartphone collects data from various vehicle sensors (engine, brake system, tire pressure, temperature, etc.). The sensor data is input, and the smartphone compiles the collected data into a single data packet. This data packet is output. Specifically, the smartphone converts the voltage and signals from the sensors into digital data and packets it in the appropriate format.

[1353] Step 2:

[1354] The smartphone sends a data packet to the cloud server. The input is the data packet generated in step 1, and the output is the data packet sent over the Internet. Specifically, the smartphone sends an HTTP request to the specified server URL using Wi-Fi or the mobile network.

[1355] Step 3:

[1356] The cloud server analyzes the received data packets. The input is the data packets and the output is the analysis results. A generative AI model is used for this analysis to detect anomalies and signs of failure. Specifically, a Python program installed on the cloud server analyzes the data packets and uses machine learning algorithms to perform pattern matching and predictions.

[1357] Step 4:

[1358] If the cloud server detects a sign of a failure based on the analysis results, it sends a notification to the smartphone. The input is the analysis results and the output is a notification message. Specifically, the cloud server generates a message based on the analysis results and sends a push notification to the smartphone using a service such as Firebase Cloud Messaging.

[1359] Step 5:

[1360] The smartphone receives the notification message and displays a warning to the driver. The input is the notification message, and the output is a voice message or a warning displayed on the screen. Specifically, the smartphone application is triggered by the notification and displays a message on the user interface.

[1361] Step 6:

[1362] The cloud server searches for the nearest repair shop based on the vehicle's current location. The input is the vehicle's GPS data, and the output is a list of repair shops. Specifically, the cloud server uses a map service such as Google Maps API to obtain the location information of repair shops based on the current location.

[1363] Step 7:

[1364] A list of repair shops is sent to the smartphone, and the driver selects one. The input is the list of repair shops, and the output is the repair shop selected by the driver. Specifically, the smartphone application displays a repair shop selection interface and provides operations for the driver to select.

[1365] Step 8:

[1366] The cloud server then completes the reservation procedure with the selected repair shop. The input is the repair shop selected by the driver and the vehicle status data, and the output is a reservation confirmation message. Specifically, the cloud server sends an HTTP request to the repair shop's reservation system to complete the reservation procedure.

[1367] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1368] The embodiments of the present invention will be specifically described below.

[1369] Sensor data collection and transmission

[1370] The device collects data from multiple sensors installed in the vehicle, including information on engine status, braking system, tire pressure, temperature, etc. The collected data is compiled into a single data packet within the device and then transmitted to a server in real time.

[1371] Data analysis and failure prediction

[1372] The server receives sensor data sent from the device. The received data is analyzed in real time using a generative model. This generative model can detect abnormal patterns and signs of faults using machine learning algorithms. The server uses the analysis results to determine whether there are signs of a fault in the vehicle.

[1373] Recognizing user emotions with an emotion engine

[1374] The device is equipped with an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's tone of voice, facial expressions, body movements, etc., and can recognize the user's emotional state in real time. The emotion engine sends the recognized emotion data to the server, which then incorporates this data into the analysis results.

[1375] Driver notification and adaptation

[1376] The server selects an appropriate notification method based on the signs of a malfunction and the emotional data obtained from the emotion engine. For example, if the server determines that the driver is under stress, it instructs the server to provide gentle voice notifications and relaxation support. The notification is sent via the device, gently informing the driver of the presence of signs of a malfunction.

[1377] Find and book a repair shop

[1378] If a malfunction is detected, the server searches for the nearest repair shop based on the vehicle's current location. As a result of the search, a list of multiple repair shops is generated and provided to the driver via the terminal. The driver can then select the desired repair shop from the provided list.

[1379] Reservation procedure

[1380] The server automatically schedules a reservation with the selected repair shop, including details such as the vehicle's condition and the driver's preferred date and time, ensuring fast and timely repairs.

[1381] Specific examples

[1382] For example, suppose an onboard sensor detects an abnormal engine temperature while a vehicle is traveling on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative model and detects signs of engine failure. As a result, the server sends information about the signs of failure to the device, and the device notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[1383] Meanwhile, the emotion engine analyzes the driver's facial expressions and voice tone to determine that the driver is under stress. Based on this information, the server instructs the device to issue a gentle voice notification to the driver. The server then searches for the nearest repair shop based on the vehicle's current location and sends a list of multiple repair shops to the device. Once the driver selects their preferred repair shop, the server automatically makes a reservation with that shop. This series of steps allows the driver to quickly discover the problem and take appropriate action.

[1384] This system significantly contributes to extending the life of a vehicle and reducing maintenance costs by analyzing various vehicle sensor data in real time and quickly detecting and notifying signs of malfunction. In addition, by combining it with an emotion engine, it provides appropriate notifications according to the driver's emotional state, realizing a more user-friendly system.

[1385] The processing flow will be explained below.

[1386] Step 1:

[1387] The device collects various data from multiple sensors installed inside the vehicle, including information on engine status, braking system, tire pressure, temperature, etc.

[1388] Step 2:

[1389] The device aggregates the collected sensor data into a single data packet, which is then sent to the server in real time.

[1390] Step 3:

[1391] The server receives the sensor data sent from the device and temporarily stores the received data within the server.

[1392] Step 4:

[1393] The server uses the generative model to analyze the incoming sensor data in real time, using machine learning algorithms to detect abnormal patterns and signs of failure.

[1394] Step 5:

[1395] The server determines whether any signs of failure have been detected from the analysis results, and if so, sends that information to the terminal.

[1396] Step 6:

[1397] The terminal receives the notification of the malfunction signs sent from the server. The terminal notifies the driver of this information. Specifically, the terminal warns the driver of the existence of the malfunction signs using a voice message or a display.

[1398] Step 7:

[1399] The device's emotion engine analyzes the user's (driver's) voice tone, facial expressions, body movements, etc. to recognize the user's emotional state in real time. The recognized emotion data is sent from the device to the server.

[1400] Step 8:

[1401] The server analyzes the emotional data sent from the emotion engine and determines whether the driver is in a stressed or relaxed state.

[1402] Step 9:

[1403] The server selects the appropriate notification method based on the malfunction signs and emotional data. For example, if it determines that the driver is under stress, it will provide gentle voice notifications or relaxation support.

[1404] Step 10:

[1405] The device notifies the driver based on instructions from the server, including voice messages, displaying information on the screen, and even playing relaxation music.

[1406] Step 11:

[1407] When a malfunction symptom is detected, the server searches for the nearest repair shop based on the vehicle's current location, and generates a list of multiple repair shops as a search result.

[1408] Step 12:

[1409] The server sends the generated repair shop list to the terminal, which displays the list to the driver, allowing the driver to select the desired repair shop.

[1410] Step 13:

[1411] The driver (user) selects the desired repair shop from a list of repair shops via the terminal, and the selected information is sent to the server.

[1412] Step 14:

[1413] The server automatically processes reservations with the selected repair shop, and the reservation includes detailed information such as the vehicle's condition and the driver's desired date and time.

[1414] Step 15:

[1415] The server notifies the terminal that the reservation procedure is complete, and the terminal notifies the driver that the reservation is complete, providing support to ensure the vehicle repair is carried out smoothly.

[1416] Example 2

[1417] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1418] Conventional vehicle operation assistance systems have the ability to detect and notify drivers of signs of malfunction based on sensor data, but they lack the ability to provide appropriate responses and notifications that take into account the user's emotional state. This can result in inadequate timing for notifications sent to stressed drivers. Furthermore, functions for searching for the nearest repair facility based on signs of malfunction and making reservations are not fully integrated, placing a significant burden on the driver. These issues need to be resolved.

[1419] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1420] In this invention, the server includes means for receiving sensor data collected from the vehicle, data analysis means including a generative model that analyzes the received sensor data in real time and detects signs of failure, emotion recognition means for recognizing the user's emotional state and collecting emotion data, notification means for notifying the driver based on the detected signs of failure and emotion data, repair facility search means for searching for the nearest repair facility when a failure is predicted, and reservation procedure means for making a reservation at the searched repair facility. This makes it possible to quickly detect and notify signs of failure and quickly take appropriate repair measures while taking the driver's emotional state into consideration.

[1421] A "vehicle" is a means of transportation used as a means of transportation, and is a general term for a vehicle that mainly includes an engine, a braking system, etc.

[1422] "Sensor data" refers to data collected from various sensors attached to a vehicle, and includes information on engine status, braking system, tire pressure, temperature, etc.

[1423] "Means for receiving" refers to a function or device that acquires data sent from outside.

[1424] A "generative model" is a model that uses machine learning algorithms to analyze data and detect abnormal patterns and signs of failure.

[1425] "Data analysis means" refers to the functions and technologies for analyzing received data and extracting meaningful information.

[1426] "Emotion recognition means" refers to devices or technologies that analyze and recognize a user's emotional state in real time from their voice tone, facial expressions, body movements, etc.

[1427] "Notification means" refers to a function or device for notifying the user of analysis results and important information.

[1428] A "repair facility" is a location that provides equipment and services for repairing vehicle breakdowns and maintenance.

[1429] "Repair facility search means" refers to a function or technology for searching for the nearest repair facility based on information such as the vehicle's current location.

[1430] "Reservation procedure means" refers to a function or system for automatically making reservations for repairs or maintenance at a repair facility.

[1431] This invention is a system that detects signs of malfunction and notifies the driver based on sensor data collected from the vehicle and user emotional data. The system aims to improve driver safety and comfort by analyzing sensor data in real time and providing appropriate notifications and responses.

[1432] Hardware Configuration

[1433] The device is installed inside the vehicle and collects data from multiple sensors (engine condition sensors, brake system sensors, tire pressure sensors, temperature sensors, etc.) It is also equipped with a camera and microphone for emotion recognition, which capture the user's voice tone, facial expressions, and body movements.

[1434] The server is a central system for receiving and analyzing collected sensor data and emotion data in real time. It is built on a computer equipped with a high-performance processor and large memory capacity. To perform data analysis, a generative AI model using a machine learning algorithm is implemented. For example, machine learning frameworks such as TensorFlow and PyTorch are used.

[1435] Software Configuration

[1436] The device contains software to collect sensor data and compile it into a single data packet. It also contains an emotion engine to analyze the user's emotional data. The emotion engine has algorithms that determine the user's emotional state, such as stress or relief, based on the user's tone of voice, facial expressions, and body movements.

[1437] The server is equipped with a generative AI model for analyzing the received data. This AI model has the ability to detect signs of malfunction from the received sensor data. It also includes an algorithm for selecting the optimal notification method for the driver based on the emotion recognition results.

[1438] Specific examples

[1439] Suppose an onboard sensor detects an abnormal engine temperature while a vehicle is traveling on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative AI model and detects signs of engine failure. As a result, the server sends information about the signs of failure to the device, and the device notifies the driver, "An abnormality has been detected in the engine. Immediate repair is required."

[1440] Meanwhile, the emotion engine analyzes the driver's facial expressions and voice tone to determine if the driver is under stress. Based on this information, the server instructs the device to issue a gentle voice notification to the driver. The server then searches for the nearest repair shop based on the vehicle's current location and sends a list of multiple repair shops to the device. Once the driver selects the desired repair shop, the server automatically completes the reservation procedure with that repair shop.

[1441] Prompt Sentence Examples

[1442] For example, the following prompt statements can be used to specify system behavior:

[1443] "What is the procedure if the onboard sensors detect an abnormal engine temperature while driving on the highway? Also, what kind of notification or support will be provided if the driver feels stressed at that time?"

[1444] In this way, the system of the present invention significantly contributes to extending the life of a vehicle and reducing maintenance costs by analyzing various sensor data of the vehicle in real time and quickly detecting and notifying signs of malfunction. In addition, by providing notifications that take the user's emotional state into consideration, it realizes a more user-friendly system.

[1445] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1446] Step 1: Collect sensor data

[1447] The terminal collects data from sensors installed inside the vehicle, such as those for the engine condition, braking system, tire pressure, and temperature. The collected data is temporarily stored in the terminal's memory. The input is data from each sensor, and the output is compiled into a single data packet. Specifically, when the tire pressure sensor detects pressure below a specified value, it records the data in memory.

[1448] Step 2: Sending sensor data

[1449] The terminal aggregates the collected sensor data into a single packet and transmits it in encrypted form to the server in real time. The input is the aggregated data packet, and the output is a transmission success message to the server. Specifically, the terminal aggregates engine status data, brake system data, tire pressure data, and temperature data, encrypts it, and transmits it to the server.

[1450] Step 3: Receiving the data

[1451] The server receives sensor data sent from the device. The input is an encrypted data packet, and the output is decrypted sensor data. Specifically, the server receives the data packet, decrypts it, and obtains the data of each sensor.

[1452] Step 4: Data analysis and failure detection

[1453] The server analyzes the received data in real time using a generative AI model to detect abnormal patterns and signs of failure. The input is decoded sensor data, and the output is the presence or absence of signs of failure and detailed data. Specifically, if the engine temperature exceeds the normal range, it is determined to be a sign of engine failure.

[1454] Step 5: Collecting emotion data

[1455] The device analyzes the user's voice tone, facial expressions, and body movements to recognize their emotional state in real time. The input is the user's voice and video data, and the output is analyzed emotional data. Specifically, the camera and microphone capture the user's facial expressions and voice, which are then analyzed by the emotion engine.

[1456] Step 6: Sending Emotion Data

[1457] The device sends the recognized emotion data to the server. The input is the analyzed emotion data, and the output is a transmission success message to the server. Specifically, the device sends data indicating that the user is in a stressed state to the server.

[1458] Step 7: Decide on notifications

[1459] The server determines the optimal notification method based on the received sensor data and emotional data. The input is failure prediction data and emotional data, and the output is the notification content and method. Specifically, if an engine abnormality is detected and the user is in a stressful state, a gentle voice notification will be selected.

[1460] Step 8: Notify drivers

[1461] The device follows instructions from the server and notifies the driver of signs of malfunction and their emotions. The input is the content and method of the notification, and the output is the notification to the driver. Specifically, it notifies the driver by voice, saying, "An abnormality has been detected in the engine. Immediate repair is required."

[1462] Step 9: Find a repair facility

[1463] The server searches for the nearest repair facility based on the vehicle's current location. The input is the vehicle's location data, and the output is a list of repair facilities. Specifically, it uses GPS data to search for nearby repair facilities and generates the list.

[1464] Step 10: Select and book a repair facility

[1465] The user selects the desired repair facility from a list of repair facilities provided by the terminal. The information of the selected repair facility is sent to the server, and the reservation procedure is carried out automatically. The input is the user's selection information, and the output is a reservation confirmation message. Specifically, the user taps the desired repair facility, and the server contacts that facility to complete the reservation procedure.

[1466] (Application example 2)

[1467] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1468] Modern cars are equipped with many sensors, but there are only a limited number of systems that can analyze this sensor data in real time and predict potential breakdowns in advance. Furthermore, there are no systems that recognize the driver's emotional state and immediately take appropriate action based on that. This creates a problem: drivers tend to feel stressed when they encounter a breakdown, making it difficult to quickly repair the vehicle.

[1469] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving sensor data collected from the vehicle, data analysis means including a generative model that analyzes the received sensor data in real time and detects signs of failure, emotion recognition means that recognizes the user's emotional state and incorporates that data into the analysis results, notification means that selects an appropriate notification method based on the detected signs of failure and the user's emotional state and notifies the driver, repair facility search means that, when a failure is predicted, searches for the nearest repair facility based on the vehicle location, and reservation procedure means that makes a reservation for the searched repair facility. This enables rapid detection of signs of failure, appropriate notification according to the driver's emotions, and immediate repair arrangements.

[1470] "Sensor data" refers to data collected from multiple sensors installed in a vehicle, and includes information on engine status, braking system, tire pressure, temperature, etc.

[1471] "Real-time analysis" refers to analyzing sensor data immediately as it is collected.

[1472] "Failure signs" are data based on collected sensor data that indicate signs of impending failure in various parts of the vehicle.

[1473] A "generative model" is a model that uses machine learning algorithms to analyze sensor data and detect abnormal patterns and signs of failure.

[1474] The "data analysis means" is a device or software for analyzing received sensor data and detecting signs of failure.

[1475] The "emotion recognition means" is a device or software that analyzes the user's tone of voice, facial expressions, body movements, etc., and recognizes the user's emotional state in real time.

[1476] The "notification means" is a device or software that selects an appropriate notification method based on the detected signs of failure and the user's emotional state, and notifies the driver.

[1477] The "repair facility search means" is a device or software for searching for the nearest repair facility based on the current location of the vehicle when a failure symptom is detected.

[1478] The "reservation procedure means" is a device or software for automatically carrying out reservation procedures with the searched repair facility.

[1479] The embodiments of the present invention will be specifically described below.

[1480] Sensor data collection and transmission

[1481] The server collects sensor data from multiple sensors installed in the vehicle, including information on engine status, braking system, tire pressure, temperature, etc. The collected data is compiled into a single data packet on the device and then transmitted to the server in real time.

[1482] Data analysis and failure prediction

[1483] The server receives sensor data sent from the device and analyzes it in real time using a generative model. This generative model uses machine learning algorithms to detect abnormal patterns and signs of malfunction. As a result of the analysis, it determines whether there are signs of a malfunction in the vehicle.

[1484] Recognizing user emotions with an emotion engine

[1485] The server uses the smartphone's camera and microphone to analyze the driver's facial expressions and tone of voice to recognize their emotional state in real time. This emotional data is then added to the analysis results and sent to the server.

[1486] Driver notification and adaptation

[1487] The server selects the appropriate notification method based on the malfunction signs and emotion recognition data. For example, if the server determines that the driver is under stress, it instructs the device to provide gentle voice notifications or relaxation support. Based on this, the device gently notifies the driver of the presence of malfunction signs.

[1488] Find and book a repair shop

[1489] If a malfunction is detected, the server searches for the nearest repair facility based on the vehicle's current location. A list of multiple repair facilities is generated as a result of the search and provided to the driver via the terminal. The driver can then select the desired repair facility from the provided list.

[1490] Reservation procedure

[1491] The server automatically schedules a reservation with the selected repair facility, including details such as the vehicle's condition and the driver's preferred date and time, ensuring fast and timely repairs.

[1492] Hardware and software used

[1493] Hardware: Sensors in the vehicle (engine sensor, tire pressure sensor, temperature sensor), smartphone (camera, microphone)

[1494] Software: Server-based machine learning algorithms, generative models, emotion recognition engines, cloud servers, notification systems

[1495] Specific examples

[1496] For example, suppose an onboard sensor detects an abnormal engine temperature while a vehicle is driving on a highway. This data is sent by the device to a server. The server analyzes the received data using a generative model to detect signs of engine failure. Based on the analysis results and the driver's emotion recognition data, the server instructs the device to send an appropriate notification. The device notifies the driver, "An abnormality has been detected in the engine. Please remain calm and take action." The server then searches for the nearest repair facility based on the vehicle's current location and sends a list of multiple repair facilities to the device. When the driver selects the desired repair facility, the server automatically processes the reservation at that facility.

[1497] Prompt Sentence Examples

[1498] Failure prediction prompt

[1499] Prompt: Detect engine pre-failure based on data sent from sensors. Example: Engine temperature is outside normal range.

[1500] Data: {"Engine Status": "Abnormal", "Brake System": "Normal", "Tire Pressure": 32, "Temperature": 110}

[1501] Result: Premature failure. Engine requires immediate repair.

[1502] Driver emotion recognition prompts

[1503] Prompt: Based on camera and microphone data, determine if the driver is stressed. Example: Talking with a stern face.

[1504] Data: {"Facial Expression": "Stern", "Voice Tone": "High-pitched"}

[1505] Result: The driver is stressed. Use a gentle tone to notify them.

[1506] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1507] Step 1:

[1508] The terminal collects sensor data from multiple sensors installed in the vehicle, such as engine status, braking system, tire pressure, and temperature, and compiles it into a data packet. The input is data from the sensors, and the output is a data packet.

[1509] Step 2:

[1510] The terminal sends the collected data packets to a server in real time. The server inputs the received sensor data into an analysis platform. Specifically, a communication protocol such as an HTTP request is used. The input is the data packets, and the output is the data sent to the server.

[1511] Step 3:

[1512] The server analyzes the received sensor data in real time using a generative model. This uses machine learning algorithms to detect abnormal patterns and signs of failure. The input is the sensor data, and the output is the analysis results.

[1513] Step 4:

[1514] The server recognizes the driver's emotional state using data provided by the smartphone's camera and microphone. Specifically, it uses image processing and voice analysis technology (e.g., OpenCV, voice analysis software). The input is data from the camera and microphone, and the output is emotional state data.

[1515] Step 5:

[1516] The server selects an appropriate notification method based on the failure signs and emotion recognition data and sends instructions to the device. For example, if the driver is in a stressful state, it will instruct a gentle voice notification. The input is the analysis results and emotional state data, and the output is notification instruction data.

[1517] Step 6:

[1518] The terminal notifies the driver based on instructions from the server. For example, it may notify the driver that "An abnormality has been detected in the engine. Please remain calm and take appropriate action." The input is notification instruction data, and the output is a notification to the user.

[1519] Step 7:

[1520] When a malfunction symptom is detected, the server searches for the nearest repair facility based on the vehicle's current location information. Here, GPS information and a database of repair facilities are used. The input is location information and malfunction symptom data, and the output is a list of repair facilities.

[1521] Step 8:

[1522] The terminal displays the list of repair facilities sent from the server to the driver and lets the driver select the desired repair facility. The input is the list of repair facilities and the output is the user's selection.

[1523] Step 9:

[1524] The server automatically processes the reservation with the selected repair facility. Specifically, it accesses the repair facility's reservation system and communicates the vehicle's status and the driver's desired date and time. The input is the user's selection and vehicle information, and the output is reservation confirmation.

[1525] Step 10:

[1526] The server sends the reservation confirmation information to the terminal, and the terminal notifies the driver. The input is the reservation confirmation data, and the output is the reservation confirmation notification to the user.

[1527] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1528] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1529] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1530] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1531] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1532] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1533] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1534] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1535] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1536] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1537] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1538] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1539] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1540] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1541] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1542] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1543] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1544] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1545] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1546] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1547] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1548] The following is further disclosed regarding the above embodiment.

[1549] (Claim 1)

[1550] means for receiving sensor data collected from a vehicle;

[1551] a data analysis means including a generative model that analyzes received sensor data in real time and detects signs of failure;

[1552] a notification means for notifying a driver of the detected failure sign;

[1553] a repair shop search means for searching for the nearest repair shop when a breakdown is expected;

[1554] A reservation procedure means for making a reservation at the searched repair shop;

[1555] A system including:

[1556] (Claim 2)

[1557] The system of claim 1 , wherein the sensor data from the vehicle includes engine conditions, braking system, tire pressure, and temperature.

[1558] (Claim 3)

[1559] The system of claim 1 , wherein the generative model uses a machine learning algorithm to detect fault precursors.

[1560] "Example 1"

[1561] (Claim 1)

[1562] means for receiving sensor data collected from a vehicle;

[1563] a data analysis means including a generative model that analyzes received sensor data in real time and detects signs of failure;

[1564] a notification means for notifying a user of the detected failure sign;

[1565] a repair facility search means for searching for the nearest repair facility when a breakdown is predicted;

[1566] A reservation procedure means for making a reservation at the searched repair facility;

[1567] A system including:

[1568] (Claim 2)

[1569] The system of claim 1 , wherein sensor data from the vehicle includes power plant status, braking system, tire pressure, and temperature.

[1570] (Claim 3)

[1571] The system of claim 1 , wherein the generative model uses a data analysis algorithm to detect fault precursors.

[1572] "Application Example 1"

[1573] (Claim 1)

[1574] means for receiving sensor data collected from a vehicle;

[1575] a data analysis means including a generative model that analyzes received sensor data in real time and detects signs of failure;

[1576] a notification means for notifying a driver of the detected failure sign;

[1577] a repair shop search means for searching for the nearest repair shop when a breakdown is expected;

[1578] A reservation procedure means for making a reservation at the searched repair shop;

[1579] a data transmission means for collecting various sensor data of the vehicle via a smartphone and transmitting the data to a cloud server;

[1580] A method for analyzing sensor data on a cloud server, detecting signs of failure and abnormalities, and notifying in real time.

[1581] A system including:

[1582] (Claim 2)

[1583] The system of claim 1 , wherein the sensor data from the vehicle includes engine conditions, braking system, tire pressure, and temperature.

[1584] (Claim 3)

[1585] The system of claim 1 , wherein the generative model uses a machine learning algorithm to detect fault precursors.

[1586] "Example 2: Combining Emotion Engines"

[1587] (Claim 1)

[1588] means for receiving sensor data collected from a vehicle;

[1589] a data analysis means including a generative model that analyzes received sensor data in real time and detects signs of failure;

[1590] emotion recognition means for recognizing an emotional state of a user and collecting emotion data;

[1591] a notification means for notifying a driver based on the detected failure signs and emotion data;

[1592] a repair facility search means for searching for the nearest repair facility when a breakdown is predicted;

[1593] A reservation procedure means for making a reservation at the searched repair facility;

[1594] A system including:

[1595] (Claim 2)

[1596] The system of claim 1 , wherein the sensor data from the vehicle includes engine conditions, braking system, tire pressure, and temperature.

[1597] (Claim 3)

[1598] The system of claim 1 , wherein the generative model uses a machine learning algorithm to detect fault precursors.

[1599] "Application example 2 when combining emotion engines"

[1600] (Claim 1)

[1601] means for receiving sensor data collected from a vehicle;

[1602] a data analysis means including a generative model that analyzes received sensor data in real time and detects signs of failure;

[1603] emotion recognition means for recognizing the user's emotional state and incorporating the data into the analysis results;

[1604] a notification means for selecting an appropriate notification method based on the detected failure sign and the user's emotional state, and notifying the driver;

[1605] a repair facility search means for searching for the nearest repair facility based on the vehicle location when a breakdown is predicted;

[1606] a reservation procedure means for performing a reservation procedure for the searched repair facility;

[1607] A system including:

[1608] (Claim 2)

[1609] The system of claim 1 , wherein the sensor data from the vehicle includes engine conditions, braking system, tire pressure, and temperature.

[1610] (Claim 3)

[1611] The system of claim 1 , wherein the generative model uses a machine learning algorithm to detect fault precursors. [Explanation of symbols]

[1612] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving sensor data collected from a vehicle; a data analysis means including a generative model that analyzes received sensor data in real time and detects signs of failure; a notification means for notifying a driver of the detected failure sign; a repair shop search means for searching for the nearest repair shop when a breakdown is expected; A reservation procedure means for making a reservation at the searched repair shop; A system including:

2. The system of claim 1 , wherein the sensor data from the vehicle includes engine conditions, braking system, tire pressure, and temperature.

3. The system of claim 1 , wherein the generative model uses a machine learning algorithm to detect fault precursors.

Images