System

Abstract

A system is provided.SOLUTION: A system, comprising: means for collecting data from a camera and a sensor mounted on a vehicle; means for analyzing the collected data by generative artificial intelligence and monitoring driving behavior; means for detecting dangerous driving based on an analysis result and sending a warning to a driver; means for recording a dangerous driving behavior and storing it in an evaluation system; and means for suspending an account of a driver who has accumulated a certain number of dangerous driving times based on the evaluation system.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The main reason ride-sharing services are not widespread in Japan is due to concerns about safety. Specifically, there is a risk of drivers with poor driving skills and inappropriate behavior in the car (sexual harassment, religious solicitation, etc.), which prevents ride-sharing services from gaining the trust of users. Another problem is the low level of hospitality provided to users, which prevents them from providing a comfortable ride-sharing experience. To resolve this situation and elevate Japan to a leading ride-sharing nation, it is necessary to provide safe and comfortable ride-sharing services. [Means for solving the problem]

[0005] The present invention solves the above problems by providing the following means: A means for collecting data from cameras and sensors mounted on vehicles is provided, and the collected data is analyzed using generative artificial intelligence to monitor driving behavior. Furthermore, a means for detecting dangerous driving based on the analysis results and sending a warning to the driver is provided, and a means for recording dangerous driving behavior and storing the information in a rating system is also provided. Based on this rating system, the account of a driver who has accumulated a certain number of dangerous driving incidents is temporarily suspended.

[0006] Furthermore, a means for collecting data from cameras and microphones in the vehicle is provided, and the collected data is analyzed using generative artificial intelligence to monitor inappropriate behavior and conversations in the vehicle. When inappropriate behavior or conversations are detected, a means for reporting them to an administrator is provided, and a means for suspending the driver's account when a report is made is also provided.

[0007] The system also includes a means for collecting profile information such as the user's age, gender, nationality, and destination, and a means for the AI ​​generator to recommend optimal conversation content and services based on the collected profile information. It also includes a means for presenting the recommended content to the driver and a means for the driver to provide services to the user based on the recommendations. In this way, the present invention realizes a safe and comfortable ride-sharing service.

[0008] A "camera" is a device for collecting visual data.

[0009] A "sensor" is a device that detects physical environmental information (e.g., vibration, speed, acceleration, etc.) and collects it as data.

[0010] "Data" is information collected by cameras and sensors.

[0011] "Generative artificial intelligence" is an algorithm or system that analyzes collected data and takes appropriate action based on the results.

[0012] "Driving behavior" refers to the pattern of operations (e.g., braking, accelerating, steering, etc.) performed by the driver.

[0013] "Dangerous driving" refers to driving behavior that deviates from safety, such as sudden braking, sudden acceleration, and swerving.

[0014] A "warning" is a notification sent to the driver when reckless driving is detected.

[0015] The "evaluation system" is a system that records the driver's driving behavior and evaluates it.

[0016] "Account suspension" is an operation that temporarily suspends the use of the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system.

[0017] An "in-vehicle camera" is a camera for collecting video data from inside a vehicle.

[0018] A "microphone" is a device for collecting voice data inside a vehicle.

[0019] "Inappropriate behavior" refers to sexual harassment, religious solicitation, and other unpleasant behavior that may occur while using a ride-sharing service.

[0020] "Reporting" is an operation that notifies an administrator when inappropriate behavior or conversation is detected.

[0021] A "user profile" is information such as the user's age, gender, nationality, and destination.

[0022] "Recommendation" means that generative artificial intelligence suggests optimal conversation content and services based on the user profile.

[0023] "Driver" means a person who provides ride-sharing services.

[0024] "Services" refers to the general benefits and support provided to rideshare users. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0026] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0029] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0030] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0031] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0033] [First embodiment]

[0034] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0035] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0036] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0038] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0041] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0043] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0044] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0045] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] Embodiment of operation monitoring function

[0047] In an embodiment of the driving monitoring function of the present invention, data is collected from cameras and various sensors mounted on the vehicle and analyzed by a generation AI. A server collects data in real time from a forward camera, speed sensor, acceleration sensor, etc. mounted on the vehicle. The collected data is analyzed by the server using a generation AI, and driving behavior is monitored.

[0048] For example, if a driver repeatedly brakes suddenly, the server will detect this driving pattern. The server will use generative AI to recognize the dangerous driving behavior and immediately send a warning message to the driver. Furthermore, this dangerous driving behavior will be recorded in the rating system and reflected in the driver's account rating. If a certain number of dangerous driving incidents are accumulated, the server will suspend the driver's account.

[0049] Embodiment of in-vehicle monitoring function

[0050] In an embodiment of the in-vehicle monitoring function of the present invention, video and audio data are collected from cameras and microphones installed in the vehicle and analyzed by generative AI. The server monitors the situation inside the vehicle and detects inappropriate behavior or remarks. For example, if a driver makes an inappropriate remark to a passenger, the server analyzes the audio data and detects that the remark is inappropriate. This data is immediately reported by the server to an administrator, and appropriate action can be taken, such as suspending the driver's account, if necessary.

[0051] For example, if a driver makes a sexually harassing remark to a passenger, the server analyzes the remark, determines it to be inappropriate, and reports it to an administrator. Once the administrator receives the report, the driver's account will be temporarily suspended.

[0052] Embodiment of hospitality function

[0053] In an embodiment of the hospitality function of the present invention, user profile information (e.g., age, gender, nationality, destination, etc.) is collected, and based on that information, a generation AI recommends optimal conversation content and services. The server collects the user's profile information and sends it to the terminal. The terminal uses the generation AI to recommend optimal conversation content and services based on this profile information and presents them to the driver.

[0054] For example, if the passenger is a tourist from abroad, the device will recommend to the driver "provide information about tourist spots in the area." The driver will then provide information about tourist spots to the passenger based on the recommendation, allowing the user to receive a comprehensive service.

[0055] As a result, the present invention provides a safe and comfortable ride-sharing service, increasing user trust and satisfaction. These functions work together to realize a consistent service as a whole.

[0056] The processing flow will be explained below.

[0057] Operation monitoring function processing steps

[0058] Step 1:

[0059] The server collects data in real time from the vehicle's front camera and various sensors.

[0060] Step 2:

[0061] The server sends the collected data to the generative AI model for analysis.

[0062] Step 3:

[0063] A generative AI model analyzes data and detects abnormal driving patterns such as hard braking, sudden acceleration, and swerving.

[0064] Step 4:

[0065] The server checks the analysis results of the driving patterns and sends a warning message to the driver if any dangerous driving behavior is detected.

[0066] Step 5:

[0067] The server records the detected risky driving behavior and stores it in a driver's rating system.

[0068] Step 6:

[0069] The server will suspend a driver's account if a certain number of dangerous driving incidents are accumulated.

[0070] In-vehicle monitoring function processing steps

[0071] Step 1:

[0072] The server collects video and audio data in real time from cameras and microphones installed inside the vehicle.

[0073] Step 2:

[0074] The server sends the collected data to the generative AI model.

[0075] Step 3:

[0076] A generative AI model analyzes data to identify inappropriate behavior and conversations in the car.

[0077] Step 4:

[0078] If the server detects any inappropriate behavior or conversation, it will immediately notify an administrator.

[0079] Step 5:

[0080] The server assists administrators who receive reports to take action such as suspending the driver's account.

[0081] Hospitality function processing steps

[0082] Step 1:

[0083] The server collects profile information such as the user's age, gender, nationality, and destination.

[0084] Step 2:

[0085] The server sends the collected profile information to the driver's terminal.

[0086] Step 3:

[0087] The device sends profile information to a generation AI, which then recommends the most suitable conversation content and services for the user.

[0088] Step 4:

[0089] The terminal then presents the generated recommendations to the driver.

[0090] Step 5:

[0091] Based on recommendations, drivers will provide users with the most appropriate services and conversations.

[0092] Example 1

[0093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] In conventional ride-sharing services, there is a need to improve the quality and safety of services due to the frequent occurrence of dangerous driving by drivers, inappropriate behavior in the vehicle, and a lack of proper hospitality for users. Furthermore, insufficient measures to address these issues have led to a decline in user satisfaction and reliability.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0096] In this invention, the server includes means for collecting data from cameras and sensors installed in the vehicle, means for analyzing the collected data using a generative AI model to monitor driving behavior, means for detecting dangerous driving based on the analysis results and sending a warning to the driver, means for recording dangerous driving behavior and saving it in a rating system, means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system, means for collecting data from cameras and microphones installed in the vehicle, means for analyzing the collected data using a generative AI model to monitor inappropriate behavior and conversation in the vehicle, means for notifying an administrator when inappropriate behavior or conversation is detected, means for collecting user profile information, means for the generative AI model to recommend optimal conversation content and services based on the collected profile information, means for presenting the recommended content to the driver, and means for the driver to provide services to the user based on the recommendations. This enables improved safety, appropriate behavior and hospitality in the vehicle, and improved overall service quality.

[0097] A "camera" is a device that detects light from an object and records the image as electronic data.

[0098] A "sensor" is a device that senses a physical quantity and converts it into an electrical signal to provide data.

[0099] A "generative AI model" is an artificial intelligence algorithm that uses machine learning and deep learning to analyze data and perform specific tasks.

[0100] "Driving behavior" refers to the driving patterns and actions of the vehicle driver, including speed, acceleration, braking, and the like.

[0101] A "warning message" is a message that notifies you when a specific condition or event occurs.

[0102] The "rating system" is a system for evaluating and recording the quality of drivers and services based on data.

[0103] "Suspending" an "account" is a process that temporarily stops the use of that account when certain conditions are met.

[0104] "Behavior" refers to the movement or behavior of a person or object in a particular situation.

[0105] "Conversational content" refers to words and sentences spoken in a specific context or situation.

[0106] "Omotenashi" refers to the act of providing service or hospitality with consideration and care by a service provider to a user.

[0107] "Profile information" refers to data including a user's personal information and attribute information, such as age, gender, nationality, and destination.

[0108] "Recommendation" refers to the act of suggesting specific information or services.

[0109] "Driver" means a person operating a vehicle.

[0110] "User" refers to a person who uses a particular service or system.

[0111] MODE FOR CARRYING OUT THE INVENTION

[0112] The present invention is a system for monitoring vehicle operation and in-vehicle conditions and providing appropriate services to drivers and users. This system uses a generative AI model to analyze collected data and take appropriate action based on specific conditions. The following describes in detail an embodiment of the present invention.

[0113] Embodiment of operation monitoring function

[0114] The server collects data in real time from the vehicle's cameras and sensors (forward camera, speed sensor, acceleration sensor). The collected data is sent to the generative AI model for analysis. Based on the analysis results, driving behavior is monitored, and if dangerous driving is detected, the server immediately sends a warning message to the driver. In addition, dangerous driving behavior is recorded in a reputation system, and if a certain number of such behaviors accumulate, the driver's account will be suspended.

[0115] As a concrete example, if the driver brakes suddenly repeatedly, the server will input the following prompt sentence into the generated AI:

[0116] "There have been frequent instances of sudden braking. Please be careful."

[0117] Embodiment of in-vehicle monitoring function

[0118] The server collects video and audio data from cameras and microphones installed in the vehicle. The collected data is analyzed by a generative AI model, and if inappropriate behavior or remarks are detected, a report is sent to an administrator. Furthermore, if inappropriate behavior or remarks are confirmed, the driver's account may be temporarily suspended.

[0119] As a specific example, if a driver makes an inappropriate remark to a passenger, the server will input the following prompt sentence into the generation AI:

[0120] "Inappropriate language detected."

[0121] Embodiment of hospitality function

[0122] The server collects the user's profile information (such as age, gender, nationality, and destination) and sends it to the device. The device then uses a generative AI model based on the collected profile information to recommend optimal conversation content and services, which are then presented to the driver. Based on these recommendations, the driver provides the user with appropriate services.

[0123] As a specific example, if the passenger is a tourist from abroad, the terminal inputs the following prompt sentence into the generation AI:

[0124] "Providing information about tourist spots in the area"

[0125] As a result, the present invention can provide a safe and comfortable ride-sharing service, increasing user trust and satisfaction. These functions work together to realize a consistent service as a whole.

[0126] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0127] Operation monitoring function processing steps

[0128] Step 1: Data collection

[0129] Input: Data from the vehicle's front camera, speed sensor, and acceleration sensor

[0130] Processing: The server collects data from these sensors in real time: video data from the camera, speed data from the speed sensor, and acceleration data from the accelerometer.

[0131] Output: Collected video data, velocity data, and acceleration data

[0132] Step 2: Send data

[0133] Input: Video data, velocity data, and acceleration data collected in step 1

[0134] Processing: The server sends these data to the generative AI model.

[0135] Output: Data sent to the generative AI model

[0136] Step 3: Data analysis

[0137] Input: Data sent in step 2

[0138] Processing: The generative AI model analyzes this data and monitors driving behavior, specifically recognizing visual driving patterns from the video data and matching them with speed and acceleration data to detect abnormal driving patterns.

[0139] Output: Analysis results (evaluation of driving behavior, identification of abnormal driving patterns)

[0140] Step 4: Sending a warning message

[0141] Input: Analysis results obtained in Step 3

[0142] Processing: The server sends a warning message to the driver based on the analysis results. Specifically, if abnormal driving such as sudden braking is detected, a warning message is generated.

[0143] Output: Warning message sent to the driver (e.g., "Frequent hard braking. Please be careful.")

[0144] Step 5: Record in the rating system

[0145] Input: Analysis results and warnings for Step 3 and Step 4

[0146] Processing: The server records this information in the rating system and reflects it in the driver's driving rating.

[0147] Output: Driving data and evaluation recorded in the evaluation system

[0148] Step 6: Suspend your account

[0149] Input: Number of dangerous driving incidents accumulated in the rating system

[0150] Action: If a certain number of dangerous driving incidents are accumulated, the server will suspend the driver's account.

[0151] Output: Information about the driver whose account was suspended

[0152] In-vehicle monitoring function processing steps

[0153] Step 1: Data collection

[0154] Input: Data from cameras and microphones installed inside the vehicle

[0155] Processing: The server collects video and audio data from these devices.

[0156] Output: Collected video and audio data

[0157] Step 2: Send data

[0158] Input: Data collected in Step 1

[0159] Processing: The server sends the collected data to the generative AI model.

[0160] Output: Data sent to the generative AI model

[0161] Step 3: Data analysis

[0162] Input: Data sent in step 2

[0163] Processing: Generative AI models analyze this data to detect inappropriate behavior and conversations.

[0164] Output: Analysis results (detection of inappropriate behavior or conversation)

[0165] Step 4: Report and Alert

[0166] Input: Analysis results obtained in Step 3

[0167] Processing: If the server detects inappropriate behavior or conversation, it will notify the administrator and send a warning message to the driver.

[0168] Output: Notification message to administrator, warning message to driver

[0169] Step 5: Suspend your account

[0170] Input: Report and warning details from Step 4

[0171] Action: Based on the administrator's instructions, the server suspends the driver's account.

[0172] Output: Information about the driver whose account was suspended

[0173] Hospitality function processing steps

[0174] Step 1: Collect profile information

[0175] Input: Information such as user's age, gender, nationality, and destination

[0176] Processing: The server collects this profile information.

[0177] Output: Collected profile information

[0178] Step 2: Submit your profile information

[0179] Input: Profile information collected in Step 1

[0180] Processing: The server sends the collected information to the device.

[0181] Output: Profile information sent to the device

[0182] Step 3: Data analysis and recommendation generation

[0183] Input: Profile information submitted in Step 2

[0184] Processing: The device uses a generative AI model based on the profile information to recommend the most appropriate conversation content and services.

[0185] Output: Recommendation content (e.g., tourist spot information)

[0186] Step 4: Present to the driver

[0187] Input: Recommendations generated in step 3

[0188] Processing: The device presents the recommendations to the driver, who then uses this information to provide the appropriate service to the user.

[0189] Output: Recommendations presented to the driver, service execution provided

[0190] Through the above steps, the present invention enables the provision of a safe and comfortable ride-sharing service by linking various monitoring functions with hospitality functions.

[0191] (Application example 1)

[0192] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0193] Conventional vehicle driving monitoring systems only monitor driving behavior and lack the ability to detect inappropriate behavior, conversations, and misconduct in the vehicle, as well as user-friendly features. As a result, safety and user satisfaction are not sufficiently improved. Furthermore, the accuracy of predictive driving warnings is limited, making it difficult to respond in real time. This calls for a comprehensive system that provides greater accuracy and added value.

[0194] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0195] In this invention, the server includes means for collecting data from cameras and sensors mounted on the vehicle, means for analyzing the collected data using generative artificial intelligence to monitor driving behavior, means for detecting dangerous driving based on the analysis results and sending a warning to the driver, means for recording dangerous driving behavior and saving the data in a rating system, means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system, means for analyzing video data from the camera using generative artificial intelligence, means for sending an alert to the server in the event of dangerous driving based on the analysis results of the video data, means for immediately issuing a warning when driving behavior exceeding a set threshold is detected, and means for recommending optimal conversation content and services based on the user's profile information.This enables comprehensive driving monitoring, improved safety, and even improved user satisfaction.

[0196] "Vehicle" means a means of transport designed to travel on roads, including those with automated driving capabilities.

[0197] A "camera" is an imaging device mounted on a vehicle to collect visual information.

[0198] A "sensor" is a device that detects physical information and collects it as data, and includes speed sensors, acceleration sensors, etc.

[0199] "Generative artificial intelligence" is a system that analyzes data using technologies such as machine learning and deep learning, and makes judgments and predictions based on the results.

[0200] "Driving behavior" refers to the driver's actions and vehicle movements when operating a vehicle, including braking, acceleration, steering, etc.

[0201] "Warning" refers to a notification or signal that alerts the driver to danger or caution, and is given by audio or visual means.

[0202] The "evaluation system" is a system that evaluates a driver's driving behavior based on collected driving data and records and manages the results.

[0203] An "account" is identification information used to identify a specific user on the system and manage their permissions and settings.

[0204] "Video data" refers to visual information captured by a camera and stored as digital data.

[0205] "Profile information" is data that includes personal information such as the user's age, gender, nationality, and destination.

[0206] "Recommendation" is the act of suggesting optimal conversation content or services based on a user's profile information and behavioral history.

[0207] An "alert" is a warning signal or message that immediately notifies you of danger or abnormality.

[0208] "Administrator" means a person or organization responsible for overseeing and managing the entire system.

[0209] In one embodiment of the present invention, the system collects data from cameras and sensors mounted on the vehicle and analyzes it using generative artificial intelligence (AI) to provide information about driving behavior, in-vehicle conditions, and services to the user.

[0210] System configuration

[0211] 1. Hardware Configuration

[0212] Camera: A device installed in front of or inside a vehicle to obtain visual information.

[0213] Sensors: Various sensors such as speed sensors and acceleration sensors are installed to collect physical operation data of the vehicle.

[0214] Microphone: A device for collecting voice data inside the vehicle.

[0215] Server: A computer equipped with powerful computing resources for performing data analysis.

[0216] 2. Software Configuration

[0217] Generative AI model (TensorFlow / Keras): A model that analyzes collected data and determines driving behavior and in-car activities.

[0218] Data collection API (requests library): An interface for collecting data from sensors and cameras and sending it to a server.

[0219] Data collection and analysis

[0220] The server collects real-time data from cameras and various sensors installed in the vehicle. This data includes video information of the area in front of the vehicle, speed information, acceleration information, and audio information from inside the vehicle. For example, video data from the camera is input into a generative artificial intelligence model to analyze driving behavior.

[0221] Dangerous driving detection

[0222] The generative AI model detects risky driving based on the analysis results. For example, if risky driving behavior such as sudden braking or sudden acceleration is detected, the server immediately sends a warning to the driver. These risky driving behaviors are also recorded in a rating system and saved as the driver's account rating. If a certain number of risky driving behaviors are accumulated, the server suspends the driver's account.

[0223] In-car monitoring

[0224] The server analyzes video and audio data collected from cameras and microphones installed in the vehicle. For example, if a driver makes inappropriate remarks to a passenger, the server analyzes the audio data and detects that the remarks are inappropriate. This data is then immediately reported to an administrator, and the driver's account is suspended if necessary.

[0225] Hospitality features

[0226] The server collects user profile information (e.g., age, gender, nationality, destination, etc.), and the generative AI uses this data to recommend the most appropriate conversation content and services. For example, if the passenger is a tourist from overseas, the server will recommend to the driver "information about tourist spots in the area." This allows the driver to provide information about tourist spots based on the recommendation, allowing the user to receive a comprehensive service.

[0227] Specific examples

[0228] An example of a specific prompt is, "The AI ​​model monitors driving behavior and immediately sends an alert if an abnormality is detected. In addition, voice analysis is used to detect inappropriate remarks and take appropriate action."

[0229] As a result, the present invention can provide a safe and comfortable ride-sharing service, increasing user trust and satisfaction.

[0230] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0231] Step 1:

[0232] The server collects data from the vehicle's cameras and sensors. Specifically, the camera collects visual information, the speed sensor and acceleration sensor collect physical movement data, and the microphone collects audio data. This data is collected in real time and sent to the server.

[0233] Step 2:

[0234] The server inputs the collected data into a pre-trained generative AI model. Specifically, visual data undergoes image processing, audio data undergoes voice analysis, and speed and acceleration data undergoes motion analysis. These analyses are used to analyze driving behavior and in-car conversations.

[0235] Step 3:

[0236] The server detects dangerous driving based on the analysis results of the generative AI model. Specifically, when dangerous driving behavior such as sudden braking or sudden acceleration is detected, the information is processed within the server. At this time, the analysis results are used as data to determine whether a warning is necessary.

[0237] Step 4:

[0238] If unsafe driving is detected, the server immediately sends a warning to the driver. The warning can be provided as an audio or visual signal to alert the driver, for example, a message such as "Sudden braking detected. Please be careful."

[0239] Step 5:

[0240] The server records risky driving behavior and stores it in a rating system. Data on driving behavior is managed within the rating system and accumulated as a driver's rating. This rating is managed based on the driver's account information.

[0241] Step 6:

[0242] Based on the rating system, the server will suspend the account of a driver who has accumulated a certain number of dangerous driving incidents. Based on the data from the rating system, the server will automatically suspend the account if the driver's driving behavior exceeds the standard.

[0243] Step 7:

[0244] The server analyzes the video and audio data collected from the camera and microphone inside the vehicle. Specifically, if the driver makes an inappropriate remark to a passenger, it is analyzed and recognized as inappropriate behavior. This data is immediately sent to the server.

[0245] Step 8:

[0246] The server will report any inappropriate comments detected to the administrator, who will then take action if necessary to temporarily suspend the driver's account.

[0247] Step 9:

[0248] The server collects user profile information and recommends optimal conversation topics and services based on that information. For example, if the user is a tourist, the server recommends that the driver provide tourist information. This process improves user satisfaction.

[0249] Step 10:

[0250] The server uses the generative AI model to present optimized conversation content and service recommendations to the driver in real time, allowing the driver to provide optimal service to the user.

[0251] As a result, the present invention makes it possible to provide a safe and comfortable ride-sharing service, thereby increasing user trust and satisfaction.

[0252] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0253] Embodiment of operation monitoring function

[0254] In an embodiment of the driving monitoring function of the present invention, data is collected from cameras and various sensors mounted on the vehicle and analyzed by a generation AI. A server collects data in real time from a forward camera, speed sensor, acceleration sensor, etc. mounted on the vehicle. The collected data is analyzed by the server using a generation AI, and driving behavior is monitored.

[0255] For example, if a driver repeatedly brakes suddenly, the server will detect this driving pattern. The server will use generative AI to recognize the dangerous driving behavior and immediately send a warning message to the driver. Furthermore, this dangerous driving behavior will be recorded in the rating system and reflected in the driver's account rating. If a certain number of dangerous driving incidents are accumulated, the server will suspend the driver's account.

[0256] Embodiment of in-vehicle monitoring function

[0257] In an embodiment of the in-vehicle monitoring function of the present invention, video and audio data are collected from cameras and microphones installed in the vehicle and analyzed by generative AI. The server monitors the situation inside the vehicle and detects inappropriate behavior or remarks. For example, if a driver makes an inappropriate remark to a passenger, the server analyzes the audio data and detects that the remark is inappropriate. This data is immediately reported by the server to an administrator, and appropriate action can be taken, such as suspending the driver's account, if necessary.

[0258] For example, if a driver makes a sexually harassing remark to a passenger, the server analyzes the remark, determines it to be inappropriate, and reports it to an administrator. Once the administrator receives the report, the driver's account will be temporarily suspended.

[0259] Embodiment of hospitality function

[0260] In an embodiment of the hospitality function of the present invention, user profile information (e.g., age, gender, nationality, destination, etc.) is collected, and based on that information, a generation AI recommends optimal conversation content and services. The server collects the user's profile information and sends it to the terminal. The terminal uses the generation AI to recommend optimal conversation content and services based on this profile information and presents them to the driver.

[0261] For example, if the passenger is a tourist from abroad, the device will recommend to the driver "provide information about tourist spots in the area." The driver will then provide information about tourist spots to the passenger based on the recommendation, allowing the user to receive a comprehensive service.

[0262] Embodiment of Emotion Engine

[0263] In an embodiment of the emotion engine of the present invention, the vehicle recognizes the user's emotions and provides appropriate feedback based on the user's emotions. The server collects data such as the user's facial expressions, voice tone, and speech content from the vehicle's cameras, microphones, and other sensors, and analyzes the data using the emotion engine. The emotion engine recognizes emotions such as stress, anxiety, and joy in real time based on the user data.

[0264] For example, if the emotion engine detects that the user is feeling stressed during the ride, the server will instruct the device to play relaxation music.The server will also recommend appropriate advice to the driver and encourage conversations that will help the user relax.

[0265] As a result, the present invention provides a safe and comfortable ride-sharing service, and increases user trust and satisfaction. These functions work together to realize a consistent service as a whole.

[0266] The processing flow will be explained below.

[0267] Emotion Engine Processing Steps

[0268] Step 1:

[0269] The server collects data in real time from cameras, microphones and other sensors installed inside the vehicle.

[0270] Step 2:

[0271] The server sends the collected data to an emotion engine, which analyzes the user's facial expressions, tone of voice, and content of their comments.

[0272] Step 3:

[0273] The emotion engine recognizes emotions (stress, anxiety, joy, etc.) in real time based on user data.

[0274] Step 4:

[0275] The server receives the user's emotion data analyzed by the emotion engine and generates appropriate feedback based on the analysis results.

[0276] Step 5:

[0277] If the emotion engine recognizes that the user is under stress, the server instructs the terminal to play relaxation music or provide advice to help the user relax.

[0278] Step 6:

[0279] Based on instructions from the server, the terminal plays relaxation music or presents appropriate advice or conversation recommendations to the driver.

[0280] Step 7:

[0281] The driver follows the advice and recommendations provided by the device, provides a relaxing conversation for the user, and provides a comfortable service.

[0282] In-vehicle monitoring function processing steps

[0283] Step 1:

[0284] The server collects video and audio data in real time from cameras and microphones installed inside the vehicle.

[0285] Step 2:

[0286] The server sends the collected data to the generative AI model.

[0287] Step 3:

[0288] A generative AI model analyzes data to identify inappropriate behavior and conversations in the car.

[0289] Step 4:

[0290] If the server detects any inappropriate behavior or conversation, it will immediately notify an administrator.

[0291] Step 5:

[0292] The server assists administrators who receive reports to take action such as suspending the driver's account.

[0293] Hospitality function processing steps

[0294] Step 1:

[0295] The server collects profile information such as the user's age, gender, nationality, and destination.

[0296] Step 2:

[0297] The server sends the collected profile information to the driver's terminal.

[0298] Step 3:

[0299] The device sends profile information to a generation AI, which then recommends the most suitable conversation content and services for the user.

[0300] Step 4:

[0301] The terminal then presents the generated recommendations to the driver.

[0302] Step 5:

[0303] Based on recommendations, drivers will provide users with the most appropriate services and conversations.

[0304] Example 2

[0305] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0306] Conventional driving monitoring systems and in-vehicle monitoring systems have had difficulty detecting inappropriate driving behavior or inappropriate behavior in the vehicle in real time and taking countermeasures. Furthermore, they have not been able to effectively provide services that respond to users' emotions and needs. This has led to problems such as an increased risk of accidents and a decrease in user satisfaction.

[0307] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting data from an image capturing device and a detection device mounted on the vehicle, means for analyzing the collected data by a generation algorithm and monitoring driving operations, means for detecting dangerous driving based on the analysis result and sending a warning to the driver, means for recording dangerous driving behavior and saving it in an evaluation system, means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the evaluation system, means for collecting data from an image capturing device and a sound collecting device in the vehicle, means for analyzing the collected data by a generation algorithm and monitoring inappropriate behavior or conversation in the vehicle, means for notifying an administrator when inappropriate behavior or conversation is detected, means for suspending the driver's account when a report is made, means for collecting attribute information of a user, means for the generation algorithm to recommend optimal conversation content or services based on the collected attribute information, means for presenting the recommended content to the driver, and means for the driver to provide a service to the user based on the recommendation. This will improve driving safety, enable proper monitoring of the in-vehicle environment, and enable the provision of services that will highly satisfy users.

[0308] An "image capture device" is a device that is mounted on a vehicle and is used to capture video data.

[0309] A "detection device" is a device for measuring physical quantities such as velocity and acceleration and acquiring data.

[0310] A "generative algorithm" refers to an artificial intelligence model used to analyze data and make predictions.

[0311] "Driving operations" refers to actions such as braking, accelerating, and steering by the driver.

[0312] "Dangerous driving" refers to unsafe driving behavior such as sudden braking or sudden acceleration.

[0313] An "evaluation system" refers to a system that evaluates drivers based on their past driving behavior.

[0314] The "voice collection device" is a device for acquiring voice data inside a vehicle.

[0315] "Inappropriate behavior" refers to inappropriate remarks or actions made in the car.

[0316] "Administrator" refers to the person in charge of monitoring and managing the entire system and taking appropriate action.

[0317] "Attribute information" refers to personal information such as the user's age, gender, nationality, and destination.

[0318] "Recommendations" refer to suggestions or advice that are optimized by a generative algorithm.

[0319] "Driver" means an individual who drives a vehicle.

[0320] "User" means an individual using a vehicle.

[0321] The present invention is a system that monitors driving operations and the in-vehicle environment by collecting data from image acquisition devices and detection devices mounted on vehicles and analyzing the data using a generation algorithm, and provides optimal services based on user attribute information. The program for this system is executed through multiple processing steps.

[0322] Hardware and software used

[0323] 1. Image capture device: A camera that collects video data from inside and outside the vehicle.

[0324] 2. Detection devices: Various sensors that measure vehicle behavior, such as speed sensors and acceleration sensors.

[0325] 3. Audio collection device: A microphone that collects audio data inside the vehicle.

[0326] 4. Server: A central processing unit that aggregates collected data and executes the generation algorithm.

[0327] 5. Generative algorithms: These are artificial intelligence models used for data analysis and prediction.

[0328] Program processing

[0329] The server collects data in real time from image capture devices and detection devices and analyzes the data using a generative algorithm. For example, it recognizes driving patterns based on video data acquired by a forward-facing camera and speed data acquired by an acceleration sensor, and detects dangerous driving such as sudden braking or sudden acceleration.

[0330] For example, if a driver repeatedly brakes suddenly, the server analyzes this driving pattern and flags it as dangerous driving. At this time, a warning message such as "You have repeatedly braked suddenly. Please drive carefully" is displayed on the driver's device. In addition, dangerous driving behavior is recorded in a rating system, and if a certain number of dangerous driving incidents are accumulated, the driver's account will be temporarily suspended.

[0331] In-car monitoring function

[0332] The server collects data from the in-car image capture and audio capture devices and analyzes it using a generative algorithm. If inappropriate behavior or conversations are detected in the car, it is reported to an administrator. The administrator will review the situation based on the report and suspend the driver's account if necessary.

[0333] For example, if a driver makes an inappropriate remark to a passenger, the server will analyze the voice data and determine that the remark is inappropriate. This information will be immediately reported to the administrator, and the driver's account will be suspended.

[0334] Hospitality features

[0335] The server collects user attribute information (such as age, gender, nationality, and destination), and based on that information, a generation algorithm recommends optimal conversation content and services. The recommendations are presented to the driver, who then provides the user with services based on them.

[0336] For example, if a passenger is a tourist from abroad, the server generates a recommendation such as "I'll show you around tourist spots in this area" based on the user's nationality and destination information. The driver follows the recommendation and provides the passenger with information about tourist spots.

[0337] Embodiment of Emotion Engine

[0338] The server collects data such as the user's facial expressions, voice tone, and speech content from the vehicle's image capture device, audio collection device, and other sensors, and analyzes it using an emotion engine to recognize the user's emotions, such as stress, anxiety, and joy, in real time and provide appropriate feedback based on that.

[0339] For example, if the emotion engine detects that the user is feeling stressed during the ride, the server will instruct the device to play relaxation music, recommend appropriate advice to the driver, and encourage conversations that will help the user relax.

[0340] Example prompt sentence:

[0341] "Please explain how the system analyzes the user's emotions and provides appropriate feedback."

[0342] This system is expected to provide a safe and comfortable driving experience and increase user satisfaction.

[0343] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0344] Embodiment of operation monitoring function

[0345] Processing Steps

[0346] Step 1:

[0347] The server collects data in real time from image capture devices and detection devices. The inputs for this step are video data from cameras and numerical data from speed sensors and acceleration sensors. Specifically, the server acquires this data and prepares it for analysis.

[0348] Step 2:

[0349] The server inputs the collected data into a generation algorithm. The output of this step is the processed data required for analysis. In this step, for example, video data is converted into an analyzable form for each frame, and velocity and acceleration data are processed as time series data.

[0350] Step 3:

[0351] The server analyzes the data using a generative algorithm to recognize driving behavior patterns. The input to this step is a series of frames of data, and the output is the extracted driving behavior patterns. Specifically, behaviors such as sudden braking and sudden acceleration are recognized at this stage.

[0352] Step 4:

[0353] The server detects dangerous driving based on the analysis results. The input of this step is the data analysis results, and the output is the detection result of dangerous driving. In this step, for example, repeated sudden braking is identified as dangerous.

[0354] Step 5:

[0355] The server sends a warning message to the driver based on the detection result of dangerous driving. The input of this step is the detection result of dangerous driving, and the output is a warning message. Specifically, the message "There have been repeated sudden braking. Please drive carefully" is displayed on the driver's device.

[0356] Step 6:

[0357] The server records the risky driving behavior in the rating system and stores the data. The input of this step is the risky driving detection data, and the output is an updated rating record. Specifically, the driver's risky driving behavior is recorded in the database.

[0358] Step 7:

[0359] The server suspends the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system. The input of this step is the accumulated rating data, and the output is a change in the account status. The specific operation is to suspend the driver's account.

[0360] Embodiment of in-vehicle monitoring function

[0361] Processing Steps

[0362] Step 1:

[0363] The server collects data from the image capture device and audio collection device inside the vehicle. The inputs in this step are video and audio data from the camera and microphone. Specifically, conversations and actions inside the vehicle are recorded.

[0364] Step 2:

[0365] The server then inputs the collected data into a generative algorithm for analysis. The output of this step is the processed data required for analysis. Specifically, the audio data is converted into text and calculations are performed to identify inappropriate behavior or comments.

[0366] Step 3:

[0367] The server analyzes the data using a generative algorithm to detect inappropriate behavior or conversation. The input to this step is the analysis results, and the output is the detection of inappropriate behavior or conversation. For example, statements identified as "sexual harassment" are detected.

[0368] Step 4:

[0369] The server reports to the administrator based on the results of the detection of inappropriate behavior or conversation. The input of this step is the detection result of inappropriate behavior or conversation, and the output is a report message. Specifically, a notification saying "The driver made an inappropriate remark" is sent to the administrator's terminal.

[0370] Step 5:

[0371] The server suspends the driver's account when a report is made. The input of this step is the administrator's decision, and the output is a change in the account status. Specifically, the driver's account is suspended.

[0372] Embodiment of hospitality function

[0373] Processing Steps

[0374] Step 1:

[0375] The server collects user attribute information. The input for this step is the user's profile data, and the output is the collected attribute information. Specifically, data entered by the user, such as age, nationality, and destination, is obtained.

[0376] Step 2:

[0377] The server executes a generation algorithm based on the collected attribute information. The output of this step is the analyzed recommendation data. Specifically, calculations are performed to generate recommendation content from the attribute information.

[0378] Step 3:

[0379] The server uses the generation algorithm to recommend optimal conversation topics and services. The input to this step is the analysis results, and the output is the recommended content. For example, a recommendation such as "Please recommend some tourist spots" may be generated.

[0380] Step 4:

[0381] The server presents the recommendation content to the driver. The input of this step is the recommendation content, and the output is the information to be presented. Specifically, a message such as "Would you like me to show you tourist spots in this area?" is displayed on the driver's terminal.

[0382] Step 5:

[0383] The driver provides the user with a service based on the recommendations. The input of this step is the presented information, and the output is the provided service. Specifically, the driver provides the user with information about tourist spots.

[0384] Embodiment of Emotion Engine

[0385] Processing Steps

[0386] Step 1:

[0387] The server collects user data from the vehicle's image capture device, audio capture device, and other sensors. The inputs for this step include facial expressions, voice tone, and speech content. Specifically, facial expressions are captured by the camera and audio is recorded by the microphone.

[0388] Step 2:

[0389] The server then inputs the collected data into a generative algorithm for analysis. The output of this step is the processed data required for analysis. Specifically, facial expression data is converted into an analyzable format, and voice tones are analyzed.

[0390] Step 3:

[0391] The server analyzes the data using an emotion engine to recognize the user's emotion. The input of this step is the analysis result, and the output is the emotion recognition result. For example, it recognizes that the user is in a stressful state.

[0392] Step 4:

[0393] The server provides appropriate feedback based on the emotion recognition results. The input of this step is the emotion recognition results, and the output is the feedback content. Specifically, an instruction to play relaxation music is sent to the terminal.

[0394] Step 5:

[0395] The server recommends appropriate advice to the driver and encourages conversation that will help the user relax. The input to this step is the emotion recognition result and feedback content, and the output is a recommendation to the driver. For example, advice such as "The user is feeling stressed. Please have a relaxing conversation" is provided to the driver.

[0396] (Application example 2)

[0397] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0398] Conventional driving monitoring systems and in-car monitoring systems focus on detecting dangerous driving and inappropriate behavior, but lack feedback on the emotions and psychological states of drivers and passengers. Furthermore, inappropriate comments are often not detected and reported in real time, making it difficult to respond immediately. As a result, improving the quality of service provided by drivers and increasing passenger comfort remains a challenge.

[0399] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from cameras and sensors mounted on the vehicle, means for analyzing the collected data using generative artificial intelligence to monitor driving behavior, means for detecting reckless driving based on the analysis results and sending a warning to the driver, means for recording reckless driving behavior and saving the recorded data in a rating system, means for suspending the account of a driver who has accumulated a certain number of reckless driving incidents based on the rating system, means for recognizing emotions in the vehicle, means for providing appropriate feedback based on the recognized emotions, and means for analyzing voice data and detecting inappropriate remarks. This not only monitors reckless driving and inappropriate behavior, but also provides feedback based on the emotions of the driver and passengers, improving service quality and ensuring a comfortable and safe riding experience.

[0400] A "camera" is a device that captures video data and is used to record and distribute the situation inside and outside a vehicle in real time.

[0401] A "sensor" is a device that measures a physical phenomenon and outputs it as a digital or analog signal, and is used to collect data such as speed, acceleration, and impact.

[0402] "Generative AI" is a technology that analyzes and predicts based on large amounts of data, and is an algorithm that automatically performs various processes such as recognizing driving behavior and emotions and detecting inappropriate remarks.

[0403] "Driving behavior" refers to a series of actions and behaviors related to driving a vehicle, including speed changes, braking operations, lane changes, etc.

[0404] "Warnings" are notifications or messages sent to inform drivers of unsafe behavior or conditions and encourage safe driving.

[0405] The "evaluation system" is a system that evaluates and records the behavior of drivers and passengers based on collected data and manages the evaluation results.

[0406] "Emotion recognition" is a technology that uses in-car cameras and microphones to analyze the facial expressions and tone of voice of passengers and estimate their emotional state.

[0407] "Feedback" refers to responses or advice provided to drivers and passengers based on the results of emotion recognition, including playing relaxing music or recommending conversations.

[0408] "Inappropriate remarks" refer to inappropriate behavior such as sexual harassment or discriminatory content that occurs inside the vehicle, and are detected from audio data analyzed using artificial intelligence.

[0409] "Reporting" is the act of immediately notifying an administrator when inappropriate behavior or remarks are detected, and signals a situation that requires appropriate action.

[0410] The system of the present invention includes several key functions to ensure safe and comfortable operation of autonomous vehicles. These include driving monitoring, in-vehicle monitoring, hospitality, and emotion recognition. The system collects data from cameras and sensors mounted on the vehicle and analyzes it using generative artificial intelligence (generative AI) to provide various services and feedback.

[0411] Embodiment of operation monitoring function

[0412] The server collects data in real time from cameras and sensors installed in the vehicle (e.g., forward-facing cameras, speed sensors, and acceleration sensors). This data is analyzed by generative AI to monitor driving behavior. For example, if repeated sudden braking is detected, the server recognizes this driving pattern as dangerous and sends a warning message to the driver. This dangerous driving behavior is recorded in the rating system and reflected in the driver's account rating. If a certain number of dangerous driving incidents are accumulated, the server suspends the driver's account.

[0413] Embodiment of in-vehicle monitoring function

[0414] The server collects video and audio data from cameras and microphones installed inside the vehicle. It uses generative artificial intelligence to analyze this data and detect inappropriate behavior or speech inside the vehicle. For example, if a driver makes inappropriate remarks to a passenger, the server analyzes the audio data and detects that the remarks are inappropriate. This data is reported to administrators in real time, and the driver's account is suspended if necessary.

[0415] Embodiment of hospitality function

[0416] The server collects profile information such as the user's age, gender, nationality, and destination, and uses generative AI to recommend the most appropriate conversation topics and services. For example, if the passenger is a foreign tourist, the server might recommend "information about tourist spots in the area." The driver provides services to the passenger based on these recommendations, allowing the user to receive a comprehensive service.

[0417] Embodiment of Emotion Recognition Function

[0418] The server collects data from the vehicle's cameras, microphones, and other sensors, and uses an emotion engine to analyze the user's facial expressions, voice tone, and speech. If the emotion engine detects that the user is feeling stressed during the ride, the server instructs the device to play relaxation music. The server also recommends appropriate advice to the driver and encourages conversations that will help the user relax.

[0419] Hardware and software used

[0420] This system is implemented using the following hardware and software:

[0421] Cameras and microphones: Devices installed in vehicles to collect video and audio data.

[0422] Generative AI model: A model using TensorFlow / Keras that performs emotion recognition and driving pattern recognition.

[0423] Server: Collects data, analyzes it, sends warning messages, and provides feedback.

[0424] Rating system: A system that evaluates and records the behavior of drivers and passengers based on collected data.

[0425] This allows the system to monitor and provide feedback in real time, significantly improving safety and comfort inside the vehicle.

[0426] Specific examples

[0427] Example prompt sentence:

[0428] If a user feels stressed while watching a movie, the emotion engine is designed to analyze that stress and provide appropriate touch feedback or music. If a user expresses gratitude, the engine will analyze the utterance and generate a corresponding response.

[0429] In this way, combining emotion recognition with adaptive feedback can improve the quality of the user experience.

[0430] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0431] Step 1:

[0432] The server collects data in real time from the cameras and sensors installed in the vehicle. The data includes various information such as video, speed, and acceleration. The data from the cameras and sensors is input, and this data is stored on the server as output.

[0433] Step 2:

[0434] The server inputs the collected data into a generative AI model to analyze driving behavior and the situation inside the vehicle. The generative AI model is trained using TensorFlow / Keras and analyzes driving patterns and driver behavior from video data, and speech content from audio data. The input is data collected from cameras and sensors, and the output is the analysis results.

[0435] Step 3:

[0436] The server detects dangerous driving and inappropriate behavior based on the analysis results of the generative AI model. For example, if sudden braking or unreasonable lane changes are detected, this is recognized as dangerous driving. It also detects inappropriate remarks in the voice data. The input is the analysis results, and the output is the detection results of dangerous driving and inappropriate behavior.

[0437] Step 4:

[0438] The server sends a warning message to the driver based on the detection results. If inappropriate behavior is detected, it also notifies the administrator. The contents of the warning message and the report are displayed on the driver's and administrator's devices. The input is the detection result, and the output is the warning message and the report.

[0439] Step 5:

[0440] The server stores records of dangerous driving and inappropriate behavior in a rating system and updates the driver's rating. If a certain number of dangerous driving incidents are accumulated, the server suspends the driver's account. The input is the detection results and rating system data, and the output is the updated rating system data and a notification of account suspension.

[0441] Step 6:

[0442] The server collects user profile information (such as age, gender, nationality, and destination) and uses a generative AI model to recommend optimal conversation topics and services. The recommendations are presented to the driver via their device. The input is the user profile information, and the output is the recommendations themselves.

[0443] Step 7:

[0444] When using the emotion recognition function, the server collects the user's facial expressions, voice tone, and speech content from the in-car camera and microphone, and analyzes them using the emotion engine. It recognizes emotions such as stress, anxiety, and joy, and generates appropriate feedback based on the input data. It sends commands to play relaxation music or recommend conversations that will help the driver relax. The input is data from the camera and microphone, and the output is feedback and recommendation commands.

[0445] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0446] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0447] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0448] [Second embodiment]

[0449] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0450] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0451] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0452] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0453] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0454] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0455] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0456] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0457] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0458] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0459] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0460] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0461] Embodiment of operation monitoring function

[0462] In an embodiment of the driving monitoring function of the present invention, data is collected from cameras and various sensors mounted on the vehicle and analyzed by a generation AI. A server collects data in real time from a forward camera, speed sensor, acceleration sensor, etc. mounted on the vehicle. The collected data is analyzed by the server using a generation AI, and driving behavior is monitored.

[0463] For example, if a driver repeatedly brakes suddenly, the server will detect this driving pattern. The server will use generative AI to recognize the dangerous driving behavior and immediately send a warning message to the driver. Furthermore, this dangerous driving behavior will be recorded in the rating system and reflected in the driver's account rating. If a certain number of dangerous driving incidents are accumulated, the server will suspend the driver's account.

[0464] Embodiment of in-vehicle monitoring function

[0465] In an embodiment of the in-vehicle monitoring function of the present invention, video and audio data are collected from cameras and microphones installed in the vehicle and analyzed by generative AI. The server monitors the situation inside the vehicle and detects inappropriate behavior or remarks. For example, if a driver makes an inappropriate remark to a passenger, the server analyzes the audio data and detects that the remark is inappropriate. This data is immediately reported by the server to an administrator, and appropriate action can be taken, such as suspending the driver's account, if necessary.

[0466] For example, if a driver makes a sexually harassing remark to a passenger, the server analyzes the remark, determines it to be inappropriate, and reports it to an administrator. Once the administrator receives the report, the driver's account will be temporarily suspended.

[0467] Embodiment of hospitality function

[0468] In an embodiment of the hospitality function of the present invention, user profile information (e.g., age, gender, nationality, destination, etc.) is collected, and based on that information, a generation AI recommends optimal conversation content and services. The server collects the user's profile information and sends it to the terminal. The terminal uses the generation AI to recommend optimal conversation content and services based on this profile information and presents them to the driver.

[0469] For example, if the passenger is a tourist from abroad, the device will recommend to the driver "provide information about tourist spots in the area." The driver will then provide information about tourist spots to the passenger based on the recommendation, allowing the user to receive a comprehensive service.

[0470] As a result, the present invention provides a safe and comfortable ride-sharing service, increasing user trust and satisfaction. These functions work together to realize a consistent service as a whole.

[0471] The processing flow will be explained below.

[0472] Operation monitoring function processing steps

[0473] Step 1:

[0474] The server collects data in real time from the vehicle's front camera and various sensors.

[0475] Step 2:

[0476] The server sends the collected data to the generative AI model for analysis.

[0477] Step 3:

[0478] A generative AI model analyzes data and detects abnormal driving patterns such as hard braking, sudden acceleration, and swerving.

[0479] Step 4:

[0480] The server checks the analysis results of the driving patterns and sends a warning message to the driver if any dangerous driving behavior is detected.

[0481] Step 5:

[0482] The server records the detected risky driving behavior and stores it in a driver's rating system.

[0483] Step 6:

[0484] The server will suspend a driver's account if a certain number of dangerous driving incidents are accumulated.

[0485] In-vehicle monitoring function processing steps

[0486] Step 1:

[0487] The server collects video and audio data in real time from cameras and microphones installed inside the vehicle.

[0488] Step 2:

[0489] The server sends the collected data to the generative AI model.

[0490] Step 3:

[0491] A generative AI model analyzes data to identify inappropriate behavior and conversations in the car.

[0492] Step 4:

[0493] If the server detects any inappropriate behavior or conversation, it will immediately notify an administrator.

[0494] Step 5:

[0495] The server assists administrators who receive reports to take action such as suspending the driver's account.

[0496] Hospitality function processing steps

[0497] Step 1:

[0498] The server collects profile information such as the user's age, gender, nationality, and destination.

[0499] Step 2:

[0500] The server sends the collected profile information to the driver's terminal.

[0501] Step 3:

[0502] The device sends profile information to a generation AI, which then recommends the most suitable conversation content and services for the user.

[0503] Step 4:

[0504] The terminal then presents the generated recommendations to the driver.

[0505] Step 5:

[0506] Based on recommendations, drivers will provide users with the most appropriate services and conversations.

[0507] Example 1

[0508] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0509] In conventional ride-sharing services, there is a need to improve the quality and safety of services due to the frequent occurrence of dangerous driving by drivers, inappropriate behavior in the vehicle, and a lack of proper hospitality for users. Furthermore, insufficient measures to address these issues have led to a decline in user satisfaction and reliability.

[0510] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0511] In this invention, the server includes means for collecting data from cameras and sensors installed in the vehicle, means for analyzing the collected data using a generative AI model to monitor driving behavior, means for detecting dangerous driving based on the analysis results and sending a warning to the driver, means for recording dangerous driving behavior and saving it in a rating system, means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system, means for collecting data from cameras and microphones installed in the vehicle, means for analyzing the collected data using a generative AI model to monitor inappropriate behavior and conversation in the vehicle, means for notifying an administrator when inappropriate behavior or conversation is detected, means for collecting user profile information, means for the generative AI model to recommend optimal conversation content and services based on the collected profile information, means for presenting the recommended content to the driver, and means for the driver to provide services to the user based on the recommendations. This enables improved safety, appropriate behavior and hospitality in the vehicle, and improved overall service quality.

[0512] A "camera" is a device that detects light from an object and records the image as electronic data.

[0513] A "sensor" is a device that senses a physical quantity and converts it into an electrical signal to provide data.

[0514] A "generative AI model" is an artificial intelligence algorithm that uses machine learning and deep learning to analyze data and perform specific tasks.

[0515] "Driving behavior" refers to the driving patterns and actions of the vehicle driver, including speed, acceleration, braking, and the like.

[0516] A "warning message" is a message that notifies you when a specific condition or event occurs.

[0517] The "rating system" is a system for evaluating and recording the quality of drivers and services based on data.

[0518] "Suspending" an "account" is a process that temporarily stops the use of that account when certain conditions are met.

[0519] "Behavior" refers to the movement or behavior of a person or object in a particular situation.

[0520] "Conversational content" refers to words and sentences spoken in a specific context or situation.

[0521] "Omotenashi" refers to the act of providing service or hospitality with consideration and care by a service provider to a user.

[0522] "Profile information" refers to data including a user's personal information and attribute information, such as age, gender, nationality, and destination.

[0523] "Recommendation" refers to the act of suggesting specific information or services.

[0524] "Driver" means a person operating a vehicle.

[0525] "User" refers to a person who uses a particular service or system.

[0526] MODE FOR CARRYING OUT THE INVENTION

[0527] The present invention is a system for monitoring vehicle operation and in-vehicle conditions and providing appropriate services to drivers and users. This system uses a generative AI model to analyze collected data and take appropriate action based on specific conditions. The following describes in detail an embodiment of the present invention.

[0528] Embodiment of operation monitoring function

[0529] The server collects data in real time from the vehicle's cameras and sensors (forward camera, speed sensor, acceleration sensor). The collected data is sent to the generative AI model for analysis. Based on the analysis results, driving behavior is monitored, and if dangerous driving is detected, the server immediately sends a warning message to the driver. In addition, dangerous driving behavior is recorded in a reputation system, and if a certain number of such behaviors accumulate, the driver's account will be suspended.

[0530] As a concrete example, if the driver brakes suddenly repeatedly, the server will input the following prompt sentence into the generated AI:

[0531] "There have been frequent instances of sudden braking. Please be careful."

[0532] Embodiment of in-vehicle monitoring function

[0533] The server collects video and audio data from cameras and microphones installed in the vehicle. The collected data is analyzed by a generative AI model, and if inappropriate behavior or remarks are detected, a report is sent to an administrator. Furthermore, if inappropriate behavior or remarks are confirmed, the driver's account may be temporarily suspended.

[0534] As a specific example, if a driver makes an inappropriate remark to a passenger, the server will input the following prompt sentence into the generation AI:

[0535] "Inappropriate language detected."

[0536] Embodiment of hospitality function

[0537] The server collects the user's profile information (such as age, gender, nationality, and destination) and sends it to the device. The device then uses a generative AI model based on the collected profile information to recommend optimal conversation content and services, which are then presented to the driver. Based on these recommendations, the driver provides the user with appropriate services.

[0538] As a specific example, if the passenger is a tourist from abroad, the terminal inputs the following prompt sentence into the generation AI:

[0539] "Providing information about tourist spots in the area"

[0540] As a result, the present invention can provide a safe and comfortable ride-sharing service, increasing user trust and satisfaction. These functions work together to realize a consistent service as a whole.

[0541] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0542] Operation monitoring function processing steps

[0543] Step 1: Data collection

[0544] Input: Data from the vehicle's front camera, speed sensor, and acceleration sensor

[0545] Processing: The server collects data from these sensors in real time: video data from the camera, speed data from the speed sensor, and acceleration data from the accelerometer.

[0546] Output: Collected video data, velocity data, and acceleration data

[0547] Step 2: Send data

[0548] Input: Video data, velocity data, and acceleration data collected in step 1

[0549] Processing: The server sends these data to the generative AI model.

[0550] Output: Data sent to the generative AI model

[0551] Step 3: Data analysis

[0552] Input: Data sent in step 2

[0553] Processing: The generative AI model analyzes this data and monitors driving behavior, specifically recognizing visual driving patterns from the video data and matching them with speed and acceleration data to detect abnormal driving patterns.

[0554] Output: Analysis results (evaluation of driving behavior, identification of abnormal driving patterns)

[0555] Step 4: Sending a warning message

[0556] Input: Analysis results obtained in Step 3

[0557] Processing: The server sends a warning message to the driver based on the analysis results. Specifically, if abnormal driving such as sudden braking is detected, a warning message is generated.

[0558] Output: Warning message sent to the driver (e.g., "Frequent hard braking. Please be careful.")

[0559] Step 5: Record in the rating system

[0560] Input: Analysis results and warnings for Step 3 and Step 4

[0561] Processing: The server records this information in the rating system and reflects it in the driver's driving rating.

[0562] Output: Driving data and evaluation recorded in the evaluation system

[0563] Step 6: Suspend your account

[0564] Input: Number of dangerous driving incidents accumulated in the rating system

[0565] Action: If a certain number of dangerous driving incidents are accumulated, the server will suspend the driver's account.

[0566] Output: Information about the driver whose account was suspended

[0567] In-vehicle monitoring function processing steps

[0568] Step 1: Data collection

[0569] Input: Data from cameras and microphones installed inside the vehicle

[0570] Processing: The server collects video and audio data from these devices.

[0571] Output: Collected video and audio data

[0572] Step 2: Send data

[0573] Input: Data collected in Step 1

[0574] Processing: The server sends the collected data to the generative AI model.

[0575] Output: Data sent to the generative AI model

[0576] Step 3: Data analysis

[0577] Input: Data sent in step 2

[0578] Processing: Generative AI models analyze this data to detect inappropriate behavior and conversations.

[0579] Output: Analysis results (detection of inappropriate behavior or conversation)

[0580] Step 4: Report and Alert

[0581] Input: Analysis results obtained in Step 3

[0582] Processing: If the server detects inappropriate behavior or conversation, it will notify the administrator and send a warning message to the driver.

[0583] Output: Notification message to administrator, warning message to driver

[0584] Step 5: Suspend your account

[0585] Input: Report and warning details from Step 4

[0586] Action: Based on the administrator's instructions, the server suspends the driver's account.

[0587] Output: Information about the driver whose account was suspended

[0588] Hospitality function processing steps

[0589] Step 1: Collect profile information

[0590] Input: Information such as user's age, gender, nationality, and destination

[0591] Processing: The server collects this profile information.

[0592] Output: Collected profile information

[0593] Step 2: Submit your profile information

[0594] Input: Profile information collected in Step 1

[0595] Processing: The server sends the collected information to the device.

[0596] Output: Profile information sent to the device

[0597] Step 3: Data analysis and recommendation generation

[0598] Input: Profile information submitted in Step 2

[0599] Processing: The device uses a generative AI model based on the profile information to recommend the most appropriate conversation content and services.

[0600] Output: Recommendation content (e.g., tourist spot information)

[0601] Step 4: Present to the driver

[0602] Input: Recommendations generated in step 3

[0603] Processing: The device presents the recommendations to the driver, who then uses this information to provide the appropriate service to the user.

[0604] Output: Recommendations presented to the driver, service execution provided

[0605] Through the above steps, the present invention enables the provision of a safe and comfortable ride-sharing service by linking various monitoring functions with hospitality functions.

[0606] (Application example 1)

[0607] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0608] Conventional vehicle driving monitoring systems only monitor driving behavior and lack the ability to detect inappropriate behavior, conversations, and misconduct in the vehicle, as well as user-friendly features. As a result, safety and user satisfaction are not sufficiently improved. Furthermore, the accuracy of predictive driving warnings is limited, making it difficult to respond in real time. This calls for a comprehensive system that provides greater accuracy and added value.

[0609] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0610] In this invention, the server includes means for collecting data from cameras and sensors mounted on the vehicle, means for analyzing the collected data using generative artificial intelligence to monitor driving behavior, means for detecting dangerous driving based on the analysis results and sending a warning to the driver, means for recording dangerous driving behavior and saving the data in a rating system, means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system, means for analyzing video data from the camera using generative artificial intelligence, means for sending an alert to the server in the event of dangerous driving based on the analysis results of the video data, means for immediately issuing a warning when driving behavior exceeding a set threshold is detected, and means for recommending optimal conversation content and services based on the user's profile information.This enables comprehensive driving monitoring, improved safety, and even improved user satisfaction.

[0611] "Vehicle" means a means of transport designed to travel on roads, including those with automated driving capabilities.

[0612] A "camera" is an imaging device mounted on a vehicle to collect visual information.

[0613] A "sensor" is a device that detects physical information and collects it as data, and includes speed sensors, acceleration sensors, etc.

[0614] "Generative artificial intelligence" is a system that analyzes data using technologies such as machine learning and deep learning, and makes judgments and predictions based on the results.

[0615] "Driving behavior" refers to the driver's actions and vehicle movements when operating a vehicle, including braking, acceleration, steering, etc.

[0616] "Warning" refers to a notification or signal that alerts the driver to danger or caution, and is given by audio or visual means.

[0617] The "evaluation system" is a system that evaluates a driver's driving behavior based on collected driving data and records and manages the results.

[0618] An "account" is identification information used to identify a specific user on the system and manage their permissions and settings.

[0619] "Video data" refers to visual information captured by a camera and stored as digital data.

[0620] "Profile information" is data that includes personal information such as the user's age, gender, nationality, and destination.

[0621] "Recommendation" is the act of suggesting optimal conversation content or services based on a user's profile information and behavioral history.

[0622] An "alert" is a warning signal or message that immediately notifies you of danger or abnormality.

[0623] "Administrator" means a person or organization responsible for overseeing and managing the entire system.

[0624] In one embodiment of the present invention, the system collects data from cameras and sensors mounted on the vehicle and analyzes it using generative artificial intelligence (AI) to provide information about driving behavior, in-vehicle conditions, and services to the user.

[0625] System configuration

[0626] 1. Hardware Configuration

[0627] Camera: A device installed in front of or inside a vehicle to obtain visual information.

[0628] Sensors: Various sensors such as speed sensors and acceleration sensors are installed to collect physical operation data of the vehicle.

[0629] Microphone: A device for collecting voice data inside the vehicle.

[0630] Server: A computer equipped with powerful computing resources for performing data analysis.

[0631] 2. Software Configuration

[0632] Generative AI model (TensorFlow / Keras): A model that analyzes collected data and determines driving behavior and in-car activities.

[0633] Data collection API (requests library): An interface for collecting data from sensors and cameras and sending it to a server.

[0634] Data collection and analysis

[0635] The server collects real-time data from cameras and various sensors installed in the vehicle. This data includes video information of the area in front of the vehicle, speed information, acceleration information, and audio information from inside the vehicle. For example, video data from the camera is input into a generative artificial intelligence model to analyze driving behavior.

[0636] Dangerous driving detection

[0637] The generative AI model detects risky driving based on the analysis results. For example, if risky driving behavior such as sudden braking or sudden acceleration is detected, the server immediately sends a warning to the driver. These risky driving behaviors are also recorded in a rating system and saved as the driver's account rating. If a certain number of risky driving behaviors are accumulated, the server suspends the driver's account.

[0638] In-car monitoring

[0639] The server analyzes video and audio data collected from cameras and microphones installed in the vehicle. For example, if a driver makes inappropriate remarks to a passenger, the server analyzes the audio data and detects that the remarks are inappropriate. This data is then immediately reported to an administrator, and the driver's account is suspended if necessary.

[0640] Hospitality features

[0641] The server collects user profile information (e.g., age, gender, nationality, destination, etc.), and the generative AI uses this data to recommend the most appropriate conversation content and services. For example, if the passenger is a tourist from overseas, the server will recommend to the driver "information about tourist spots in the area." This allows the driver to provide information about tourist spots based on the recommendation, allowing the user to receive a comprehensive service.

[0642] Specific examples

[0643] An example of a specific prompt is, "The AI ​​model monitors driving behavior and immediately sends an alert if an abnormality is detected. In addition, voice analysis is used to detect inappropriate remarks and take appropriate action."

[0644] As a result, the present invention can provide a safe and comfortable ride-sharing service, increasing user trust and satisfaction.

[0645] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0646] Step 1:

[0647] The server collects data from the vehicle's cameras and sensors. Specifically, the camera collects visual information, the speed sensor and acceleration sensor collect physical movement data, and the microphone collects audio data. This data is collected in real time and sent to the server.

[0648] Step 2:

[0649] The server inputs the collected data into a pre-trained generative AI model. Specifically, visual data undergoes image processing, audio data undergoes voice analysis, and speed and acceleration data undergoes motion analysis. These analyses are used to analyze driving behavior and in-car conversations.

[0650] Step 3:

[0651] The server detects dangerous driving based on the analysis results of the generative AI model. Specifically, when dangerous driving behavior such as sudden braking or sudden acceleration is detected, the information is processed within the server. At this time, the analysis results are used as data to determine whether a warning is necessary.

[0652] Step 4:

[0653] If unsafe driving is detected, the server immediately sends a warning to the driver. The warning can be provided as an audio or visual signal to alert the driver, for example, a message such as "Sudden braking detected. Please be careful."

[0654] Step 5:

[0655] The server records risky driving behavior and stores it in a rating system. Data on driving behavior is managed within the rating system and accumulated as a driver's rating. This rating is managed based on the driver's account information.

[0656] Step 6:

[0657] Based on the rating system, the server will suspend the account of a driver who has accumulated a certain number of dangerous driving incidents. Based on the data from the rating system, the server will automatically suspend the account if the driver's driving behavior exceeds the standard.

[0658] Step 7:

[0659] The server analyzes the video and audio data collected from the camera and microphone inside the vehicle. Specifically, if the driver makes an inappropriate remark to a passenger, it is analyzed and recognized as inappropriate behavior. This data is immediately sent to the server.

[0660] Step 8:

[0661] The server will report any inappropriate comments detected to the administrator, who will then take action if necessary to temporarily suspend the driver's account.

[0662] Step 9:

[0663] The server collects user profile information and recommends optimal conversation topics and services based on that information. For example, if the user is a tourist, the server recommends that the driver provide tourist information. This process improves user satisfaction.

[0664] Step 10:

[0665] The server uses the generative AI model to present optimized conversation content and service recommendations to the driver in real time, allowing the driver to provide optimal service to the user.

[0666] As a result, the present invention makes it possible to provide a safe and comfortable ride-sharing service, thereby increasing user trust and satisfaction.

[0667] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0668] Embodiment of operation monitoring function

[0669] In an embodiment of the driving monitoring function of the present invention, data is collected from cameras and various sensors mounted on the vehicle and analyzed by a generation AI. A server collects data in real time from a forward camera, speed sensor, acceleration sensor, etc. mounted on the vehicle. The collected data is analyzed by the server using a generation AI, and driving behavior is monitored.

[0670] For example, if a driver repeatedly brakes suddenly, the server will detect this driving pattern. The server will use generative AI to recognize the dangerous driving behavior and immediately send a warning message to the driver. Furthermore, this dangerous driving behavior will be recorded in the rating system and reflected in the driver's account rating. If a certain number of dangerous driving incidents are accumulated, the server will suspend the driver's account.

[0671] Embodiment of in-vehicle monitoring function

[0672] In an embodiment of the in-vehicle monitoring function of the present invention, video and audio data are collected from cameras and microphones installed in the vehicle and analyzed by generative AI. The server monitors the situation inside the vehicle and detects inappropriate behavior or remarks. For example, if a driver makes an inappropriate remark to a passenger, the server analyzes the audio data and detects that the remark is inappropriate. This data is immediately reported by the server to an administrator, and appropriate action can be taken, such as suspending the driver's account, if necessary.

[0673] For example, if a driver makes a sexually harassing remark to a passenger, the server analyzes the remark, determines it to be inappropriate, and reports it to an administrator. Once the administrator receives the report, the driver's account will be temporarily suspended.

[0674] Embodiment of hospitality function

[0675] In an embodiment of the hospitality function of the present invention, user profile information (e.g., age, gender, nationality, destination, etc.) is collected, and based on that information, a generation AI recommends optimal conversation content and services. The server collects the user's profile information and sends it to the terminal. The terminal uses the generation AI to recommend optimal conversation content and services based on this profile information and presents them to the driver.

[0676] For example, if the passenger is a tourist from abroad, the device will recommend to the driver "provide information about tourist spots in the area." The driver will then provide information about tourist spots to the passenger based on the recommendation, allowing the user to receive a comprehensive service.

[0677] Embodiment of Emotion Engine

[0678] In an embodiment of the emotion engine of the present invention, the vehicle recognizes the user's emotions and provides appropriate feedback based on the user's emotions. The server collects data such as the user's facial expressions, voice tone, and speech content from the vehicle's cameras, microphones, and other sensors, and analyzes the data using the emotion engine. The emotion engine recognizes emotions such as stress, anxiety, and joy in real time based on the user data.

[0679] For example, if the emotion engine detects that the user is feeling stressed during the ride, the server will instruct the device to play relaxation music.The server will also recommend appropriate advice to the driver and encourage conversations that will help the user relax.

[0680] As a result, the present invention provides a safe and comfortable ride-sharing service, and increases user trust and satisfaction. These functions work together to realize a consistent service as a whole.

[0681] The processing flow will be explained below.

[0682] Emotion Engine Processing Steps

[0683] Step 1:

[0684] The server collects data in real time from cameras, microphones and other sensors installed inside the vehicle.

[0685] Step 2:

[0686] The server sends the collected data to an emotion engine, which analyzes the user's facial expressions, tone of voice, and content of their comments.

[0687] Step 3:

[0688] The emotion engine recognizes emotions (stress, anxiety, joy, etc.) in real time based on user data.

[0689] Step 4:

[0690] The server receives the user's emotion data analyzed by the emotion engine and generates appropriate feedback based on the analysis results.

[0691] Step 5:

[0692] If the emotion engine recognizes that the user is under stress, the server instructs the terminal to play relaxation music or provide advice to help the user relax.

[0693] Step 6:

[0694] Based on instructions from the server, the terminal plays relaxation music or presents appropriate advice or conversation recommendations to the driver.

[0695] Step 7:

[0696] The driver follows the advice and recommendations provided by the device, provides a relaxing conversation for the user, and provides a comfortable service.

[0697] In-vehicle monitoring function processing steps

[0698] Step 1:

[0699] The server collects video and audio data in real time from cameras and microphones installed inside the vehicle.

[0700] Step 2:

[0701] The server sends the collected data to the generative AI model.

[0702] Step 3:

[0703] A generative AI model analyzes data to identify inappropriate behavior and conversations in the car.

[0704] Step 4:

[0705] If the server detects any inappropriate behavior or conversation, it will immediately notify an administrator.

[0706] Step 5:

[0707] The server assists administrators who receive reports to take action such as suspending the driver's account.

[0708] Hospitality function processing steps

[0709] Step 1:

[0710] The server collects profile information such as the user's age, gender, nationality, and destination.

[0711] Step 2:

[0712] The server sends the collected profile information to the driver's terminal.

[0713] Step 3:

[0714] The device sends profile information to a generation AI, which then recommends the most suitable conversation content and services for the user.

[0715] Step 4:

[0716] The terminal then presents the generated recommendations to the driver.

[0717] Step 5:

[0718] Based on recommendations, drivers will provide users with the most appropriate services and conversations.

[0719] Example 2

[0720] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0721] Conventional driving monitoring systems and in-vehicle monitoring systems have had difficulty detecting inappropriate driving behavior or inappropriate behavior in the vehicle in real time and taking countermeasures. Furthermore, they have not been able to effectively provide services that respond to users' emotions and needs. This has led to problems such as an increased risk of accidents and a decrease in user satisfaction.

[0722] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting data from an image capturing device and a detection device mounted on the vehicle, means for analyzing the collected data by a generation algorithm and monitoring driving operations, means for detecting dangerous driving based on the analysis result and sending a warning to the driver, means for recording dangerous driving behavior and saving it in an evaluation system, means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the evaluation system, means for collecting data from an image capturing device and a sound collecting device in the vehicle, means for analyzing the collected data by a generation algorithm and monitoring inappropriate behavior or conversation in the vehicle, means for notifying an administrator when inappropriate behavior or conversation is detected, means for suspending the driver's account when a report is made, means for collecting attribute information of a user, means for the generation algorithm to recommend optimal conversation content or services based on the collected attribute information, means for presenting the recommended content to the driver, and means for the driver to provide a service to the user based on the recommendation. This will improve driving safety, enable proper monitoring of the in-vehicle environment, and enable the provision of services that will highly satisfy users.

[0723] An "image capture device" is a device that is mounted on a vehicle and is used to capture video data.

[0724] A "detection device" is a device for measuring physical quantities such as velocity and acceleration and acquiring data.

[0725] A "generative algorithm" refers to an artificial intelligence model used to analyze data and make predictions.

[0726] "Driving operations" refers to actions such as braking, accelerating, and steering by the driver.

[0727] "Dangerous driving" refers to unsafe driving behavior such as sudden braking or sudden acceleration.

[0728] An "evaluation system" refers to a system that evaluates drivers based on their past driving behavior.

[0729] The "voice collection device" is a device for acquiring voice data inside a vehicle.

[0730] "Inappropriate behavior" refers to inappropriate remarks or actions made in the car.

[0731] "Administrator" refers to the person in charge of monitoring and managing the entire system and taking appropriate action.

[0732] "Attribute information" refers to personal information such as the user's age, gender, nationality, and destination.

[0733] "Recommendations" refer to suggestions or advice that are optimized by a generative algorithm.

[0734] "Driver" means an individual who drives a vehicle.

[0735] "User" means an individual using a vehicle.

[0736] The present invention is a system that monitors driving operations and the in-vehicle environment by collecting data from image acquisition devices and detection devices mounted on vehicles and analyzing the data using a generation algorithm, and provides optimal services based on user attribute information. The program for this system is executed through multiple processing steps.

[0737] Hardware and software used

[0738] 1. Image capture device: A camera that collects video data from inside and outside the vehicle.

[0739] 2. Detection devices: Various sensors that measure vehicle behavior, such as speed sensors and acceleration sensors.

[0740] 3. Audio collection device: A microphone that collects audio data inside the vehicle.

[0741] 4. Server: A central processing unit that aggregates collected data and executes the generation algorithm.

[0742] 5. Generative algorithms: These are artificial intelligence models used for data analysis and prediction.

[0743] Program processing

[0744] The server collects data in real time from image capture devices and detection devices and analyzes the data using a generative algorithm. For example, it recognizes driving patterns based on video data acquired by a forward-facing camera and speed data acquired by an acceleration sensor, and detects dangerous driving such as sudden braking or sudden acceleration.

[0745] For example, if a driver repeatedly brakes suddenly, the server analyzes this driving pattern and flags it as dangerous driving. At this time, a warning message such as "You have repeatedly braked suddenly. Please drive carefully" is displayed on the driver's device. In addition, dangerous driving behavior is recorded in a rating system, and if a certain number of dangerous driving incidents are accumulated, the driver's account will be temporarily suspended.

[0746] In-car monitoring function

[0747] The server collects data from the in-car image capture and audio capture devices and analyzes it using a generative algorithm. If inappropriate behavior or conversations are detected in the car, it is reported to an administrator. The administrator will review the situation based on the report and suspend the driver's account if necessary.

[0748] For example, if a driver makes an inappropriate remark to a passenger, the server will analyze the voice data and determine that the remark is inappropriate. This information will be immediately reported to the administrator, and the driver's account will be suspended.

[0749] Hospitality features

[0750] The server collects user attribute information (such as age, gender, nationality, and destination), and based on that information, a generation algorithm recommends optimal conversation content and services. The recommendations are presented to the driver, who then provides the user with services based on them.

[0751] For example, if a passenger is a tourist from abroad, the server generates a recommendation such as "I'll show you around tourist spots in this area" based on the user's nationality and destination information. The driver follows the recommendation and provides the passenger with information about tourist spots.

[0752] Embodiment of Emotion Engine

[0753] The server collects data such as the user's facial expressions, voice tone, and speech content from the vehicle's image capture device, audio collection device, and other sensors, and analyzes it using an emotion engine to recognize the user's emotions, such as stress, anxiety, and joy, in real time and provide appropriate feedback based on that.

[0754] For example, if the emotion engine detects that the user is feeling stressed during the ride, the server will instruct the device to play relaxation music, recommend appropriate advice to the driver, and encourage conversations that will help the user relax.

[0755] Example prompt sentence:

[0756] "Please explain how the system analyzes the user's emotions and provides appropriate feedback."

[0757] This system is expected to provide a safe and comfortable driving experience and increase user satisfaction.

[0758] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0759] Embodiment of operation monitoring function

[0760] Processing Steps

[0761] Step 1:

[0762] The server collects data in real time from image capture devices and detection devices. The inputs for this step are video data from cameras and numerical data from speed sensors and acceleration sensors. Specifically, the server acquires this data and prepares it for analysis.

[0763] Step 2:

[0764] The server inputs the collected data into a generation algorithm. The output of this step is the processed data required for analysis. In this step, for example, video data is converted into an analyzable form for each frame, and velocity and acceleration data are processed as time series data.

[0765] Step 3:

[0766] The server analyzes the data using a generative algorithm to recognize driving behavior patterns. The input to this step is a series of frames of data, and the output is the extracted driving behavior patterns. Specifically, behaviors such as sudden braking and sudden acceleration are recognized at this stage.

[0767] Step 4:

[0768] The server detects dangerous driving based on the analysis results. The input of this step is the data analysis results, and the output is the detection result of dangerous driving. In this step, for example, repeated sudden braking is identified as dangerous.

[0769] Step 5:

[0770] The server sends a warning message to the driver based on the detection result of dangerous driving. The input of this step is the detection result of dangerous driving, and the output is a warning message. Specifically, the message "There have been repeated sudden braking. Please drive carefully" is displayed on the driver's device.

[0771] Step 6:

[0772] The server records the risky driving behavior in the rating system and stores the data. The input of this step is the risky driving detection data, and the output is an updated rating record. Specifically, the driver's risky driving behavior is recorded in the database.

[0773] Step 7:

[0774] The server suspends the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system. The input of this step is the accumulated rating data, and the output is a change in the account status. The specific operation is to suspend the driver's account.

[0775] Embodiment of in-vehicle monitoring function

[0776] Processing Steps

[0777] Step 1:

[0778] The server collects data from the image capture device and audio collection device inside the vehicle. The inputs in this step are video and audio data from the camera and microphone. Specifically, conversations and actions inside the vehicle are recorded.

[0779] Step 2:

[0780] The server then inputs the collected data into a generative algorithm for analysis. The output of this step is the processed data required for analysis. Specifically, the audio data is converted into text and calculations are performed to identify inappropriate behavior or comments.

[0781] Step 3:

[0782] The server analyzes the data using a generative algorithm to detect inappropriate behavior or conversation. The input to this step is the analysis results, and the output is the detection of inappropriate behavior or conversation. For example, statements identified as "sexual harassment" are detected.

[0783] Step 4:

[0784] The server reports to the administrator based on the results of the detection of inappropriate behavior or conversation. The input of this step is the detection result of inappropriate behavior or conversation, and the output is a report message. Specifically, a notification saying "The driver made an inappropriate remark" is sent to the administrator's terminal.

[0785] Step 5:

[0786] The server suspends the driver's account when a report is made. The input of this step is the administrator's decision, and the output is a change in the account status. Specifically, the driver's account is suspended.

[0787] Embodiment of hospitality function

[0788] Processing Steps

[0789] Step 1:

[0790] The server collects user attribute information. The input for this step is the user's profile data, and the output is the collected attribute information. Specifically, data entered by the user, such as age, nationality, and destination, is obtained.

[0791] Step 2:

[0792] The server executes a generation algorithm based on the collected attribute information. The output of this step is the analyzed recommendation data. Specifically, calculations are performed to generate recommendation content from the attribute information.

[0793] Step 3:

[0794] The server uses the generation algorithm to recommend optimal conversation topics and services. The input to this step is the analysis results, and the output is the recommended content. For example, a recommendation such as "Please recommend some tourist spots" may be generated.

[0795] Step 4:

[0796] The server presents the recommendation content to the driver. The input of this step is the recommendation content, and the output is the information to be presented. Specifically, a message such as "Would you like me to show you tourist spots in this area?" is displayed on the driver's terminal.

[0797] Step 5:

[0798] The driver provides the user with a service based on the recommendations. The input of this step is the presented information, and the output is the provided service. Specifically, the driver provides the user with information about tourist spots.

[0799] Embodiment of Emotion Engine

[0800] Processing Steps

[0801] Step 1:

[0802] The server collects user data from the vehicle's image capture device, audio capture device, and other sensors. The inputs for this step include facial expressions, voice tone, and speech content. Specifically, facial expressions are captured by the camera and audio is recorded by the microphone.

[0803] Step 2:

[0804] The server then inputs the collected data into a generative algorithm for analysis. The output of this step is the processed data required for analysis. Specifically, facial expression data is converted into an analyzable format, and voice tones are analyzed.

[0805] Step 3:

[0806] The server analyzes the data using an emotion engine to recognize the user's emotion. The input of this step is the analysis result, and the output is the emotion recognition result. For example, it recognizes that the user is in a stressful state.

[0807] Step 4:

[0808] The server provides appropriate feedback based on the emotion recognition results. The input of this step is the emotion recognition results, and the output is the feedback content. Specifically, an instruction to play relaxation music is sent to the terminal.

[0809] Step 5:

[0810] The server recommends appropriate advice to the driver and encourages conversation that will help the user relax. The input to this step is the emotion recognition result and feedback content, and the output is a recommendation to the driver. For example, advice such as "The user is feeling stressed. Please have a relaxing conversation" is provided to the driver.

[0811] (Application example 2)

[0812] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0813] Conventional driving monitoring systems and in-car monitoring systems focus on detecting dangerous driving and inappropriate behavior, but lack feedback on the emotions and psychological states of drivers and passengers. Furthermore, inappropriate comments are often not detected and reported in real time, making it difficult to respond immediately. As a result, improving the quality of service provided by drivers and increasing passenger comfort remains a challenge.

[0814] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from cameras and sensors mounted on the vehicle, means for analyzing the collected data using generative artificial intelligence to monitor driving behavior, means for detecting reckless driving based on the analysis results and sending a warning to the driver, means for recording reckless driving behavior and saving the recorded data in a rating system, means for suspending the account of a driver who has accumulated a certain number of reckless driving incidents based on the rating system, means for recognizing emotions in the vehicle, means for providing appropriate feedback based on the recognized emotions, and means for analyzing voice data and detecting inappropriate remarks. This not only monitors reckless driving and inappropriate behavior, but also provides feedback based on the emotions of the driver and passengers, improving service quality and ensuring a comfortable and safe riding experience.

[0815] A "camera" is a device that captures video data and is used to record and distribute the situation inside and outside a vehicle in real time.

[0816] A "sensor" is a device that measures a physical phenomenon and outputs it as a digital or analog signal, and is used to collect data such as speed, acceleration, and impact.

[0817] "Generative AI" is a technology that analyzes and predicts based on large amounts of data, and is an algorithm that automatically performs various processes such as recognizing driving behavior and emotions and detecting inappropriate remarks.

[0818] "Driving behavior" refers to a series of actions and behaviors related to driving a vehicle, including speed changes, braking operations, lane changes, etc.

[0819] "Warnings" are notifications or messages sent to inform drivers of unsafe behavior or conditions and encourage safe driving.

[0820] The "evaluation system" is a system that evaluates and records the behavior of drivers and passengers based on collected data and manages the evaluation results.

[0821] "Emotion recognition" is a technology that uses in-car cameras and microphones to analyze the facial expressions and tone of voice of passengers and estimate their emotional state.

[0822] "Feedback" refers to responses or advice provided to drivers and passengers based on the results of emotion recognition, including playing relaxing music or recommending conversations.

[0823] "Inappropriate remarks" refer to inappropriate behavior such as sexual harassment or discriminatory content that occurs inside the vehicle, and are detected from audio data analyzed using artificial intelligence.

[0824] "Reporting" is the act of immediately notifying an administrator when inappropriate behavior or remarks are detected, and signals a situation that requires appropriate action.

[0825] The system of the present invention includes several key functions to ensure safe and comfortable operation of autonomous vehicles. These include driving monitoring, in-vehicle monitoring, hospitality, and emotion recognition. The system collects data from cameras and sensors mounted on the vehicle and analyzes it using generative artificial intelligence (generative AI) to provide various services and feedback.

[0826] Embodiment of operation monitoring function

[0827] The server collects data in real time from cameras and sensors installed in the vehicle (e.g., forward-facing cameras, speed sensors, and acceleration sensors). This data is analyzed by generative AI to monitor driving behavior. For example, if repeated sudden braking is detected, the server recognizes this driving pattern as dangerous and sends a warning message to the driver. This dangerous driving behavior is recorded in the rating system and reflected in the driver's account rating. If a certain number of dangerous driving incidents are accumulated, the server suspends the driver's account.

[0828] Embodiment of in-vehicle monitoring function

[0829] The server collects video and audio data from cameras and microphones installed inside the vehicle. It uses generative artificial intelligence to analyze this data and detect inappropriate behavior or speech inside the vehicle. For example, if a driver makes inappropriate remarks to a passenger, the server analyzes the audio data and detects that the remarks are inappropriate. This data is reported to administrators in real time, and the driver's account is suspended if necessary.

[0830] Embodiment of hospitality function

[0831] The server collects profile information such as the user's age, gender, nationality, and destination, and uses generative AI to recommend the most appropriate conversation topics and services. For example, if the passenger is a foreign tourist, the server might recommend "information about tourist spots in the area." The driver provides services to the passenger based on these recommendations, allowing the user to receive a comprehensive service.

[0832] Embodiment of Emotion Recognition Function

[0833] The server collects data from the vehicle's cameras, microphones, and other sensors, and uses an emotion engine to analyze the user's facial expressions, voice tone, and speech. If the emotion engine detects that the user is feeling stressed during the ride, the server instructs the device to play relaxation music. The server also recommends appropriate advice to the driver and encourages conversations that will help the user relax.

[0834] Hardware and software used

[0835] This system is implemented using the following hardware and software:

[0836] Cameras and microphones: Devices installed in vehicles to collect video and audio data.

[0837] Generative AI model: A model using TensorFlow / Keras that performs emotion recognition and driving pattern recognition.

[0838] Server: Collects data, analyzes it, sends warning messages, and provides feedback.

[0839] Rating system: A system that evaluates and records the behavior of drivers and passengers based on collected data.

[0840] This allows the system to monitor and provide feedback in real time, significantly improving safety and comfort inside the vehicle.

[0841] Specific examples

[0842] Example prompt sentence:

[0843] If a user feels stressed while watching a movie, the emotion engine is designed to analyze that stress and provide appropriate touch feedback or music. If a user expresses gratitude, the engine will analyze the utterance and generate a corresponding response.

[0844] In this way, combining emotion recognition with adaptive feedback can improve the quality of the user experience.

[0845] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0846] Step 1:

[0847] The server collects data in real time from the cameras and sensors installed in the vehicle. The data includes various information such as video, speed, and acceleration. The data from the cameras and sensors is input, and this data is stored on the server as output.

[0848] Step 2:

[0849] The server inputs the collected data into a generative AI model to analyze driving behavior and the situation inside the vehicle. The generative AI model is trained using TensorFlow / Keras and analyzes driving patterns and driver behavior from video data, and speech content from audio data. The input is data collected from cameras and sensors, and the output is the analysis results.

[0850] Step 3:

[0851] The server detects dangerous driving and inappropriate behavior based on the analysis results of the generative AI model. For example, if sudden braking or unreasonable lane changes are detected, this is recognized as dangerous driving. It also detects inappropriate remarks in the voice data. The input is the analysis results, and the output is the detection results of dangerous driving and inappropriate behavior.

[0852] Step 4:

[0853] The server sends a warning message to the driver based on the detection results. If inappropriate behavior is detected, it also notifies the administrator. The contents of the warning message and the report are displayed on the driver's and administrator's devices. The input is the detection result, and the output is the warning message and the report.

[0854] Step 5:

[0855] The server stores records of dangerous driving and inappropriate behavior in a rating system and updates the driver's rating. If a certain number of dangerous driving incidents are accumulated, the server suspends the driver's account. The input is the detection results and rating system data, and the output is the updated rating system data and a notification of account suspension.

[0856] Step 6:

[0857] The server collects user profile information (such as age, gender, nationality, and destination) and uses a generative AI model to recommend optimal conversation topics and services. The recommendations are presented to the driver via their device. The input is the user profile information, and the output is the recommendations themselves.

[0858] Step 7:

[0859] When using the emotion recognition function, the server collects the user's facial expressions, voice tone, and speech content from the in-car camera and microphone, and analyzes them using the emotion engine. It recognizes emotions such as stress, anxiety, and joy, and generates appropriate feedback based on the input data. It sends commands to play relaxation music or recommend conversations that will help the driver relax. The input is data from the camera and microphone, and the output is feedback and recommendation commands.

[0860] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0861] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0862] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0863] [Third embodiment]

[0864] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0865] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0866] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0867] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0868] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0869] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0870] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0871] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0872] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0873] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0874] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0875] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0876] Embodiment of operation monitoring function

[0877] In an embodiment of the driving monitoring function of the present invention, data is collected from cameras and various sensors mounted on the vehicle and analyzed by a generation AI. A server collects data in real time from a forward camera, speed sensor, acceleration sensor, etc. mounted on the vehicle. The collected data is analyzed by the server using a generation AI, and driving behavior is monitored.

[0878] For example, if a driver repeatedly brakes suddenly, the server will detect this driving pattern. The server will use generative AI to recognize the dangerous driving behavior and immediately send a warning message to the driver. Furthermore, this dangerous driving behavior will be recorded in the rating system and reflected in the driver's account rating. If a certain number of dangerous driving incidents are accumulated, the server will suspend the driver's account.

[0879] Embodiment of in-vehicle monitoring function

[0880] In an embodiment of the in-vehicle monitoring function of the present invention, video and audio data are collected from cameras and microphones installed in the vehicle and analyzed by generative AI. The server monitors the situation inside the vehicle and detects inappropriate behavior or remarks. For example, if a driver makes an inappropriate remark to a passenger, the server analyzes the audio data and detects that the remark is inappropriate. This data is immediately reported by the server to an administrator, and appropriate action can be taken, such as suspending the driver's account, if necessary.

[0881] For example, if a driver makes a sexually harassing remark to a passenger, the server analyzes the remark, determines it to be inappropriate, and reports it to an administrator. Once the administrator receives the report, the driver's account will be temporarily suspended.

[0882] Embodiment of hospitality function

[0883] In an embodiment of the hospitality function of the present invention, user profile information (e.g., age, gender, nationality, destination, etc.) is collected, and based on that information, a generation AI recommends optimal conversation content and services. The server collects the user's profile information and sends it to the terminal. The terminal uses the generation AI to recommend optimal conversation content and services based on this profile information and presents them to the driver.

[0884] For example, if the passenger is a tourist from abroad, the device will recommend to the driver "provide information about tourist spots in the area." The driver will then provide information about tourist spots to the passenger based on the recommendation, allowing the user to receive a comprehensive service.

[0885] As a result, the present invention provides a safe and comfortable ride-sharing service, increasing user trust and satisfaction. These functions work together to realize a consistent service as a whole.

[0886] The processing flow will be explained below.

[0887] Operation monitoring function processing steps

[0888] Step 1:

[0889] The server collects data in real time from the vehicle's front camera and various sensors.

[0890] Step 2:

[0891] The server sends the collected data to the generative AI model for analysis.

[0892] Step 3:

[0893] A generative AI model analyzes data and detects abnormal driving patterns such as hard braking, sudden acceleration, and swerving.

[0894] Step 4:

[0895] The server checks the analysis results of the driving patterns and sends a warning message to the driver if any dangerous driving behavior is detected.

[0896] Step 5:

[0897] The server records the detected risky driving behavior and stores it in a driver's rating system.

[0898] Step 6:

[0899] The server will suspend a driver's account if a certain number of dangerous driving incidents are accumulated.

[0900] In-vehicle monitoring function processing steps

[0901] Step 1:

[0902] The server collects video and audio data in real time from cameras and microphones installed inside the vehicle.

[0903] Step 2:

[0904] The server sends the collected data to the generative AI model.

[0905] Step 3:

[0906] A generative AI model analyzes data to identify inappropriate behavior and conversations in the car.

[0907] Step 4:

[0908] If the server detects any inappropriate behavior or conversation, it will immediately notify an administrator.

[0909] Step 5:

[0910] The server assists administrators who receive reports to take action such as suspending the driver's account.

[0911] Hospitality function processing steps

[0912] Step 1:

[0913] The server collects profile information such as the user's age, gender, nationality, and destination.

[0914] Step 2:

[0915] The server sends the collected profile information to the driver's terminal.

[0916] Step 3:

[0917] The device sends profile information to a generation AI, which then recommends the most suitable conversation content and services for the user.

[0918] Step 4:

[0919] The terminal then presents the generated recommendations to the driver.

[0920] Step 5:

[0921] Based on recommendations, drivers will provide users with the most appropriate services and conversations.

[0922] Example 1

[0923] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0924] In conventional ride-sharing services, there is a need to improve the quality and safety of services due to the frequent occurrence of dangerous driving by drivers, inappropriate behavior in the vehicle, and a lack of proper hospitality for users. Furthermore, insufficient measures to address these issues have led to a decline in user satisfaction and reliability.

[0925] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0926] In this invention, the server includes means for collecting data from cameras and sensors installed in the vehicle, means for analyzing the collected data using a generative AI model to monitor driving behavior, means for detecting dangerous driving based on the analysis results and sending a warning to the driver, means for recording dangerous driving behavior and saving it in a rating system, means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system, means for collecting data from cameras and microphones installed in the vehicle, means for analyzing the collected data using a generative AI model to monitor inappropriate behavior and conversation in the vehicle, means for notifying an administrator when inappropriate behavior or conversation is detected, means for collecting user profile information, means for the generative AI model to recommend optimal conversation content and services based on the collected profile information, means for presenting the recommended content to the driver, and means for the driver to provide services to the user based on the recommendations. This enables improved safety, appropriate behavior and hospitality in the vehicle, and improved overall service quality.

[0927] A "camera" is a device that detects light from an object and records the image as electronic data.

[0928] A "sensor" is a device that senses a physical quantity and converts it into an electrical signal to provide data.

[0929] A "generative AI model" is an artificial intelligence algorithm that uses machine learning and deep learning to analyze data and perform specific tasks.

[0930] "Driving behavior" refers to the driving patterns and actions of the vehicle driver, including speed, acceleration, braking, and the like.

[0931] A "warning message" is a message that notifies you when a specific condition or event occurs.

[0932] The "rating system" is a system for evaluating and recording the quality of drivers and services based on data.

[0933] "Suspending" an "account" is a process that temporarily stops the use of that account when certain conditions are met.

[0934] "Behavior" refers to the movement or behavior of a person or object in a particular situation.

[0935] "Conversational content" refers to words and sentences spoken in a specific context or situation.

[0936] "Omotenashi" refers to the act of providing service or hospitality with consideration and care by a service provider to a user.

[0937] "Profile information" refers to data including a user's personal information and attribute information, such as age, gender, nationality, and destination.

[0938] "Recommendation" refers to the act of suggesting specific information or services.

[0939] "Driver" means a person operating a vehicle.

[0940] "User" refers to a person who uses a particular service or system.

[0941] MODE FOR CARRYING OUT THE INVENTION

[0942] The present invention is a system for monitoring vehicle operation and in-vehicle conditions and providing appropriate services to drivers and users. This system uses a generative AI model to analyze collected data and take appropriate action based on specific conditions. The following describes in detail an embodiment of the present invention.

[0943] Embodiment of operation monitoring function

[0944] The server collects data in real time from the vehicle's cameras and sensors (forward camera, speed sensor, acceleration sensor). The collected data is sent to the generative AI model for analysis. Based on the analysis results, driving behavior is monitored, and if dangerous driving is detected, the server immediately sends a warning message to the driver. In addition, dangerous driving behavior is recorded in a reputation system, and if a certain number of such behaviors accumulate, the driver's account will be suspended.

[0945] As a concrete example, if the driver brakes suddenly repeatedly, the server will input the following prompt sentence into the generated AI:

[0946] "There have been frequent instances of sudden braking. Please be careful."

[0947] Embodiment of in-vehicle monitoring function

[0948] The server collects video and audio data from cameras and microphones installed in the vehicle. The collected data is analyzed by a generative AI model, and if inappropriate behavior or remarks are detected, a report is sent to an administrator. Furthermore, if inappropriate behavior or remarks are confirmed, the driver's account may be temporarily suspended.

[0949] As a specific example, if a driver makes an inappropriate remark to a passenger, the server will input the following prompt sentence into the generation AI:

[0950] "Inappropriate language detected."

[0951] Embodiment of hospitality function

[0952] The server collects the user's profile information (such as age, gender, nationality, and destination) and sends it to the device. The device then uses a generative AI model based on the collected profile information to recommend optimal conversation content and services, which are then presented to the driver. Based on these recommendations, the driver provides the user with appropriate services.

[0953] As a specific example, if the passenger is a tourist from abroad, the terminal inputs the following prompt sentence into the generation AI:

[0954] "Providing information about tourist spots in the area"

[0955] As a result, the present invention can provide a safe and comfortable ride-sharing service, increasing user trust and satisfaction. These functions work together to realize a consistent service as a whole.

[0956] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0957] Operation monitoring function processing steps

[0958] Step 1: Data collection

[0959] Input: Data from the vehicle's front camera, speed sensor, and acceleration sensor

[0960] Processing: The server collects data from these sensors in real time: video data from the camera, speed data from the speed sensor, and acceleration data from the accelerometer.

[0961] Output: Collected video data, velocity data, and acceleration data

[0962] Step 2: Send data

[0963] Input: Video data, velocity data, and acceleration data collected in step 1

[0964] Processing: The server sends these data to the generative AI model.

[0965] Output: Data sent to the generative AI model

[0966] Step 3: Data analysis

[0967] Input: Data sent in step 2

[0968] Processing: The generative AI model analyzes this data and monitors driving behavior, specifically recognizing visual driving patterns from the video data and matching them with speed and acceleration data to detect abnormal driving patterns.

[0969] Output: Analysis results (evaluation of driving behavior, identification of abnormal driving patterns)

[0970] Step 4: Sending a warning message

[0971] Input: Analysis results obtained in Step 3

[0972] Processing: The server sends a warning message to the driver based on the analysis results. Specifically, if abnormal driving such as sudden braking is detected, a warning message is generated.

[0973] Output: Warning message sent to the driver (e.g., "Frequent hard braking. Please be careful.")

[0974] Step 5: Record in the rating system

[0975] Input: Analysis results and warnings for Step 3 and Step 4

[0976] Processing: The server records this information in the rating system and reflects it in the driver's driving rating.

[0977] Output: Driving data and evaluation recorded in the evaluation system

[0978] Step 6: Suspend your account

[0979] Input: Number of dangerous driving incidents accumulated in the rating system

[0980] Action: If a certain number of dangerous driving incidents are accumulated, the server will suspend the driver's account.

[0981] Output: Information about the driver whose account was suspended

[0982] In-vehicle monitoring function processing steps

[0983] Step 1: Data collection

[0984] Input: Data from cameras and microphones installed inside the vehicle

[0985] Processing: The server collects video and audio data from these devices.

[0986] Output: Collected video and audio data

[0987] Step 2: Send data

[0988] Input: Data collected in Step 1

[0989] Processing: The server sends the collected data to the generative AI model.

[0990] Output: Data sent to the generative AI model

[0991] Step 3: Data analysis

[0992] Input: Data sent in step 2

[0993] Processing: Generative AI models analyze this data to detect inappropriate behavior and conversations.

[0994] Output: Analysis results (detection of inappropriate behavior or conversation)

[0995] Step 4: Report and Alert

[0996] Input: Analysis results obtained in Step 3

[0997] Processing: If the server detects inappropriate behavior or conversation, it will notify the administrator and send a warning message to the driver.

[0998] Output: Notification message to administrator, warning message to driver

[0999] Step 5: Suspend your account

[1000] Input: Report and warning details from Step 4

[1001] Action: Based on the administrator's instructions, the server suspends the driver's account.

[1002] Output: Information about the driver whose account was suspended

[1003] Hospitality function processing steps

[1004] Step 1: Collect profile information

[1005] Input: Information such as user's age, gender, nationality, and destination

[1006] Processing: The server collects this profile information.

[1007] Output: Collected profile information

[1008] Step 2: Submit your profile information

[1009] Input: Profile information collected in Step 1

[1010] Processing: The server sends the collected information to the device.

[1011] Output: Profile information sent to the device

[1012] Step 3: Data analysis and recommendation generation

[1013] Input: Profile information submitted in Step 2

[1014] Processing: The device uses a generative AI model based on the profile information to recommend the most appropriate conversation content and services.

[1015] Output: Recommendation content (e.g., tourist spot information)

[1016] Step 4: Present to the driver

[1017] Input: Recommendations generated in step 3

[1018] Processing: The device presents the recommendations to the driver, who then uses this information to provide the appropriate service to the user.

[1019] Output: Recommendations presented to the driver, service execution provided

[1020] Through the above steps, the present invention enables the provision of a safe and comfortable ride-sharing service by linking various monitoring functions with hospitality functions.

[1021] (Application example 1)

[1022] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1023] Conventional vehicle driving monitoring systems only monitor driving behavior and lack the ability to detect inappropriate behavior, conversations, and misconduct in the vehicle, as well as user-friendly features. As a result, safety and user satisfaction are not sufficiently improved. Furthermore, the accuracy of predictive driving warnings is limited, making it difficult to respond in real time. This calls for a comprehensive system that provides greater accuracy and added value.

[1024] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1025] In this invention, the server includes means for collecting data from cameras and sensors mounted on the vehicle, means for analyzing the collected data using generative artificial intelligence to monitor driving behavior, means for detecting dangerous driving based on the analysis results and sending a warning to the driver, means for recording dangerous driving behavior and saving the data in a rating system, means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system, means for analyzing video data from the camera using generative artificial intelligence, means for sending an alert to the server in the event of dangerous driving based on the analysis results of the video data, means for immediately issuing a warning when driving behavior exceeding a set threshold is detected, and means for recommending optimal conversation content and services based on the user's profile information.This enables comprehensive driving monitoring, improved safety, and even improved user satisfaction.

[1026] "Vehicle" means a means of transport designed to travel on roads, including those with automated driving capabilities.

[1027] A "camera" is an imaging device mounted on a vehicle to collect visual information.

[1028] A "sensor" is a device that detects physical information and collects it as data, and includes speed sensors, acceleration sensors, etc.

[1029] "Generative artificial intelligence" is a system that analyzes data using technologies such as machine learning and deep learning, and makes judgments and predictions based on the results.

[1030] "Driving behavior" refers to the driver's actions and vehicle movements when operating a vehicle, including braking, acceleration, steering, etc.

[1031] "Warning" refers to a notification or signal that alerts the driver to danger or caution, and is given by audio or visual means.

[1032] The "evaluation system" is a system that evaluates a driver's driving behavior based on collected driving data and records and manages the results.

[1033] An "account" is identification information used to identify a specific user on the system and manage their permissions and settings.

[1034] "Video data" refers to visual information captured by a camera and stored as digital data.

[1035] "Profile information" is data that includes personal information such as the user's age, gender, nationality, and destination.

[1036] "Recommendation" is the act of suggesting optimal conversation content or services based on a user's profile information and behavioral history.

[1037] An "alert" is a warning signal or message that immediately notifies you of danger or abnormality.

[1038] "Administrator" means a person or organization responsible for overseeing and managing the entire system.

[1039] In one embodiment of the present invention, the system collects data from cameras and sensors mounted on the vehicle and analyzes it using generative artificial intelligence (AI) to provide information about driving behavior, in-vehicle conditions, and services to the user.

[1040] System configuration

[1041] 1. Hardware Configuration

[1042] Camera: A device installed in front of or inside a vehicle to obtain visual information.

[1043] Sensors: Various sensors such as speed sensors and acceleration sensors are installed to collect physical operation data of the vehicle.

[1044] Microphone: A device for collecting voice data inside the vehicle.

[1045] Server: A computer equipped with powerful computing resources for performing data analysis.

[1046] 2. Software Configuration

[1047] Generative AI model (TensorFlow / Keras): A model that analyzes collected data and determines driving behavior and in-car activities.

[1048] Data collection API (requests library): An interface for collecting data from sensors and cameras and sending it to a server.

[1049] Data collection and analysis

[1050] The server collects real-time data from cameras and various sensors installed in the vehicle. This data includes video information of the area in front of the vehicle, speed information, acceleration information, and audio information from inside the vehicle. For example, video data from the camera is input into a generative artificial intelligence model to analyze driving behavior.

[1051] Dangerous driving detection

[1052] The generative AI model detects risky driving based on the analysis results. For example, if risky driving behavior such as sudden braking or sudden acceleration is detected, the server immediately sends a warning to the driver. These risky driving behaviors are also recorded in a rating system and saved as the driver's account rating. If a certain number of risky driving behaviors are accumulated, the server suspends the driver's account.

[1053] In-car monitoring

[1054] The server analyzes video and audio data collected from cameras and microphones installed in the vehicle. For example, if a driver makes inappropriate remarks to a passenger, the server analyzes the audio data and detects that the remarks are inappropriate. This data is then immediately reported to an administrator, and the driver's account is suspended if necessary.

[1055] Hospitality features

[1056] The server collects user profile information (e.g., age, gender, nationality, destination, etc.), and the generative AI uses this data to recommend the most appropriate conversation content and services. For example, if the passenger is a tourist from overseas, the server will recommend to the driver "information about tourist spots in the area." This allows the driver to provide information about tourist spots based on the recommendation, allowing the user to receive a comprehensive service.

[1057] Specific examples

[1058] An example of a specific prompt is, "The AI ​​model monitors driving behavior and immediately sends an alert if an abnormality is detected. In addition, voice analysis is used to detect inappropriate remarks and take appropriate action."

[1059] As a result, the present invention can provide a safe and comfortable ride-sharing service, increasing user trust and satisfaction.

[1060] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1061] Step 1:

[1062] The server collects data from the vehicle's cameras and sensors. Specifically, the camera collects visual information, the speed sensor and acceleration sensor collect physical movement data, and the microphone collects audio data. This data is collected in real time and sent to the server.

[1063] Step 2:

[1064] The server inputs the collected data into a pre-trained generative AI model. Specifically, visual data undergoes image processing, audio data undergoes voice analysis, and speed and acceleration data undergoes motion analysis. These analyses are used to analyze driving behavior and in-car conversations.

[1065] Step 3:

[1066] The server detects dangerous driving based on the analysis results of the generative AI model. Specifically, when dangerous driving behavior such as sudden braking or sudden acceleration is detected, the information is processed within the server. At this time, the analysis results are used as data to determine whether a warning is necessary.

[1067] Step 4:

[1068] If unsafe driving is detected, the server immediately sends a warning to the driver. The warning can be provided as an audio or visual signal to alert the driver, for example, a message such as "Sudden braking detected. Please be careful."

[1069] Step 5:

[1070] The server records risky driving behavior and stores it in a rating system. Data on driving behavior is managed within the rating system and accumulated as a driver's rating. This rating is managed based on the driver's account information.

[1071] Step 6:

[1072] Based on the rating system, the server will suspend the account of a driver who has accumulated a certain number of dangerous driving incidents. Based on the data from the rating system, the server will automatically suspend the account if the driver's driving behavior exceeds the standard.

[1073] Step 7:

[1074] The server analyzes the video and audio data collected from the camera and microphone inside the vehicle. Specifically, if the driver makes an inappropriate remark to a passenger, it is analyzed and recognized as inappropriate behavior. This data is immediately sent to the server.

[1075] Step 8:

[1076] The server will report any inappropriate comments detected to the administrator, who will then take action if necessary to temporarily suspend the driver's account.

[1077] Step 9:

[1078] The server collects user profile information and recommends optimal conversation topics and services based on that information. For example, if the user is a tourist, the server recommends that the driver provide tourist information. This process improves user satisfaction.

[1079] Step 10:

[1080] The server uses the generative AI model to present optimized conversation content and service recommendations to the driver in real time, allowing the driver to provide optimal service to the user.

[1081] As a result, the present invention makes it possible to provide a safe and comfortable ride-sharing service, thereby increasing user trust and satisfaction.

[1082] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1083] Embodiment of operation monitoring function

[1084] In an embodiment of the driving monitoring function of the present invention, data is collected from cameras and various sensors mounted on the vehicle and analyzed by a generation AI. A server collects data in real time from a forward camera, speed sensor, acceleration sensor, etc. mounted on the vehicle. The collected data is analyzed by the server using a generation AI, and driving behavior is monitored.

[1085] For example, if a driver repeatedly brakes suddenly, the server will detect this driving pattern. The server will use generative AI to recognize the dangerous driving behavior and immediately send a warning message to the driver. Furthermore, this dangerous driving behavior will be recorded in the rating system and reflected in the driver's account rating. If a certain number of dangerous driving incidents are accumulated, the server will suspend the driver's account.

[1086] Embodiment of in-vehicle monitoring function

[1087] In an embodiment of the in-vehicle monitoring function of the present invention, video and audio data are collected from cameras and microphones installed in the vehicle and analyzed by generative AI. The server monitors the situation inside the vehicle and detects inappropriate behavior or remarks. For example, if a driver makes an inappropriate remark to a passenger, the server analyzes the audio data and detects that the remark is inappropriate. This data is immediately reported by the server to an administrator, and appropriate action can be taken, such as suspending the driver's account, if necessary.

[1088] For example, if a driver makes a sexually harassing remark to a passenger, the server analyzes the remark, determines it to be inappropriate, and reports it to an administrator. Once the administrator receives the report, the driver's account will be temporarily suspended.

[1089] Embodiment of hospitality function

[1090] In an embodiment of the hospitality function of the present invention, user profile information (e.g., age, gender, nationality, destination, etc.) is collected, and based on that information, a generation AI recommends optimal conversation content and services. The server collects the user's profile information and sends it to the terminal. The terminal uses the generation AI to recommend optimal conversation content and services based on this profile information and presents them to the driver.

[1091] For example, if the passenger is a tourist from abroad, the device will recommend to the driver "provide information about tourist spots in the area." The driver will then provide information about tourist spots to the passenger based on the recommendation, allowing the user to receive a comprehensive service.

[1092] Embodiment of Emotion Engine

[1093] In an embodiment of the emotion engine of the present invention, the vehicle recognizes the user's emotions and provides appropriate feedback based on the user's emotions. The server collects data such as the user's facial expressions, voice tone, and speech content from the vehicle's cameras, microphones, and other sensors, and analyzes the data using the emotion engine. The emotion engine recognizes emotions such as stress, anxiety, and joy in real time based on the user data.

[1094] For example, if the emotion engine detects that the user is feeling stressed during the ride, the server will instruct the device to play relaxation music.The server will also recommend appropriate advice to the driver and encourage conversations that will help the user relax.

[1095] As a result, the present invention provides a safe and comfortable ride-sharing service, and increases user trust and satisfaction. These functions work together to realize a consistent service as a whole.

[1096] The processing flow will be explained below.

[1097] Emotion Engine Processing Steps

[1098] Step 1:

[1099] The server collects data in real time from cameras, microphones and other sensors installed inside the vehicle.

[1100] Step 2:

[1101] The server sends the collected data to an emotion engine, which analyzes the user's facial expressions, tone of voice, and content of their comments.

[1102] Step 3:

[1103] The emotion engine recognizes emotions (stress, anxiety, joy, etc.) in real time based on user data.

[1104] Step 4:

[1105] The server receives the user's emotion data analyzed by the emotion engine and generates appropriate feedback based on the analysis results.

[1106] Step 5:

[1107] If the emotion engine recognizes that the user is under stress, the server instructs the terminal to play relaxation music or provide advice to help the user relax.

[1108] Step 6:

[1109] Based on instructions from the server, the terminal plays relaxation music or presents appropriate advice or conversation recommendations to the driver.

[1110] Step 7:

[1111] The driver follows the advice and recommendations provided by the device, provides a relaxing conversation for the user, and provides a comfortable service.

[1112] In-vehicle monitoring function processing steps

[1113] Step 1:

[1114] The server collects video and audio data in real time from cameras and microphones installed inside the vehicle.

[1115] Step 2:

[1116] The server sends the collected data to the generative AI model.

[1117] Step 3:

[1118] A generative AI model analyzes data to identify inappropriate behavior and conversations in the car.

[1119] Step 4:

[1120] If the server detects any inappropriate behavior or conversation, it will immediately notify an administrator.

[1121] Step 5:

[1122] The server assists administrators who receive reports to take action such as suspending the driver's account.

[1123] Hospitality function processing steps

[1124] Step 1:

[1125] The server collects profile information such as the user's age, gender, nationality, and destination.

[1126] Step 2:

[1127] The server sends the collected profile information to the driver's terminal.

[1128] Step 3:

[1129] The device sends profile information to a generation AI, which then recommends the most suitable conversation content and services for the user.

[1130] Step 4:

[1131] The terminal then presents the generated recommendations to the driver.

[1132] Step 5:

[1133] Based on recommendations, drivers will provide users with the most appropriate services and conversations.

[1134] Example 2

[1135] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1136] Conventional driving monitoring systems and in-vehicle monitoring systems have had difficulty detecting inappropriate driving behavior or inappropriate behavior in the vehicle in real time and taking countermeasures. Furthermore, they have not been able to effectively provide services that respond to users' emotions and needs. This has led to problems such as an increased risk of accidents and a decrease in user satisfaction.

[1137] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting data from an image capturing device and a detection device mounted on the vehicle, means for analyzing the collected data by a generation algorithm and monitoring driving operations, means for detecting dangerous driving based on the analysis result and sending a warning to the driver, means for recording dangerous driving behavior and saving it in an evaluation system, means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the evaluation system, means for collecting data from an image capturing device and a sound collecting device in the vehicle, means for analyzing the collected data by a generation algorithm and monitoring inappropriate behavior or conversation in the vehicle, means for notifying an administrator when inappropriate behavior or conversation is detected, means for suspending the driver's account when a report is made, means for collecting attribute information of a user, means for the generation algorithm to recommend optimal conversation content or services based on the collected attribute information, means for presenting the recommended content to the driver, and means for the driver to provide a service to the user based on the recommendation. This will improve driving safety, enable proper monitoring of the in-vehicle environment, and enable the provision of services that will highly satisfy users.

[1138] An "image capture device" is a device that is mounted on a vehicle and is used to capture video data.

[1139] A "detection device" is a device for measuring physical quantities such as velocity and acceleration and acquiring data.

[1140] A "generative algorithm" refers to an artificial intelligence model used to analyze data and make predictions.

[1141] "Driving operations" refers to actions such as braking, accelerating, and steering by the driver.

[1142] "Dangerous driving" refers to unsafe driving behavior such as sudden braking or sudden acceleration.

[1143] An "evaluation system" refers to a system that evaluates drivers based on their past driving behavior.

[1144] The "voice collection device" is a device for acquiring voice data inside a vehicle.

[1145] "Inappropriate behavior" refers to inappropriate remarks or actions made in the car.

[1146] "Administrator" refers to the person in charge of monitoring and managing the entire system and taking appropriate action.

[1147] "Attribute information" refers to personal information such as the user's age, gender, nationality, and destination.

[1148] "Recommendations" refer to suggestions or advice that are optimized by a generative algorithm.

[1149] "Driver" means an individual who drives a vehicle.

[1150] "User" means an individual using a vehicle.

[1151] The present invention is a system that monitors driving operations and the in-vehicle environment by collecting data from image acquisition devices and detection devices mounted on vehicles and analyzing the data using a generation algorithm, and provides optimal services based on user attribute information. The program for this system is executed through multiple processing steps.

[1152] Hardware and software used

[1153] 1. Image capture device: A camera that collects video data from inside and outside the vehicle.

[1154] 2. Detection devices: Various sensors that measure vehicle behavior, such as speed sensors and acceleration sensors.

[1155] 3. Audio collection device: A microphone that collects audio data inside the vehicle.

[1156] 4. Server: A central processing unit that aggregates collected data and executes the generation algorithm.

[1157] 5. Generative algorithms: These are artificial intelligence models used for data analysis and prediction.

[1158] Program processing

[1159] The server collects data in real time from image capture devices and detection devices and analyzes the data using a generative algorithm. For example, it recognizes driving patterns based on video data acquired by a forward-facing camera and speed data acquired by an acceleration sensor, and detects dangerous driving such as sudden braking or sudden acceleration.

[1160] For example, if a driver repeatedly brakes suddenly, the server analyzes this driving pattern and flags it as dangerous driving. At this time, a warning message such as "You have repeatedly braked suddenly. Please drive carefully" is displayed on the driver's device. In addition, dangerous driving behavior is recorded in a rating system, and if a certain number of dangerous driving incidents are accumulated, the driver's account will be temporarily suspended.

[1161] In-car monitoring function

[1162] The server collects data from the in-car image capture and audio capture devices and analyzes it using a generative algorithm. If inappropriate behavior or conversations are detected in the car, it is reported to an administrator. The administrator will review the situation based on the report and suspend the driver's account if necessary.

[1163] For example, if a driver makes an inappropriate remark to a passenger, the server will analyze the voice data and determine that the remark is inappropriate. This information will be immediately reported to the administrator, and the driver's account will be suspended.

[1164] Hospitality features

[1165] The server collects user attribute information (such as age, gender, nationality, and destination), and based on that information, a generation algorithm recommends optimal conversation content and services. The recommendations are presented to the driver, who then provides the user with services based on them.

[1166] For example, if a passenger is a tourist from abroad, the server generates a recommendation such as "I'll show you around tourist spots in this area" based on the user's nationality and destination information. The driver follows the recommendation and provides the passenger with information about tourist spots.

[1167] Embodiment of Emotion Engine

[1168] The server collects data such as the user's facial expressions, voice tone, and speech content from the vehicle's image capture device, audio collection device, and other sensors, and analyzes it using an emotion engine to recognize the user's emotions, such as stress, anxiety, and joy, in real time and provide appropriate feedback based on that.

[1169] For example, if the emotion engine detects that the user is feeling stressed during the ride, the server will instruct the device to play relaxation music, recommend appropriate advice to the driver, and encourage conversations that will help the user relax.

[1170] Example prompt sentence:

[1171] "Please explain how the system analyzes the user's emotions and provides appropriate feedback."

[1172] This system is expected to provide a safe and comfortable driving experience and increase user satisfaction.

[1173] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1174] Embodiment of operation monitoring function

[1175] Processing Steps

[1176] Step 1:

[1177] The server collects data in real time from image capture devices and detection devices. The inputs for this step are video data from cameras and numerical data from speed sensors and acceleration sensors. Specifically, the server acquires this data and prepares it for analysis.

[1178] Step 2:

[1179] The server inputs the collected data into a generation algorithm. The output of this step is the processed data required for analysis. In this step, for example, video data is converted into an analyzable form for each frame, and velocity and acceleration data are processed as time series data.

[1180] Step 3:

[1181] The server analyzes the data using a generative algorithm to recognize driving behavior patterns. The input to this step is a series of frames of data, and the output is the extracted driving behavior patterns. Specifically, behaviors such as sudden braking and sudden acceleration are recognized at this stage.

[1182] Step 4:

[1183] The server detects dangerous driving based on the analysis results. The input of this step is the data analysis results, and the output is the detection result of dangerous driving. In this step, for example, repeated sudden braking is identified as dangerous.

[1184] Step 5:

[1185] The server sends a warning message to the driver based on the detection result of dangerous driving. The input of this step is the detection result of dangerous driving, and the output is a warning message. Specifically, the message "There have been repeated sudden braking. Please drive carefully" is displayed on the driver's device.

[1186] Step 6:

[1187] The server records the risky driving behavior in the rating system and stores the data. The input of this step is the risky driving detection data, and the output is an updated rating record. Specifically, the driver's risky driving behavior is recorded in the database.

[1188] Step 7:

[1189] The server suspends the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system. The input of this step is the accumulated rating data, and the output is a change in the account status. The specific operation is to suspend the driver's account.

[1190] Embodiment of in-vehicle monitoring function

[1191] Processing Steps

[1192] Step 1:

[1193] The server collects data from the image capture device and audio collection device inside the vehicle. The inputs in this step are video and audio data from the camera and microphone. Specifically, conversations and actions inside the vehicle are recorded.

[1194] Step 2:

[1195] The server then inputs the collected data into a generative algorithm for analysis. The output of this step is the processed data required for analysis. Specifically, the audio data is converted into text and calculations are performed to identify inappropriate behavior or comments.

[1196] Step 3:

[1197] The server analyzes the data using a generative algorithm to detect inappropriate behavior or conversation. The input to this step is the analysis results, and the output is the detection of inappropriate behavior or conversation. For example, statements identified as "sexual harassment" are detected.

[1198] Step 4:

[1199] The server reports to the administrator based on the results of the detection of inappropriate behavior or conversation. The input of this step is the detection result of inappropriate behavior or conversation, and the output is a report message. Specifically, a notification saying "The driver made an inappropriate remark" is sent to the administrator's terminal.

[1200] Step 5:

[1201] The server suspends the driver's account when a report is made. The input of this step is the administrator's decision, and the output is a change in the account status. Specifically, the driver's account is suspended.

[1202] Embodiment of hospitality function

[1203] Processing Steps

[1204] Step 1:

[1205] The server collects user attribute information. The input for this step is the user's profile data, and the output is the collected attribute information. Specifically, data entered by the user, such as age, nationality, and destination, is obtained.

[1206] Step 2:

[1207] The server executes a generation algorithm based on the collected attribute information. The output of this step is the analyzed recommendation data. Specifically, calculations are performed to generate recommendation content from the attribute information.

[1208] Step 3:

[1209] The server uses the generation algorithm to recommend optimal conversation topics and services. The input to this step is the analysis results, and the output is the recommended content. For example, a recommendation such as "Please recommend some tourist spots" may be generated.

[1210] Step 4:

[1211] The server presents the recommendation content to the driver. The input of this step is the recommendation content, and the output is the information to be presented. Specifically, a message such as "Would you like me to show you tourist spots in this area?" is displayed on the driver's terminal.

[1212] Step 5:

[1213] The driver provides the user with a service based on the recommendations. The input of this step is the presented information, and the output is the provided service. Specifically, the driver provides the user with information about tourist spots.

[1214] Embodiment of Emotion Engine

[1215] Processing Steps

[1216] Step 1:

[1217] The server collects user data from the vehicle's image capture device, audio capture device, and other sensors. The inputs for this step include facial expressions, voice tone, and speech content. Specifically, facial expressions are captured by the camera and audio is recorded by the microphone.

[1218] Step 2:

[1219] The server then inputs the collected data into a generative algorithm for analysis. The output of this step is the processed data required for analysis. Specifically, facial expression data is converted into an analyzable format, and voice tones are analyzed.

[1220] Step 3:

[1221] The server analyzes the data using an emotion engine to recognize the user's emotion. The input of this step is the analysis result, and the output is the emotion recognition result. For example, it recognizes that the user is in a stressful state.

[1222] Step 4:

[1223] The server provides appropriate feedback based on the emotion recognition results. The input of this step is the emotion recognition results, and the output is the feedback content. Specifically, an instruction to play relaxation music is sent to the terminal.

[1224] Step 5:

[1225] The server recommends appropriate advice to the driver and encourages conversation that will help the user relax. The input to this step is the emotion recognition result and feedback content, and the output is a recommendation to the driver. For example, advice such as "The user is feeling stressed. Please have a relaxing conversation" is provided to the driver.

[1226] (Application example 2)

[1227] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1228] Conventional driving monitoring systems and in-car monitoring systems focus on detecting dangerous driving and inappropriate behavior, but lack feedback on the emotions and psychological states of drivers and passengers. Furthermore, inappropriate comments are often not detected and reported in real time, making it difficult to respond immediately. As a result, improving the quality of service provided by drivers and increasing passenger comfort remains a challenge.

[1229] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from cameras and sensors mounted on the vehicle, means for analyzing the collected data using generative artificial intelligence to monitor driving behavior, means for detecting reckless driving based on the analysis results and sending a warning to the driver, means for recording reckless driving behavior and saving the recorded data in a rating system, means for suspending the account of a driver who has accumulated a certain number of reckless driving incidents based on the rating system, means for recognizing emotions in the vehicle, means for providing appropriate feedback based on the recognized emotions, and means for analyzing voice data and detecting inappropriate remarks. This not only monitors reckless driving and inappropriate behavior, but also provides feedback based on the emotions of the driver and passengers, improving service quality and ensuring a comfortable and safe riding experience.

[1230] A "camera" is a device that captures video data and is used to record and distribute the situation inside and outside a vehicle in real time.

[1231] A "sensor" is a device that measures a physical phenomenon and outputs it as a digital or analog signal, and is used to collect data such as speed, acceleration, and impact.

[1232] "Generative AI" is a technology that analyzes and predicts based on large amounts of data, and is an algorithm that automatically performs various processes such as recognizing driving behavior and emotions and detecting inappropriate remarks.

[1233] "Driving behavior" refers to a series of actions and behaviors related to driving a vehicle, including speed changes, braking operations, lane changes, etc.

[1234] "Warnings" are notifications or messages sent to inform drivers of unsafe behavior or conditions and encourage safe driving.

[1235] The "evaluation system" is a system that evaluates and records the behavior of drivers and passengers based on collected data and manages the evaluation results.

[1236] "Emotion recognition" is a technology that uses in-car cameras and microphones to analyze the facial expressions and tone of voice of passengers and estimate their emotional state.

[1237] "Feedback" refers to responses or advice provided to drivers and passengers based on the results of emotion recognition, including playing relaxing music or recommending conversations.

[1238] "Inappropriate remarks" refer to inappropriate behavior such as sexual harassment or discriminatory content that occurs inside the vehicle, and are detected from audio data analyzed using artificial intelligence.

[1239] "Reporting" is the act of immediately notifying an administrator when inappropriate behavior or remarks are detected, and signals a situation that requires appropriate action.

[1240] The system of the present invention includes several key functions to ensure safe and comfortable operation of autonomous vehicles. These include driving monitoring, in-vehicle monitoring, hospitality, and emotion recognition. The system collects data from cameras and sensors mounted on the vehicle and analyzes it using generative artificial intelligence (generative AI) to provide various services and feedback.

[1241] Embodiment of operation monitoring function

[1242] The server collects data in real time from cameras and sensors installed in the vehicle (e.g., forward-facing cameras, speed sensors, and acceleration sensors). This data is analyzed by generative AI to monitor driving behavior. For example, if repeated sudden braking is detected, the server recognizes this driving pattern as dangerous and sends a warning message to the driver. This dangerous driving behavior is recorded in the rating system and reflected in the driver's account rating. If a certain number of dangerous driving incidents are accumulated, the server suspends the driver's account.

[1243] Embodiment of in-vehicle monitoring function

[1244] The server collects video and audio data from cameras and microphones installed inside the vehicle. It uses generative artificial intelligence to analyze this data and detect inappropriate behavior or speech inside the vehicle. For example, if a driver makes inappropriate remarks to a passenger, the server analyzes the audio data and detects that the remarks are inappropriate. This data is reported to administrators in real time, and the driver's account is suspended if necessary.

[1245] Embodiment of hospitality function

[1246] The server collects profile information such as the user's age, gender, nationality, and destination, and uses generative AI to recommend the most appropriate conversation topics and services. For example, if the passenger is a foreign tourist, the server might recommend "information about tourist spots in the area." The driver provides services to the passenger based on these recommendations, allowing the user to receive a comprehensive service.

[1247] Embodiment of Emotion Recognition Function

[1248] The server collects data from the vehicle's cameras, microphones, and other sensors, and uses an emotion engine to analyze the user's facial expressions, voice tone, and speech. If the emotion engine detects that the user is feeling stressed during the ride, the server instructs the device to play relaxation music. The server also recommends appropriate advice to the driver and encourages conversations that will help the user relax.

[1249] Hardware and software used

[1250] This system is implemented using the following hardware and software:

[1251] Cameras and microphones: Devices installed in vehicles to collect video and audio data.

[1252] Generative AI model: A model using TensorFlow / Keras that performs emotion recognition and driving pattern recognition.

[1253] Server: Collects data, analyzes it, sends warning messages, and provides feedback.

[1254] Rating system: A system that evaluates and records the behavior of drivers and passengers based on collected data.

[1255] This allows the system to monitor and provide feedback in real time, significantly improving safety and comfort inside the vehicle.

[1256] Specific examples

[1257] Example prompt sentence:

[1258] If a user feels stressed while watching a movie, the emotion engine is designed to analyze that stress and provide appropriate touch feedback or music. If a user expresses gratitude, the engine will analyze the utterance and generate a corresponding response.

[1259] In this way, combining emotion recognition with adaptive feedback can improve the quality of the user experience.

[1260] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1261] Step 1:

[1262] The server collects data in real time from the cameras and sensors installed in the vehicle. The data includes various information such as video, speed, and acceleration. The data from the cameras and sensors is input, and this data is stored on the server as output.

[1263] Step 2:

[1264] The server inputs the collected data into a generative AI model to analyze driving behavior and the situation inside the vehicle. The generative AI model is trained using TensorFlow / Keras and analyzes driving patterns and driver behavior from video data, and speech content from audio data. The input is data collected from cameras and sensors, and the output is the analysis results.

[1265] Step 3:

[1266] The server detects dangerous driving and inappropriate behavior based on the analysis results of the generative AI model. For example, if sudden braking or unreasonable lane changes are detected, this is recognized as dangerous driving. It also detects inappropriate remarks in the voice data. The input is the analysis results, and the output is the detection results of dangerous driving and inappropriate behavior.

[1267] Step 4:

[1268] The server sends a warning message to the driver based on the detection results. If inappropriate behavior is detected, it also notifies the administrator. The contents of the warning message and the report are displayed on the driver's and administrator's devices. The input is the detection result, and the output is the warning message and the report.

[1269] Step 5:

[1270] The server stores records of dangerous driving and inappropriate behavior in a rating system and updates the driver's rating. If a certain number of dangerous driving incidents are accumulated, the server suspends the driver's account. The input is the detection results and rating system data, and the output is the updated rating system data and a notification of account suspension.

[1271] Step 6:

[1272] The server collects user profile information (such as age, gender, nationality, and destination) and uses a generative AI model to recommend optimal conversation topics and services. The recommendations are presented to the driver via their device. The input is the user profile information, and the output is the recommendations themselves.

[1273] Step 7:

[1274] When using the emotion recognition function, the server collects the user's facial expressions, voice tone, and speech content from the in-car camera and microphone, and analyzes them using the emotion engine. It recognizes emotions such as stress, anxiety, and joy, and generates appropriate feedback based on the input data. It sends commands to play relaxation music or recommend conversations that will help the driver relax. The input is data from the camera and microphone, and the output is feedback and recommendation commands.

[1275] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1276] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1277] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1278] [Fourth embodiment]

[1279] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1280] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1281] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1282] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1283] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1284] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1285] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1286] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1287] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1288] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1289] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1290] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1291] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1292] Embodiment of operation monitoring function

[1293] In an embodiment of the driving monitoring function of the present invention, data is collected from cameras and various sensors mounted on the vehicle and analyzed by a generation AI. A server collects data in real time from a forward camera, speed sensor, acceleration sensor, etc. mounted on the vehicle. The collected data is analyzed by the server using a generation AI, and driving behavior is monitored.

[1294] For example, if a driver repeatedly brakes suddenly, the server will detect this driving pattern. The server will use generative AI to recognize the dangerous driving behavior and immediately send a warning message to the driver. Furthermore, this dangerous driving behavior will be recorded in the rating system and reflected in the driver's account rating. If a certain number of dangerous driving incidents are accumulated, the server will suspend the driver's account.

[1295] Embodiment of in-vehicle monitoring function

[1296] In an embodiment of the in-vehicle monitoring function of the present invention, video and audio data are collected from cameras and microphones installed in the vehicle and analyzed by generative AI. The server monitors the situation inside the vehicle and detects inappropriate behavior or remarks. For example, if a driver makes an inappropriate remark to a passenger, the server analyzes the audio data and detects that the remark is inappropriate. This data is immediately reported by the server to an administrator, and appropriate action can be taken, such as suspending the driver's account, if necessary.

[1297] For example, if a driver makes a sexually harassing remark to a passenger, the server analyzes the remark, determines it to be inappropriate, and reports it to an administrator. Once the administrator receives the report, the driver's account will be temporarily suspended.

[1298] Embodiment of hospitality function

[1299] In an embodiment of the hospitality function of the present invention, user profile information (e.g., age, gender, nationality, destination, etc.) is collected, and based on that information, a generation AI recommends optimal conversation content and services. The server collects the user's profile information and sends it to the terminal. The terminal uses the generation AI to recommend optimal conversation content and services based on this profile information and presents them to the driver.

[1300] For example, if the passenger is a tourist from abroad, the device will recommend to the driver "provide information about tourist spots in the area." The driver will then provide information about tourist spots to the passenger based on the recommendation, allowing the user to receive a comprehensive service.

[1301] As a result, the present invention provides a safe and comfortable ride-sharing service, increasing user trust and satisfaction. These functions work together to realize a consistent service as a whole.

[1302] The processing flow will be explained below.

[1303] Operation monitoring function processing steps

[1304] Step 1:

[1305] The server collects data in real time from the vehicle's front camera and various sensors.

[1306] Step 2:

[1307] The server sends the collected data to the generative AI model for analysis.

[1308] Step 3:

[1309] A generative AI model analyzes data and detects abnormal driving patterns such as hard braking, sudden acceleration, and swerving.

[1310] Step 4:

[1311] The server checks the analysis results of the driving patterns and sends a warning message to the driver if any dangerous driving behavior is detected.

[1312] Step 5:

[1313] The server records the detected risky driving behavior and stores it in a driver's rating system.

[1314] Step 6:

[1315] The server will suspend a driver's account if a certain number of dangerous driving incidents are accumulated.

[1316] In-vehicle monitoring function processing steps

[1317] Step 1:

[1318] The server collects video and audio data in real time from cameras and microphones installed inside the vehicle.

[1319] Step 2:

[1320] The server sends the collected data to the generative AI model.

[1321] Step 3:

[1322] A generative AI model analyzes data to identify inappropriate behavior and conversations in the car.

[1323] Step 4:

[1324] If the server detects any inappropriate behavior or conversation, it will immediately notify an administrator.

[1325] Step 5:

[1326] The server assists administrators who receive reports to take action such as suspending the driver's account.

[1327] Hospitality function processing steps

[1328] Step 1:

[1329] The server collects profile information such as the user's age, gender, nationality, and destination.

[1330] Step 2:

[1331] The server sends the collected profile information to the driver's terminal.

[1332] Step 3:

[1333] The device sends profile information to a generation AI, which then recommends the most suitable conversation content and services for the user.

[1334] Step 4:

[1335] The terminal then presents the generated recommendations to the driver.

[1336] Step 5:

[1337] Based on recommendations, drivers will provide users with the most appropriate services and conversations.

[1338] Example 1

[1339] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1340] In conventional ride-sharing services, there is a need to improve the quality and safety of services due to the frequent occurrence of dangerous driving by drivers, inappropriate behavior in the vehicle, and a lack of proper hospitality for users. Furthermore, insufficient measures to address these issues have led to a decline in user satisfaction and reliability.

[1341] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1342] In this invention, the server includes means for collecting data from cameras and sensors installed in the vehicle, means for analyzing the collected data using a generative AI model to monitor driving behavior, means for detecting dangerous driving based on the analysis results and sending a warning to the driver, means for recording dangerous driving behavior and saving it in a rating system, means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system, means for collecting data from cameras and microphones installed in the vehicle, means for analyzing the collected data using a generative AI model to monitor inappropriate behavior and conversation in the vehicle, means for notifying an administrator when inappropriate behavior or conversation is detected, means for collecting user profile information, means for the generative AI model to recommend optimal conversation content and services based on the collected profile information, means for presenting the recommended content to the driver, and means for the driver to provide services to the user based on the recommendations. This enables improved safety, appropriate behavior and hospitality in the vehicle, and improved overall service quality.

[1343] A "camera" is a device that detects light from an object and records the image as electronic data.

[1344] A "sensor" is a device that senses a physical quantity and converts it into an electrical signal to provide data.

[1345] A "generative AI model" is an artificial intelligence algorithm that uses machine learning and deep learning to analyze data and perform specific tasks.

[1346] "Driving behavior" refers to the driving patterns and actions of the vehicle driver, including speed, acceleration, braking, and the like.

[1347] A "warning message" is a message that notifies you when a specific condition or event occurs.

[1348] The "rating system" is a system for evaluating and recording the quality of drivers and services based on data.

[1349] "Suspending" an "account" is a process that temporarily stops the use of that account when certain conditions are met.

[1350] "Behavior" refers to the movement or behavior of a person or object in a particular situation.

[1351] "Conversational content" refers to words and sentences spoken in a specific context or situation.

[1352] "Omotenashi" refers to the act of providing service or hospitality with consideration and care by a service provider to a user.

[1353] "Profile information" refers to data including a user's personal information and attribute information, such as age, gender, nationality, and destination.

[1354] "Recommendation" refers to the act of suggesting specific information or services.

[1355] "Driver" means a person operating a vehicle.

[1356] "User" refers to a person who uses a particular service or system.

[1357] MODE FOR CARRYING OUT THE INVENTION

[1358] The present invention is a system for monitoring vehicle operation and in-vehicle conditions and providing appropriate services to drivers and users. This system uses a generative AI model to analyze collected data and take appropriate action based on specific conditions. The following describes in detail an embodiment of the present invention.

[1359] Embodiment of operation monitoring function

[1360] The server collects data in real time from the vehicle's cameras and sensors (forward camera, speed sensor, acceleration sensor). The collected data is sent to the generative AI model for analysis. Based on the analysis results, driving behavior is monitored, and if dangerous driving is detected, the server immediately sends a warning message to the driver. In addition, dangerous driving behavior is recorded in a reputation system, and if a certain number of such behaviors accumulate, the driver's account will be suspended.

[1361] As a concrete example, if the driver brakes suddenly repeatedly, the server will input the following prompt sentence into the generated AI:

[1362] "There have been frequent instances of sudden braking. Please be careful."

[1363] Embodiment of in-vehicle monitoring function

[1364] The server collects video and audio data from cameras and microphones installed in the vehicle. The collected data is analyzed by a generative AI model, and if inappropriate behavior or remarks are detected, a report is sent to an administrator. Furthermore, if inappropriate behavior or remarks are confirmed, the driver's account may be temporarily suspended.

[1365] As a specific example, if a driver makes an inappropriate remark to a passenger, the server will input the following prompt sentence into the generation AI:

[1366] "Inappropriate language detected."

[1367] Embodiment of hospitality function

[1368] The server collects the user's profile information (such as age, gender, nationality, and destination) and sends it to the device. The device then uses a generative AI model based on the collected profile information to recommend optimal conversation content and services, which are then presented to the driver. Based on these recommendations, the driver provides the user with appropriate services.

[1369] As a specific example, if the passenger is a tourist from abroad, the terminal inputs the following prompt sentence into the generation AI:

[1370] "Providing information about tourist spots in the area"

[1371] As a result, the present invention can provide a safe and comfortable ride-sharing service, increasing user trust and satisfaction. These functions work together to realize a consistent service as a whole.

[1372] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1373] Operation monitoring function processing steps

[1374] Step 1: Data collection

[1375] Input: Data from the vehicle's front camera, speed sensor, and acceleration sensor

[1376] Processing: The server collects data from these sensors in real time: video data from the camera, speed data from the speed sensor, and acceleration data from the accelerometer.

[1377] Output: Collected video data, velocity data, and acceleration data

[1378] Step 2: Send data

[1379] Input: Video data, velocity data, and acceleration data collected in step 1

[1380] Processing: The server sends these data to the generative AI model.

[1381] Output: Data sent to the generative AI model

[1382] Step 3: Data analysis

[1383] Input: Data sent in step 2

[1384] Processing: The generative AI model analyzes this data and monitors driving behavior, specifically recognizing visual driving patterns from the video data and matching them with speed and acceleration data to detect abnormal driving patterns.

[1385] Output: Analysis results (evaluation of driving behavior, identification of abnormal driving patterns)

[1386] Step 4: Sending a warning message

[1387] Input: Analysis results obtained in Step 3

[1388] Processing: The server sends a warning message to the driver based on the analysis results. Specifically, if abnormal driving such as sudden braking is detected, a warning message is generated.

[1389] Output: Warning message sent to the driver (e.g., "Frequent hard braking. Please be careful.")

[1390] Step 5: Record in the rating system

[1391] Input: Analysis results and warnings for Step 3 and Step 4

[1392] Processing: The server records this information in the rating system and reflects it in the driver's driving rating.

[1393] Output: Driving data and evaluation recorded in the evaluation system

[1394] Step 6: Suspend your account

[1395] Input: Number of dangerous driving incidents accumulated in the rating system

[1396] Action: If a certain number of dangerous driving incidents are accumulated, the server will suspend the driver's account.

[1397] Output: Information about the driver whose account was suspended

[1398] In-vehicle monitoring function processing steps

[1399] Step 1: Data collection

[1400] Input: Data from cameras and microphones installed inside the vehicle

[1401] Processing: The server collects video and audio data from these devices.

[1402] Output: Collected video and audio data

[1403] Step 2: Send data

[1404] Input: Data collected in Step 1

[1405] Processing: The server sends the collected data to the generative AI model.

[1406] Output: Data sent to the generative AI model

[1407] Step 3: Data analysis

[1408] Input: Data sent in step 2

[1409] Processing: Generative AI models analyze this data to detect inappropriate behavior and conversations.

[1410] Output: Analysis results (detection of inappropriate behavior or conversation)

[1411] Step 4: Report and Alert

[1412] Input: Analysis results obtained in Step 3

[1413] Processing: If the server detects inappropriate behavior or conversation, it will notify the administrator and send a warning message to the driver.

[1414] Output: Notification message to administrator, warning message to driver

[1415] Step 5: Suspend your account

[1416] Input: Report and warning details from Step 4

[1417] Action: Based on the administrator's instructions, the server suspends the driver's account.

[1418] Output: Information about the driver whose account was suspended

[1419] Hospitality function processing steps

[1420] Step 1: Collect profile information

[1421] Input: Information such as user's age, gender, nationality, and destination

[1422] Processing: The server collects this profile information.

[1423] Output: Collected profile information

[1424] Step 2: Submit your profile information

[1425] Input: Profile information collected in Step 1

[1426] Processing: The server sends the collected information to the device.

[1427] Output: Profile information sent to the device

[1428] Step 3: Data analysis and recommendation generation

[1429] Input: Profile information submitted in Step 2

[1430] Processing: The device uses a generative AI model based on the profile information to recommend the most appropriate conversation content and services.

[1431] Output: Recommendation content (e.g., tourist spot information)

[1432] Step 4: Present to the driver

[1433] Input: Recommendations generated in step 3

[1434] Processing: The device presents the recommendations to the driver, who then uses this information to provide the appropriate service to the user.

[1435] Output: Recommendations presented to the driver, service execution provided

[1436] Through the above steps, the present invention enables the provision of a safe and comfortable ride-sharing service by linking various monitoring functions with hospitality functions.

[1437] (Application example 1)

[1438] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1439] Conventional vehicle driving monitoring systems only monitor driving behavior and lack the ability to detect inappropriate behavior, conversations, and misconduct in the vehicle, as well as user-friendly features. As a result, safety and user satisfaction are not sufficiently improved. Furthermore, the accuracy of predictive driving warnings is limited, making it difficult to respond in real time. This calls for a comprehensive system that provides greater accuracy and added value.

[1440] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1441] In this invention, the server includes means for collecting data from cameras and sensors mounted on the vehicle, means for analyzing the collected data using generative artificial intelligence to monitor driving behavior, means for detecting dangerous driving based on the analysis results and sending a warning to the driver, means for recording dangerous driving behavior and saving the data in a rating system, means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system, means for analyzing video data from the camera using generative artificial intelligence, means for sending an alert to the server in the event of dangerous driving based on the analysis results of the video data, means for immediately issuing a warning when driving behavior exceeding a set threshold is detected, and means for recommending optimal conversation content and services based on the user's profile information.This enables comprehensive driving monitoring, improved safety, and even improved user satisfaction.

[1442] "Vehicle" means a means of transport designed to travel on roads, including those with automated driving capabilities.

[1443] A "camera" is an imaging device mounted on a vehicle to collect visual information.

[1444] A "sensor" is a device that detects physical information and collects it as data, and includes speed sensors, acceleration sensors, etc.

[1445] "Generative artificial intelligence" is a system that analyzes data using technologies such as machine learning and deep learning, and makes judgments and predictions based on the results.

[1446] "Driving behavior" refers to the driver's actions and vehicle movements when operating a vehicle, including braking, acceleration, steering, etc.

[1447] "Warning" refers to a notification or signal that alerts the driver to danger or caution, and is given by audio or visual means.

[1448] The "evaluation system" is a system that evaluates a driver's driving behavior based on collected driving data and records and manages the results.

[1449] An "account" is identification information used to identify a specific user on the system and manage their permissions and settings.

[1450] "Video data" refers to visual information captured by a camera and stored as digital data.

[1451] "Profile information" is data that includes personal information such as the user's age, gender, nationality, and destination.

[1452] "Recommendation" is the act of suggesting optimal conversation content or services based on a user's profile information and behavioral history.

[1453] An "alert" is a warning signal or message that immediately notifies you of danger or abnormality.

[1454] "Administrator" means a person or organization responsible for overseeing and managing the entire system.

[1455] In one embodiment of the present invention, the system collects data from cameras and sensors mounted on the vehicle and analyzes it using generative artificial intelligence (AI) to provide information about driving behavior, in-vehicle conditions, and services to the user.

[1456] System configuration

[1457] 1. Hardware Configuration

[1458] Camera: A device installed in front of or inside a vehicle to obtain visual information.

[1459] Sensors: Various sensors such as speed sensors and acceleration sensors are installed to collect physical operation data of the vehicle.

[1460] Microphone: A device for collecting voice data inside the vehicle.

[1461] Server: A computer equipped with powerful computing resources for performing data analysis.

[1462] 2. Software Configuration

[1463] Generative AI model (TensorFlow / Keras): A model that analyzes collected data and determines driving behavior and in-car activities.

[1464] Data collection API (requests library): An interface for collecting data from sensors and cameras and sending it to a server.

[1465] Data collection and analysis

[1466] The server collects real-time data from cameras and various sensors installed in the vehicle. This data includes video information of the area in front of the vehicle, speed information, acceleration information, and audio information from inside the vehicle. For example, video data from the camera is input into a generative artificial intelligence model to analyze driving behavior.

[1467] Dangerous driving detection

[1468] The generative AI model detects risky driving based on the analysis results. For example, if risky driving behavior such as sudden braking or sudden acceleration is detected, the server immediately sends a warning to the driver. These risky driving behaviors are also recorded in a rating system and saved as the driver's account rating. If a certain number of risky driving behaviors are accumulated, the server suspends the driver's account.

[1469] In-car monitoring

[1470] The server analyzes video and audio data collected from cameras and microphones installed in the vehicle. For example, if a driver makes inappropriate remarks to a passenger, the server analyzes the audio data and detects that the remarks are inappropriate. This data is then immediately reported to an administrator, and the driver's account is suspended if necessary.

[1471] Hospitality features

[1472] The server collects user profile information (e.g., age, gender, nationality, destination, etc.), and the generative AI uses this data to recommend the most appropriate conversation content and services. For example, if the passenger is a tourist from overseas, the server will recommend to the driver "information about tourist spots in the area." This allows the driver to provide information about tourist spots based on the recommendation, allowing the user to receive a comprehensive service.

[1473] Specific examples

[1474] An example of a specific prompt is, "The AI ​​model monitors driving behavior and immediately sends an alert if an abnormality is detected. In addition, voice analysis is used to detect inappropriate remarks and take appropriate action."

[1475] As a result, the present invention can provide a safe and comfortable ride-sharing service, increasing user trust and satisfaction.

[1476] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1477] Step 1:

[1478] The server collects data from the vehicle's cameras and sensors. Specifically, the camera collects visual information, the speed sensor and acceleration sensor collect physical movement data, and the microphone collects audio data. This data is collected in real time and sent to the server.

[1479] Step 2:

[1480] The server inputs the collected data into a pre-trained generative AI model. Specifically, visual data undergoes image processing, audio data undergoes voice analysis, and speed and acceleration data undergoes motion analysis. These analyses are used to analyze driving behavior and in-car conversations.

[1481] Step 3:

[1482] The server detects dangerous driving based on the analysis results of the generative AI model. Specifically, when dangerous driving behavior such as sudden braking or sudden acceleration is detected, the information is processed within the server. At this time, the analysis results are used as data to determine whether a warning is necessary.

[1483] Step 4:

[1484] If unsafe driving is detected, the server immediately sends a warning to the driver. The warning can be provided as an audio or visual signal to alert the driver, for example, a message such as "Sudden braking detected. Please be careful."

[1485] Step 5:

[1486] The server records risky driving behavior and stores it in a rating system. Data on driving behavior is managed within the rating system and accumulated as a driver's rating. This rating is managed based on the driver's account information.

[1487] Step 6:

[1488] Based on the rating system, the server will suspend the account of a driver who has accumulated a certain number of dangerous driving incidents. Based on the data from the rating system, the server will automatically suspend the account if the driver's driving behavior exceeds the standard.

[1489] Step 7:

[1490] The server analyzes the video and audio data collected from the camera and microphone inside the vehicle. Specifically, if the driver makes an inappropriate remark to a passenger, it is analyzed and recognized as inappropriate behavior. This data is immediately sent to the server.

[1491] Step 8:

[1492] The server will report any inappropriate comments detected to the administrator, who will then take action if necessary to temporarily suspend the driver's account.

[1493] Step 9:

[1494] The server collects user profile information and recommends optimal conversation topics and services based on that information. For example, if the user is a tourist, the server recommends that the driver provide tourist information. This process improves user satisfaction.

[1495] Step 10:

[1496] The server uses the generative AI model to present optimized conversation content and service recommendations to the driver in real time, allowing the driver to provide optimal service to the user.

[1497] As a result, the present invention makes it possible to provide a safe and comfortable ride-sharing service, thereby increasing user trust and satisfaction.

[1498] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1499] Embodiment of operation monitoring function

[1500] In an embodiment of the driving monitoring function of the present invention, data is collected from cameras and various sensors mounted on the vehicle and analyzed by a generation AI. A server collects data in real time from a forward camera, speed sensor, acceleration sensor, etc. mounted on the vehicle. The collected data is analyzed by the server using a generation AI, and driving behavior is monitored.

[1501] For example, if a driver repeatedly brakes suddenly, the server will detect this driving pattern. The server will use generative AI to recognize the dangerous driving behavior and immediately send a warning message to the driver. Furthermore, this dangerous driving behavior will be recorded in the rating system and reflected in the driver's account rating. If a certain number of dangerous driving incidents are accumulated, the server will suspend the driver's account.

[1502] Embodiment of in-vehicle monitoring function

[1503] In an embodiment of the in-vehicle monitoring function of the present invention, video and audio data are collected from cameras and microphones installed in the vehicle and analyzed by generative AI. The server monitors the situation inside the vehicle and detects inappropriate behavior or remarks. For example, if a driver makes an inappropriate remark to a passenger, the server analyzes the audio data and detects that the remark is inappropriate. This data is immediately reported by the server to an administrator, and appropriate action can be taken, such as suspending the driver's account, if necessary.

[1504] For example, if a driver makes a sexually harassing remark to a passenger, the server analyzes the remark, determines it to be inappropriate, and reports it to an administrator. Once the administrator receives the report, the driver's account will be temporarily suspended.

[1505] Embodiment of hospitality function

[1506] In an embodiment of the hospitality function of the present invention, user profile information (e.g., age, gender, nationality, destination, etc.) is collected, and based on that information, a generation AI recommends optimal conversation content and services. The server collects the user's profile information and sends it to the terminal. The terminal uses the generation AI to recommend optimal conversation content and services based on this profile information and presents them to the driver.

[1507] For example, if the passenger is a tourist from abroad, the device will recommend to the driver "provide information about tourist spots in the area." The driver will then provide information about tourist spots to the passenger based on the recommendation, allowing the user to receive a comprehensive service.

[1508] Embodiment of Emotion Engine

[1509] In an embodiment of the emotion engine of the present invention, the vehicle recognizes the user's emotions and provides appropriate feedback based on the user's emotions. The server collects data such as the user's facial expressions, voice tone, and speech content from the vehicle's cameras, microphones, and other sensors, and analyzes the data using the emotion engine. The emotion engine recognizes emotions such as stress, anxiety, and joy in real time based on the user data.

[1510] For example, if the emotion engine detects that the user is feeling stressed during the ride, the server will instruct the device to play relaxation music.The server will also recommend appropriate advice to the driver and encourage conversations that will help the user relax.

[1511] As a result, the present invention provides a safe and comfortable ride-sharing service, and increases user trust and satisfaction. These functions work together to realize a consistent service as a whole.

[1512] The processing flow will be explained below.

[1513] Emotion Engine Processing Steps

[1514] Step 1:

[1515] The server collects data in real time from cameras, microphones and other sensors installed inside the vehicle.

[1516] Step 2:

[1517] The server sends the collected data to an emotion engine, which analyzes the user's facial expressions, tone of voice, and content of their comments.

[1518] Step 3:

[1519] The emotion engine recognizes emotions (stress, anxiety, joy, etc.) in real time based on user data.

[1520] Step 4:

[1521] The server receives the user's emotion data analyzed by the emotion engine and generates appropriate feedback based on the analysis results.

[1522] Step 5:

[1523] If the emotion engine recognizes that the user is under stress, the server instructs the terminal to play relaxation music or provide advice to help the user relax.

[1524] Step 6:

[1525] Based on instructions from the server, the terminal plays relaxation music or presents appropriate advice or conversation recommendations to the driver.

[1526] Step 7:

[1527] The driver follows the advice and recommendations provided by the device, provides a relaxing conversation for the user, and provides a comfortable service.

[1528] In-vehicle monitoring function processing steps

[1529] Step 1:

[1530] The server collects video and audio data in real time from cameras and microphones installed inside the vehicle.

[1531] Step 2:

[1532] The server sends the collected data to the generative AI model.

[1533] Step 3:

[1534] A generative AI model analyzes data to identify inappropriate behavior and conversations in the car.

[1535] Step 4:

[1536] If the server detects any inappropriate behavior or conversation, it will immediately notify an administrator.

[1537] Step 5:

[1538] The server assists administrators who receive reports to take action such as suspending the driver's account.

[1539] Hospitality function processing steps

[1540] Step 1:

[1541] The server collects profile information such as the user's age, gender, nationality, and destination.

[1542] Step 2:

[1543] The server sends the collected profile information to the driver's terminal.

[1544] Step 3:

[1545] The device sends profile information to a generation AI, which then recommends the most suitable conversation content and services for the user.

[1546] Step 4:

[1547] The terminal then presents the generated recommendations to the driver.

[1548] Step 5:

[1549] Based on recommendations, drivers will provide users with the most appropriate services and conversations.

[1550] Example 2

[1551] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1552] Conventional driving monitoring systems and in-vehicle monitoring systems have had difficulty detecting inappropriate driving behavior or inappropriate behavior in the vehicle in real time and taking countermeasures. Furthermore, they have not been able to effectively provide services that respond to users' emotions and needs. This has led to problems such as an increased risk of accidents and a decrease in user satisfaction.

[1553] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting data from an image capturing device and a detection device mounted on the vehicle, means for analyzing the collected data by a generation algorithm and monitoring driving operations, means for detecting dangerous driving based on the analysis result and sending a warning to the driver, means for recording dangerous driving behavior and saving it in an evaluation system, means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the evaluation system, means for collecting data from an image capturing device and a sound collecting device in the vehicle, means for analyzing the collected data by a generation algorithm and monitoring inappropriate behavior or conversation in the vehicle, means for notifying an administrator when inappropriate behavior or conversation is detected, means for suspending the driver's account when a report is made, means for collecting attribute information of a user, means for the generation algorithm to recommend optimal conversation content or services based on the collected attribute information, means for presenting the recommended content to the driver, and means for the driver to provide a service to the user based on the recommendation. This will improve driving safety, enable proper monitoring of the in-vehicle environment, and enable the provision of services that will highly satisfy users.

[1554] An "image capture device" is a device that is mounted on a vehicle and is used to capture video data.

[1555] A "detection device" is a device for measuring physical quantities such as velocity and acceleration and acquiring data.

[1556] A "generative algorithm" refers to an artificial intelligence model used to analyze data and make predictions.

[1557] "Driving operations" refers to actions such as braking, accelerating, and steering by the driver.

[1558] "Dangerous driving" refers to unsafe driving behavior such as sudden braking or sudden acceleration.

[1559] An "evaluation system" refers to a system that evaluates drivers based on their past driving behavior.

[1560] The "voice collection device" is a device for acquiring voice data inside a vehicle.

[1561] "Inappropriate behavior" refers to inappropriate remarks or actions made in the car.

[1562] "Administrator" refers to the person in charge of monitoring and managing the entire system and taking appropriate action.

[1563] "Attribute information" refers to personal information such as the user's age, gender, nationality, and destination.

[1564] "Recommendations" refer to suggestions or advice that are optimized by a generative algorithm.

[1565] "Driver" means an individual who drives a vehicle.

[1566] "User" means an individual using a vehicle.

[1567] The present invention is a system that monitors driving operations and the in-vehicle environment by collecting data from image acquisition devices and detection devices mounted on vehicles and analyzing the data using a generation algorithm, and provides optimal services based on user attribute information. The program for this system is executed through multiple processing steps.

[1568] Hardware and software used

[1569] 1. Image capture device: A camera that collects video data from inside and outside the vehicle.

[1570] 2. Detection devices: Various sensors that measure vehicle behavior, such as speed sensors and acceleration sensors.

[1571] 3. Audio collection device: A microphone that collects audio data inside the vehicle.

[1572] 4. Server: A central processing unit that aggregates collected data and executes the generation algorithm.

[1573] 5. Generative algorithms: These are artificial intelligence models used for data analysis and prediction.

[1574] Program processing

[1575] The server collects data in real time from image capture devices and detection devices and analyzes the data using a generative algorithm. For example, it recognizes driving patterns based on video data acquired by a forward-facing camera and speed data acquired by an acceleration sensor, and detects dangerous driving such as sudden braking or sudden acceleration.

[1576] For example, if a driver repeatedly brakes suddenly, the server analyzes this driving pattern and flags it as dangerous driving. At this time, a warning message such as "You have repeatedly braked suddenly. Please drive carefully" is displayed on the driver's device. In addition, dangerous driving behavior is recorded in a rating system, and if a certain number of dangerous driving incidents are accumulated, the driver's account will be temporarily suspended.

[1577] In-car monitoring function

[1578] The server collects data from the in-car image capture and audio capture devices and analyzes it using a generative algorithm. If inappropriate behavior or conversations are detected in the car, it is reported to an administrator. The administrator will review the situation based on the report and suspend the driver's account if necessary.

[1579] For example, if a driver makes an inappropriate remark to a passenger, the server will analyze the voice data and determine that the remark is inappropriate. This information will be immediately reported to the administrator, and the driver's account will be suspended.

[1580] Hospitality features

[1581] The server collects user attribute information (such as age, gender, nationality, and destination), and based on that information, a generation algorithm recommends optimal conversation content and services. The recommendations are presented to the driver, who then provides the user with services based on them.

[1582] For example, if a passenger is a tourist from abroad, the server generates a recommendation such as "I'll show you around tourist spots in this area" based on the user's nationality and destination information. The driver follows the recommendation and provides the passenger with information about tourist spots.

[1583] Embodiment of Emotion Engine

[1584] The server collects data such as the user's facial expressions, voice tone, and speech content from the vehicle's image capture device, audio collection device, and other sensors, and analyzes it using an emotion engine to recognize the user's emotions, such as stress, anxiety, and joy, in real time and provide appropriate feedback based on that.

[1585] For example, if the emotion engine detects that the user is feeling stressed during the ride, the server will instruct the device to play relaxation music, recommend appropriate advice to the driver, and encourage conversations that will help the user relax.

[1586] Example prompt sentence:

[1587] "Please explain how the system analyzes the user's emotions and provides appropriate feedback."

[1588] This system is expected to provide a safe and comfortable driving experience and increase user satisfaction.

[1589] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1590] Embodiment of operation monitoring function

[1591] Processing Steps

[1592] Step 1:

[1593] The server collects data in real time from image capture devices and detection devices. The inputs for this step are video data from cameras and numerical data from speed sensors and acceleration sensors. Specifically, the server acquires this data and prepares it for analysis.

[1594] Step 2:

[1595] The server inputs the collected data into a generation algorithm. The output of this step is the processed data required for analysis. In this step, for example, video data is converted into an analyzable form for each frame, and velocity and acceleration data are processed as time series data.

[1596] Step 3:

[1597] The server analyzes the data using a generative algorithm to recognize driving behavior patterns. The input to this step is a series of frames of data, and the output is the extracted driving behavior patterns. Specifically, behaviors such as sudden braking and sudden acceleration are recognized at this stage.

[1598] Step 4:

[1599] The server detects dangerous driving based on the analysis results. The input of this step is the data analysis results, and the output is the detection result of dangerous driving. In this step, for example, repeated sudden braking is identified as dangerous.

[1600] Step 5:

[1601] The server sends a warning message to the driver based on the detection result of dangerous driving. The input of this step is the detection result of dangerous driving, and the output is a warning message. Specifically, the message "There have been repeated sudden braking. Please drive carefully" is displayed on the driver's device.

[1602] Step 6:

[1603] The server records the risky driving behavior in the rating system and stores the data. The input of this step is the risky driving detection data, and the output is an updated rating record. Specifically, the driver's risky driving behavior is recorded in the database.

[1604] Step 7:

[1605] The server suspends the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system. The input of this step is the accumulated rating data, and the output is a change in the account status. The specific operation is to suspend the driver's account.

[1606] Embodiment of in-vehicle monitoring function

[1607] Processing Steps

[1608] Step 1:

[1609] The server collects data from the image capture device and audio collection device inside the vehicle. The inputs in this step are video and audio data from the camera and microphone. Specifically, conversations and actions inside the vehicle are recorded.

[1610] Step 2:

[1611] The server then inputs the collected data into a generative algorithm for analysis. The output of this step is the processed data required for analysis. Specifically, the audio data is converted into text and calculations are performed to identify inappropriate behavior or comments.

[1612] Step 3:

[1613] The server analyzes the data using a generative algorithm to detect inappropriate behavior or conversation. The input to this step is the analysis results, and the output is the detection of inappropriate behavior or conversation. For example, statements identified as "sexual harassment" are detected.

[1614] Step 4:

[1615] The server reports to the administrator based on the results of the detection of inappropriate behavior or conversation. The input of this step is the detection result of inappropriate behavior or conversation, and the output is a report message. Specifically, a notification saying "The driver made an inappropriate remark" is sent to the administrator's terminal.

[1616] Step 5:

[1617] The server suspends the driver's account when a report is made. The input of this step is the administrator's decision, and the output is a change in the account status. Specifically, the driver's account is suspended.

[1618] Embodiment of hospitality function

[1619] Processing Steps

[1620] Step 1:

[1621] The server collects user attribute information. The input for this step is the user's profile data, and the output is the collected attribute information. Specifically, data entered by the user, such as age, nationality, and destination, is obtained.

[1622] Step 2:

[1623] The server executes a generation algorithm based on the collected attribute information. The output of this step is the analyzed recommendation data. Specifically, calculations are performed to generate recommendation content from the attribute information.

[1624] Step 3:

[1625] The server uses the generation algorithm to recommend optimal conversation topics and services. The input to this step is the analysis results, and the output is the recommended content. For example, a recommendation such as "Please recommend some tourist spots" may be generated.

[1626] Step 4:

[1627] The server presents the recommendation content to the driver. The input of this step is the recommendation content, and the output is the information to be presented. Specifically, a message such as "Would you like me to show you tourist spots in this area?" is displayed on the driver's terminal.

[1628] Step 5:

[1629] The driver provides the user with a service based on the recommendations. The input of this step is the presented information, and the output is the provided service. Specifically, the driver provides the user with information about tourist spots.

[1630] Embodiment of Emotion Engine

[1631] Processing Steps

[1632] Step 1:

[1633] The server collects user data from the vehicle's image capture device, audio capture device, and other sensors. The inputs for this step include facial expressions, voice tone, and speech content. Specifically, facial expressions are captured by the camera and audio is recorded by the microphone.

[1634] Step 2:

[1635] The server then inputs the collected data into a generative algorithm for analysis. The output of this step is the processed data required for analysis. Specifically, facial expression data is converted into an analyzable format, and voice tones are analyzed.

[1636] Step 3:

[1637] The server analyzes the data using an emotion engine to recognize the user's emotion. The input of this step is the analysis result, and the output is the emotion recognition result. For example, it recognizes that the user is in a stressful state.

[1638] Step 4:

[1639] The server provides appropriate feedback based on the emotion recognition results. The input of this step is the emotion recognition results, and the output is the feedback content. Specifically, an instruction to play relaxation music is sent to the terminal.

[1640] Step 5:

[1641] The server recommends appropriate advice to the driver and encourages conversation that will help the user relax. The input to this step is the emotion recognition result and feedback content, and the output is a recommendation to the driver. For example, advice such as "The user is feeling stressed. Please have a relaxing conversation" is provided to the driver.

[1642] (Application example 2)

[1643] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1644] Conventional driving monitoring systems and in-car monitoring systems focus on detecting dangerous driving and inappropriate behavior, but lack feedback on the emotions and psychological states of drivers and passengers. Furthermore, inappropriate comments are often not detected and reported in real time, making it difficult to respond immediately. As a result, improving the quality of service provided by drivers and increasing passenger comfort remains a challenge.

[1645] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from cameras and sensors mounted on the vehicle, means for analyzing the collected data using generative artificial intelligence to monitor driving behavior, means for detecting reckless driving based on the analysis results and sending a warning to the driver, means for recording reckless driving behavior and saving the recorded data in a rating system, means for suspending the account of a driver who has accumulated a certain number of reckless driving incidents based on the rating system, means for recognizing emotions in the vehicle, means for providing appropriate feedback based on the recognized emotions, and means for analyzing voice data and detecting inappropriate remarks. This not only monitors reckless driving and inappropriate behavior, but also provides feedback based on the emotions of the driver and passengers, improving service quality and ensuring a comfortable and safe riding experience.

[1646] A "camera" is a device that captures video data and is used to record and distribute the situation inside and outside a vehicle in real time.

[1647] A "sensor" is a device that measures a physical phenomenon and outputs it as a digital or analog signal, and is used to collect data such as speed, acceleration, and impact.

[1648] "Generative AI" is a technology that analyzes and predicts based on large amounts of data, and is an algorithm that automatically performs various processes such as recognizing driving behavior and emotions and detecting inappropriate remarks.

[1649] "Driving behavior" refers to a series of actions and behaviors related to driving a vehicle, including speed changes, braking operations, lane changes, etc.

[1650] "Warnings" are notifications or messages sent to inform drivers of unsafe behavior or conditions and encourage safe driving.

[1651] The "evaluation system" is a system that evaluates and records the behavior of drivers and passengers based on collected data and manages the evaluation results.

[1652] "Emotion recognition" is a technology that uses in-car cameras and microphones to analyze the facial expressions and tone of voice of passengers and estimate their emotional state.

[1653] "Feedback" refers to responses or advice provided to drivers and passengers based on the results of emotion recognition, including playing relaxing music or recommending conversations.

[1654] "Inappropriate remarks" refer to inappropriate behavior such as sexual harassment or discriminatory content that occurs inside the vehicle, and are detected from audio data analyzed using artificial intelligence.

[1655] "Reporting" is the act of immediately notifying an administrator when inappropriate behavior or remarks are detected, and signals a situation that requires appropriate action.

[1656] The system of the present invention includes several key functions to ensure safe and comfortable operation of autonomous vehicles. These include driving monitoring, in-vehicle monitoring, hospitality, and emotion recognition. The system collects data from cameras and sensors mounted on the vehicle and analyzes it using generative artificial intelligence (generative AI) to provide various services and feedback.

[1657] Embodiment of operation monitoring function

[1658] The server collects data in real time from cameras and sensors installed in the vehicle (e.g., forward-facing cameras, speed sensors, and acceleration sensors). This data is analyzed by generative AI to monitor driving behavior. For example, if repeated sudden braking is detected, the server recognizes this driving pattern as dangerous and sends a warning message to the driver. This dangerous driving behavior is recorded in the rating system and reflected in the driver's account rating. If a certain number of dangerous driving incidents are accumulated, the server suspends the driver's account.

[1659] Embodiment of in-vehicle monitoring function

[1660] The server collects video and audio data from cameras and microphones installed inside the vehicle. It uses generative artificial intelligence to analyze this data and detect inappropriate behavior or speech inside the vehicle. For example, if a driver makes inappropriate remarks to a passenger, the server analyzes the audio data and detects that the remarks are inappropriate. This data is reported to administrators in real time, and the driver's account is suspended if necessary.

[1661] Embodiment of hospitality function

[1662] The server collects profile information such as the user's age, gender, nationality, and destination, and uses generative AI to recommend the most appropriate conversation topics and services. For example, if the passenger is a foreign tourist, the server might recommend "information about tourist spots in the area." The driver provides services to the passenger based on these recommendations, allowing the user to receive a comprehensive service.

[1663] Embodiment of Emotion Recognition Function

[1664] The server collects data from the vehicle's cameras, microphones, and other sensors, and uses an emotion engine to analyze the user's facial expressions, voice tone, and speech. If the emotion engine detects that the user is feeling stressed during the ride, the server instructs the device to play relaxation music. The server also recommends appropriate advice to the driver and encourages conversations that will help the user relax.

[1665] Hardware and software used

[1666] This system is implemented using the following hardware and software:

[1667] Cameras and microphones: Devices installed in vehicles to collect video and audio data.

[1668] Generative AI model: A model using TensorFlow / Keras that performs emotion recognition and driving pattern recognition.

[1669] Server: Collects data, analyzes it, sends warning messages, and provides feedback.

[1670] Rating system: A system that evaluates and records the behavior of drivers and passengers based on collected data.

[1671] This allows the system to monitor and provide feedback in real time, significantly improving safety and comfort inside the vehicle.

[1672] Specific examples

[1673] Example prompt sentence:

[1674] If a user feels stressed while watching a movie, the emotion engine is designed to analyze that stress and provide appropriate touch feedback or music. If a user expresses gratitude, the engine will analyze the utterance and generate a corresponding response.

[1675] In this way, combining emotion recognition with adaptive feedback can improve the quality of the user experience.

[1676] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1677] Step 1:

[1678] The server collects data in real time from the cameras and sensors installed in the vehicle. The data includes various information such as video, speed, and acceleration. The data from the cameras and sensors is input, and this data is stored on the server as output.

[1679] Step 2:

[1680] The server inputs the collected data into a generative AI model to analyze driving behavior and the situation inside the vehicle. The generative AI model is trained using TensorFlow / Keras and analyzes driving patterns and driver behavior from video data, and speech content from audio data. The input is data collected from cameras and sensors, and the output is the analysis results.

[1681] Step 3:

[1682] The server detects dangerous driving and inappropriate behavior based on the analysis results of the generative AI model. For example, if sudden braking or unreasonable lane changes are detected, this is recognized as dangerous driving. It also detects inappropriate remarks in the voice data. The input is the analysis results, and the output is the detection results of dangerous driving and inappropriate behavior.

[1683] Step 4:

[1684] The server sends a warning message to the driver based on the detection results. If inappropriate behavior is detected, it also notifies the administrator. The contents of the warning message and the report are displayed on the driver's and administrator's devices. The input is the detection result, and the output is the warning message and the report.

[1685] Step 5:

[1686] The server stores records of dangerous driving and inappropriate behavior in a rating system and updates the driver's rating. If a certain number of dangerous driving incidents are accumulated, the server suspends the driver's account. The input is the detection results and rating system data, and the output is the updated rating system data and a notification of account suspension.

[1687] Step 6:

[1688] The server collects user profile information (such as age, gender, nationality, and destination) and uses a generative AI model to recommend optimal conversation topics and services. The recommendations are presented to the driver via their device. The input is the user profile information, and the output is the recommendations themselves.

[1689] Step 7:

[1690] When using the emotion recognition function, the server collects the user's facial expressions, voice tone, and speech content from the in-car camera and microphone, and analyzes them using the emotion engine. It recognizes emotions such as stress, anxiety, and joy, and generates appropriate feedback based on the input data. It sends commands to play relaxation music or recommend conversations that will help the driver relax. The input is data from the camera and microphone, and the output is feedback and recommendation commands.

[1691] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1692] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1693] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1694] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1695] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1696] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1697] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1698] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1699] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1700] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1701] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1702] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1703] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1704] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1705] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1706] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1707] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1708] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1709] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1710] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1711] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1712] The following is further disclosed regarding the above embodiment.

[1713] (Claim 1)

[1714] means for collecting data from cameras and sensors mounted on the vehicle;

[1715] A means for analyzing the collected data using artificial intelligence to monitor driving behavior;

[1716] a means for detecting risky driving based on the analysis result and sending a warning to the driver;

[1717] a means for recording and storing in a rating system the risky driving behavior;

[1718] A system that includes a means to suspend the account of a driver who has accumulated a certain number of dangerous driving incidents based on a rating system.

[1719] (Claim 2)

[1720] means for collecting data from cameras and microphones within the vehicle;

[1721] The collected data will be analyzed using artificial intelligence to monitor inappropriate behavior and conversations in the car.

[1722] A means to report inappropriate behavior or conversations to an administrator;

[1723] 10. The system of claim 1, further comprising means for suspending the driver's account if a report is made.

[1724] (Claim 3)

[1725] A means of collecting profile information such as the user's age, gender, nationality, and destination;

[1726] A means for artificial intelligence to recommend optimal conversation content and services based on collected profile information,

[1727] A means of presenting the recommendations to the driver,

[1728] 10. The system of claim 1, further comprising means for the driver to provide services to the user based on the recommendations.

[1729] "Example 1"

[1730] (Claim 1)

[1731] means for collecting data from cameras and sensors mounted on the vehicle;

[1732] A means of analyzing the collected data using a generative AI model and monitoring driving behavior;

[1733] a means for detecting risky driving based on the analysis result and sending a warning to the driver;

[1734] a means for recording and storing in a rating system the risky driving behavior;

[1735] A measure to suspend the accounts of drivers who have accumulated a certain number of dangerous driving incidents based on a rating system;

[1736] means for collecting data from cameras and microphones installed within the vehicle;

[1737] The collected data will be analyzed using a generative AI model to monitor inappropriate behavior and conversations in the car.

[1738] A means to report inappropriate behavior or conversations to an administrator;

[1739] means of collecting user profile information;

[1740] A means for a generative AI model to recommend optimal conversation content and services based on collected profile information, and

[1741] A means of presenting the recommendations to the driver,

[1742] A system that includes a means for drivers to provide services to users based on recommendations.

[1743] (Claim 2)

[1744] means for collecting data from cameras and microphones within the vehicle;

[1745] The collected data will be analyzed using a generative AI model to monitor inappropriate behavior and conversations in the car.

[1746] A means to report inappropriate behavior or conversations to an administrator;

[1747] 10. The system of claim 1, further comprising means for suspending the driver's account if a report is made.

[1748] (Claim 3)

[1749] A means of collecting profile information such as the user's age, gender, nationality, and destination;

[1750] A means for a generative AI model to recommend optimal conversation content and services based on collected profile information, and

[1751] A means of presenting the recommendations to the driver,

[1752] 10. The system of claim 1, further comprising means for the driver to provide services to the user based on the recommendations.

[1753] "Application Example 1"

[1754] (Claim 1)

[1755] means for collecting data from cameras and sensors mounted on the vehicle;

[1756] A means for analyzing the collected data using artificial intelligence to monitor driving behavior;

[1757] a means for detecting risky driving based on the analysis result and sending a warning to the driver;

[1758] a means for recording and storing in a rating system the risky driving behavior;

[1759] A measure to suspend the accounts of drivers who have accumulated a certain number of dangerous driving incidents based on a rating system;

[1760] A means for analyzing video data from the camera using artificial intelligence;

[1761] a means for sending an alert to a server when dangerous driving occurs based on the analysis result of the video data;

[1762] means for issuing an immediate warning when a driving behavior exceeding a set threshold is detected;

[1763] A system that includes a means for recommending optimal conversation content and services based on a user's profile information.

[1764] (Claim 2)

[1765] means for collecting data from cameras and microphones within the vehicle;

[1766] The collected data will be analyzed using artificial intelligence to monitor inappropriate behavior and conversations in the car.

[1767] A means to report inappropriate behavior or conversations to an administrator;

[1768] A means to suspend a driver's account if a report is made;

[1769] A means for detecting inappropriate remarks by analyzing the collected voice data;

[1770] 2. The system according to claim 1, further comprising means for immediately notifying an administrator when a detected inappropriate comment is determined to be inappropriate.

[1771] (Claim 3)

[1772] A means of collecting profile information such as the user's age, gender, nationality, and destination;

[1773] A means for artificial intelligence to recommend optimal conversation content and services based on collected profile information,

[1774] A means of presenting the recommendations to the driver,

[1775] A means for drivers to provide services to users based on recommendations;

[1776] The system according to claim 1, further comprising means for providing optimal conversation content and services to the driver in real time.

[1777] "Example 2: Combining Emotion Engines"

[1778] (Claim 1)

[1779] means for collecting data from image capture and detection devices mounted on the vehicle;

[1780] a means for analyzing the collected data using a generating algorithm and monitoring driving operations;

[1781] a means for detecting dangerous driving based on the analysis result and sending a warning to the driver;

[1782] a means for recording and storing in a rating system the risky driving behavior;

[1783] and means for suspending the account of a driver who has accumulated a certain number of dangerous driving incidents based on the rating system.

[1784] (Claim 2)

[1785] means for collecting data from image capture devices and audio collection devices within the vehicle;

[1786] A means of analyzing the collected data using a generative algorithm to monitor inappropriate behavior and conversations in the car;

[1787] A means to report inappropriate behavior or conversations to an administrator;

[1788] 10. The system of claim 1, further comprising means for suspending the driver's account if a report is made.

[1789] (Claim 3)

[1790] A means for collecting user attribute information;

[1791] A means for a generation algorithm to recommend optimal conversation content and services based on collected attribute information, and

[1792] A means for presenting the recommendation content to the driver;

[1793] 2. The system according to claim 1, further comprising means for the driver to provide services to the user based on the recommendations.

[1794] "Application example 2 when combining emotion engines"

[1795] (Claim 1)

[1796] means for collecting data from cameras and sensors mounted on the vehicle;

[1797] A means for analyzing the collected data using artificial intelligence to monitor driving behavior;

[1798] a means for detecting risky driving based on the analysis result and sending a warning to the driver;

[1799] a means for recording and storing in a rating system the risky driving behavior;

[1800] A measure to suspend the accounts of drivers who have accumulated a certain number of dangerous driving incidents based on a rating system;

[1801] a means for recognizing emotions within the vehicle;

[1802] a means for providing appropriate feedback based on the perceived emotions;

[1803] A means for analyzing the audio data and detecting inappropriate remarks;

[1804] A system including:

[1805] (Claim 2)

[1806] means for collecting data from cameras and microphones within the vehicle;

[1807] The collected data will be analyzed using artificial intelligence to monitor inappropriate behavior and conversations in the car.

[1808] ...

Claims

1. means for collecting data from cameras and sensors mounted on the vehicle; A means for analyzing the collected data using artificial intelligence to monitor driving behavior; a means for detecting risky driving based on the analysis result and sending a warning to the driver; a means for recording and storing in a rating system the risky driving behavior; A system that includes a means to suspend the account of a driver who has accumulated a certain number of dangerous driving incidents based on a rating system.

2. means for collecting data from cameras and microphones within the vehicle; The collected data will be analyzed using artificial intelligence to monitor inappropriate behavior and conversations in the car. A means to report inappropriate behavior or conversations to an administrator; 10. The system of claim 1, further comprising means for suspending a driver's account if a report is made.

3. A means of collecting profile information such as the user's age, gender, nationality, and destination; A means for artificial intelligence to recommend optimal conversation content and services based on collected profile information, A means of presenting the recommendations to the driver, 10. The system of claim 1, further comprising means for the driver to provide services to the user based on the recommendations.

Images