Personalized driving control method for a vehicle, personalized driving system, and vehicle providing personalized driving control service

Abstract

The personalized driving control method for vehicles of the present invention discloses a step of identifying information about objects and driving conditions around the vehicle based on vehicle sensors and external map-related information during vehicle operation. Furthermore, the personalized driving control method includes: a step of monitoring the gaze and emotional state of passengers in response to the identified information and the vehicle's movement; and a step of controlling at least one of the vehicle's driving and route based on the monitoring to reflect the passengers' emotional state. Additionally, the personalized driving control method includes a step of training a vehicle driving model based on the control results reflecting the passengers' emotional state. Thus, the driver or passenger can experience a more personalized driving experience, and a personalized driving model can be managed solely using driver / passenger information, independent of the vehicle itself.

Description

Technical Field

[0001] This invention relates to a personalized driving control method for a vehicle, and more specifically, to a personalized driving control method for a vehicle that takes into account the emotional state of the passengers, a personalized driving control system, and a vehicle that provides personalized driving control services. Background Technology

[0002] A vehicle is a device that enables its users to move in the desired direction. A car can be taken as a representative example.

[0003] On the other hand, there is a trend towards installing various sensors and electronic devices to enhance user convenience. In particular, research is actively underway on Advanced Driver Assistance Systems (ADAS) to improve driving convenience for users. Furthermore, autonomous vehicles are being actively developed.

[0004] Automatic driving refers to a system that enables a vehicle to make its own judgments and drive itself. As mentioned above, automatic driving can be divided into progressive steps from non-automation to full automation, based on the degree of system involvement in driving and the degree of control the driver has over the vehicle.

[0005] However, as mentioned above, even when driver assistance systems are used to help the driver drive or to perform fully automated driving based on the level of system involvement, the individual state of the driver or passengers is not taken into account, which may result in a sense of alienation in the driving experience.

[0006] For example, while the vehicle is in motion, continuous exposure to pedestrians or other pedestrians on the roadside can lead to greater feelings of unease, depending on the driver or passenger. Alternatively, some drivers or passengers may experience persistent discomfort when other vehicles overtake them. Or, even when the vehicle is traveling at the legally permitted speed, some drivers or passengers may feel more fear.

[0007] Relatedly, WO1999FR002571 (hereinafter referred to as "Prior Document 1") discloses detecting changes in the shape of the driver's eyes or double eyelids to determine whether the driver is drowsy, and thereby issuing warnings about the driving environment or situation. However, Prior Document 1 only warns about the driver's state and has the limitation of not participating in driving control.

[0008] Another document, US2022 / 0126863 (hereinafter referred to as "Prior Document 2"), discloses a technique for determining vehicle driving modes using vehicle sensor information, and vehicle driving modes include driving modes such as ride comfort / arrival time priority / fuel consumption priority.

[0009] However, existing literature 2 has limitations in reflecting individual differences among passengers. In addition, existing literature 2 operates according to a predetermined driving mode under matched driving conditions, thus presenting difficulties in implementing personalized driving for specific driving conditions. Summary of the Invention

[0010] The problem that the invention aims to solve

[0011] The purpose of this invention is to solve the above-mentioned problems and other issues.

[0012] According to some embodiments of the present invention, the object is to provide a personalized driving control method, a personalized driving system, and a vehicle that provides personalized driving control services, which can minimize the driving heterogeneity that the driver or passengers may feel when the vehicle is driving autonomously.

[0013] Furthermore, according to some embodiments of the present invention, the object is to provide a personalized driving control method, a personalized driving system, and a vehicle that adaptively controls driving during autonomous driving by reflecting the emotional state felt by the driver or passenger.

[0014] Furthermore, according to some embodiments of the present invention, the objective is to provide a personalized driving control method, a personalized driving system, and a vehicle that generates a personalized driving model by reflecting the emotional state of the driver or passenger of the vehicle, as well as a vehicle through personalized driving control services.

[0015] Technical solutions to the problem

[0016] Therefore, the personalized driving control method for vehicles according to embodiments of the present invention can reflect the emotional state of the driver / passenger based on information identified during vehicle driving, thereby adaptively adjusting the vehicle's driving or route changes.

[0017] Additionally, the control results reflecting the driver's / passenger's emotional state in response to the identified information can be used as filtered input data to train the driving model. This allows for management using customized driving models tailored to each driver / passenger.

[0018] Specifically, the personalized driving control method for a vehicle according to embodiments of the present invention may include the following steps. The personalized driving control method may include: identifying information about objects and driving conditions around the vehicle based on vehicle sensors and external map information during vehicle operation; monitoring the gaze and emotional state of passengers in response to the identified information and the vehicle's movement; and controlling at least one of the vehicle's driving and route based on the monitoring to reflect the passengers' emotional state. Additionally, the personalized driving control method may further include training a vehicle driving model based on control results reflecting the passengers' emotional state.

[0019] Additionally, according to an embodiment, the monitoring step may include: collecting data related to the passenger's gaze and emotional state using at least one of the DMS and IMS included in the vehicle; and determining, based on the collected data, whether the identified information and the emotional state of the passenger in relation to the vehicle's operation are in a state of stress.

[0020] Additionally, according to an embodiment, the monitoring step may include: if the identified information is about objects around the vehicle, then monitoring the emotional state based on data related to the passenger's line of sight; if the identified information is about attributes different from those of objects around the vehicle, then monitoring the emotional state based on data about changes in the passenger's facial temperature and heart rate.

[0021] Additionally, according to an embodiment, the monitoring step may include determining the identified information, including composite information, that matches the pressure state.

[0022] Additionally, according to an embodiment, the control step may be a step of controlling the driving or route of each vehicle matched with each of the composite information in a direction that alleviates the stress state of the passenger.

[0023] Additionally, according to an embodiment, the control step may be a step of weighted control of the driving or route of a vehicle matched with one of the composite information in a direction that alleviates the stress state of the passenger.

[0024] Additionally, according to an embodiment, the step of training the driving model may include training the driving model based on the control results differently according to the attributes of the identified information.

[0025] Additionally, according to an embodiment, the step of training the driving model may include: generating first driving information based on the identified information using a pre-trained driving model, generating second driving information about the first driving information based on the emotional state of the passenger in the first driving information, and training the driving model based on the second driving information.

[0026] Additionally, according to an embodiment, the monitoring step may include: identifying the presence of other vehicles traveling alongside the vehicle in its lane, and monitoring the emotional state of passengers affected by the identified information; the control step may include: adjusting the vehicle's deviation from the road away from the other vehicles while they are traveling alongside the vehicle; and if the other vehicles are no longer traveling alongside the vehicle, reverting the vehicle's deviation from the road back to its previous state and continuing to travel.

[0027] Additionally, according to an embodiment, the monitoring step may include: identifying that the vehicle is traveling in an area that meets a first condition, and monitoring the stress emotions of the passengers based on the identified information; the control step may include: the vehicle changing its route to avoid traveling in the area that meets the first condition, or further adjusting at least one of the travel interval and travel speed during the period when traveling in the area that meets the first condition.

[0028] Additionally, according to an embodiment, the monitoring step may include: monitoring the passenger's stress level while the vehicle is traveling on a steep road; the control step may include: further reducing the vehicle's speed in accordance with the degree of the passenger's stress level.

[0029] Additionally, according to an embodiment, the monitoring step may include: monitoring the stress emotions of a passenger overtaking the vehicle in front; the control step may include: resetting the distance between the vehicle and the vehicle in front to a narrower and driving in order to reflect the stress emotions of the passenger.

[0030] Furthermore, the personalized driving system for a vehicle according to embodiments of the present invention may include an interface unit that receives sensor data from the vehicle's sensors and external map information during vehicle operation, an identification unit that identifies information about objects around the vehicle and driving conditions based on the received sensor data and external map information, a judgment unit, and a control unit. The judgment unit may monitor the gaze and emotional state of the occupants in response to the identified information and the vehicle's movement, and determine whether to change at least one of the vehicle's driving direction and route based on the monitoring and the occupants' emotional state. Alternatively, the control unit may control at least one of the vehicle's driving direction and route based on the determination, and control the vehicle to train a driving model based on the control result reflecting the occupants' emotional state.

[0031] Furthermore, the computer program recorded in the recording medium in embodiments of the present invention can be combined with a computing device comprising a memory, a transceiver, and a processor for processing instructions stored in the memory, so that the processor performs the following actions. These actions may include: identifying information about objects and driving conditions around the vehicle based on vehicle sensors and external map information during vehicle operation; monitoring the gaze and emotional state of passengers in response to the identified information and the vehicle's movement; controlling at least one of the vehicle's movement and route based on the monitoring to reflect the passengers' emotional state; and training a driving model based on the control results reflecting the passengers' emotional state.

[0032] Invention Effects

[0033] The effects of the personalized driving control method, system, and computer recording medium of the present invention will be described below.

[0034] According to at least some embodiments of the present invention, when a vehicle is in autonomous driving mode, the driving or route is adjusted to reflect the emotional state of the driver or passenger in relation to the driving conditions, thereby minimizing the sense of alienation during driving.

[0035] In addition, according to at least some embodiments of the present invention, the driving model is updated based on the results reflecting the emotional state of the driver or passenger when recognizing the driving conditions during autonomous driving, so that the driver or passenger can experience a more personalized driving and riding experience.

[0036] Furthermore, according to at least some embodiments of the present invention, personalized driving models can be managed by driver or passenger, so that personalized driving can be experienced solely based on driver / passenger information, regardless of the vehicle. Attached Figure Description

[0037] Figure 1This is a diagram illustrating an example of a vehicle related to an embodiment of the present invention.

[0038] Figure 2 These are diagrams showing vehicles from various angles as described in embodiments of the present invention.

[0039] Figure 3 and Figure 4 This is a diagram showing the interior of a vehicle according to an embodiment of the present invention.

[0040] Figure 5 and Figure 6 The diagram is a reference to various objects related to the driving of a vehicle in relation to an embodiment of the present invention.

[0041] Figure 7 This is a block diagram illustrating a vehicle and personalized driving system with reference to embodiments of the present invention.

[0042] Figure 8 This is a block diagram illustrating the conceptual configuration of a personalized driving system in relation to embodiments of the present invention.

[0043] Figure 9 This is a representative flowchart illustrating a personalized driving control method for a vehicle according to embodiments of the present invention.

[0044] Figure 10a and Figure 10b This diagram illustrates how an embodiment of the invention identifies the passenger's stressful emotional state and reflects it in the vehicle's driving control.

[0045] Figure 11 This is a flowchart illustrating a method for matching composite identification information and passenger emotional state in relation to embodiments of the present invention.

[0046] Figure 12a , Figure 12b , Figure 13a , Figure 13b , Figure 13c , Figure 14a , Figure 14b , Figure 15a , Figure 15b , Figure 16a , Figure 16b These are diagrams illustrating various embodiments of controlling vehicle movement or route by utilizing feedback from the emotional state of passengers under various driving conditions.

[0047] Figure 17 This diagram illustrates a method for determining the control of vehicle movement or route that reflects the emotional state of passengers when there are multiple passengers, according to an embodiment of the present invention. Detailed Implementation

[0048] Figure 1 and Figure 2 This pertains to the appearance of a vehicle in accordance with an embodiment of the present invention. Figure 3 and Figure 4 This is a diagram showing the interior of a vehicle according to an embodiment of the present invention.

[0049] Figures 5 to 6 This is a diagram illustrating various objects related to the driving of a vehicle in relation to an embodiment of the present invention.

[0050] Figure 7 This is a block diagram illustrating a vehicle with reference to an embodiment of the present invention. Figure 7 This is a block diagram illustrating a vehicle according to an embodiment of the present invention.

[0051] Reference Figures 1 to 7 The vehicle 100 may include wheels that are rotated by a power source and a steering input device 510 for adjusting the direction of travel of the vehicle 100.

[0052] Vehicle 100 may be an autonomous vehicle. Vehicle 100 may switch between autonomous driving mode and manual mode based on user input. For example, vehicle 100 may switch from manual mode to autonomous driving mode, or from autonomous driving mode to manual mode, based on user input received through a user interface device (hereinafter, which may be referred to as "user terminal") 200.

[0053] Vehicle 100 can switch between autonomous driving mode and manual mode based on driving condition information. The driving condition information can be generated based on object information provided by object detection device 300. For example, vehicle 100 can switch from manual mode to autonomous driving mode, or vice versa, based on driving condition information generated by object detection device 300. Alternatively, vehicle 100 can switch from manual mode to autonomous driving mode, or vice versa, based on driving condition information received via communication device 400.

[0054] Vehicle 100 can switch from manual mode to autonomous driving mode, or from autonomous driving mode to manual mode, based on information, data, and signals provided by external devices.

[0055] When vehicle 100 is operating in autonomous driving mode, autonomous vehicle 100 can operate based on operating system 700. For example, autonomous vehicle 100 can operate based on information, data or signals generated in driving system 710, vehicle dispatch system 740, and parking system 750.

[0056] When the vehicle 100 is operating in manual mode, the autonomous vehicle 100 can receive user input for driving via the driving control device 500. The vehicle 100 can operate based on the user input received via the driving control device 500.

[0057] Overall length refers to the length from the front to the rear of vehicle 100, overall width refers to the width of vehicle 100, and overall height refers to the length from the bottom of the wheels to the roof. In the following description, the overall length direction L can refer to the direction based on the overall length measurement of vehicle 100, the overall width direction W can refer to the direction based on the overall width measurement of vehicle 100, and the overall height direction H can refer to the direction based on the overall height measurement of vehicle 100.

[0058] like Figure 7 As exemplified, vehicle 100 may include a user interface device (hereinafter, which may be referred to as "user terminal") 200, an object detection device 300, a communication device 400, a driving operation device 500, a vehicle drive device 600, an operating system 700, a navigation system 770, a sensing unit 120, a vehicle interface unit 130, a memory 140, a control unit 170, and a power supply unit 190.

[0059] According to the embodiments, the vehicle 100 may include other constituent elements in addition to those described in this specification, or may exclude some of the constituent elements described.

[0060] The user interface device 200 is a means for communication between the vehicle 100 and the user. The user interface device 200 can receive user input and provide the user with information generated in the vehicle 100. The vehicle 100 can implement UI (User Interfaces) or UX (User Experience) through the user interface device (hereinafter, may be referred to as "user terminal") 200.

[0061] The user interface device 200 may include an input unit 210, an internal camera 220, a biometric detection unit 230, an output unit 250, and a processor 270. According to an embodiment, the user interface device 200 may include other components besides those described, or may exclude some of the described components.

[0062] The input unit 210 is used to receive information from the user. The data collected in the input unit 210 can be analyzed by the processor 270 and processed into control commands for the user.

[0063] The input unit 210 can be configured inside the vehicle. For example, the input unit 210 can be configured in an area of ​​the steering wheel, an area of ​​the instrument panel, an area of ​​the seat, an area of ​​each pillar, an area of ​​the door, an area of ​​the center console, an area of ​​the headlining, an area of ​​the sun visor, an area of ​​the windshield, or an area of ​​the window, etc.

[0064] The input unit 210 may include a voice input unit 211, a gesture input unit 212, a touch input unit 213, and a mechanical input unit 214.

[0065] The voice input unit 211 can convert the user's voice input into an electrical signal. The converted electrical signal can be provided to the processor 270 or the control unit 170. The voice input unit 211 may include one or more microphones.

[0066] The gesture input unit 212 can convert the user's gesture input into an electrical signal. The converted electrical signal can be provided to the processor 270 or the control unit 170.

[0067] The gesture input unit 212 may include at least one of an infrared sensor and an image sensor for detecting user gesture input. According to an embodiment, the gesture input unit 212 can detect three-dimensional gesture input from the user. For this purpose, the gesture input unit 212 may include a light output unit that outputs a fraction of infrared light or a plurality of image sensors.

[0068] The gesture input unit 212 can detect the user's three-dimensional gesture input through TOF (Time of Flight), structured light, or disparity methods.

[0069] The touch input unit 213 can convert the user's touch input into an electrical signal. The converted electrical signal can be provided to the processor 270 or the control unit 170.

[0070] The touch input unit 213 may include a touch sensor for detecting user touch input. According to an embodiment, the touch input unit 213 is integrated with the display unit 251, thereby realizing a touchscreen. This touchscreen can together provide both an input interface and an output interface between the vehicle 100 and the user.

[0071] The mechanical input unit 214 may include at least one of a button, a dome switch, a rotary knob, and a rotary switch. The electrical signal generated by the mechanical input unit 214 can be provided to the processor 270 or the control unit 170. The mechanical input unit 214 may be configured in the steering wheel, central instrument panel, center console, cockpit module, door, etc.

[0072] The interior camera 220 can acquire images of the vehicle's interior. The processor 270 can detect the user's state based on the images of the vehicle's interior. The processor 270 can acquire the user's gaze information from the images of the vehicle's interior. The processor 270 can detect the user's gestures from the images of the vehicle's interior.

[0073] The biometric detection unit 230 can acquire a user's biometric information. The biometric detection unit 230 may include sensors capable of acquiring the user's biometric information, such as fingerprints and heart rate data. This biometric information can be used for user authentication.

[0074] The output unit 250 is used to generate outputs related to vision, hearing, or touch. The output unit 250 may include at least one of a display unit 251, a sound output unit 252, and a tactile output unit 253.

[0075] Display unit 251 can display graphic objects corresponding to various information. Display unit 251 may include at least one of liquid crystal display (LCD), thin film transistor-liquid crystal display (TFT LCD), organic light-emitting diode (OLED), flexible display, 3D display, and e-ink display.

[0076] The display unit 251 and the touch input unit 213 form a layer structure or are integrated together, thereby realizing a touch screen.

[0077] The display unit 251 can be implemented by a HUD (Head-Up Display). When the display unit 251 is implemented by a HUD, the display unit 251 can be provided with a projection module to output information by projecting an image onto a windshield or window.

[0078] Display unit 251 may include a transparent display. The transparent display may be attached to a windshield or window. The transparent display may have a specified transparency and display a specified image. To achieve transparency, the transparent display may include at least one of the following: transparent TFEL (Thin Film Electroluminescent), transparent OLED (Organic Light-Emitting Diode), transparent LCD (Liquid Crystal Display), transmissive transparent display, and transparent LED (Light Emitting Diode) display. The transparency of the transparent display is adjustable.

[0079] On the other hand, the user interface device 200 may include a plurality of display units 251a to 251g.

[0080] The display unit 251 can be configured in a region of the steering wheel, a region of the instrument panel 521a, 251b, 251e, a region of the seat 251d, a region of each pillar 251f, a region of the door 251g, a region of the center console, a region of the roof, a region of the sun visor, or it can be implemented in a region of the windshield 251c or a region of the window 251h.

[0081] The sound output unit 252 converts the electrical signals provided by the processor 270 or the control unit 170 into audio signals and outputs them. For this purpose, the sound output unit 252 may include more than one speaker.

[0082] The tactile output unit 253 generates tactile output. For example, the tactile output unit 253 can be activated by causing the steering wheel, seat belt, and seats 110FL, 110FR, 110RL, and 110RR to vibrate so that the user can recognize the output.

[0083] The processor (hereinafter, may be referred to as the "control unit") 270 can control the overall operation of each unit of the user interface device 200. According to an embodiment, the user interface device 200 may include a plurality of processors 270, or may not include any processors 270.

[0084] If the user interface device 200 does not include the processor 270, the user interface device 200 can operate according to the control of the processor or control unit 170 of other devices in the vehicle 100.

[0085] On the other hand, the user interface device 200 can be named a vehicle display device. The user interface device 200 can operate according to the control of the control unit 170.

[0086] The object detection device 300 is a device for detecting objects located outside the vehicle 100. The objects can be various objects related to the operation of the vehicle 100. (See reference...) Figures 5 to 6 Object O can include lane OB10, other vehicles OB11, pedestrians OB12, two-wheeled vehicles OB13, traffic signals OB14, OB15, light, roads, structures, speed bumps, terrain features, animals, etc.

[0087] Lane OB10 can be a driving lane, a lane adjacent to a driving lane, or a lane for oncoming vehicles. Lane OB10 can also include the concept of the left and right side lines that form the lane.

[0088] Other vehicles OB11 can be vehicles traveling around vehicle 100. Other vehicles can be vehicles located within a specified distance of vehicle 100. For example, other vehicles OB11 can be vehicles that travel before or after vehicle 100.

[0089] Pedestrian OB12 can be a person located around vehicle 100. Pedestrian OB12 can be a person located within a specified distance from vehicle 100. For example, pedestrian OB12 can be a person located on a sidewalk or driveway.

[0090] Two-wheeled vehicles OB12 can be located around vehicle 100 and can refer to riding that uses two wheels for movement. Two-wheeled vehicles OB12 can be two-wheeled vehicles located within a specified distance from vehicle 100. For example, two-wheeled vehicles OB13 can be motorcycles or bicycles located on sidewalks or driveways.

[0091] Traffic signals may include traffic lights OB15, traffic signs OB14, and patterns or text painted on the road surface.

[0092] Light can be generated by lights installed on other vehicles. Light can be generated by streetlights. Light can be sunlight.

[0093] Roads can include road surfaces, curves, uphill slopes, downhill slopes, and other ramps.

[0094] Structures can be located around roads or can be objects fixed to the ground. For example, structures can include streetlights, trees, buildings, utility poles, traffic lights, and bridges.

[0095] Topographic features can include mountains, hills, etc.

[0096] On the other hand, objects can be classified into moving objects and stationary objects. For example, moving objects can include concepts such as other vehicles and pedestrians. Stationary objects can include concepts such as traffic signals, roads, and structures.

[0097] The object detection device 300 may include a camera 310, a radar 320, a lidar 330, an ultrasonic sensor 340, an infrared sensor 350, and a processor 370.

[0098] According to the embodiments, the object detection device 300 may include other components in addition to the components described, or may exclude some of the components described.

[0099] Camera 310 can be located at an appropriate position outside the vehicle to acquire images of the vehicle's exterior. Camera 310 can be a monocular camera, a stereo camera 310a, an around-view monitoring (AVM) camera 310b, or a 360-degree camera.

[0100] For example, camera 310 can be positioned inside the vehicle near the windshield to capture an image of the area in front of the vehicle. Alternatively, camera 310 can be positioned around the front bumper or radiator grille.

[0101] For example, camera 310 can be positioned inside the vehicle near the rear window to capture images of the area behind the vehicle. Alternatively, camera 310 can be positioned around the rear bumper, trunk, or tailgate.

[0102] For example, camera 310 may be configured in at least one of the side windows inside the vehicle to acquire images of the side of the vehicle. Alternatively, camera 310 may be configured around a side mirror, fender, or door.

[0103] The camera 310 can provide the acquired images to the processor 370.

[0104] Radar 320 may include an electromagnetic wave transmitter and a receiver. Radar 320 can be implemented as a pulse radar or a continuous wave radar, based on the principle of electromagnetic wave transmission. In the continuous wave radar mode, radar 320 can be implemented as a frequency-modulated continuous wave (FMCW) or frequency-shift keying (FSK) radar, depending on the signal waveform.

[0105] Radar 320 can detect objects using electromagnetic waves as a medium, based on TOF (Time of Flight) or phase-shift methods. It can detect the position of the detected object, the distance between the detected objects, and the relative speed.

[0106] The radar 320 can be configured at appropriate locations on the exterior of a vehicle to detect objects located in front of, behind, or to the side of the vehicle.

[0107] The lidar 330 may include a laser emitter and a receiver. The lidar 330 may be implemented in a TOF (Time of Flight) mode or a phase-shift mode.

[0108] The LiDAR 330 can be implemented in either driven or non-driven mode.

[0109] In a driven configuration, the lidar 330 can be rotated by a motor to detect objects around the vehicle 100.

[0110] In a non-driven implementation, the lidar 330 can detect objects within a specified range relative to the vehicle 100 via optical steering. The vehicle 100 may include a plurality of non-driven lidars 330.

[0111] The LiDAR 330 can detect objects using laser as a medium, based on TOF (Time of Flight) or phase-shift methods. It can detect the position of the detected object, the distance between the detected objects, and the relative speed.

[0112] The LiDAR 330 can be configured at appropriate locations on the exterior of a vehicle to detect objects located in front of, behind, or to the side of the vehicle.

[0113] The ultrasonic sensor 340 may include an ultrasonic transmitter and a receiver. The ultrasonic sensor 340 can detect objects based on ultrasonic waves, and can detect the position of the detected object, the distance between the detected objects, and the relative speed.

[0114] The ultrasonic sensor 340 can be configured at an appropriate location on the exterior of the vehicle to detect objects located in front of, behind, or to the side of the vehicle.

[0115] Infrared sensor 350 may include an infrared emitter and a receiver. Infrared sensor 340 can detect objects based on infrared light, and can detect the position of the detected object, the distance between the two objects, and the relative speed.

[0116] The infrared sensor 350 can be configured at an appropriate location on the exterior of the vehicle to detect objects located in front of, behind, or to the side of the vehicle.

[0117] The processor 370 can control the overall operation of each unit of the object detection device 300.

[0118] Processor 370 can detect and track objects based on the acquired images. Processor 370 can perform actions such as distance calculation and relative velocity calculation between itself and the object using image processing algorithms.

[0119] Processor 370 can detect and track objects based on the electromagnetic waves reflected back from them. Processor 370 can also perform actions such as distance calculation and relative velocity calculation based on these electromagnetic waves.

[0120] Processor 370 can detect and track objects based on the reflected laser light from the emitted laser. Processor 370 can also perform actions such as distance calculation and relative velocity calculation based on the laser light.

[0121] Processor 370 can detect and track objects based on reflected ultrasonic waves from emitted ultrasonic waves. Processor 370 can also perform actions such as distance calculation and relative speed calculation based on these ultrasonic waves.

[0122] Processor 370 can detect and track objects based on the reflected infrared light from the emitted infrared light. Processor 370 can also perform actions such as distance calculation and relative speed calculation based on the infrared light.

[0123] According to an embodiment, the object detection device 300 may include a plurality of processors 370, or may not include processors 370. For example, the camera 310, radar 320, lidar 330, ultrasonic sensor 340, and infrared sensor 350 may each include a processor individually.

[0124] If the object detection device 300 does not include the processor 370, the object detection device 300 can operate according to the control of the processor or the control unit 170 of the device inside the vehicle 100.

[0125] The object detection device 400 can operate under the control of the control unit 170.

[0126] The communication device 400 is a device for performing communication with external devices. Here, the external device may be another vehicle, a mobile terminal, or a server.

[0127] The communication device 400 may include at least one of a transmitting antenna, a receiving antenna, a radio frequency (RF) circuit capable of implementing various communication protocols, and an RF element to perform communication.

[0128] The communication device 400 may include a short-range communication unit 410, a location information unit 420, a V2X communication unit 430, an optical communication unit 440, a broadcast transceiver unit 450, and a processor 470.

[0129] According to the embodiments, the communication device 400 may include other components in addition to the components described, or may exclude some of the components described.

[0130] The short-range communication unit 410 is a unit for short-range communication. The short-range communication unit 410 can support short-range communication using at least one of the following technologies: Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Near Field Communication (NFC), Wi-Fi, Wi-Fi Direct, and Wireless Universal Serial Bus.

[0131] The short-range communication unit 410 can form a short-range wireless communication network to perform short-range communication between the vehicle 100 and at least one external device.

[0132] The location information unit 420 is a unit used to acquire the location information of the vehicle 100. For example, the location information unit 420 may include a Global Positioning System (GPS) module or a Differential Global Positioning System (DGPS) module.

[0133] The V2X communication unit 430 is a unit used to perform wireless communication with a server (V2I: Vehicle to Infra), other vehicles (V2V: Vehicle to Vehicle), or pedestrians (V2P: Vehicle to Pedestrian). The V2X communication unit 430 may include RF circuitry capable of implementing communication protocols for infrastructure (V2I), vehicle-to-vehicle (V2V), and pedestrian-to-pedestrian (V2P).

[0134] The optical communication unit 440 is a unit for performing communication with external devices using light as a medium. The optical communication unit 440 may include: an optical transmitter that converts electrical signals into optical signals and transmits them to the outside; and an optical receiver that converts received optical signals into electrical signals.

[0135] According to an embodiment, the light emitting part can be integrated with the lamps included in the vehicle 100.

[0136] The broadcast transceiver unit 450 is a unit used to receive broadcast signals from an external broadcast management server or to send broadcast signals to the broadcast management server via a broadcast channel. The broadcast channel may include a satellite channel or a terrestrial channel. The broadcast signal may include TV broadcast signals, radio broadcast signals, or data broadcast signals.

[0137] The processor 470 can control the overall operation of each unit of the communication device 400.

[0138] According to an embodiment, the communication device 400 may include a plurality of processors 470, or may not include processors 470.

[0139] In the absence of a processor 470, the communication device 400 may operate under the control of a processor or control unit 170 of other devices within the vehicle 100.

[0140] On the other hand, the communication device 400 can be used together with the user interface device 200 to implement a vehicle display device. In this case, the vehicle display device can be named a telematics device or an audio-visual navigation (AVN) device.

[0141] The communication device 400 can operate according to the control of the control unit 170.

[0142] The driving control device 500 is a device that receives user input for driving.

[0143] In manual mode, vehicle 100 can operate based on signals provided by driving control device 500.

[0144] The driving control device 500 may include a steering input device 510, an acceleration input device 530, and a braking input device 570.

[0145] The steering input device 510 can receive the driving direction input of the vehicle 100 from the user. The steering input device 510 is preferably configured in the form of a steering wheel so that steering input can be achieved by rotation. According to an embodiment, the steering input device may also be configured as a touch screen, touchpad, or button.

[0146] The accelerator input device 530 can receive input from the user for accelerating the vehicle 100. The brake input device 570 can receive input from the user for decelerating the vehicle 100. The accelerator input device 530 and the brake input device 570 are preferably configured as pedals. According to an embodiment, the accelerator input device or the brake input device may also be configured as a touchscreen, touchpad, or button.

[0147] The driving control device 500 can operate according to the control unit 170.

[0148] The vehicle drive unit 600 is a device that electrically controls the drive of various devices within the vehicle 100.

[0149] The vehicle drive unit 600 may include a powertrain drive unit 610, a chassis drive unit 620, a door / window drive unit 630, a safety device drive unit 640, a light drive unit 650, and an air conditioning drive unit 660.

[0150] According to the embodiments, the vehicle drive unit 600 may include other components in addition to the components described, or may exclude some of the components described.

[0151] On the other hand, the vehicle drive unit 600 may include a processor. Each unit of the vehicle drive unit 600 may individually include a processor.

[0152] The powertrain drive unit 610 can control the operation of the powertrain unit.

[0153] The powertrain drive unit 610 may include a power source drive unit 611 and a transmission drive unit 612.

[0154] The power source drive unit 611 can perform control of the power source of the vehicle 100.

[0155] For example, when a fossil fuel-based engine is used as the power source, the power source drive unit 610 can perform electronic control of the engine. This allows for control of the engine's output torque, etc. The power source drive unit 611 can adjust the engine's output torque according to the control unit 170.

[0156] For example, when an electric motor is used as the power source, the power source drive unit 610 can control the motor. The power source drive unit 610 can adjust the motor's rotational speed, torque, etc., according to the control unit 170.

[0157] The transmission drive unit 612 can control the transmission. The transmission drive unit 612 can adjust the state of the transmission. The transmission drive unit 612 can adjust the state of the transmission to forward (D), reverse (R), neutral (N), or park (P).

[0158] On the other hand, when the engine is the power source, the transmission drive unit 612 can adjust the gear engagement state in the forward D state.

[0159] The chassis drive unit 620 can control the movement of the chassis assembly. The chassis drive unit 620 may include a steering drive unit 621, a braking drive unit 622, and a suspension drive unit 623.

[0160] The steering drive unit 621 can perform electronic control of the steering apparatus within the vehicle 100. The steering drive unit 621 can change the direction of travel of the vehicle.

[0161] The brake drive unit 622 can perform electronic control of the brake apparatus within the vehicle 100. For example, the speed of the vehicle 100 can be reduced by controlling the braking action of the brakes configured on the wheels.

[0162] On the other hand, the brake drive unit 622 can individually control each of the plurality of brakes. The brake drive unit 622 can control the braking force applied to the plurality of wheels differently.

[0163] The suspension drive unit 623 can perform electronic control of the suspension apparatus within the vehicle 100. For example, the suspension drive unit 623 can control the suspension apparatus to reduce vibration of the vehicle 100 when the road surface is uneven. On the other hand, the suspension drive unit 623 can individually control each of the plurality of suspensions.

[0164] The door / window drive unit 630 can perform electronic control of the door apparatus or window apparatus inside the vehicle 100.

[0165] The door / window drive unit 630 may include a door drive unit 631 and a window drive unit 632.

[0166] The door drive unit 631 can control the door mechanism. The door drive unit 631 can control the opening and closing of a plurality of doors included in the vehicle 100. The door drive unit 631 can control the opening and closing of the trunk or tailgate. The door drive unit 631 can control the opening and closing of the sunroof.

[0167] The window drive unit 632 can perform electronic control of the window apparatus. It can control the opening or closing of a plurality of windows included in the vehicle 100.

[0168] The safety device drive unit 640 can perform electronic control of various safety devices within the vehicle 100.

[0169] The safety device drive unit 640 may include an airbag drive unit 641, a seat belt drive unit 642, and a pedestrian protection device drive unit 643.

[0170] The airbag actuator 641 can perform electronic control of the airbag apparatus within the vehicle 100. For example, the airbag actuator 641 can control the deployment of the airbag when a hazard is detected.

[0171] The seatbelt drive unit 642 can perform electronic control of the seatbelt apparatus within the vehicle 100. For example, the seatbelt drive unit 642 can control the use of seatbelts to secure passengers to seats 110FL, 110FR, 110RL, and 110RR when a hazard is detected.

[0172] The pedestrian protection device drive unit 643 can perform electronic control of the hood lifting mechanism and the pedestrian airbag. For example, the pedestrian protection device drive unit 643 can control the hood lifting mechanism to lift and the pedestrian airbag to deploy when a collision with a pedestrian is detected.

[0173] The lamp drive unit 650 can perform electronic control of various lamp apparatuses within the vehicle 100.

[0174] The air conditioning drive unit 660 can perform electronic control of the air conditioning unit within the vehicle 100. For example, when the temperature inside the vehicle is high, the air conditioning drive unit 660 can control the air conditioning unit to operate and supply cool air to the vehicle interior.

[0175] The vehicle drive unit 600 may include a processor. Each unit of the vehicle drive unit 600 may individually include a processor.

[0176] The vehicle drive unit 600 can operate according to the control of the control unit 170.

[0177] The operating system 700 is a system that controls various operations of the vehicle 100. The operating system 700 can operate in automatic driving mode.

[0178] The operating system 700 may include a driving system 710, a vehicle dispatching system 740, and a parking system 750.

[0179] According to the embodiments, the operating system 700 may include other components in addition to the components described, or may exclude some of the components described.

[0180] On the other hand, the operating system 700 may include a processor. Each unit of the operating system 700 may individually include a processor.

[0181] On the other hand, according to the embodiment, when the running system 700 is implemented as software, it can also be a subordinate concept of the control unit 170.

[0182] On the other hand, according to the embodiment, the operating system 700 may be a concept including at least one of the following: user interface device 200, object detection device 300, communication device 400, vehicle drive device 600, and control unit 170.

[0183] The driving system 710 can drive the vehicle 100.

[0184] The driving system 710 can receive navigation information from the navigation system 770 and provide control signals to the vehicle drive unit 600, thereby driving the vehicle 100. The driving system 710 can also receive object information from the object detection device 300 and provide control signals to the vehicle drive unit 600, thereby driving the vehicle 100. Furthermore, the driving system 710 can receive signals from external devices via the communication device 400 and provide control signals to the vehicle drive unit 600, thereby driving the vehicle 100.

[0185] The vehicle dispatch system 740 can dispatch vehicle 100.

[0186] The vehicle dispatch system 740 can receive navigation information from the navigation system 770 and provide control signals to the vehicle drive unit 600, thereby dispatching the vehicle 100. The vehicle dispatch system 740 can also receive object information from the object detection device 300 and provide control signals to the vehicle drive unit 600, thereby dispatching the vehicle 100. Furthermore, the vehicle dispatch system 740 can receive signals from external devices via the communication device 400 and provide control signals to the vehicle drive unit 600, thereby dispatching the vehicle 100.

[0187] The parking system 750 can perform parking for 100 vehicles.

[0188] The parking system 750 can receive navigation information from the navigation system 770 and provide control signals to the vehicle drive unit 600 to perform parking of the vehicle 100. The parking system 750 can also receive object information from the object detection device 300 and provide control signals to the vehicle drive unit 600 to perform parking of the vehicle 100. Furthermore, the parking system 750 can receive signals from external devices via the communication device 400 and provide control signals to the vehicle drive unit 600 to perform parking of the vehicle 100.

[0189] The navigation system 770 can provide navigation information. The navigation information may include at least one of the following: map information, set destination information, route information set according to the destination, information about various objects on the route, lane information, and the vehicle's current location information.

[0190] The navigation system 770 may include a memory and a processor. The memory can store navigation information. The processor can control the operation of the navigation system 770.

[0191] According to an embodiment, the navigation system 770 can receive information from an external device via the communication device 400, thereby updating the stored information.

[0192] According to the embodiments, the navigation system 770 can also be classified as a subordinate component of the user interface device 200.

[0193] The sensing unit 120 can sense the state of the vehicle. The sensing unit 120 may include posture sensors (e.g., yaw sensor, roll sensor, pitch sensor), collision sensors, wheel sensors, speed sensors, tilt sensors, weight detection sensors, heading sensors, yaw sensors, gyroscope sensors, position modules, vehicle forward / reverse sensors, battery sensors, fuel sensors, tire sensors, steering sensors that utilize steering wheel rotation, vehicle interior temperature sensors, vehicle interior humidity sensors, ultrasonic sensors, illuminance sensors, accelerator pedal position sensors, brake pedal position sensors, etc.

[0194] The sensing unit 120 can acquire sensing signals regarding vehicle posture information, vehicle collision information, vehicle direction information, vehicle position information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward / reverse information, battery information, fuel information, tire information, vehicle light information, vehicle interior temperature information, vehicle interior humidity information, steering wheel rotation angle, vehicle exterior illuminance, pressure applied to the accelerator pedal, and pressure applied to the brake pedal.

[0195] In addition, the sensing unit 120 may also include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a coolant temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0196] The vehicle interface unit 130 can function as a conduit for various types of external devices connected to the vehicle 100. For example, the vehicle interface unit 130 may be provided with a port that can connect to a mobile terminal, through which the vehicle interface unit 130 can exchange data with the mobile terminal.

[0197] On the other hand, the vehicle interface unit 130 can function as a pathway to supply power to the connected mobile terminal. When the mobile terminal is electrically connected to the vehicle interface unit 130, the vehicle interface unit 130 can supply power from the unit 190 to the mobile terminal under the control of the control unit 170.

[0198] The memory 140 is electrically connected to the control unit 170. The memory 140 can store basic data about the unit, control data for controlling the unit's operation, and input and output data. The memory 140 can be a various storage device such as ROM, RAM, EPROM, flash memory drive, or hard disk. The memory 140 can store programs for processing or controlling the control unit 170, and various data related to the overall operation of the vehicle 100.

[0199] According to the embodiment, the memory 140 may be integrated with the control unit 170, or it may be implemented as a subordinate component of the control unit 170.

[0200] The control unit 170 can control the overall operation of various units within the vehicle 100. The control unit 170 can be named an electronic control unit (ECU).

[0201] The power supply unit 190 can supply the power required for the operation of each component according to the control of the control unit 170. In particular, the power supply unit 190 can receive power from the battery or the like inside the vehicle.

[0202] The vehicle 100 includes one or more processors and control units 170 that can be implemented using at least one of application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions.

[0203] On the other hand, in embodiments of the present invention, "driver", "passenger", "driver or passenger (driver / passenger)" refers to a driver, passenger, fellow passenger, or other person who is riding in a vehicle capable of performing personalized driving control according to embodiments of the present invention, and there may be one or more such persons.

[0204] Additionally, in embodiments of the present invention, "vehicle" is used to mean a vehicle capable of autonomous driving by artificial intelligence (AI) or a vehicle capable of receiving such services.

[0205] Additionally, in embodiments of the present invention, "personalized driving system" is used to mean a system that interacts with all vehicles including Advanced Driver Assistance Systems (ADAS), Autonomous Driving Systems (ADS), and / or Full Self Driving Systems (FSD), or additional systems / devices / servers capable of providing such services.

[0206] like Figure 7 As shown, the personalized driving system 800 of an embodiment of the present invention can be provided in the vehicle 100.

[0207] The personalized driving system 800 can adjust the vehicle's driving or route by reflecting the emotional state of the occupants monitored during autonomous driving of the vehicle 100. Furthermore, the personalized driving system 800 can train or update the vehicle 100's driving model based on control results reflecting the occupants' emotional states.

[0208] According to an embodiment, the personalized driving system 800 may be one of the ADAS system, ADS system, and FSD system described above, or a part thereof.

[0209] The Personalized Driving System 800 can control or link with a pre-trained driving model to execute driving model training from the left.

[0210] Here, the pre-trained driving model can be a driving model trained for autonomous driving technology using artificial intelligence (AI) machine learning, reinforcement learning (e.g., high-density deep reinforcement learning using deep learning techniques, reinforcement learning using simulators, etc.), end-to-end learning, etc. However, the pre-trained driving model can obviously be trained using other learning methods besides those described herein.

[0211] Data used in machine learning for artificial intelligence (AI) that can be used in autonomous driving of vehicle 100 may include sensor values ​​collected by various types of sensors installed in vehicle 100. For example, data used in machine learning for artificial intelligence (AI) that can be used in autonomous driving of said vehicle may include data from cameras, radars, lidar, ultrasonic sensors, etc. installed in the vehicle.

[0212] The personalized driving system 800 monitors and identifies the passenger's emotional state based on sensors within the vehicle 100 and external data related to driving. Here, monitoring the passenger's emotional state refers to detecting the passenger's stress level through sensors inside the vehicle 100 during driving, based on the identified information.

[0213] The personalized driving system 800 monitors the emotional state of the occupants to filter and identify information that triggers stress in them. The personalized driving system 800 then adjusts the driving and / or route of the vehicle 100 based on this determination to alleviate the passengers' stress.

[0214] Here, adjusting the driving and / or route of vehicle 100 may include all of the following: generating and / or changing driving control values ​​related to the driving of the vehicle through the personalized driving system 800, and / or bypassing / avoiding at least a portion of the vehicle's current driving route or generating a new route.

[0215] The personalized driving system 800 can use the results of controlling the driving and / or route of the vehicle 100 in accordance with the emotional state of the passengers as input values ​​to train the pre-trained driving model. In this invention, the driving model that has been additionally trained as described above to reflect the emotional state of the passengers is named the "personalized driving model".

[0216] It can be said that the "personalized driving model" uses end-to-end learning of artificial intelligence (AI) in its training to immediately reflect the emotional state of the passengers. Furthermore, it can be said that the "personalized driving model" does not use all the identified information, but rather uses high-density deep reinforcement learning (D2RL) to enhance the data by using identified information that matches the passengers' stress levels. However, the learning of the "personalized driving model" is not limited to the learning methods described above.

[0217] The Personalized Driving System 800 can match, store, and manage passenger information and personalized driving models. Therefore, even if a passenger drives / rides in another vehicle, the system can still provide a personalized driving model based solely on the passenger's experience by recognizing that passenger.

[0218] Figure 8 This is a block diagram illustrating the conceptual configuration of a personalized driving system 800 according to an embodiment of the present invention. Figure 8 The personalized driving system 800 can be like Figure 7 The system shown is implemented within vehicle 100, but it can obviously also be implemented as a system separate from vehicle 100.

[0219] The personalized driving system 800 operates by adaptively adjusting the vehicle's driving mode and route based on the emotional state of the driver / passenger in the autonomous vehicle. For this purpose, the personalized driving system 800 generally includes a recognition unit (A), a judgment unit (B), and a control unit (C). Additionally, the personalized driving system 800 may include an interface unit (not shown) for receiving / sending data / information from / to an external source.

[0220] The identification unit (A) may include a passenger identification module 10 and a driving condition identification module 20. The passenger identification module 10 identifies information about the passenger and the passenger's emotional state, while the driving condition identification module 20 identifies information related to the vehicle's driving.

[0221] For this purpose, the identification unit (A) receives data collected by the occupant identification module 10 and the driving condition identification module 20 during vehicle operation from the vehicle 100 or externally via the interface unit.

[0222] The occupant recognition module 10 can use in-vehicle / external sensors to identify occupant information, the gaze of the occupant while in motion, and the emotional state of the occupant while in motion. Specifically, the occupant recognition module 10 can use the in-vehicle driver monitoring system (DMS) and / or interior monitoring system (IMS) to identify the driver / occupant's emotional state based on information about the driver / occupant, their gaze, behavior, and facial expression changes, as well as heart rate and / or other biosignals based on facial blood flow.

[0223] Additionally, although no detailed illustration is provided, the occupant recognition module 10 can also identify the driver's emotional state by monitoring driver behavior through other additional sensors inside the vehicle. For example, by combining the intensity, frequency, and interval of the driver's grip / touch of the steering wheel (steering wheel) detected by the grip / touch sensor located on the vehicle, if it is determined that the driver suddenly grips the steering wheel, it can be determined that the driver is in a state of tension. Or, for example, by combining the intensity, frequency, and interval of the vehicle's braking, if it is determined that the driver suddenly brakes, it can be determined that the driver is in a state of tension. This helps to identify driver tension states that cannot be detected using only DMS and / or IMS.

[0224] On the other hand, DMS (Driver monitoring system) and / or IMS (Interior monitoring system) serve as driver / passenger monitoring systems. For example, driver / passenger cameras equipped with LED light sources or lasers can be used to identify the driver / passenger's face and track their eyes.

[0225] The passenger identification module 10 can use DMS and / or IMS to confirm the driver / passenger's face and verify the registered information. If the driver / passenger is registered, it can check whether there is a stored personalized driving model and retrieve it. If the result of confirming the driver / passenger's face is that they are not registered, it can confirm the driver / passenger's basic information and status (e.g., whether they are working, drowsy, etc.) to retrieve a pre-trained basic driving model or identify and retrieve a similar personalized driving style.

[0226] DMS and / or IMS can monitor the state of the driver / passenger based on data collected through driver / passenger cameras (e.g., blinking, head angle, facial blood flow, whether looking straight ahead, yawning, drowsiness, smoking, or being in a distracted state). For example, DMS and / or IMS can monitor emotional states such as tension, fear, and stress based on confirmed changes in facial expressions and facial blood flow of the driver / passenger.

[0227] The driving condition recognition module 20 can use external vehicle sensors and externally provided data to identify information about objects around the vehicle and driving conditions.

[0228] Specifically, the driving condition recognition module 20 can use external cameras and LiDAR to identify objects around the moving vehicle (e.g., vehicles, pedestrians, motorcycles, lanes, guardrails, traffic lights, traffic signs, etc.).

[0229] Here, the lidar can be positioned on the vehicle 100 and emit laser pulses around the vehicle 100, detecting light reflected back from objects located around the vehicle 100, thereby generating a three-dimensional image (more accurately, corresponding 3D point data) of the surroundings of the vehicle 100. Additionally, multiple external cameras can be positioned on the vehicle 100 to acquire two-dimensional images of the surroundings. In another embodiment, all sensors included in an Advanced Driver Assistance System (ADAS), such as radar, ultrasonic sensors, lidar, and cameras, can be used as external sensors.

[0230] Furthermore, the driving condition recognition module 20 can acquire driving / traffic-related information by utilizing external map information such as MAP (ADASIS) and / or vehicle network applications such as V2X and 5G. For example, the driving condition recognition module 20 can use MAP (ADASIS), V2X, 5G, etc. to identify information such as accident-prone areas, real-time traffic (LiveTraffic) information, construction zone information, disaster information, and traffic control information due to events (e.g., celebrations / marathons). In addition, for example, the driving condition recognition module 20 can not only use MAP (ADASIS), V2X, 5G, etc. to identify driving-related information such as the presence or absence of roadside, time, traffic volume and driving speed, emergency braking and traffic jams, hazard light information, etc., but also identify weather information (e.g., snow / rain (windshield wiper / black ice detection) etc.).

[0231] The judgment unit (B) can determine whether to change the driving mode that reflects the passenger's emotional state based on the information identified by the recognition unit (A) as described above and the monitoring 820 of the passenger's gaze and emotional state regarding the driving (or driving mode / scenario) of the vehicle 100. Here, changing the driving mode means changing at least one of the vehicle's driving method (e.g., driving speed, distance from the vehicle in front, distance from the adjacent lane / guardrail, etc.) or driving route to alleviate the passenger's emotional state.

[0232] The judgment unit (B) can perform a monitoring 820 that matches the information identified by the recognition unit (A) with the vehicle's movement to determine whether the monitored passenger's gaze and emotional state are in a state of stress or comfort.

[0233] Here, a state of stress, defined as the activation of the sympathetic nervous system, refers to negative emotional states such as unease, discomfort, tension, anger, irritability, and fear experienced by the passenger (e.g., based on facial blood flow). For example, a state of stress can be identified when there is an increase in facial blood flow to the driver / passenger monitored by DMS and / or IMS, uneasy eye movements, or sudden steering wheel grip or braking actions detected by additional sensors (however, if these events occur more than a certain number of times, it can be considered a state of stress). Conversely, a state of comfort, defined as the activation of the parasympathetic nervous system within the autonomic nervous system, refers to a state where the passenger is in a normal, unaffected emotional state.

[0234] The judgment unit (B) integrates the information identified by the recognition unit (A) (i.e., information about objects / driving conditions around the vehicle) with the monitoring results of the driver's / passenger's gaze and emotional state to obtain the driver's / passenger's emotional state towards a specific object or driving condition (e.g., stress state based on blood flow). The judgment unit (B) can determine whether to change the current driving mode based on the obtained results.

[0235] The control unit (C) can operate to control at least one of the vehicle's driving and route based on the determination (830). Additionally, the control unit (C) can be controlled to perform artificial intelligence (AI) learning on the driving model based on control results reflecting the driver's / passenger's emotional state (850).

[0236] Specifically, the control unit (C) can control the vehicle's driving / route based on information filtered from the surrounding objects of the vehicle 100 and the emotional state of the passengers under stress monitored from external information, and train the driving model based on this information to generate or update a personalized driving model.

[0237] In this case, the driving model is a pre-trained model that performs reinforcement learning based on selected information rather than all identified information, thus requiring no large amount of input data. The selected information can be input data used to perform end-to-end learning that immediately changes the vehicle's driving behavior based on it. That is, the selected information does not refer to simple sensor or external data, but rather to personalized information on driving control values ​​or route control values ​​adjusted according to the passenger's gaze and emotional state. Therefore, the selected information differs from specific control values ​​or control values ​​for a specific driving mode, even though it is based on the same sensed / external data values.

[0238] As described above, the personalized driving system 800 according to an embodiment of the present invention can eliminate the feeling of alienation in a vehicle by training a driving model personalized for each driver / passenger. However, the personalized driving model may be a model that has learned a portion of a plurality of neural networks connected to a large neural network or a model that has learned an updated version of the original version. In this case, the personalized driving model can be managed in a manner that it is sent together with passenger information or deleted together with passenger information.

[0239] Figure 9 This is a representative flowchart illustrating a personalized driving control method for a vehicle according to embodiments of the present invention.

[0240] Figure 9 The steps shown can be comprised of the components of the personalized driving system 800. Figure 8 The identification unit (A), judgment unit (B), control unit (C) or controller / processor (not shown) of the vehicle 100 providing personalized driving control services, or the service platform providing personalized driving control services, are executed by this system. However, for ease of explanation, it will be referred to below as personalized driving system 800. Additionally, Figure 9 The personalized driving control method shown can be activated when the driving mode of vehicle 100 is activated, and executed based on the personalized driving mode action.

[0241] The personalized driving method of the present invention includes a driving-related information identification step (910), a passenger emotional state monitoring step (920), a driving / route control step based on the fusion of the identified information and the passenger emotional state (930), and a driving model training step based on the control results (940).

[0242] The step of identifying driving-related information (910) is performed to identify information about objects around the vehicle and driving conditions based on the vehicle's sensors and external map information during the vehicle's operation.

[0243] The vehicle can utilize a pre-trained artificial intelligence (AI) driving model to perform autonomous driving. Furthermore, the vehicle can perform autonomous driving according to a driving mode that matches the identified information, based on the execution of the autonomous driving mode.

[0244] Furthermore, during driving, vehicles can utilize external sensors and externally provided data to identify various information about objects and driving conditions around the vehicle. Specifically, external cameras and LiDAR can be used to identify objects around the vehicle (e.g., vehicles, pedestrians, motorcycles, lanes, guardrails, traffic lights, traffic signs, etc.). Additionally, information about driving / traffic-related conditions can be identified using external map information such as MAP (ADASIS) and vehicle network applications such as V2X and 5G.

[0245] According to an embodiment, when a vehicle detects a passenger entering the vehicle, it can identify the passenger information using in-vehicle sensors (e.g., IMS, etc.). For example, the registered driver / passenger can be identified based on the passenger's face confirmed by the in-vehicle sensors. Therefore, the memory of the personalized driving system 800 or the memory of a device / server / system linked to it can store facial information pre-stored by the user.

[0246] Next, the personalized driving system 800 performs a step (920) to monitor information identified during vehicle operation and the gaze and emotional state of the occupants while the vehicle is in motion. This can be performed simultaneously with the aforementioned identification of information about objects around the vehicle and driving conditions.

[0247] According to an embodiment, the step (920) of monitoring the passenger's gaze and emotional state can fuse the identified information with the monitoring results of the passenger's gaze and emotional state to determine the passenger's emotional state in relation to a specific object or specific driving situation. That is, the passenger's emotional state can be determined by fusing the aforementioned identified information with the monitoring results of the passenger's gaze and emotional state.

[0248] Fusion refers to a method of combining information from different sensors on the vehicle / external environment, including data on surrounding objects, driving / traffic-related information, and the emotional state of passengers while the vehicle is in motion. Fusion utilizes the passenger's reactions to surrounding objects and driving / traffic-related information gathered by various sensors to filter information that may cause stressful emotional states in passengers. Alternatively, fusion can include the action of combining various pieces of information using known algorithms to calculate filtering information that produces stressful emotional states in a specific passenger; it can also be used as terms such as combining, integrating, fusing, or as part of a synthesis.

[0249] On the other hand, the personalized driving system 800 measures passenger behavior, gaze and pupil, facial expressions and emotional changes, facial blood flow and heart rate through in-vehicle driver monitoring system (DMS), interior monitoring system (IMS), and occupant monitoring system (OMS), thereby determining the passenger's gaze and emotional state (e.g., stress state, tension state, etc.) under specific objects and / or specific driving conditions.

[0250] Heart beat measurement based on facial blood flow uses devices such as cameras and lidar included in DMS, IMS, and OMS to shine light onto the passenger's face and reflect it back onto the passenger's face, thereby obtaining information through the diffusion and reflection of blood flow in the face.

[0251] If heart rate data is obtained in this non-contact manner, heart rate variability (HRV), which is a periodic change in heart rate over time, can be confirmed, thereby determining whether the rider's emotional state is under stress.

[0252] Normally, when a person is under stress, the sympathetic nervous system is activated, resulting in an increased heart rate. Furthermore, while HRV variability is large and complex in healthy individuals, this complexity decreases significantly under stress. Therefore, an increased heart rate and decreased HRV variability and complexity indicate that the passenger is under stress.

[0253] For example, if the driver's gaze, monitored by DMS and IMS, and the position of an object captured by the vehicle's front camera remain consistent (matched) for a certain period of time / number of times during vehicle operation, and an increase in heart rate based on facial blood flow is detected, the personalized driving system 800 can determine that the passenger is under stress in relation to the corresponding object. Alternatively, for example, if the increase in heart rate based on facial blood flow monitored by DMS and IMS during vehicle operation matches fused information about the gradient of the driving route and the current driving speed, it can be determined that the passenger is under tension in relation to this type of driving.

[0254] On the other hand, radar-based methods can measure heart rate based on facial blood flow using frequency-modulated continuous wave (FMCW) or photoplethysmogram (PPG) sensors, as well as various features extraction techniques employed by Blaze Face. Additionally, other sensors / algorithms not mentioned here can be used to determine whether the passenger is experiencing stress.

[0255] In another embodiment, the personalized driving system 800 can match information identified before a predetermined time, based on the point in time when the passenger's gaze and emotional state are determined to be under stress, to information indicating that the passenger's gaze and emotional state are under stress or tension. This is based on the premise that the occurrence of the passenger's stress state can be monitored some time after the actual occurrence of a specific object or situation.

[0256] Next, the personalized driving system 800 can perform a step (930) to control at least one of the vehicle's driving and route based on the monitoring to reflect the emotional state of the occupants.

[0257] Specifically, the personalized driving system 800 can act by changing one or more of the driving modes and routes associated with a specific situation based on the detection that the passenger is under stress towards a specific object or driving condition.

[0258] For example, the distance between the vehicle and the vehicle in front can be adjusted to be shorter if the system detects that the passengers are under stress due to frequent overtaking maneuvers ahead. Alternatively, if the system detects that the passengers are stressed while the vehicle is traveling at a prescribed speed on a sloping downhill road, the route can be modified to take a detour to a gentler slope and / or the vehicle speed can be further reduced than the prescribed speed.

[0259] Next, the personalized driving system 800 can also perform the step (940) of training the vehicle driving model based on the control results that reflect the emotional state of the passengers.

[0260] Here, the vehicle driving model can be a driving model pre-trained through artificial intelligence (AI) machine learning or reinforcement learning.

[0261] The input data used in step (940) of training the vehicle driving model consists of selected specific information rather than all identified information, thus eliminating the need for a large amount of input data. Furthermore, in step (940), reinforcement learning, as artificial intelligence (AI), can be trained through end-to-end learning, thereby immediately reflecting the control results based on the selected information back to the driving model. This allows for the generation and updating of personalized driving models tailored to each passenger.

[0262] the following, Figure 10a and Figure 10b This is an example diagram used to illustrate how the identification of a passenger's stressful emotional state is reflected in the vehicle's driving control.

[0263] The personalized driving system 800 can use at least one of the DMS and IMS included in the vehicle 100 to collect data related to the occupant's gaze and emotional state.

[0264] The personalized driving system 800 can determine, based on the data collected as described above, the vehicle's sensors, and externally provided data, whether the emotional state of the passengers is under stress in response to the identified information and the actual driving of the vehicle.

[0265] The identified information may include various objects around the vehicle and various driving / traffic-related information. For example, it may include overtaking situations of vehicles next to the vehicle, traffic congestion, parking situations at intersections, pedestrians crossing the road, parked vehicles on the roadside, road surface and weather conditions, empty roads, guardrails, large vehicles such as trucks, etc.

[0266] Reference Figure 10a (A) can confirm that the driver's gaze R1 is focused on a large vehicle in front of the vehicle during vehicle operation, monitored by, for example, a DMS (i.e., a driver monitoring system R2 inside the vehicle). The DMS can use a camera equipped with an LED light source or a laser to monitor whether the driver's gaze is frequently focused on a large vehicle in front of the vehicle and whether there is an increase in heart rate based on facial blood flow.

[0267] The Personalized Driving System 800 can integrate driver information and emotional state with information from large vehicles identified as objects in the vehicle's vicinity to understand the driver's emotional state. For example, referring to... Figure 10a (B) By recognizing facial expressions based on cameras, various emotions can be identified, such as happiness, anger, sadness, and disgust. For example, if anger or disgust is detected, the driver's emotion can be matched as a state of stress.

[0268] In other words, by fusing driver gaze information with the recognition results of objects in front of the vehicle, information and emotional state about the specific object the driver is monitoring can be obtained and matched. This can be used as input for filtering information to train a pre-trained driving model into a personalized driving model.

[0269] By fusing driver information and emotional state with information from large vehicles identified as objects in the vehicle's vicinity, it is possible to control the vehicle's driving / route to reflect the driver's emotional state.

[0270] Examples of vehicle driving control could include longitudinal / lateral acceleration (autopilot / comfort / economy), desired distance, driving speed, auto lane change, lane offset driving, etc.

[0271] Examples of vehicle route control could include curves (Rolling), motion sickness (traffic lights) minimization of route selection, pedestrian minimization of route selection, etc.

[0272] Specifically, for example, automatic lane changes can be performed to avoid dangerous vehicles (e.g., trucks in front / behind, intoxicated / fatigued drivers). Additionally, for example, the driver's emotional state can be detected in relation to pedestrians on the roadside or objects crossing the road illegally, prompting a reduction in speed or a change of route.

[0273] Following the vehicle's driving / route control as described above, monitoring of the driver's gaze and emotional state will continue. If the driver's stress level does not improve, driving control can be implemented step-by-step.

[0274] In this regard, Figure 10b Examples of driving control steps 1001, 1002, and 1003 are shown, which are based on the driver's emotional state of stress perceived from surrounding objects or vehicle driving conditions while the driver is looking ahead during vehicle operation. That is, driving control is executed step by step by continuously monitoring whether the driver's stress state is relieved.

[0275] In each step, it can be confirmed that the magnitude of the change is different when the vehicle decelerates / accelerates and rotates left / right. Figure 10b The diagram shows that as the level of driving control increases, the degree of deceleration / acceleration increases, while the magnitude of changes in left / right turns decreases. However, it is not limited to this; obviously, other control methods / levels can be changed. Furthermore, this does not mean controlling driving with a specific value, but rather changing the driving control values ​​step by step to find a driving style suitable for each driver.

[0276] For example, if the driver's emotional state is observed to be high stress level during Level 1001 driving control, the system can switch to Level 21002 driving control and proceed with driving. Subsequently, if the driver's emotional state is continuously alleviated, Level 21002 driving control is maintained; otherwise, it can switch to Level 31003 driving control.

[0277] Figure 11 This is a flowchart illustrating a method for matching composite recognition information and passenger emotional state information based on the actions of the personalized driving system 800.

[0278] Depending on the type / attribute of the identified information, the focus of determining the passenger's emotional state will vary. Furthermore, when multiple / complex pieces of information are identified simultaneously, the basis for determining vehicle movement / routing or how the vehicle is driven / controlled will differ.

[0279] Reference Figure 11 The personalized driving system 800 determines whether the identified information is relevant to objects around the vehicle based on the vehicle's sensors or external data (1101).

[0280] According to an embodiment, if the information identified based on the vehicle's sensors or external data is related to objects around the vehicle, the emotional state can be monitored based on data related to the passenger's line of sight (1102).

[0281] For example, the personalized driving system 800 can assign a weighted value to the score of the monitoring results of the passenger's gaze to determine whether the passenger is in a state of tension.

[0282] On the other hand, according to an embodiment, if the information identified based on the vehicle's sensors or external data is related to attributes different from those of objects around the vehicle, then emotional state can be monitored based on data related to changes in the occupant's facial temperature and heart rate (1103).

[0283] For example, the personalized driving system 800 can assign a weighted value to a score based on the monitoring results of heart rate based on facial blood flow in the passenger to determine whether the passenger is under stress. In this case, a stress index matched with the monitored heart rate can be pre-stored in the form of a table or the like, and the personalized driving system 800 can match the stress index-weighted score to determine the passenger's stress state / level.

[0284] Next, according to an embodiment, the step (1104) may include determining whether the identified information that matches the stress state of the passenger includes composite information.

[0285] Based on the determination (1104), if the identified information and the passenger's emotional state are not combined, the vehicle or route can be controlled by a specific driving mode determined by fusing the specific information with the emotional state (1106). For example, if the passenger's emotional state reflects the situation of other vehicles overtaking ahead, the distance between vehicle 100 and the vehicle in front can be set to be narrower, thereby eliminating the situation of other vehicles overtaking ahead.

[0286] On the other hand, according to the determination (1104), when the identified information and the passenger's emotional state are combined, the vehicle's driving / route can be controlled according to action A (1105). Embodiments of action A (1105) can be varied.

[0287] As one embodiment of action A (1105), the personalized driving system 800 can control the driving or route of each vehicle matching each of the identified composite information in a direction that alleviates the stress state of the passenger. For example, if the vehicle 100 is driving in an area with many parked vehicles on the roadside ("Identification Information 1") and the corresponding area is a steep downhill road ("Identification Information 2"), if it is determined that the passenger is in a stress state, the offset interval with the adjacent lane can be set to be wider ("Control 1") while the vehicle speed is reduced to below the permissible speed for driving ("Control 2").

[0288] As another embodiment of action A (1105), the personalized driving system 800 can selectively and weightedly control the driving or route of a vehicle that matches one of the identified composite information in a direction that alleviates the stress state of the passengers. For example, in the case of identification information 1 and 2 in the example above, if control 2 is selected, then the reduction in the vehicle's driving speed can be further weighted and controlled.

[0289] As another embodiment of action A (1105), it can also be a method of controlling the driving / route in the best way selected by performing multiple control methods that are different from each other and monitoring the degree of stress relief of the passenger.

[0290] On the other hand, according to embodiments of the present invention, a pre-trained driving model is trained to reflect the results of action A (1105) or vehicle driving / control (1106), thereby enabling the generation or updating of a personalized driving model.

[0291] Pre-trained driving models can process various open datasets, such as data collected by various sensors (e.g., cameras, LiDAR, etc.) set up in autonomous vehicles, and use this data as training data for autonomous driving. Thus, vehicle 100 can use the pre-trained driving model to automatically generate images of the position, orientation, level, and attributes of various objects such as other vehicles, pedestrians, traffic lights, and signs.

[0292] Personalized driving models are driving models generated or updated based on the aforementioned pre-trained driving models. They can be trained using high-density deep reinforcement learning that selectively learns new information (i.e., filtered information) based on the emotional state of the passengers identified.

[0293] Such a personalized driving model can be a standalone driving model trained based on the emotional state of the passenger as identified, or it can be one of a plurality of smaller models connected to a complex / base model (e.g., a large neural network model for autonomous driving).

[0294] Furthermore, there can be multiple personalized driving models. In this case, each of the multiple personalized driving models can be trained on each other. Even when there is only one personalized driving model, it is also possible to generate multiple versions of the personalized driving model trained on a base driving model in different ways than each other.

[0295] According to an embodiment, a personalized driving model can be generated or updated by training the driving model differently based on the attributes of the identified information and the control results that reflect the emotional state of the passenger.

[0296] For example, after immediately controlling the identified information, comparing the results of the control based on the passenger's emotional state can determine the input data for updating the driving model. Additionally, based on the type / characteristics of the identified information, it's possible to distinguish between a passenger's emotional state that is an immediate, one-off situation (e.g., the sudden appearance of an object) and a continuous situation (e.g., a large vehicle driving alongside in the adjacent lane), which may be reflected differently in the vehicle's driving / route or determine whether to update the personalized driving model.

[0297] According to an embodiment, the personalized driving system 800 uses a pre-trained driving model to generate first driving information based on the identified information. Here, the first driving information includes all of the common automatic driving mode or the control information of automatic driving based on the selected driving mode (e.g., ride comfort priority mode, fuel consumption priority mode), such as driving speed, driving distance from the vehicle in front, automatic lane change, etc.

[0298] According to an embodiment, the personalized driving system 800 can generate second driving information related to the first driving information based on the emotional state of the passenger regarding the first driving information, and can train a previous driving model based on the generated second driving information. Here, the second driving information includes all control information for information changes selected based on the emotional state of the passenger regarding specific identification information (e.g., decrease / increase in driving speed, decrease in driving distance from the vehicle in front, adjustment of offset distance from adjacent lanes, etc.).

[0299] At this point, the modified control information refers to specific control values ​​for each specific identification information. Furthermore, the modified control information can be set so that even when passengers are under the same stress level regarding the same identification information, they can have different specific control values ​​based on continuous monitoring of subsequent relief levels. Thus, it is possible to generate autonomous driving and driving models that are not designed for a specific purpose (e.g., prioritizing passenger comfort, fuel efficiency, etc.) or stable average control, but are suitable for the driving style of each passenger.

[0300] Figure 12a , Figure 12b , Figure 13a , Figure 13b , Figure 13c , Figure 14a , Figure 14b , Figure 15a , Figure 15b , Figure 16a , Figure 16b These are diagrams illustrating various embodiments of controlling vehicle movement or route by utilizing feedback from the emotional state of passengers under various driving conditions.

[0301] Specifically, Figure 12a and Figure 12b This is a specific embodiment of a system that responds to the feelings of threat felt by the occupants of a vehicle and controls the vehicle's movement when a large vehicle is traveling alongside the vehicle 100.

[0302] The personalized driving system 800 can identify information such as the presence of a large vehicle 1210 traveling alongside the vehicle 100 in the adjacent lane 1201 of the vehicle's driving lane using a front-facing camera and lidar. The personalized driving system 800 can determine, based on the identified information, such as the occupant's emotional state as monitored by DMS and / or IMS, that they are in a state of perceived threat (e.g., eye contact towards the large vehicle 1210 and / or increased heart rate based on facial blood flow). The personalized driving system 800 controls itself to increase the offset from the driving lane 1201 and widen the distance from the large vehicle 1210, i.e., adjust the distance and drive away from the large vehicle 1210, according to the determination.

[0303] To address this, the personalized driving system 800 can use local path planning to control driving. Local path planning refers to a path planner method that plans a local route given a global path or a local objective, primarily used for avoidance or deviation driving. For example, when vehicle 100 is driving autonomously, there might be situations where road cones are blocking the road due to construction or vehicles blocking part of the driving path due to illegal parking. In such cases, the vehicle could stop and then resume following the global path after the situation ends, but using local path planning allows for immediate resolution and flexible response.

[0304] The personalized driving system 800 can determine whether the emotional state of the passenger has been alleviated after subsequent monitoring (e.g., reduced eye gaze and / or decreased heart rate based on facial blood flow), and based on the determination, generate (store) or update the personalized driving model with the control results according to the local path planning method.

[0305] After being controlled by the local path planning method, the personalized driving system 800 can restore the vehicle 100's offset to its previous state and continue driving if it recognizes that the large vehicle 1210 is no longer driving alongside the vehicle.

[0306] As yet another example, Figures 13a to 13c as well as Figure 15a and Figure 15b A specific implementation example of controlling driving / route based on the emotional state of passengers monitored during the vehicle's travel within a specific area is shown.

[0307] The personalized driving system 800 can identify when the vehicle 100 is driving within a range that meets specific conditions (e.g., a first condition), and determine that the passenger is in a state of stress by fusing the monitoring results of the identified information with the passenger's emotional state. Here, the range that meets the first condition can include all sorts of situations where a specific condition occurs within a certain range, such as alleyways with many parked cars or narrow alleyways, or areas with guardrails next to the driving lane.

[0308] In this situation, the personalized driving system 800 can change the vehicle's route to avoid driving in the area that meets the first condition. Alternatively, the personalized driving system 800 can further adjust at least one of the vehicle's driving interval and speed during driving in the area that meets the first condition, thereby addressing the stress experienced by the passengers.

[0309] Reference Figures 13a to 13c If, during the driving of vehicle 100, the driver / passenger is continuously exposed to pedestrians P1 or illegally parked vehicles 1311, 1312, and 1313 on the roadside and is determined to be disturbed, then the personalized driving system 800 can, based on this fused information, [further details are needed]. Figure 13b The diagram shows an alternative route 1322 set for driving route 1321 to prioritize roads with fewer pedestrians. At this time, the personalized driving system 800 can ask the driver / passenger whether to change the driving route to the alternative route 1322 via a display inside the vehicle 100.

[0310] Here, avoiding route 1322 can be a route modification using the Global Path Planning method. Global Path Planning refers to a route planning method from the vehicle's current position to the destination. Each time a new route is planned, the vehicle uses a global planner to plan and set the prerequisite route.

[0311] Alternatively, the personalized driving system 800 can be like... Figure 13c Instead of changing the driving route according to the global route planning method, the driving route is adjusted in the direction of increasing the offset 1324 between the roadside 1302 and the driving lane 1301 and / or the driving speed 1323 of the vehicle 100 is further reduced.

[0312] On the other hand, in order to determine whether a specific situation identified as the passenger's stress state applies to an area that meets the first condition mentioned above, the personalized driving system 800 can determine this by integrating the various emotional states of the passenger monitored at the time point when the vehicle 100 enters the area that meets the first condition (or, the first position), the time point after a certain period of time has elapsed after entering the area (or, the second position), and the time point when it leaves the area (or, the third position). In this case, the personalized driving system 800 can then specificize the corresponding or similar areas and reflect this in the personalized driving model, so that it is not included in the driving route setting.

[0313] Figure 15a The illustration shows that during the period when vehicle 100 is traveling next to guardrail 1511 (a), it is determined that the occupants of the vehicle are under stress, and the distance (i.e., offset) between the vehicle and guardrail 1511 is increased from a first distance 1521 to a second distance 1522 (b). Herein, the second distance 1522 refers to the degree to which the vehicle's driving lane 1501 is moved away from guardrail 1511 compared to the first distance 1521.

[0314] If the passenger's emotional state is alleviated during the second interval 1522 of driving with the offset adjusted, the personalized driving system 800 can reflect the corresponding adjustment result and store it as a personalized driving model. Alternatively, the personalized driving system 800 can update the vehicle's local path planning model so that it can automatically adjust the offset and drive next to a guardrail when carrying the same passengers in the future.

[0315] Figure 15b This is an example of controlling the vehicle's movement by detecting the driver's / passenger's emotional state in relation to the curve conditions while the vehicle 100 is in motion. The personalized driving system 800 can control the vehicle to adjust the offset 1523 and continue driving (a) when a curve section is identified. Alternatively, the personalized driving system 800 can further reduce the vehicle's speed while the vehicle 100 is traveling in the corresponding curve section, adjusting in a direction that further reduces the graphic (GG Diagram) limit value when turning left or right.

[0316] As yet another example, Figure 14a and Figure 14b This is an example of controlling vehicle movement based on monitoring the emotional state of passengers under complex conditions such as driving on steep roads and having many vehicles parked on the roadside.

[0317] Even when the vehicle 100 is traveling at a safe speed (e.g., 60 km / h) on a steep road (or, similarly, at night), if the system detects that the passenger is experiencing stress, the personalized driving system 800 can reflect the passenger's emotional state and further reduce the driving speed (e.g., 40 km / h). Here, the identification 1412 of whether it is a steep road can be inferred, for example, by using external data from a map or sensing data from the vehicle 100's ESP ECU, inertial sensors, etc.

[0318] On the other hand, refer to Figure 14a In the complex situation 1411 where vehicle 100 is on a road with a large slope and there are many parked vehicles on the roadside, 1) the vehicle can drive according to the control value for each of the complex situations, or 2) the vehicle can drive according to the weighted application of the control value for a selected situation.

[0319] In case 1), vehicle offset and speed adjustments can be performed simultaneously, and the results can be reflected in the personalized driving model. In case 2), such as Figure 14b As shown, the vehicle speed can be reduced by 1420 km / h, and the reduction can be further increased by weighted application. For example, in one scenario of the above embodiment, if the speed is reduced to 40 km / h, then in the composite scenario, the speed can be further reduced to 30 km / h or below 30 km / h and the vehicle can continue driving.

[0320] Figure 16a and Figure 16b This example illustrates how, in a traffic jam, the vehicle's driving control is adjusted based on information about overtaking ahead and the passenger's emotional state of discomfort.

[0321] Specifically, the personalized driving system 800 can identify situations where overtaking 1610 is continuously occurring during traffic jams and confirm the driver's / passenger's distressed emotional state monitored by the vehicle's DMS and IMS. In this case, the personalized driving system 800 can adjust the distance between vehicle 100 and the vehicle in front 1620 from a first distance 1621 to a second distance 1622 and reflect this result in the personalized driving model. Here, the second distance 1622 is obtained by reducing the first distance 1621 by a predetermined value.

[0322] Figure 17 This invention illustrates a method for determining the control of vehicle movement or route that reflects the emotional state of passengers when there are multiple passengers, according to an embodiment of the invention.

[0323] According to an embodiment of the present invention, when the identified information and the emotional state of the passenger are fused, in the case of multiple passengers, the personalized driving system 800 can determine which passenger's emotional state should be reflected.

[0324] When a driver is present and the vehicle is operating in semi-autonomous driving mode, the personalized driving system 800 can recommend vehicle controls or display notifications that reflect the driver's emotional state. When only passengers are present and the vehicle is operating in autonomous driving mode (e.g., in an autonomous taxi), the personalized driving system 800 can execute vehicle controls that reflect the passengers' emotional state.

[0325] According to an embodiment, when the vehicle is carrying multiple passengers 1701, 1702, 1703, and 1704, including the driver, the personalized driving system 800 generates a driving model for each passenger.

[0326] When multiple passengers 1701, 1702, 1703, and 1704 are simultaneously on board, the personalized driving system 800 can perform vehicle control based on passengers selected according to specific criteria. These specific criteria may include priority for vulnerable groups such as children / the elderly and infirm, or specific seating criteria.

[0327] In another embodiment, when multiple passengers 1701, 1702, 1703, and 1704 are simultaneously on board, the personalized driving system 800 can perform vehicle driving with a weighted control value by taking into account the emotional state of all passengers, or it can perform vehicle driving based on an average value.

[0328] As described above, the personalized driving control method, personalized driving system, and vehicle providing personalized driving control services according to embodiments of the present invention adjust driving or route based on the emotional state of the driver or passenger identified in the driving situation during autonomous driving, thereby minimizing the sense of heterogeneity in driving. Furthermore, the driving model is updated based on the results reflecting the emotional state of the driver or passenger identified in the driving situation during autonomous driving, allowing the driver or passenger to experience a more personalized driving experience. Moreover, personalized driving models can be managed according to the driver or passenger, thus enabling a personalized driving experience using only driver / passenger information, independent of the vehicle.

[0329] The foregoing invention can be implemented by computer-readable code in a storage medium. Computer-readable media include all kinds of storage devices that store data readable by a computer system. Examples of computer-readable media include HDDs (Hard Disk Drives), SSDs (Solid State Disks), SDDs (Silicon Disk Drives), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also include those implemented as carrier waves (e.g., transmitted via the Internet). Additionally, the computer may include a processor or control unit of a personal driving system 800. Therefore, the detailed description above should not be construed as limiting in all respects, but rather as exemplary. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

1. A method for personalized driving control of a vehicle, wherein, include: The steps involved in identifying objects and driving conditions around the vehicle based on information from the vehicle's sensors and external maps during vehicle operation. The steps of monitoring the gaze and emotional state of the occupants in relation to the identified information and the vehicle's movement; The step of controlling at least one of the vehicle's driving and route based on the monitoring to reflect the emotional state of the passenger; as well as The steps involve training a vehicle's driving model based on control results that reflect the emotional state of the passengers.

2. The personalized driving control method for a vehicle according to claim 1, wherein, The monitoring steps include: The steps of collecting data related to the occupant's gaze and emotional state using at least one of the DMS and IMS included in the vehicle; and The steps involve determining whether the identified information and the emotional state of the passengers in relation to the vehicle's operation are in a state of stress based on the collected data.

3. The personalized driving control method for a vehicle according to claim 2, wherein, include: If the identified information is about objects around the vehicle, then the emotional state is monitored based on data related to the passenger's line of sight; if the identified information is about attributes different from those of objects around the vehicle, then the emotional state is monitored based on data about changes in the passenger's facial temperature and heart rate.

4. The personalized driving control method for a vehicle according to claim 3, characterized in that, The monitoring steps include determining identified information, including composite information, that matches the pressure state.

5. The personalized driving control method for a vehicle according to claim 4, characterized in that, The control step is to control the driving or route of each vehicle that matches each of the composite information in the direction of relieving the stress state of the passenger.

6. The personalized driving control method for a vehicle according to claim 4, characterized in that, The control step is a step of weighting the driving or route of a vehicle matched with one of the composite information in a direction that alleviates the stress state of the passenger.

7. The personalized driving control method for a vehicle according to claim 1, characterized in that, The steps of training the driving model include training the driving model based on the control results differently according to the attributes of the identified information.

8. The personalized driving control method for a vehicle according to claim 1, characterized in that, The steps for training the driving model include: The steps are: generating first driving information based on the identified information using a pre-trained driving model; generating second driving information about the first driving information based on the emotional state of the passenger in the first driving information; and training the driving model based on the second driving information.

9. The personalized driving control method for a vehicle according to claim 1, characterized in that, The monitoring steps include: The steps include identifying the presence of other vehicles traveling alongside the vehicle in its lane and monitoring the stress and emotional state of the passengers in response to the identified information. The control steps include: The step of adjusting the vehicle's deviation from the driving path away from the other vehicles while they are driving alongside; and If it is detected that the other vehicles are no longer driving alongside, the vehicle's deviation from the driving road is restored to its previous state and the driving process resumes.

10. The personalized driving control method for a vehicle according to claim 1, characterized in that, The monitoring steps include: The steps include identifying the vehicle traveling within an area that meets a first condition and monitoring the stress and emotional state of the passengers based on the identified information. The control steps include: The steps include changing the vehicle's route to avoid traveling in the section that meets the first condition, or further adjusting at least one of the following during the period when traveling in the section that meets the first condition:

11. The personalized driving control method for a vehicle according to claim 1, characterized in that, The monitoring steps include: The procedure involves monitoring the passengers' stress levels while the vehicle is traveling on a steep road. The control steps include: The step of further reducing the vehicle speed in accordance with the level of stress experienced by the passenger.

12. The personalized driving control method for a vehicle according to claim 1, characterized in that, The monitoring steps include: Steps for monitoring the stress and emotional state of passengers when overtaking the vehicle in front of it; The control steps include: The step of resetting the distance between the vehicle and the vehicle in front to a narrower level and driving is to reflect the passenger's stressful mood.

13. A personalized driving system for a vehicle, wherein, include: The interface unit receives sensor data from the vehicle's sensors and external map-related information while the vehicle is in motion. The identification unit identifies information about objects and driving conditions around the vehicle based on the received sensing data and external map-related information; The judgment unit monitors the gaze and emotional state of the passengers in relation to the identified information and the vehicle's movement, and determines, based on the monitoring, whether to change at least one of the vehicle's movement and route to reflect the passengers' emotional state. as well as The control unit controls at least one of the vehicle's driving and route based on the determination, and controls the vehicle's driving model to be trained based on the control results reflecting the emotional state of the passengers.

14. The personalized driving system for a vehicle according to claim 13, characterized in that, The control unit controls the training of a driving model based on the control results according to the different attributes of the identified information.

15. The personalized driving system for a vehicle according to claim 13, characterized in that, The control unit controls the generation of first driving information based on the identified information using a pre-trained driving model; The control unit controls the generation of second driving information related to the first driving information based on the emotional state of the passenger in relation to the first driving information. The control unit controls the training of the driving model based on the second driving information.

16. A computer program recorded on a recording medium, characterized in that, A computing device comprising a memory, a transceiver, and a processor for processing instructions stored in the memory is combined such that the processor performs the following steps: The steps involved in identifying objects and driving conditions around the vehicle based on information from the vehicle's sensors and external maps during vehicle operation. The steps of monitoring the gaze and emotional state of the occupants in relation to the identified information and the vehicle's movement; The step of controlling at least one of the vehicle's driving and route based on the monitoring to reflect the emotional state of the passenger; as well as The steps involve training a driving model based on control results that reflect the emotional state of the passengers.

Images