Apparatus for controlling autonomous driving of vehicle and method for determining state of driver

Abstract

The invention relates to an apparatus for controlling autonomous driving of a vehicle and a method for determining a state of a driver. An example of the present invention provides an autonomous driving control apparatus including: a processor; a memory storing instructions that, when executed by the processor, may cause the apparatus to: communicate with a mobile device during driving of the vehicle in an autonomous driving mode; determining whether a driver state is an available state for a manual driving mode based on communication with the mobile device; and based on the determination, may cause the vehicle to remain in the autonomous driving mode or may cause the mobile device to send a take-over request (TD) to switch a driving mode of the vehicle from the autonomous driving mode to the manual driving mode.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of Korean Patent Application No. 10-2023-0079181 filed in the Korean Intellectual Property Office on June 20, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to an autonomous driving control device and a driver state determination method thereof, and more particularly to a technology for accurately determining a driver's state using a mobile device during autonomous driving. Background Art

[0004] As electronic technology of vehicles develops, interest in autonomous driving vehicles that drive to a destination by recognizing the driving environment of the vehicle itself without driver manipulation is becoming increasingly greater.

[0005] An autonomous vehicle refers to a vehicle that controls itself by recognizing the surrounding environment using various types of sensors (e.g., lidar, camera, radar, ultrasonic sensor, etc.) and vehicle-to-everything (V2X) and determining the driving situation without driver intervention based on location information and road information from the Global Positioning System (GPS).

[0006] During conditional autonomous driving, accuracy may be lower because only in-vehicle camera information may be used to determine the driver's state (e.g., drowsy state, carelessly looking forward, etc.) Specifically, in response to a situation where the driver is looking at a smartphone, the driver may be incorrectly determined to be in a drowsy state depending on the driver's gaze angle relative to the smartphone. Summary of the invention

[0007] According to the present invention, a device for controlling a driving mode of a vehicle is proposed, and the device may include: a processor; a memory, the memory storing instructions, and the instructions, when executed by the processor, may enable the device to: communicate with a mobile device during the vehicle's driving in an autonomous driving mode; determine whether the driver's state is an available state for a manual driving mode based on the communication with the mobile device; and based on the determination, the vehicle may be maintained in the autonomous driving mode or the mobile device may send a takeover request (transition demand, TD) to switch the vehicle's driving mode from the autonomous driving mode to the manual driving mode.

[0008] For the device, the instructions, when executed by the processor, may further cause the device to determine that the driver state is at a conditional autonomous driving level among a plurality of autonomous driving levels.

[0009] For the apparatus, the instructions, when executed by the processor, may further cause the apparatus to determine the driver status by using information sensors within the vehicle and the mobile device.

[0010] For the device, the instructions, when executed by the processor, may further cause the device to determine whether the mobile device is in a communication connection state with the vehicle based on the information sensor in the vehicle determining that the driver state is not the available state.

[0011] For the device, the instructions, when executed by the processor, may further cause the device to determine whether the mobile device is a mobile device owned, handled or used by the driver of the vehicle based on determining that the mobile device is in a communication connection state with the vehicle.

[0012] For the apparatus, the instructions, when executed by the processor, may further cause the apparatus to determine that the mobile device is a mobile device owned, handled, or used by the driver based on the mobile device being located near the driver's seat.

[0013] For the apparatus, wherein the instructions, when executed by the processor, may further cause the apparatus to determine the driver status using the mobile device based on the mobile device being a mobile device owned, handled, or used by the driver.

[0014] For the device, the instructions, when executed by the processor, may also cause the device to execute a TD procedure for notifying the driver of TD based on determining that the mobile device is communicatively connected to the vehicle and the driver of the vehicle does not own or handle the mobile device, or based on determining that the mobile device is not communicatively connected to the vehicle.

[0015] For the device, wherein the instructions, when executed by the processor, may further cause the device to: output a TD notification message on a display within the vehicle; or notify the driver of TD via the mobile device based on the execution of the TD procedure.

[0016] For the device, the instructions, when executed by the processor, may also cause the device to cause the vehicle to operate in the manual driving mode based on determining that the driver state is the available state.

[0017] For the device, the instructions, when executed by the processor, may also cause the device to determine that the driver state is an unavailable state in which TD is not sent based on determining that the mobile device is not located near the driver's seat or the mobile device is located in a passenger seat or a back seat.

[0018] For the device, the instructions, when executed by the processor, may also enable the device to determine whether the manipulation of the mobile device is performed at a predetermined interval; and based on determining that the manipulation of the mobile device is not performed at the predetermined interval, determine that the driver state is an unavailable state in which TD will not be sent.

[0019] For the device, the instructions, when executed by the processor, may also enable the device to determine whether the driver's pupils are detected via the front camera of the mobile device; and based on determining that the driver's pupils are not detected via the front camera of the mobile device, determine that the driver's state is an unavailable state in which TD will not be sent.

[0020] For the device, the instructions, when executed by the processor, may also cause the device to output a touch notification through the mobile device; determine whether the driver performs a touch manipulation of the mobile device based on the touch notification; and based on determining that the driver has not performed a touch manipulation of the mobile device, determine that the driver state is an unavailable state in which TD will not be sent.

[0021] For the device, the instructions, when executed by the processor, may also cause the device, based on determining that the mobile device is located near the driver's seat: setting an automatic screen off mode for the mobile device; and based on the screen of the mobile device being automatically turned off, determining that the driver's state is an unavailable state in which TD will not be sent.

[0022] For the device, the instructions, when executed by the processor, may also enable the device to use camera information of the mobile device and information from information sensors within the vehicle to estimate the location of the mobile device; and determine whether the mobile device is a mobile device owned, handled or used by the driver based on the estimated location.

[0023] According to the present invention, a method for controlling a driving mode of a vehicle is proposed, and the method may include: a processor communicating with a mobile device during the vehicle's driving in an autonomous driving mode; the processor determining, based on the communication with the mobile device, whether the driver's state is an available state for a manual driving mode; and based on the determination, causing the vehicle to remain in the autonomous driving mode or causing the mobile device to send a takeover request (TD) to switch the vehicle's driving mode from the autonomous driving mode to the manual driving mode.

[0024] For the method, determining whether the driver state is the available state may include determining, by the processor, that the driver state is at a conditional autonomous driving level among a plurality of autonomous driving levels.

[0025] For the method, determining whether the driver state is the available state may include determining, by the processor, the driver state using an information sensor within the vehicle and the mobile device.

[0026] For the method, determining whether the driver state is the available state may include: determining, by the processor, whether manipulation of the mobile device is performed at a predetermined interval; and determining, by the processor and based on determining that manipulation of the mobile device is not performed at the predetermined interval, that the driver state is an unavailable state in which TD will not be sent. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 An exemplary configuration of a vehicle system including an autonomous driving control device is shown.

[0028] Figure 2 An exemplary processor is shown.

[0029] Figure 3a An example of using a camera to determine the driver's state is shown.

[0030] Figure 3b An example of using a mobile device to determine a driver's status is shown.

[0031] Figure 4 An exemplary method for determining a driver's state using a camera of a mobile device of an autonomous driving control device is shown.

[0032] Figure 5 An exemplary method for determining a driver's status by an auto-off function of a mobile device using an autonomous driving control device is shown.

[0033] Figure 6 An exemplary driver state determination method using a mobile device during autonomous driving control is shown.

[0034] Figure 7 An exemplary computing system is shown. DETAILED DESCRIPTION

[0035] Below, some examples of the present invention will be described in detail with reference to the exemplary drawings. It should be noted that when adding reference numerals to the constituent elements of each drawing, even if the same constituent elements are indicated on different drawings, the same constituent elements are given the same reference numerals as much as possible. When describing an example, if it is determined that the detailed description of a well-known configuration or function associated with the example may obscure the gist of the present invention, it will be omitted.

[0036] When describing the constituent elements according to the examples, terms such as first, second, A, B, (a) and (b) may be used. These terms are only used to distinguish constituent elements from other constituent elements, and the nature, order or sequence of the constituent elements are not limited by these terms. In addition, all terms including technical scientific terms used in this article have the same meaning as the meaning generally understood by the technicians (technical personnel in the field) of the technical field to which the examples of the present invention belong, unless they are defined differently. The terms defined in the commonly used dictionaries should be interpreted as having the meanings that match the terms in the context of the relevant field, and should not be interpreted as having idealized or overly formal meanings, unless they are clearly defined in this specification.

[0037] Next, we will refer to Figures 1 to 7 Various examples of the present invention are described in detail.

[0038] There are six levels of autonomous driving, with no automated functions applied at level 0, driver assistance at level 1, partial autonomous driving at level 2, and conditional autonomous driving at level 3. Advanced autonomous driving is performed at level 4, and fully autonomous driving is performed at level 5.

[0039] SUMMARY Arrangements for determining driver availability during conditional autonomous driving are disclosed.

[0040] Figure 1 An exemplary configuration of a vehicle system including an autonomous driving control device is shown.

[0041] refer to Figure 1 According to an example of the present invention, a vehicle system may include an autonomous driving control device 100, a vehicle external information sensor 210, a vehicle internal information sensor 220, a GPS receiver 230, a map database 240, a direction indicator 250, an emergency flashing indicator 260, a braking control device 270, a driving control device 280 and a steering control device 290.

[0042] According to an example of the present invention, the autonomous driving control device 100 may be implemented in a vehicle or separately from the vehicle. In this case, the autonomous driving control device 100 may be formed integrally with the internal control unit of the vehicle, or may be implemented as a separate hardware device to be connected to the control unit of the vehicle through a connecting device. For example, the autonomous driving control device 100 may be implemented integrally with the vehicle, may be implemented in a form of being installed or attached to the vehicle as a configuration separate from the vehicle, or a portion thereof may be implemented integrally with the vehicle, and another portion may be implemented in a form of being installed or attached to the vehicle as a configuration separate from the vehicle.

[0043] The autonomous driving control device 100 may include an advanced driver assistance system (ADAS), an automatic lane keeping system (ALKS), a forward collision avoidance assist (FCA) system, a lane keeping assist (LKA) system, a blind spot collision avoidance assist (BCA) system, and a smart cruise control (SCC) system.

[0044] The autonomous driving control device 100 should be able to detect whether the driver is in a proper driving position and whether the driver is available to respond to a takeover request (TD) by monitoring the driver. The proper driving position of the driver refers to a posture in which the driver is sitting in a chair and the driver's back is not lying down. In addition or alternatively, the driver available state indicates a state in which the driver can receive TD at any time, and the driver unavailable state indicates a state in which the driver cannot receive TD (e.g., due to inattention and not looking forward, drowsiness, etc.).

[0045] By law, vehicle manufacturers can demonstrate to certification bodies the vehicle's driver-friendly testing performance for receiving TD.

[0046] The autonomous driving control device 100 may determine whether the following conditions are satisfied in order to determine whether the driver is in an available state.

[0047] 1) A driver may be determined to be available if the driver does not generate an affirmative signal indicating that the driver is available through at least two driver availability factors (e.g., driver's gaze, driver's posture, driver's nodding, driver's eyes closed, etc.).

[0048] 2) At any time, the autonomous driving control device 100 may determine that the driver is unavailable.

[0049] 3) If it is determined that the driver is unavailable or if less than two driver availability factors are positive, the autonomous driving control device 100 may determine that the driver is in an unavailable state and immediately generate a warning signal. The warning signal is generated until the driver's appropriate action (action) is detected or TD is initiated. TD is issued in response to the warning lasting 15 seconds.

[0050] The autonomous driving control device 100 may use the mobile device 300 to determine whether the driver is in a state that can receive a TD during autonomous driving.

[0051] The autonomous driving control device 100 may include a communication device 110 , a storage device 120 , an interface device 130 , and a processor 140 .

[0052] The communication device 110 is a hardware device implemented by various electronic circuits to transmit and receive signals through wireless or wired connections, and can transmit and receive information based on in-vehicle devices and in-vehicle network communication technologies. As an example, the in-vehicle network communication technology may include controller area network (CAN) communication, local interconnect network (LIN) communication, flex-ray communication, etc.

[0053] Additionally or alternatively, the communication device 110 may communicate with a server, infrastructure, or a third vehicle outside the vehicle, etc., via wireless Internet access or short-range communication technology. In this context, wireless communication technology may include wireless LAN (WLAN), wireless broadband (Wibro), Wi-Fi, Worldwide Interoperability for Microwave Access (Wimax), etc. Additionally or alternatively, short-range communication technology may include Bluetooth, ZigBee, ultra-wideband (UWB), radio frequency identification (RFID), infrared data association (IrDA), etc.

[0054] The communication device 110 may perform V2X communication. V2X communication may include communication between a vehicle and all entities, such as V2V (vehicle-to-vehicle) communication referring to communication between vehicles, V2I (vehicle-to-infrastructure) communication referring to communication between a vehicle and an eNB or a roadside unit (RSU), V2P (vehicle-to-pedestrian) communication referring to communication between a user equipment (UE) held by a vehicle and an individual (pedestrian, cyclist, vehicle driver or occupant), and V2N (vehicle-to-network) communication.

[0055] Additionally or alternatively, the communication device 110 may share information for checking the driver's status by performing Bluetooth or ultra-wideband (UWB) communication with the mobile device 300 .

[0056] The storage device 120 may store sensing results of the vehicle exterior information sensor 210 and the vehicle interior information sensor 220 , location information received from the GPS receiver 230 , map information received from the map database 240 , and data and / or algorithms for the processor 140 to operate.

[0057] The storage device 120 may include at least one type of storage medium such as the following types of memory: flash memory, hard disk, micro memory, card (e.g., secure digital (SD) card or extreme digital (XD) card), random access memory (RAM), static RAM (SRAM), read-only memory (ROM), programmable ROM (PROM), electrically erasable PROM (EEPROM), magnetic memory (MRAM), magnetic disk, and optical disk.

[0058] The interface device 130 may include an input device for receiving a control command from a user and an output device for outputting an operation state of the device 100 and a result thereof. Herein, the input device may include a key, and may include a mouse, a joystick, a single-key shuttle, a stylus pen, etc. In addition, the input device may include a soft key implemented on a display.

[0059] The interface device 130 may be implemented as a head-up display (HUD), an instrument panel, an audio video navigation (AVN), or a human machine interface (HM), a human machine interface (HMI).

[0060] The output device may include a display, and may also include a voice output device such as a speaker. In the present case, in response to a touch sensor formed of a touch film, a touch sheet, or a touch pad being provided on the display, the display may be operated as a touch screen, and may be implemented in a form in which an input device and an output device are integrated. In the present invention, the output device may output autonomous driving information. In this case, the autonomous driving information may include driver state determination information, autonomous driving state information, information related to the remaining time until a takeover request (TD) occurs, and the like.

[0061] In this case, the display may include at least one of a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFTLCD), an organic light emitting diode display (OLED display), a flexible display, a field emission display (FED), a 3D display, or any combination thereof.

[0062] The processor 140 may be electrically connected to the communication device 110 , the storage device 120 , the interface device 130 , etc., may electrically control each component, and may be a circuit that executes software commands, thereby performing various data processing and calculations as described below.

[0063] The processor 140 may process signals transferred between components of the autonomous driving control device 100 to perform overall control so that each component may perform its function normally. The processor 140 may be implemented in the form of hardware, software, or a combination of hardware and software. For example, the processor 140 may be implemented as a microprocessor, but the present invention is not limited thereto. For example, it may be, for example, an electronic control unit (ECU), a microcontroller unit (MCU), or other sub-controllers installed in a vehicle.

[0064] The processor 140 may use the mobile device 300 during autonomous driving to determine whether the driver is in a state in which a TD may be received.

[0065] The processor 140 may determine that the driver state is at a conditional autonomous driving level among the autonomous driving levels.

[0066] The processor 140 may determine the driver's state by using information sensors within the vehicle and the mobile device 300 . Figure 3a An example of using camera 231 to determine the driver's state is shown, and Figure 3b An example of using mobile device 300 to determine a driver's state is shown.

[0067] The processor 140 may determine whether the mobile device 300 is in a communication connection state with the vehicle in response to the vehicle interior information sensor 220 determining that the driver is not in a state in which the TD can be received.

[0068] The processor 140 may determine whether the mobile device 300 is a mobile device 300 owned, handled, or used by the driver in response to the mobile device 300 being in a communication connection state with the vehicle.

[0069] The processor 140 may determine that the mobile device 300 belongs to the driver in response to the mobile device 300 being located near the driver's seat.

[0070] The processor 140 may determine the driver status by using the mobile device 300 .

[0071] The processor 140 may execute a TD procedure for executing TD in response to a situation where the mobile device 300 is not connected to the vehicle or the mobile device 300 is owned, handled, or used by the driver.

[0072] The processor 140 may notify the driver of TD by outputting a TD notification phrase on a display in the vehicle in response to a situation in which the TD procedure is executed, or may notify the driver of TD through the mobile device 300 .

[0073] The processor 140 may continue the autonomous driving mode in response to a situation where the driver is determined to be in a state in which a TD may be received.

[0074] The processor 140 may determine that the driver is in a state in which the TD cannot be received in response to a situation where the mobile device 300 is not located around the driver's seat and the mobile device 300 is located in a passenger seat or a rear seat.

[0075] The processor 140 may determine whether the mobile device 300 is located around the driver's seat and whether manipulation of the mobile device 300 is performed at predetermined intervals.

[0076] The processor 140 may determine whether to check the driver's pupils through the front camera of the mobile device 300 in response to a situation where the manipulation of the mobile device 300 is not performed at predetermined intervals.

[0077] The processor 140 outputs a touch notification through the mobile device 300 in response to a situation where the driver's pupils are not recognized through the front camera of the mobile device 300, and may determine whether the driver performs a touch manipulation of the mobile device 300 in response to the touch notification.

[0078] The processor 140 may determine that the driver is in a state in which the TD cannot be received in response to a case where the driver does not perform a touch manipulation of the mobile device 300 according to the touch notification.

[0079] The processor 140 may set the screen of the mobile device 300 to automatically turn off in response to the mobile device 300 being located around the driver's seat, and may determine that the driver is in a state in which TD cannot be received in response to the screen of the mobile device 300 automatically turning off.

[0080] The processor 140 may estimate the location of the mobile device 300 using camera information of the mobile device 300 and information from the information sensor 220 within the vehicle, and may determine whether the mobile device 300 is a mobile device 300 owned, handled, or used by the driver.

[0081] The vehicle exterior information sensors 210 may include one or more sensors that sense obstacles located around the host vehicle (eg, a vehicle ahead) and measure the distance to the obstacles and / or their relative speed.

[0082] The vehicle external information sensor 210 may include a plurality of sensors to sense external objects of the vehicle, thereby obtaining information related to the position of the external object, the speed of the external object, the moving direction of the external object, and / or the type of the external object (e.g., a vehicle, a pedestrian, a bicycle, or a motorcycle, etc.). To this end, the vehicle external information sensor 210 may include an ultrasonic sensor, a radar, an external camera, a laser scanner and / or a corner radar, a laser radar, an acceleration sensor, a yaw rate sensor, a torque measurement sensor, and / or a wheel speed sensor, a steering angle sensor, etc.

[0083] The vehicle interior information sensor 220 may include a camera for photographing the interior of the vehicle and the user.

[0084] The GPS receiver 230 receives a GPS signal from a satellite and transmits it to the autonomous driving control device 100. Therefore, the position of the host vehicle of the autonomous driving control device 100 can be grasped.

[0085] Although the map database 240 is illustrated as being included in the vehicle system, the autonomous driving control device 100 may be located outside the vehicle to receive map information from the map database 240. In this case, the map data may include a precise map.

[0086] The direction indicator 250 is controlled by the autonomous driving control device 100 to turn on in response to turning left or right.

[0087] The emergency flashing indicator 260 flashes in response to an emergency or for warning.

[0088] The brake control device 270 may be configured to control braking of the vehicle, and may include a controller that controls brakes thereof.

[0089] The driving control device 280 may be configured to control an engine or motor drive of the vehicle, and may include a controller that controls a speed of the vehicle.

[0090] The steering control device 290 may be configured to control a steering angle of a vehicle, and may include a steering wheel, an actuator interlocked with the steering wheel, and a controller that controls the actuator.

[0091] In response to a situation where there is no driver input such as driving / braking / steering during operation of the autonomous driving control device 100, acceleration / deceleration control and steering control are performed according to a situation predetermined by the driver or determined by the autonomous driving control device 100 as a general conditional autonomous driving situation.

[0092] The mobile device 300 may include a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a netbook computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a mobile communication terminal, an electronic notebook, etc., as well as any device on which a navigation application can be installed.

[0093] Figure 2 An exemplary processor is shown.

[0094] refer to Figure 2 The processor 140 may include an external situation recognition device 141, a driving state recognition device 142, a display controller 143, a lighting controller 144, a vehicle position recognition device 145, a driving path creator 146, a collision determination prediction device 147, a drive controller 148 and a vehicle internal state determination device 149.

[0095] The external situation recognition device 141 recognizes external situations during autonomous driving based on the sensing information of the vehicle external information sensor 210. In this case, the external situation may include the inter-vehicle distance to the front vehicle, the speed of the front vehicle, cut-in vehicle information, and the like.

[0096] The driving state recognition device 142 recognizes the driving state of the vehicle. That is, the driving state may include states such as an autonomous driving mode and a manual driving mode.

[0097] The display controller 143 may control the operation of the interface device 130 .

[0098] The lighting controller 144 may control the flashing operation of the emergency flashing indicator 260 .

[0099] The vehicle location identification device 145 may determine the location of the vehicle based on GPS information of the GPS receiver 230 .

[0100] The driving path creator 146 may create a path for autonomous driving or correct the created path.

[0101] The collision determination prediction device 147 may predict a collision with an external object around the vehicle (e.g., a preceding vehicle, etc.). Additionally or alternatively, the collision determination prediction device 147 may calculate the likelihood of a collision for an area around the host vehicle on the current travel path, and predict a collision determination in a general autonomous driving situation.

[0102] The driving controller 148 may adjust the actuator output value by calculating the vehicle position and driving path predicted by the vehicle position identification device 145 and determining the driver's steering / braking / driving amount.

[0103] The vehicle interior state determination device 149 may determine the driver state using the vehicle interior information sensor 220 and the mobile device 300 to determine whether the driver state is available or unavailable, and may request the driver to receive a TD.

[0104] Thus, according to the present invention, during conditional autonomous driving, the driver state is monitored so that the driver can respond immediately in response to TD.

[0105] The following will refer to Figures 4 to 6 A driver state determination method of an autonomous driving control apparatus according to an example of the present invention is described in detail. Figure 4 An exemplary method for determining a driver's state using a camera of a mobile device of an autonomous driving control device is shown, Figure 5 An exemplary method for determining a driver's status by an automatic shutoff function of a mobile device using an autonomous driving control device is shown, and Figure 6 An exemplary driver state determination method using a mobile device during autonomous driving control is shown.

[0106] In the following, it is assumed Figure 1 The autonomous driving control device 100 performs Figures 4 to 6 Additionally or alternatively, Figures 4 to 6 In the description, operations described as being performed by the device may be understood as being controlled by the processor 140 of the autonomous driving control device 100.

[0107] refer to Figure 4 , the autonomous driving control device 100 determines whether the mobile device 300 is located around the driver's seat in response to the situation where the driver availability determination starts (S101). In this case, the autonomous driving control device 100 may perform Bluetooth or ultra-wideband (UWB) communication with the mobile device 300 to determine whether the mobile device 300 is located around the driver's seat. Additionally or alternatively, the autonomous driving control device 100 may determine whether the mobile device 300 is located around the driver's seat by interlocking with an application (APP) installed in the mobile device 300. Additionally or alternatively, determining whether the mobile device 300 is located around the driver's seat is determining whether the mobile device 300 is the driver's mobile device.

[0108] In response to the situation where the mobile device 300 is not located around the driver's seat (i.e., the situation where the mobile device 300 is located in the passenger seat or the rear passenger seat (rear seat)), the driver is in a position different from the driver's seat, and therefore it can be determined that the driver is in an unavailable state (S105).

[0109] On the other hand, if the mobile device 300 is located around the driver's seat, the autonomous driving control device 100 determines whether the manipulation of the mobile device 300 is periodically performed (S102). In this case, the autonomous driving control device 100 may determine whether the mobile device 300 is manipulated by communicating with the mobile device 300.

[0110] In response to the situation that the manipulation of the mobile device 300 is periodically performed in the above step S102, the autonomous driving control device 100 determines that the driver state is available (S103) and returns to the initial step of determining the availability. That is, in response to the situation that the manipulation of the mobile device 300 is periodically performed, the autonomous driving control device 100 determines a state in which TD can be received, which is a state in which the driver is manipulating the mobile device 300 in the driver's seat, that is, a state in which there is availability.

[0111] Meanwhile, in order to check the driver's state in more detail in response to a situation where the manipulation of the mobile device 300 is not periodically performed, the autonomous driving control device 100 determines whether the driver's pupils are checked by the front camera of the mobile device 300 ( S103 ).

[0112] In response to the front camera of the mobile device 300 detecting the driver's pupils, the driver is using the mobile device 300 and is not drowsy or asleep, and thus the autonomous driving control device 100 may determine that the driver's status is available.

[0113] On the other hand, in response to the situation where the front camera of the mobile device 300 does not detect the driver's pupils, in order to determine the driver's state (such as the driver's closed eyes) and check the driver's state in more detail, the mobile device 300 generates a touch notification to check whether the driver performs a touch operation (S104).

[0114] The autonomous driving control device 100 determines that the driver's state is unavailable in response to a case where a touch manipulation is not performed for the touch notification of the mobile device 300 (S105). On the other hand, the autonomous driving control device 100 determines that the driver is in an available state in response to a case where the driver performs a touch manipulation for the touch notification of the mobile device 300.

[0115] In this way, according to the present invention, it is possible to accurately determine whether the driver is in an available state by determining whether the mobile device 300 is located around the driver's seat, and determining whether periodic manipulation of the mobile device 300 is performed, use of the front camera of the mobile device 300, whether touch manipulation is performed through touch notification of the mobile device 300, etc.

[0116] refer to Figure 5 , the autonomous driving control device 100 determines whether the mobile device 300 is located around the driver's seat ( S201 ).

[0117] In response to the mobile device 300 being located around the driver's seat, the autonomous driving control device 100 automatically changes the automatic screen off time of the mobile device 300 to a predetermined time (e.g., 15 seconds), and changes the screen of the mobile device 300 to automatically turn off when watching a video (S202).

[0118] The autonomous driving control device 100 determines whether the mobile device 300 is not automatically turned off ( S203 ), and in response to the automatic turning off, determines that the driver status is unavailable ( S204 ).

[0119] On the other hand, in response to a case where the mobile device 300 is not automatically turned off, it is because the driver continues to operate the mobile device 300 , and thus the autonomous driving control device 100 determines the driver status as available and continues to periodically determine the driver availability.

[0120] like Figure 4 and Figure 5 As shown, according to the present invention, in order to determine the driver state, the driver state determination logic may be started after determining whether the mobile device 300 connected to the vehicle is the driver's mobile device. In addition or alternatively, in response to satisfying Figure 4 Steps S102, S103 and S104 and Figure 5The driver status can be determined to be available if at least one of the conditions for determining the driver status in step S203 is met, or a combination of two or more conditions is met.

[0121] refer to Figure 6 , the autonomous driving control device 100 determines the driver availability through the in-vehicle camera in response to the situation where the conditional autonomous driving mode starts ( S301 ).

[0122] Since the driver availability is determined by the in-vehicle camera, the autonomous driving control device 100 continues the conditional autonomous driving mode in response to the driver status being available.

[0123] On the other hand, since the driver availability is determined by the in-vehicle camera, the autonomous driving control device 100 determines whether the mobile device 300 is in a communication connection state with the vehicle in response to the driver status being unavailable (S302). That is, the autonomous driving control device 100 may determine whether the mobile device 300 is connected to the communication device 110 of the autonomous driving control device 100 through Bluetooth communication.

[0124] In response to the mobile device 300 not being in a communication connection state with the vehicle, the autonomous driving control device 100 continues the TD procedure to execute TD to the driver (S305) because the driver status is unavailable. Subsequently, the autonomous driving control device 100 notifies the execution of TD through the instrument panel (not shown), the multimedia system (not shown), etc. (S306).

[0125] On the other hand, in response to the situation that the mobile device 300 is in a communication connection state with the vehicle, the autonomous driving control device 100 determines whether the connected mobile device 300 is a mobile device 300 owned, handled, or used by the driver (S303). In this case, the autonomous driving control device 100 may determine whether the mobile device 300 belongs to the driver based on the information exchanged with the mobile device 300. In addition or alternatively, the autonomous driving control device 100 may determine whether the mobile device 300 is owned, handled, or used by the driver by determining whether the mobile device 300 is located around the driver's seat.

[0126] In response to the case where the connected mobile device 300 is not a mobile device 300 owned, handled, or used by the driver, the autonomous driving control device 100 continues the TD procedure to execute TD to the driver because the driver status is unavailable (S305). Subsequently, the autonomous driving control device 100 notifies the execution of TD through the instrument panel (not shown), the multimedia system (not shown), the mobile device 300, etc. (S306).

[0127] On the other hand, the autonomous driving control device 100 determines the driver availability using the mobile device 300 in response to the connected mobile device 300 being a mobile device 300 owned, handled, or used by the driver ( S304 ), and continues the conditional autonomous driving mode in response to the driver status being available.

[0128] In the above Figures 4 to 6 In the example of using the mobile device 300 and the indoor camera of the vehicle to determine whether the driver status is available, respectively, the driver status may be determined by configuring various combinations of conditions for determining the corresponding driver status.

[0129] That is, in response to checking whether the mobile device 300 is located around the driver's seat, whether the mobile device 300 is the driver's mobile device, whether the mobile device 300 is connected to the vehicle, etc., the autonomous driving control device 100 determines the driver's status by using at least one of the following: whether the mobile device 300 is manipulated using the vehicle's internal camera, whether the driver's pupils are checked through the front camera of the mobile device 300, whether a touch operation is performed through the touch alarm of the mobile device 300, whether the mobile device 300 is automatically turned off, or a combination thereof.

[0130] In addition or alternatively, the autonomous driving control device 100 may use camera information of the mobile device 300 and information related to a camera in the vehicle to estimate the position of the mobile device 300 and determine whether it is the driver's mobile device, without checking whether the mobile device 300 is located around the driver's seat, whether the mobile device 300 is the driver's mobile device, whether the mobile device 300 is connected to the vehicle, etc., and may determine whether the driver is using the mobile device 300 by using a front camera in the vehicle.

[0131] In this way, according to the present technology, the accuracy of driver availability determination can be improved by using a mobile device as well as an in-vehicle camera to determine driver availability during conditional autonomous driving, thereby improving driver convenience.

[0132] Figure 7 An exemplary computing system is shown.

[0133] refer to Figure 7 The computing system 1000 includes at least one processor 1100 , a memory 1300 , a user interface input device 1400 , a user interface output device 1500 , a storage device 1600 , and a network interface 1700 connected via a bus 1200 .

[0134] The processor 1100 may be a central processing unit (CPU) or a semiconductor device that performs processing on commands stored in the memory 1300 and / or the storage device 1600. The memory 1300 and the storage device 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include a read-only memory (ROM) 1310 and a random access memory (RAM) 1320.

[0135] Therefore, the steps of the method or algorithm described in conjunction with the examples disclosed herein may be implemented directly by hardware, software modules, or a combination of the two executed by the processor 1100. The software module may reside in a storage medium (i.e., the memory 1300 and / or the storage device 1600), such as a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, and a CD-ROM.

[0136] The storage medium is coupled to the processor 1100, and the processor can read information from the storage medium and write information to the storage medium. Alternatively or additionally, the storage medium can be integrated with the processor 1100. The processor and the storage medium can reside in an application specific integrated circuit (ASIC). The ASIC can reside in a user terminal. Alternatively or additionally, the processor and the storage medium can reside in a user terminal as separate components.

[0137] Examples of the present invention attempt to provide an autonomous driving control apparatus and a driver state determination method thereof, which can enable a driver to immediately act on a takeover request by accurately monitoring the driver's state during conditional autonomous driving.

[0138] Additionally or alternatively, examples of the present invention attempt to provide an autonomous driving control apparatus and a driver state determination method thereof that are capable of accurately determining a driver's state using a mobile device during conditional autonomous driving.

[0139] Examples of the present invention also attempt to provide an autonomous driving control device and a driver state determination method thereof, which are capable of accurately determining the driver's state using a mobile device and an in-vehicle camera during conditional autonomous driving.

[0140] The technical objectives of the present invention are not limited to the above objectives, and those skilled in the art can clearly understand other technical objectives not mentioned from the description of the claims.

[0141] An example of the present invention provides an autonomous driving control device, comprising: a communication device configured to communicate with a mobile device during autonomous driving; a processor configured to determine whether a driver state is a state in which the mobile device will be used to receive a takeover request (TD), and to continue the autonomous driving or execute TD based on the driver state; and a memory configured to store data and algorithms driven by the processor.

[0142] In an example of the present invention, the processor may be configured to determine that the driver state is at a conditional autonomous driving level among the autonomous driving levels.

[0143] In an example of the present invention, the processor may be configured to determine the driver state by using information sensors within the vehicle and the mobile device.

[0144] In an example of the present invention, in response to the information sensor in the vehicle determining that the driver state is not a state in which TD is to be received, the processor may be configured to determine whether the mobile device is in a communication connection state with the vehicle.

[0145] In an example of the present invention, in response to the mobile device being in a communication connection state with the vehicle, the processor may be configured to determine whether the mobile device is a mobile device owned, handled or used by the driver.

[0146] In an example of the present invention, in response to the mobile device being located around a driver's seat, the processor may be configured to determine whether the mobile device is a mobile device owned, handled or used by the driver.

[0147] In an example of the invention, in response to the mobile device being a mobile device owned, handled or used by the driver, the processor may be configured to determine the driver status using the mobile device.

[0148] In an example of the present invention, in response to a situation where the mobile device connected to the communication is not a mobile device owned, handled or used by the driver (although the mobile device is not communicatively connected to the vehicle or the mobile device is communicatively connected to the vehicle), the processor may be configured to execute a takeover request (TD) procedure for receiving a TD from the driver.

[0149] In an example of the present invention, the processor may be configured to output a TD notification message on an in-vehicle display to notify the driver of TD in response to execution of the TD procedure or notify the driver of TD through the mobile device.

[0150] In an example of the present invention, the processor may be configured to continue the autonomous driving in response to determining that the driver state is a state in which a TD is to be received.

[0151] In an example of the present invention, the processor may be configured to determine that the driver is in a state in which TD will not be received in response to the mobile device not being located around the driver's seat and the mobile device being located in a passenger seat or a rear seat.

[0152] In an example of the present invention, the processor may be configured to determine whether manipulation of the mobile device is performed at predetermined intervals, and determine that the driver state is a state in which TD will not be received in response to a situation in which manipulation of the mobile device is not performed at predetermined intervals.

[0153] In an example of the present invention, the processor may be configured to determine whether the driver's pupils are checked through the front camera of the mobile device, and in response to the driver's pupils not being checked through the front camera of the mobile device, determine that the driver state is a state in which TD will not be received.

[0154] In an example of the present invention, the processor may be configured to output a touch notification through the mobile device, determine whether the driver performs a touch manipulation of the mobile device in response to the touch notification, and determine that the driver state is a state in which TD will not be received in response to the driver not performing a touch manipulation of the mobile device according to the touch notification.

[0155] In an example of the present invention, the processor may be configured to set automatic screen off of the mobile device in response to the mobile device being located around a driver's seat, and determine that the driver state is a state in which TD will not be received in response to the screen of the mobile device being automatically turned off.

[0156] In an example of the present invention, the processor may be configured to estimate the location of the mobile device using camera information of the mobile device and information from in-vehicle information sensors, and determine whether the mobile device is a mobile device owned, handled or used by the driver.

[0157] An example of the present invention provides a method for determining a driver state of an autonomous driving control device, the method comprising: a processor communicating with a mobile device during autonomous driving; the processor determining whether the driver state is a state in which the mobile device will be used to receive a takeover request (TD); and the processor continuing autonomous driving or executing TD based on the driver state.

[0158] In an example of the present invention, the determining whether the driver state is a state in which TD is to be received includes determining that the driver state is at a conditional autonomous driving level among autonomous driving levels.

[0159] In an example of the present invention, the determining whether the driver state is a state in which a TD is to be received includes determining, by the processor, the driver state by using an information sensor within the vehicle and the mobile device.

[0160] In an example of the present invention, determining whether the driver state is a state in which TD will be received includes: determining, by the processor, whether manipulation of the mobile device is performed at predetermined intervals; and determining, by the processor, in response to a situation in which manipulation of the mobile device is not performed at predetermined intervals, that the driver state is a state in which TD will not be received.

[0161] According to the present technology, it is possible to enable a driver to immediately act on a takeover request by accurately monitoring the driver's state during conditional autonomous driving.

[0162] According to the present technology, driver convenience can be increased by accurately determining the driver's state using a mobile device during conditional autonomous driving.

[0163] Additionally or alternatively, according to the present technology, the accuracy of driver availability determinations may be improved by using a mobile device and an in-vehicle camera to determine driver availability during conditional autonomous driving.

[0164] Furthermore, various effects that can be directly or indirectly recognized through the file can be provided.

[0165] The above description is merely an illustration of the technical idea of ​​the present invention, and those skilled in the art to which the present invention belongs may make various modifications and variations without departing from the essential characteristics of the present invention.

[0166] Therefore, the examples disclosed in the present invention are not intended to limit the technical ideas of the present invention but to explain them, and the scope of the technical ideas of the present invention is not limited to these examples. The protection scope of the present invention should be interpreted by the attached claims, and all technical ideas within the equivalent scope should be interpreted as included in the scope of the present invention.

Claims

1. A device for controlling a driving mode of a vehicle, the device include: processor; a memory storing instructions that, when executed by the processor, cause the apparatus to: communicating with a mobile device while the vehicle is traveling in the autonomous driving mode; determining, based on communication with the mobile device, whether the driver status is an available state for a manual driving mode; as well as Based on the determination, the vehicle is caused to remain in the autonomous driving mode or the mobile device is caused to send a takeover request TD to switch the driving mode of the vehicle from the autonomous driving mode to the manual driving mode. 2 . The apparatus of claim 1 , wherein the instructions, when executed by the processor, further cause the apparatus to determine that the driver state is at a conditional autonomous driving level among a plurality of autonomous driving levels. 3 . The apparatus of claim 1 , wherein the instructions, when executed by the processor, further cause the apparatus to determine the driver status by using information sensors within the vehicle and the mobile device.

4. The device according to claim 3, wherein the instructions, when executed by the processor, further cause the device to determine whether the mobile device is in a communication connection state with the vehicle based on a determination by the information sensor within the vehicle that the driver state is not the available state.

5. The apparatus of claim 4, wherein the instructions, when executed by the processor, further cause the apparatus to determine whether the mobile device is a mobile device owned by a driver of the vehicle based on determining that the mobile device is in a communication connection state with the vehicle. 6 . The apparatus of claim 5 , wherein the instructions, when executed by the processor, further cause the apparatus to determine that the mobile device is a mobile device owned by the driver based on the mobile device being located near a driver's seat. 7 . The apparatus of claim 5 , wherein the instructions, when executed by the processor, further cause the apparatus to determine the driver status using the mobile device based on the mobile device being a mobile device owned by the driver.

8. The apparatus of claim 2, wherein the instructions, when executed by the processor, further cause the apparatus to: Based on a determination that the mobile device is communicatively connected to the vehicle and a driver of the vehicle does not own the mobile device, or based on a determination that the mobile device is not communicatively connected to the vehicle, a TD procedure is executed for notifying the driver of the TD.

9. The apparatus of claim 8, wherein the instructions, when executed by the processor, further cause the apparatus to: Outputting a TD notification message on a display in the vehicle; or The driver is notified of the TD through the mobile device based on the execution of the TD procedure.

10. The apparatus of claim 1, wherein the instructions, when executed by the processor, further cause the apparatus to cause the vehicle to operate in the manual driving mode based on determining that the driver state is the available state.

11. The apparatus according to claim 1, wherein the instructions, when executed by the processor, further cause the apparatus to: determine that the driver state is an unavailable state in which the TD is not sent based on determining that the mobile device is not located near a driver's seat or that the mobile device is located in a passenger seat or a back seat.

12. The apparatus of claim 1, wherein the instructions, when executed by the processor, further cause the apparatus to: determining whether manipulation of the mobile device is performed at predetermined intervals; and Based on determining that the manipulation of the mobile device is not performed at the predetermined interval, it is determined that the driver state is an unavailable state in which the TD is not transmitted.

13. The apparatus of claim 1, wherein the instructions, when executed by the processor, further cause the apparatus to: determining whether to detect the driver's pupils via a front-facing camera of the mobile device; and Based on determining that the driver's pupils are not detected via the front camera of the mobile device, it is determined that the driver state is an unavailable state in which the TD is not sent.

14. The apparatus of claim 1, wherein the instructions, when executed by the processor, further cause the apparatus to: outputting a touch notification via the mobile device; determining whether the driver performs a touch manipulation of the mobile device based on the touch notification; as well as Based on determining that the driver does not perform a touch manipulation of the mobile device, it is determined that the driver state is an unavailable state in which the TD is not transmitted.

15. The apparatus of claim 1, wherein the instructions, when executed by the processor, further cause the apparatus to: Setting an automatic screen off mode for the mobile device; and Based on the screen of the mobile device being automatically turned off, it is determined that the driver state is an unavailable state in which the TD is not sent.

16. The apparatus of claim 1, wherein the instructions, when executed by the processor, further cause the apparatus to: estimating the location of the mobile device using camera information of the mobile device and information from information sensors within the vehicle; and A determination is made based on the estimated location whether the mobile device is a mobile device owned by a driver.

17. A method for controlling a driving mode of a vehicle, the method include: communicating, by a processor, with a mobile device while the vehicle is traveling in an autonomous driving mode; determining, by the processor based on communication with the mobile device, whether a driver state is an available state for a manual driving mode; as well as Based on the determination, the vehicle is caused to remain in the autonomous driving mode or the mobile device is caused to send a takeover request TD to switch the driving mode of the vehicle from the autonomous driving mode to the manual driving mode. 18 . The method of claim 17 , wherein determining whether the driver state is the available state comprises determining, by the processor, that the driver state is at a conditional autonomous driving level among a plurality of autonomous driving levels.

19. The method of claim 17, wherein determining whether the driver state is the available state comprises determining, by the processor, the driver state using information sensors within the vehicle and the mobile device.

20. The method according to claim 17, wherein determining whether the driver status is the available status include: determining, by the processor, whether manipulation of the mobile device is performed at predetermined intervals; as well as The processor determines, based on determining that manipulation of the mobile device is not performed at the predetermined interval, that the driver state is an unavailable state in which the TD is not transmitted.

Images