Mobile terminal identification system

Abstract

To provide a system which determines a portable terminal of a driver.SOLUTION: An automatic driving device 100 includes an automatic driving ECU 10 mounted on a vehicle VA and provided with a first imaging unit (driver monitor camera) for capturing an image of a driver of the vehicle and a portable terminal 110 provided with a second imaging unit (front camera). The automatic driving ECU determines whether the portable terminal is a portable terminal of the driver or not by performing comparison processing of comparing a first image captured by the first imaging unit and a second image captured by the second image unit with each other.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to a system for determining a mobile device of a driver. [Background technology]

[0002] Conventionally, a system has been proposed that provides services to a driver by connecting a mobile terminal carried by the driver to a vehicle (see, for example, Patent Document 1). An example of such a service is a service that determines whether the driver is in an awake state. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-115983 A Summary of the Invention

[0004] A mobile terminal and a vehicle are usually connected via wireless communication. However, a vehicle may have passengers other than the driver. If the passenger carries a mobile terminal, the passenger's mobile terminal may be connected to the vehicle via wireless communication. In this way, when multiple mobile terminals are connected to the vehicle, the system needs to determine (identify) which mobile terminal is the driver's mobile terminal. If the system cannot identify the driver's mobile terminal, it cannot provide services to the driver.

[0005] One object of the present disclosure is to provide a technique capable of identifying a driver's mobile terminal in a system in which a mobile terminal and a vehicle are connected.

[0006] In one or more embodiments, the mobile terminal determination system includes: a first imaging unit (75) mounted in a vehicle (VA) and configured to capture an image of a driver of the vehicle; A mobile terminal (110) including a second imaging unit (202); a control device (10) configured to determine whether the mobile terminal is a mobile terminal of the driver by executing a comparison process that compares a first image (500) acquired by the first imaging unit with a second image (600) acquired by the second imaging unit; Equipped with.

[0007] The second image captured by the second imaging unit of the portable device carried by the driver includes the driver. On the other hand, the second image captured by the second imaging unit of the portable device carried by a passenger other than the driver does not include the driver. Therefore, the control device having the above configuration can determine which portable device is the driver's portable device by comparing the first image and the second image.

[0008] In one or more embodiments, the comparison process comprises: comparing the faces recognized in the first image with the faces recognized in the second image; A process of comparing the behavior of the occupant recognized in the first image with the behavior of the occupant recognized in the second image; and comparing a change in luminance in the first image with a change in luminance in the second image; Includes at least one of the following.

[0009] In one or more embodiments, the mobile terminal determination system includes: A first communication unit mounted in a vehicle (VA); A mobile terminal (110) having a second communication unit (204); a control device (10) configured to determine whether the mobile terminal is a mobile terminal of a driver based on a communication state between the first communication unit and the second communication unit; Equipped with.

[0010] The control device having the above configuration can determine which mobile terminal is the driver's mobile terminal, based on the communication status between the first communication unit and the second communication unit.

[0011] In one or more embodiments, the first communication unit is a short-range communication unit (401) installed near the driver's seat.

[0012] For example, the second communication unit of the portable terminal carried by the driver can establish communication with the short-range communication unit, but the second communication unit of the portable terminal carried by a passenger other than the driver cannot establish communication with the short-range communication unit. Taking such a situation into consideration, the control device having the above configuration can determine which portable terminal is the driver's portable terminal.

[0013] In one or more embodiments, The first communication unit includes a plurality of short-range communication units installed separately within the vehicle. Each of the plurality of short-range communication units measures a receiving strength of a signal transmitted from the second communication unit. The control device is configured to determine whether the mobile terminal is a driver's mobile terminal based on the difference in reception strength among the multiple short-range communication units.

[0014] For example, assume that a first short-range communication unit among the multiple short-range communication units is installed near the driver's seat, and a second short-range communication unit among the multiple short-range communication units is installed in the back seat. When the first short-range communication unit and the second short-range communication unit each receive a signal from the second communication unit of a mobile terminal carried by the driver, the reception strength of the signal received by the first short-range communication unit is high, and the reception strength of the signal received by the second short-range communication unit is low. The control device having the above configuration can determine which mobile terminal is the driver's mobile terminal based on such a difference in reception strength.

[0015] In one or more embodiments, the control device may be implemented by at least one processor programmed to perform one or more functions described herein. In one or more embodiments, the control device may be implemented in whole or in part by hardware configured by one or more application specific integrated circuits, i.e., ASICs. In the above description, the names and / or symbols used in the embodiments are added in parentheses to components corresponding to one or more embodiments described below. However, each component is not limited to the embodiment defined by the name and / or symbol. [Brief description of the drawings]

[0016] [Figure 1] 1 is a schematic configuration diagram of a mobile terminal determination system according to a first embodiment. [Diagram 2] 1 is an external view of a mobile terminal according to a first embodiment. [Diagram 3] 1 is a schematic configuration diagram of a mobile terminal according to a first embodiment. [Figure 4] FIG. 13 is a diagram showing a situation in which a driver operates a mobile terminal while automatic driving control is being executed. [Diagram 5] 1 is an example of an image (first image) captured by a driver monitor camera. [Figure 6] 13 is an example of an image (second image) captured by the mobile terminal. [Figure 7] FIG. 2 is a sequence diagram of an automatic driving device (automatic driving ECU) and mobile terminals (first mobile terminal and second mobile terminal). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] First Embodiment As shown in FIG. 1, the mobile terminal determination system includes an automatic driving device 100 mounted (applied) on a vehicle VA, and a mobile terminal 110.

[0018] (Configuration of the automatic driving device) The autonomous driving device 100 includes an autonomous driving ECU 10, an engine ECU 20, a brake ECU 30, a steering ECU 40, a meter ECU 50, and a navigation ECU 60. These ECUs are connected to each other via a controller area network (CAN) 90 so as to be able to transmit and receive data to and from each other.

[0019] ECU is an abbreviation of Electronic Control Unit, and is an electronic control circuit having a microcomputer as a main component. The microcomputer includes a CPU, ROM, RAM, an interface, a non-volatile memory, etc. The CPU realizes various functions by executing instructions (routines, programs) stored in the ROM. For example, an autonomous driving ECU 10 includes a microcomputer including a CPU 10a, a ROM 10b, a RAM 10c, an interface (I / F) 10d, a non-volatile memory 10e, etc.

[0020] The engine ECU 20 is connected to an engine actuator 21. The engine actuator 21 includes a throttle valve actuator that changes the opening degree of a throttle valve of the internal combustion engine 22. The engine ECU 20 can change the torque generated by the internal combustion engine 22 by driving the engine actuator 21. The torque generated by the internal combustion engine 22 is transmitted to driving wheels via a transmission 23 and a driving force transmission mechanism (not shown). Therefore, the engine ECU 20 can control the driving force of the vehicle VA by controlling the engine actuator 21.

[0021] If the vehicle VA is a hybrid vehicle, the engine ECU 20 can control the driving force generated by either or both of the "internal combustion engine and the electric motor" as the driving source. Furthermore, if the vehicle VA is an electric vehicle, the engine ECU 20 can control the driving force generated by the electric motor as the driving source.

[0022] The brake ECU 30 is connected to a brake actuator 31. The brake actuator 31 includes a known hydraulic circuit. The hydraulic circuit includes a reservoir, an oil pump, various valve devices, and the like. The brake actuator 31 adjusts the hydraulic pressure (i.e., braking pressure) supplied to the wheel cylinder 32 in response to an instruction from the brake ECU 30. The frictional braking force generated on the wheel changes in response to the braking pressure. Therefore, the brake ECU 30 can control the braking force of the vehicle VA by controlling the brake actuator 31.

[0023] The steering ECU 40 is connected to an assist motor (M) 41. The assist motor 41 is incorporated in a steering mechanism 42. The steering mechanism 42 is a mechanism for steering the steered wheels by rotating the steering wheel SW. The steering mechanism 42 includes a steering wheel SW, a steering shaft US connected to the steering wheel SW, and a steering gear mechanism (not shown). The steering ECU 40 detects the steering torque input to the steering wheel SW by the driver using a steering torque sensor (not shown) provided on the steering shaft US, and drives the assist motor 41 based on this steering torque. The steering ECU 40 applies a steering torque (steering assist torque) to the steering mechanism 42 by driving the assist motor 41, thereby assisting the steering operation of the driver.

[0024] In addition, the steering ECU 40 drives the assist motor 41 in accordance with a steering command (including an automatic steering torque, described later) from the automatic driving ECU 10 while the automatic driving control described below is being executed. This automatic steering torque is different from the steering assist torque described above, and represents a torque that is applied to the steering mechanism 42 by a steering command from the automatic driving ECU 10 without requiring a steering operation by the driver. The automatic steering torque changes the steering angle of the steered wheels of the vehicle VA.

[0025] The meter ECU 50 is connected to an alarm device 51. The alarm device 51 includes a buzzer and a display device. The display device is, for example, a multi-information display provided in front of the driver's seat. In response to an instruction from the autonomous driving ECU 10, the meter ECU 50 can sound a buzzer to alert the driver or display an alert mark (warning lamp) on the display device.

[0026] The navigation ECU 60 is connected to a GPS receiver 61 and a map storage unit 62. The GPS receiver 61 receives a GPS signal for detecting the "latitude and longitude" of the location where the vehicle VA is located. The map storage unit 62 stores map information. The map information includes road information. The road information includes the road type, road width, road curvature, etc. In this example, the road type is either a motorway (expressway) or a general road. The navigation ECU 60 performs various calculation processes based on the latitude and longitude of the location where the vehicle VA is located and the map information, etc., and displays the position of the vehicle VA on the map on a touch panel (not shown).

[0027] The vehicle speed sensor 71 detects the traveling speed (vehicle speed) SPD of the vehicle VA, and outputs a signal indicating the vehicle speed SPD to the automatic driving ECU 10.

[0028] The surrounding sensor 72 is adapted to obtain information about the road around the vehicle VA (including the lane in which the vehicle VA is traveling) and information about three-dimensional objects existing on the road. The three-dimensional objects include, for example, moving objects such as pedestrians, four-wheeled vehicles and two-wheeled vehicles, as well as fixed objects such as guardrails and fences. Hereinafter, these three-dimensional objects will be simply referred to as "objects." The surrounding sensor 72 includes a radar sensor 73 and a camera sensor 74.

[0029] The radar sensor 73, for example, emits millimeter wave band radio waves (hereinafter referred to as "millimeter waves") to a surrounding area including at least the area in front of the vehicle VA, and receives millimeter waves (i.e., reflected waves) reflected by objects present within the emission range. The radar sensor 73 then determines whether or not an object is present, and calculates information indicating the relative relationship between the vehicle VA and the object. The information indicating the relative relationship between the vehicle VA and the object includes the distance between the vehicle VA and the object, the orientation (or position) of the object with respect to the vehicle VA, and the relative speed between the vehicle VA and the object. Hereinafter, the information indicating the relative relationship between the vehicle VA and the object will be simply referred to as "object information."

[0030] The camera sensor 74 photographs the scenery ahead of the vehicle VA to obtain image data. Based on the image data, the camera sensor 74 recognizes the left and right lane lines (e.g., left and right white lines) that define the driving lane, and calculates parameters that indicate the shape of the driving lane (e.g., curvature) and parameters that indicate the positional relationship between the vehicle VA and the driving lane. The parameter that indicates the positional relationship between the vehicle VA and the driving lane is, for example, the distance from the left or right lane line to the center position of the vehicle VA in the vehicle width direction. The information obtained by the camera sensor 74 is referred to as "lane information." The camera sensor 74 may be configured to determine the presence or absence of an object based on the image data and calculate object information.

[0031] The surrounding sensor 72 outputs information regarding the surrounding conditions of the vehicle VA, including "object information and lane information", to the autonomous driving ECU 10 as "vehicle surrounding information".

[0032] The driver monitor camera (first imaging unit) 75 is disposed above a column cover (so-called steering column cover) of the steering wheel SW. The driver monitor camera 75 captures an image of the driver to acquire image data. Hereinafter, the image data acquired by the driver monitor camera 75 is referred to as a "first image." The position and orientation of the driver monitor camera 75 are set so that at least the face of the driver is included in the first image. The driver monitor camera 75 outputs the first image to the autonomous driving ECU 10.

[0033] The driver monitor camera 75 may be set at another position as long as it can capture the face of the driver inside the vehicle VA. For example, the driver monitor camera 75 may be provided near the dashboard or the windshield.

[0034] Furthermore, the autonomous driving ECU 10 is connected to a communication unit 80. The communication unit 80 is configured to be capable of transmitting and receiving information to and from the mobile terminal 110 via short-range wireless communication, which will be described later.

[0035] The automatic driving switch 81 is a switch operated by the driver. By operating the automatic driving switch 81, the driver can set the operation state of the automatic driving control, which will be described later.

[0036] (Autonomous Driving Control Overview) Hereinafter, the automatic driving ECU 10 is simply referred to as "ECU 10". The ECU 10 is configured to execute automatic driving control. The level of automatic driving control (so-called automatic driving level) is classified into a plurality of levels 0 to 5 by the Society of Automotive Engineers (SAE). In this example, the automatic driving level is level 3. Level 3 is a level at which all driving operations are executed by automatic driving control under a specific environment, and when a request for changing driving operations (hereinafter simply referred to as "driving operation change request") occurs, the driver is required to perform driving operations. Therefore, automatic driving control at level 3 may be referred to as "conditional automatic driving control". Automatic driving control includes speed control and steering control.

[0037] An example of speed control is adaptive cruise control (hereinafter, simply referred to as "ACC"). ACC itself is publicly known (see, for example, JP 2014-148293 A, JP 2006-315491 A, and the specification of Japanese Patent No. 4172434).

[0038] ACC includes two types of control: constant speed cruise control and preceding vehicle following control. Constant speed cruise control is a control that adjusts the speed of the vehicle VA so that the traveling speed of the vehicle VA matches a target speed Vset without the need to operate the accelerator pedal or brake pedal. Preceding vehicle following control is a control that makes the vehicle VA follow the preceding vehicle (the vehicle to be followed) while maintaining the inter-vehicle distance Dset between the vehicle VA and the preceding vehicle (the vehicle to be followed) at a target inter-vehicle distance Dset without the need to operate the accelerator pedal or brake pedal. The vehicle to be followed is a vehicle that is traveling in the area ahead of the vehicle VA and immediately before the vehicle VA.

[0039] One example of steering control is lane keeping assist control (hereinafter, simply referred to as "LKA"). LKA itself is publicly known (see, for example, JP 2008-195402 A, JP 2009-190464 A, JP 2010-6279 A, ​​and Japanese Patent No. 4349210, etc.).

[0040] LKA is a control that applies automatic steering torque to the steering mechanism 42 to change the steering angle of the steering wheels of the vehicle VA so that the vehicle VA travels along a target driving line that is set using either a lane marking or the driving trajectory of the preceding vehicle (i.e., the preceding vehicle trajectory), or both.

[0041] The ECU 10 can execute the automatic driving control under a specific situation. The ECU 10 executes the automatic driving control when a predetermined automatic driving execution condition is satisfied. For example, the automatic driving execution condition is satisfied when all of the following conditions A1 to A3 are satisfied. (Condition A1): The vehicle VA is traveling on a highway. (Condition A2): The vehicle speed SPD is equal to or less than a predetermined speed threshold SPDth. (Condition A3): The driver presses the automatic driving switch 81 to set the operation state of the automatic driving control from the OFF state to the ON state.

[0042] The speed threshold SPDth may be set to a relatively small value. In this case, the situation in which the autonomous driving execution condition is satisfied means that the vehicle VA is traveling in a congested section. In such a situation, the ECU 10 may execute the autonomous driving control. Note that the autonomous driving execution condition is not limited to the above example, and may be other conditions.

[0043] When a predetermined automatic driving end condition is satisfied, the ECU 10 ends the automatic driving control. For example, the automatic driving end condition is satisfied when at least one of the following conditions B1 to B4 is satisfied. (Condition B1): The vehicle VA arrives at the exit of a freeway. (Condition B2): The vehicle speed SPD becomes greater than the speed threshold SPDth. (Condition B3): The driver presses the automatic driving switch 81 to set the operation state of the automatic driving control from the on state to the off state. (Condition B4): An event has occurred that makes it impossible to continue autonomous driving control (e.g., a system error). The autonomous driving end condition is not limited to the above example, and may be another condition.

[0044] (Mobile device configuration) The mobile terminal 110 is a device carried by a passenger, and in this example, is a smartphone as shown in Fig. 2. The mobile terminal 110 may be another device (for example, a tablet terminal or a wearable terminal) as long as it is carried by the passenger.

[0045] The mobile terminal 110 includes a display 201. The display 201 is a touch panel type display, and is capable of detecting a touch operation on the display surface (operation surface) of the display 201.

[0046] Mobile terminal 110 includes a front camera (second imaging unit) 202 on the surface facing display 201. Mobile terminal 110 may include a rear camera on the surface opposite to display 201 (i.e., the back surface). Mobile terminal 110 further includes an illumination unit 203 (e.g., an LED) on the surface facing display 201. Illumination unit 203 is configured to emit light.

[0047] Furthermore, the mobile terminal 110 includes a communication unit 204 therein. The communication unit 204 is a communication module for communicating with an external device via short-range wireless communication. The short-range wireless communication includes various wireless communication such as Bluetooth (registered trademark), a wireless ID tag (RFID: Radio Frequency Identification), a near-field wireless communication (NFC: Near Field Communication), and WiFi.

[0048] 3, the mobile terminal 110 includes a CPU 301, a memory 302, and a non-volatile memory 303. The CPU 301, the memory 302, the non-volatile memory 303, the display 201, the front camera 202, the illumination unit 203, and the communication unit 204 are connected to each other via an internal bus 310.

[0049] The CPU 301 includes at least one processor or circuit. The memory 302 includes, for example, a RAM. The non-volatile memory 303 includes, for example, a flash memory and a ROM. The CPU 301 uses the memory 302 as a working memory to execute instructions (routines, programs) stored in the non-volatile memory 303.

[0050] (Outline of wakefulness state determination process) In level 3 autonomous driving control, if the ECU 10 cannot continue autonomous driving control (if a request to take over driving operations occurs), the driver must take over driving operations. Therefore, the driver is not permitted to fall asleep. On the other hand, the driver is permitted to look away from the vehicle. Therefore, while the ECU 10 is executing level 3 autonomous driving control, the driver can perform various actions (such as operating a mobile device or reading a book).

[0051] For example, as shown in FIG. 4, there is a case where the driver DR operates the mobile terminal 110 while the ECU 10 is executing the automatic driving control. In such a situation, the ECU 10 cooperates with the mobile terminal 110 to provide a specific service. In this example, the ECU 10 provides a service for determining whether the driver DR is in an awake state so that the driver can immediately take over the driving operation when a driving operation changeover request occurs. If the driver DR is not in an awake state, the ECU 10 calls the driver's attention. This allows the driver DR to return to an awake state.

[0052] Specifically, during execution of the automatic driving control, the ECU 10 determines whether the driver DR is in an awake state by using information acquired by the mobile terminal 110. Hereinafter, such processing is referred to as an "awake state determination processing." Furthermore, the information acquired by the mobile terminal 110 is referred to as "terminal side information."

[0053] For the above configuration, as shown in Fig. 3, the non-volatile memory 303 of the mobile terminal 110 stores a driver monitoring application 311. The driver monitoring application 311 is a program for monitoring the state of the driver DR.

[0054] The CPU 301 executes the driver monitoring application 311 to acquire information to be used in the wakefulness determination process, and transmits the acquired information to the automatic driving device 100 as terminal-side information. Specifically, the CPU 301 acquires image data (hereinafter referred to as a "second image") from the front camera 202. The second image includes the face of the driver DR. The CPU 301 transmits the second image to the automatic driving device 100 as terminal-side information.

[0055] The ECU 10 receives the terminal side information via the communication unit 80. The ECU 10 executes an awakening state determination process based on the second image. Specifically, the ECU 10 determines the awakening degree of the driver DR according to a known method. In this example, the awakening degree includes a first degree, a second degree, and a third degree. The first degree indicates that the driver DR is in an awakening state. The second degree indicates that the driver DR is not in a drowsy state, but is feeling drowsy. The third degree indicates that the driver DR is in a drowsy state.

[0056] For example, the degree of wakefulness is determined based on at least one of the following: eye closure rate, blinking frequency, eye open state (distance between upper eyelid and lower eyelid), eye movement, and the presence or absence of yawning behavior, etc. The eye closure rate will be briefly described below.

[0057] The ECU 10 extracts the eyes of the driver DR from the second image by inputting the second image to a classifier for eye detection or by performing template matching on the second image. The ECU 10 determines whether the eyes are closed based on, for example, the aspect ratio of the extracted eye portion. The ECU 10 calculates the ratio of time the eyes are closed within a predetermined time as an eye closure rate. If the eye closure rate is equal to or less than a first threshold, the ECU 10 determines that the degree of wakefulness of the driver DR is a first degree. If the eye closure rate is greater than the first threshold and equal to or less than a second threshold, the ECU 10 determines that the degree of wakefulness of the driver DR is a second degree. If the eye closure rate is greater than the second threshold, the ECU 10 determines that the degree of wakefulness of the driver DR is a third degree.

[0058] When the ECU 10 determines that the wakefulness level of the driver DR is the third level, the ECU 10 causes the notification device 51 to execute a predetermined attention-calling process. Specifically, the ECU 10 uses the meter ECU 50 to sound a buzzer and to display a warning lamp on the display device.

[0059] The terminal side information used in the awake state determination process is not limited to the second image. The CPU 301 may obtain operation information (touch operation on the display 201 and / or operation on various buttons) in the mobile terminal 110 and / or acceleration information in the mobile terminal 110 as terminal side information and transmit it to the automatic driving device 100. The ECU 10 may execute the awake state determination process using these pieces of information.

[0060] (Overview of operation) There may be passengers other than the driver DR in the vehicle VA. For example, assume that there is a passenger in the back seat. Hereinafter, the mobile terminal 110 carried by the driver DR may be referred to as the "first mobile terminal 110A," and the mobile terminal 110 carried by the passenger in the back seat may be referred to as the "second mobile terminal 110B." When there is no need to distinguish between the first mobile terminal 110A and the second mobile terminal 110B, they will be collectively referred to as the "mobile terminal 110."

[0061] It is assumed that the first portable terminal 110A and the second portable terminal 110B are each connected to the ECU 10 via wireless communication. When the ECU 10 executes the wakefulness state determination process in this situation, the ECU 10 needs to determine (identify) which portable terminal is the first portable terminal 110A of the driver DR.

[0062] Therefore, the ECU 10 executes a comparison process to compare the first image acquired by the driver monitor camera 75 with the second image acquired by the front camera 202 of the mobile terminal 110, thereby determining which mobile terminal is the first mobile terminal 110A of the driver DR. Hereinafter, such a process is referred to as a "terminal determination process."

[0063] In this example, the comparison process is a process of comparing a face recognized in a first image with a face recognized in a second image.

[0064] For the above configuration, as shown in Fig. 3, the non-volatile memory 303 of the mobile terminal 110 stores a terminal determination application 312. The terminal determination application 312 is a program for acquiring a second image.

[0065] The ECU 10 acquires a first image from the driver monitor camera 75. Meanwhile, the CPU 301 of the mobile terminal 110 executes the terminal determination application 312 to transmit a second image as terminal-side information to the automatic driving device 100. The ECU 10 receives the second image as terminal-side information.

[0066] The ECU 10 executes the terminal determination process as follows. The ECU 10 extracts characteristic elements (feature points) of the face included in the first image. In this example, the characteristic elements include the positional relationship of the center points and / or end points of the parts constituting the face. The parts constituting the face include the eyes, eyebrows, nose, mouth, etc., and are hereinafter simply referred to as "facial parts". In another example, the characteristic elements may include the size of the facial parts and / or the aspect ratio of the facial parts. In yet another example, the characteristic elements may be the length of the straight lines connecting the center points and / or end points of the facial parts, or the angle formed by two straight lines extracted from the straight lines. Similarly, the ECU 10 extracts characteristic elements of the face included in the second image.

[0067] As a comparison process, the ECU 10 compares a characteristic element extracted from the first image (hereinafter referred to as a "first characteristic element") with a characteristic element extracted from the second image (hereinafter referred to as a "second characteristic element"). If the degree of match between the first characteristic element and the second characteristic element is greater than a predetermined degree, the ECU 10 determines that the mobile terminal that transmitted the second image is the first mobile terminal 110A of the driver DR.

[0068] For example, assume that the first image is the first image 500 shown in Fig. 5, and the second image is the second image 600 shown in Fig. 6. The second image 600 is an image acquired by the first mobile terminal 110A. In this case, since the degree of match between the first characteristic element and the second characteristic element is greater than a predetermined degree, the ECU 10 determines that the mobile terminal that transmitted the second image 600 is the first mobile terminal 110A of the driver DR. After determining the first mobile terminal 110A of the driver DR in this way, the ECU 10 continues to acquire terminal side information (second image) from the first mobile terminal 110A and executes the wakefulness state determination process.

[0069] On the other hand, assume that the second mobile terminal 110B of the passenger in the back seat transmits a second image to the automatic driving device 100. In this case, the degree of match between the first characteristic element and the second characteristic element becomes equal to or lower than a predetermined degree, so that the ECU 10 determines that the mobile terminal that transmitted the second image is not the first mobile terminal 110A of the driver DR. After this determination, the ECU 10 does not acquire terminal side information (second image) from the second mobile terminal 110B.

[0070] (Example of operation) A specific process flow will be described with reference to Fig. 7. As in the above, an example in which a driver is present in the driver's seat and passengers are present in the rear seats will be used for the description.

[0071] The driver DR gets into the driver's seat of the vehicle VA, and a passenger gets into the back seat of the vehicle VA. Short-distance wireless communication between the ECU 10 and the first mobile terminal 110A is started. In response to this, the first mobile terminal 110A starts the terminal determination application 312 (step 701). Furthermore, short-distance wireless communication between the ECU 10 and the second mobile terminal 110B is started. In response to this, the second mobile terminal 110B starts the terminal determination application 312 (step 701).

[0072] The first portable terminal 110A transmits the terminal side information (second image) to the ECU 10 (step 702). Furthermore, the second portable terminal 110B transmits the terminal side information (second image) to the ECU 10 (step 703).

[0073] The ECU 10 executes the terminal determination process as described above (step 704). Specifically, the ECU 10 acquires a first image from the driver monitor camera 75. The ECU 10 executes a comparison process to compare a first characteristic element extracted from the first image with a second characteristic element extracted from the second image of the first portable terminal 110A. Furthermore, the ECU 10 executes a comparison process to compare a first characteristic element extracted from the first image with a second characteristic element extracted from the second image of the second portable terminal 110B. As a result, the ECU 10 determines that the first portable terminal 110A is the portable terminal of the driver DR. On the other hand, the ECU 10 determines that the second portable terminal 110B is not the portable terminal of the driver DR.

[0074] After that, the driver DR drives the vehicle VA. The vehicle VA is traveling on a motorway at a vehicle speed SPD equal to or less than the speed threshold SPDth. In this situation, the driver DR presses the automatic driving switch 81. The ECU 10 determines that the automatic driving execution condition is satisfied, and starts automatic driving control (level 3) (step 705). The ECU 10 transmits a start instruction signal only to the first mobile terminal 110A (step 706). The start instruction signal is a signal that instructs the start of the driver monitoring application 311. Since the second mobile terminal 110B is not the mobile terminal of the driver DR, the ECU 10 does not transmit the start instruction signal to the second mobile terminal 110B. In response to the start instruction signal, the first mobile terminal 110A starts the driver monitoring application 311 (step 707).

[0075] The first mobile terminal 110A transmits the terminal side information (second image) to the ECU 10 every time a predetermined time dT elapses (step 708). The ECU 10 executes an awake state determination process using the terminal side information (second image) every time a predetermined time dT elapses (step 709). The processes of steps 708 and 709 are repeatedly executed until the automatic driving control ends.

[0076] After that, the above-mentioned automatic driving end condition is satisfied. The ECU 10 ends the automatic driving control (step 710). Then, the ECU 10 transmits an end instruction signal to the first mobile terminal 110A (step 711). The end instruction signal is a signal that instructs the driver monitoring application 311 to end. In response to the end instruction signal, the first mobile terminal 110A ends the driver monitoring application 311 (step 712).

[0077] The mobile terminal determination system having the above configuration can determine (identify) which mobile terminal is the first mobile terminal 110A of the driver DR. This allows the ECU 10 to provide various services (awake state determination processing and attention calling processing) to the driver DR.

[0078] The comparison process may be a process of comparing the behavior of the occupant recognized in the first image with the behavior of the occupant recognized in the second image. Here, the "occupant behavior" refers to the orientation and / or movement of the occupant's upper body. The "occupant behavior" includes the orientation and movement of the face, and the orientation and movement of facial parts. An example in which the "face orientation" is used as the occupant behavior will be described. When the orientation of the face of the occupant recognized in the first image matches the orientation of the face of the occupant recognized in the second image, the ECU 10 determines that the portable terminal that transmitted the second image is the first portable terminal 110A of the driver DR.

[0079] The comparison process may be a process of comparing a change in luminance in the first image with a change in luminance in the second image. In this configuration, an illumination unit 76 (e.g., an LED, see FIG. 4) is provided near the driver monitor camera 75 and facing the driver's seat. The illumination unit 76 is configured to emit light toward the driver's seat. When the ECU 10 starts the terminal determination process, it activates the illumination unit 76. This causes the face of the driver DR to be illuminated by the light. As the luminance in the first image increases, the luminance in the second image of the first mobile terminal 110A also increases. Then, both the luminance in the first image and the luminance in the second image of the first mobile terminal 110A fall within a predetermined luminance range. In consideration of this, when both the luminance in the first image and the luminance in the second image fall within a predetermined luminance range, the ECU 10 determines that the mobile terminal that transmitted the second image is the first mobile terminal 110A of the driver DR.

[0080] On the other hand, the passenger in the back seat carrying the second portable terminal 110B is not illuminated by light, so the luminance of the second image of the second portable terminal 110B is low. That is, although the luminance of the first image is within a predetermined luminance range, the luminance of the second image of the second portable terminal 110B is lower than the predetermined luminance range. In consideration of this, when the luminance of the second image is outside the predetermined luminance range, the ECU 10 determines that the portable terminal that transmitted the second image is not the first portable terminal 110A of the driver DR.

[0081] In the terminal determination process, the illumination unit 203 of the first portable terminal 110A and the illumination unit 203 of the second portable terminal 110B may be activated. In this case, the ECU 10 transmits instructions to the first portable terminal 110A and the second portable terminal 110B so that the light intensity of the illumination unit 203 of the first portable terminal 110A differs from the light intensity of the illumination unit 203 of the second portable terminal 110B. This allows the ECU 10 to determine which portable terminal is the first portable terminal 110A of the driver DR based on the difference between the change in luminance in the first image and the change in luminance in the second image.

[0082] Two or more of the above comparison processes may be combined as appropriate.

[0083] The comparison process may be a process of comparing the second image with a preset reference image. The reference image is the first image acquired by the driver monitor camera 75 at an arbitrary timing before the terminal determination process is executed.

[0084] A combination of a plurality of occupants who use the vehicle VA and an identifier for identifying each occupant's mobile device may be registered in the non-volatile memory 10e. In this case, the non-volatile memory 10e may further include information on the driving schedule of each occupant. The ECU 10 may determine which occupant is sitting in the driver's seat according to the driving schedule, and determine that the mobile device corresponding to that occupant is the driver's mobile device.

[0085] <Second embodiment> Next, a mobile terminal determination system according to the second embodiment will be described. In this embodiment, the terminal determination application 312 acquires information on the communication status between the first communication unit mounted on the vehicle VA and the communication unit 204 of the mobile terminal 110. The mobile terminal 110 transmits the information on the communication status to the ECU 10. The ECU 10 determines which mobile terminal is the first mobile terminal 110A of the driver DR based on the information on the communication status.

[0086] In this example, the first communication unit is a short-range communication unit 401 (see FIG. 4) installed near the driver's seat. The short-range communication unit 401 is incorporated in the steering wheel SW. The short-range communication unit 401 is a device capable of wireless communication with a mobile terminal by NFC. NFC is a wireless communication technology using a frequency band of 13.56 MHz, and its communication range is about 10 cm. It is considered that the first mobile terminal 110A of the driver DR is highly likely to be within the communication range of NFC. The mobile terminal 110 transmits information on the communication status between the short-range communication unit 401 and the communication unit 204 (i.e., information indicating whether wireless communication is established) to the ECU 10. The ECU 10 determines that the mobile terminal with which wireless communication is established with the short-range communication unit 401 is the first mobile terminal 110A of the driver DR.

[0087] On the other hand, the second portable terminal 110B carried by the passenger in the back seat is not within the communication distance range of NFC. The ECU 10 can determine that the portable terminal with which wireless communication has not been established with the short-range communication unit 401 is not the first portable terminal 110A of the driver DR.

[0088] The mobile terminal determination system having the above configuration can determine (identify) which mobile terminal is the first mobile terminal 110A of the driver DR based on information regarding the communication status between the first communication unit (short-range communication unit 401) and the mobile terminal 110.

[0089] The first communication unit may include a plurality of short-range communication units installed separately in the vehicle VA. In this case, the plurality of short-range communication units include at least a first short-range communication unit and a second short-range communication unit. The first short-range communication unit and the second short-range communication unit are devices capable of wireless communication with the mobile terminal 110 by Bluetooth (registered trademark). The first short-range communication unit is installed near the driver's seat, and the second short-range communication unit is installed in the back seat. The first short-range communication unit and the second short-range communication unit each measure the reception strength of a signal transmitted from the communication unit 204 of the mobile terminal 110. The first short-range communication unit and the second short-range communication unit each transmit information on the reception strength to the ECU 10. The ECU 10 determines which mobile terminal is the first mobile terminal 110A of the driver DR based on the difference in reception strength. For example, when the first short-range communication unit and the second short-range communication unit each receive a signal from the communication unit 204 of the first mobile terminal 110A, the reception strength of the signal received at the first short-range communication unit is large, and the reception strength of the signal received at the second short-range communication unit is small. On the other hand, when the first short-range communication unit and the second short-range communication unit each receive a signal from the communication unit 204 of the second mobile terminal 110B, the reception strength of the signal received at the first short-range communication unit is small, and the reception strength of the signal received at the second short-range communication unit is large. Based on the difference in reception strength as described above, the ECU 10 can determine which mobile terminal is the first mobile terminal 110A of the driver DR.

[0090] The present disclosure is not limited to the above-described embodiments, and various modifications can be made within the scope of the present disclosure.

[0091] Another ECU may be disposed between the ECU 10 and the mobile terminal 110. The other ECU may execute the terminal determination process and the awake state determination process.

[0092] The ECU 10 may transmit the first image to the first mobile terminal 110A and the second mobile terminal 110B. In this case, each of the first mobile terminal 110A and the second mobile terminal 110B executes a terminal determination process. Each of the first mobile terminal 110A and the second mobile terminal 110B may transmit a determination result of the terminal determination process to the automatic driving device 100 (i.e., the ECU 10).

[0093] The terminal determination process may be applied to a service other than the wakefulness state determination process. Therefore, the timing of the terminal determination process is not limited to before the automatic driving control is executed, and may be before the start of the service to be executed. [Explanation of symbols]

[0094] 10...automatic driving ECU, 20...engine ECU, 30...brake ECU, 40...steering ECU, 50...meter ECU, 60...navigation ECU, 100...automatic driving device, 110...mobile terminal.

Claims

[Claim 1] A first imaging unit mounted in a vehicle and configured to capture an image of a driver of the vehicle; A portable terminal including a second imaging unit; a control device configured to execute a comparison process of comparing a first image acquired by the first imaging unit with a second image acquired by the second imaging unit, thereby determining whether the mobile terminal is a mobile terminal of the driver; and A mobile terminal determination system comprising: Further, a lighting unit that emits light toward a driver's seat of the vehicle is provided, The control device includes: Activating the lighting unit to illuminate the face of the driver; The comparison process is configured to compare a luminance in the first image with a luminance in the second image, and determine whether or not both the luminance in the first image and the luminance in the second image are within a predetermined luminance range, thereby determining whether or not the mobile terminal is a mobile terminal of the driver. Mobile terminal identification system.

Images