Monitoring device, computer program product and method

Abstract

The invention relates to a monitoring device, a computer program product and a monitoring method. The monitoring device is provided with: a determination unit that determines, on the basis of posture information indicating the posture of the driver acquired when the vehicle is driven by automatic control, whether the degree to which the driver prepares to start manual control is a first degree to which manual control of the vehicle can be performed, a second degree to which manual control of the vehicle can be performed, and a third degree to which manual control of the vehicle can be performed; the second degree is permissible when the vehicle is driven by the automatic control, and the third degree is not permissible when the vehicle is driven by the automatic control; and a notification control unit that, when the determination unit determines that the degree of preparation for starting the manual control is the third degree, issues a predetermined notification via a notification unit such that the driver increases the degree of preparation for starting the manual control. As a result, when it is determined that the degree to which the driver is ready to start the manual control is low, the driver is notified so as to increase the degree to which the driver is ready to start the manual control.

Description

Technical Field

[0001] The present disclosure relates to a monitoring device, a computer program product including a computer program for monitoring, and a monitoring method. Background Art

[0002] The automatic control system of the vehicle does not require the driver to always be in a state where the vehicle can be driven when the vehicle is driven by automatic control. When the vehicle is driven by automatic control, the driver can, for example, move the seat position to take a relaxed posture.

[0003] On the other hand, when the automatic control system determines that the vehicle cannot be driven safely by automatic control, it requests the driver to switch the driving of the vehicle from automatic control to manual control (TOR: Take Over Request).

[0004] For example, Patent Document 1 proposes a driver status detection device, which accumulates in advance the correspondence between the difference of the driver's physical information exceeding the standard range and the time until it returns to the standard range, estimates the recovery time until the acquired physical information returns to the standard range based on the correspondence, and if the estimated recovery time exceeds a predetermined allowable time, it is determined that the driver is in a state where he cannot drive the vehicle. The standard range represents the physical information when the driver is able to drive the vehicle.

[0005] Prior art literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2017-210013 Summary of the invention

[0007] Problems to be solved by the invention

[0008] The driver has to start manual control of the vehicle in accordance with the takeover request. The automatic control system performs a transition of control to hand over the control of the vehicle to the driver within a predetermined time (eg, 10 seconds) from the point in time when the takeover request is notified.

[0009] Therefore, depending on the driver's posture, when the automatic control system notifies the driver of a takeover request, the time required for the driver to return to a state where the vehicle can be manually controlled may exceed the time allowed for the transition.

[0010] On the other hand, from the perspective of allowing the driver to take a relaxed posture as much as possible, it is preferable to allow a posture within a range in which the driver can return to a state where the vehicle can be manually controlled within a predetermined time after the automatic control system notifies the driver of a takeover request.

[0011] As in Patent Document 1, when the driver's posture is not a posture capable of driving the vehicle, the driver is immediately required to change to a posture capable of driving the vehicle, which may prevent the driver from taking a relaxed posture.

[0012] That is, when the vehicle is driven by automatic control, although the driver is not required to always be in a state capable of driving the vehicle by manual control, the driver is required to be prepared to start manual control to the extent that he can start driving the vehicle by manual control within a predetermined time.

[0013] Therefore, the purpose of the present disclosure is to provide a monitoring device that does not notify the driver when the driver's posture-based readiness level for starting manual control is at a level that allows manual control of the vehicle or at a level that is permitted when the vehicle is driven by automatic control, but instead notifies the driver to increase the driver's readiness to start manual control when the above-mentioned readiness level is at a level that is not permitted when the vehicle is driven by automatic control.

[0014] Technical solutions to solve problems

[0015] (1) According to one embodiment, a monitoring device is provided. The monitoring device includes: a determination unit that determines whether the driver's degree of readiness to start manual control is at a first degree, a second degree, or a third degree, based on posture information indicating the driver's posture obtained when a vehicle is driven by automatic control, the first degree being a degree at which manual control of the vehicle is possible, the second degree being a degree that is permitted when the vehicle is driven by automatic control, and the third degree being a degree that is not permitted when the vehicle is driven by automatic control; and a notification control unit that, when the determination unit determines that the degree of readiness to start manual control is at the third degree, issues a predetermined notification via a notification unit so as to cause the driver to increase the degree of readiness to start manual control.

[0016] (2) In the monitoring device of (1), preferably, the determination unit estimates the transition time based on the posture information, the transition time being the time required for the driver to become able to start driving the vehicle in the manual driving mode after the notification unit notifies the driver of the transition request, the transition request being a request to switch from the automatic driving mode of driving the vehicle by automatic control to the manual driving mode of driving the vehicle by manual control, and the determination unit determines that the driver is ready to start manual control to a first degree when the transition time is zero, determines that the driver is ready to start manual control to a second degree when the transition time does not exceed a predetermined reference time, and determines that the driver is ready to start manual control to a third degree when the transition time exceeds the predetermined reference time.

[0017] (3) In the monitoring device of (2), preferably, the predetermined reference time is determined based on a time required for the driver to start driving the vehicle in the manual driving mode after the driver is notified of the transition request.

[0018] (4) In the monitoring device of (2), preferably, the posture information includes seat position information indicating the position of a driver's seat in which the driver is seated, and the determination unit estimates the transition time based on the seat position information.

[0019] (5) In the monitoring device of (1), preferably, the posture information includes an image showing the driver, and the determination unit determines that the degree of readiness to start manual control is the third degree when the driver shown in the image is in a predetermined posture.

[0020] (6) In the monitoring device of (5), preferably, the predetermined posture includes a posture of crossing arms and a posture of crossing legs.

[0021] (7) In the monitoring device of any one of (1) to (6), it is preferred that the device comprises a second determination unit for determining whether the driver's level of readiness to start manual control is low based on a cognitive state indicating the driver's level of involvement in driving, and a notification control unit for issuing a predetermined notification to the driver when the determination unit determines that the driver's level of readiness to start manual control is at the third level based on the posture information and the second determination unit determines that the driver's level of readiness to start manual control is low based on the cognitive state.

[0022] (8) According to another embodiment, a non-transitory storage medium storing a monitoring computer program is provided. The monitoring computer program causes a processor to perform the following processing: based on posture information indicating the posture of the driver obtained when the vehicle is driven by automatic control, determine whether the driver's degree of readiness to start manual control is at a first degree, a second degree, or a third degree, wherein the first degree is a degree at which manual control of the vehicle is possible, the second degree is a degree that is permitted when the vehicle is driven by automatic control, and the third degree is a degree that is not permitted when the vehicle is driven by automatic control; and when it is determined that the degree of readiness to start manual control is at the third degree, issue a predetermined notification via a notification unit so that the driver increases the degree of readiness to start manual control.

[0023] (9) According to another embodiment, a monitoring method is provided. In the monitoring method, a monitoring device performs: based on posture information indicating the posture of the driver obtained when the vehicle is driven by automatic control, determining whether the driver's degree of readiness to start manual control is at a first degree, a second degree, or a third degree, wherein the first degree is a degree at which manual control of the vehicle can be performed, the second degree is a degree that is permitted when the vehicle is driven by automatic control, and the third degree is a degree that is not permitted when the vehicle is driven by automatic control; and when it is determined that the degree of readiness to start manual control is at the third degree, issuing a predetermined notification via a notification unit so that the driver increases the degree of readiness to start manual control.

[0024] Effects of the Invention

[0025] The monitoring device involved in the present disclosure does not notify the driver when the driver's posture-based preparation level for starting manual control is a level that allows manual control of the vehicle, or a level that is permitted when the vehicle is driven by automatic control. However, when it is determined that the above-mentioned preparation level is a level that is not permitted when the vehicle is driven by automatic control, the driver is notified to increase the driver's preparation level for starting manual control. Therefore, the driver can maintain a state of relaxation while being able to return to a state where manual control is possible within a predetermined time. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 (A) to (C) illustrate an overview of the operation of the monitoring device of the present embodiment, where (A) is a diagram showing a vehicle, (B) is a diagram showing a driver's seat in a manual driving mode, and (C) is a diagram showing a driver's seat in an automatic driving mode.

[0027] Figure 2 This is a hardware configuration diagram of a vehicle on which the monitoring device according to the present embodiment is mounted.

[0028] Figure 3 This is an example of an operation flow chart related to the monitoring process of the monitoring device according to this embodiment.

[0029] Figure 4 This is an example of an operation flow chart related to the determination processing of the monitoring device according to this embodiment.

[0030] Figure 5 This is an example of another operation flowchart related to the determination processing of the monitoring device according to this embodiment.

[0031] Figure 6 This is an example of an operation flowchart related to a modification of the monitoring process of the monitoring device according to the present embodiment.

[0032] Figure 7This is an example of an operation flow chart related to the estimation process of the monitoring device according to this embodiment.

[0033] Description of symbols

[0034] 1 monitoring system; 2 monitoring camera; 3 seat operating unit; 4 seat driving unit; 5 seat position detection sensor; 6 user interface (UI); 6a display device; 10 vehicle; 11 automatic control device; 12 monitoring device; 21 communication interface (I / F); 22 memory; 23 processor; 231 control unit; 232 determination unit; 14 in-vehicle network; 30 vehicle interior (cockpit); 31 driver's seat; 31a seat back; 31b seat cushion; 31c headrest; 32 steering wheel; 33 brake pedal; 34 accelerator pedal. DETAILED DESCRIPTION

[0035] Figure 1 (A) to (C) of FIG. 1 describe the outline of the operation of the monitoring device according to the present embodiment. Figure 1 (A) is a diagram showing a vehicle 10, Figure 1 (B) is a diagram showing the driver's seat 31 in the manual driving mode. Figure 1 (C) is a diagram showing the driver's seat 31 in the automatic driving mode.

[0036] like Figure 1 As shown in (A), the vehicle 10 has an automatic control device 11 and a monitoring device 12. The automatic control device 11 has an automatic driving mode (for example, a driving mode of levels L3 to 5) in which the automatic control device 11 is used as the main body to drive the vehicle 10, and a manual driving mode (for example, a driving mode of levels L0 to 2) in which the driver 40 is used as the main body to drive the vehicle 10.

[0037] In the manual driving mode, the monitoring device 12 moves the seat position of the driver's seat 31 according to the operation of the seat operating unit 3 by the driver 40. The monitoring device 12 stores the seat position of the driver's seat 31 set by the driver 40 in the manual driving mode (hereinafter also referred to as the manual driving position).

[0038] like Figure 1 As shown in FIG. 1B , in the manual driving mode, the automatic control device 11 controls the operation of the vehicle 10 based on the operation of the driver 40 . The driver 40 sits in the driver's seat 31 in the vehicle cabin 30 so as to be able to operate a steering wheel 32 , a brake pedal 33 , and an accelerator pedal 34 .

[0039] In the automatic driving mode, the driver 40 is not required to be in a state of being able to drive the vehicle 10 all the time. In the automatic driving mode, the monitoring device 12 moves the seat position of the driver's seat 31 according to the operation of the seat operating unit 3 by the driver 40. The monitoring device 12 stores the seat position of the driver's seat 31. Figure 1 As shown in (C), the driver 40 can move the seat position of the driver's seat 31 to take a relaxed posture.

[0040] In the automatic driving mode, when the automatic control device 11 determines that the vehicle 10 cannot be driven safely by automatic control, it notifies the driver 40 of a control change request to change the driving subject of the vehicle 10 from the automatic control device 11 to the driver 40 via the user interface (UI) 6. The automatic control device 11 performs the control change so as to hand over the control of the vehicle 10 to the driver within the control change time from the time point when the control change request is notified to the driver 40. The control change time is, for example, 10 seconds.

[0041] The monitoring device 12 moves the driver's seat 31 to the manual driving position when notifying the driver 40 of the control change request. As a result, the driver 40 starts driving the vehicle 10 in the manual driving mode after performing a predetermined operation to approve the control change request. The control change time includes the moving time for moving the driver's seat 31 from the current seat position to the manual driving position and the approval time required for the driver 40 to perform the operation to approve the control change request.

[0042] The approval time can be regarded as a substantially constant time, but the moving time required for the driver's seat 31 to move to the manual driving position varies depending on the seat position of the driver's seat 31 in the automatic driving mode. The moving time for moving the driver's seat 31 from the current seat position to the manual driving position corresponds to the preparation transition time required for the driver 40 to be in a state where he can start driving the vehicle 10 in the manual driving mode.

[0043] Therefore, in the automatic driving mode, the monitoring device 12 determines whether the driver 40 is low in readiness to start manual control based on the posture information indicating the posture of the driver 40 .

[0044] The seat position of the driver's seat 31 leaving the manual driving position means that the driver's 40 hands are off the steering wheel 32 and / or the driver's 40 feet are off the brake pedal 33 and the accelerator pedal 34. In these postures, it can be said that the driver's 40 is not ready to start manual control. The seat position of the driver's seat 31 is an example of posture information.

[0045] The monitoring device 12 determines the degree to which the driver 40 is ready to start manual control based on the seat position of the driver's seat 31. First, the monitoring device 12 obtains the preparation transition time required for the driver 40 to become able to start driving the vehicle 10 in the manual driving mode based on the seat position of the driver's seat 31.

[0046] Specifically, the monitoring device 12 determines whether the driver 40 is at a first level enabling manual control of the vehicle 10, a second level allowing the vehicle to be driven under automatic control, or a third level not allowing the vehicle to be driven under automatic control based on the preparation transition time.

[0047] When the preparation transition time is zero, the monitoring device 12 determines that the degree of readiness to start manual control is high. The driver's seat 31 being in the manual driving position is an example in which the degree of readiness to start manual control by the driver 40 is a first degree in which manual control of the vehicle 10 is possible.

[0048] When the preparation transition time exceeds the predetermined reference time, the monitoring device 12 determines that the driver 40 is less prepared to start manual control. The position of the driver's seat 31 where the preparation transition time exceeds the reference time is an example of the driver 40 being prepared to start manual control at a third level that is not permitted when the vehicle 10 is driven by automatic control.

[0049] When the preparation transition time is not zero but does not exceed the predetermined reference time, the monitoring device 12 determines that the driver 40 is highly prepared to start manual control. The driver's seat 31 being in a position where the preparation transition time does not exceed the reference time is an example of the driver's 40 being prepared to start manual control to the second degree allowed when the vehicle 10 is driven by automatic control. In addition, when the preparation transition time is zero, the monitoring device 12 determines that the driver 40 is highly prepared to start manual control. The preparation transition time of zero means that the driver's seat 31 is in the manual driving position. The driver's seat 31 being in the manual driving position is an example of the driver's 40 being prepared to start manual control to the first degree that allows manual control of the vehicle 10.

[0050] When the driver 40 is not ready to start manual control, the monitoring device 12 issues a driving readiness notification via the UI 6 until the driver 40 assumes a posture that is permitted when the vehicle 10 is driven under automatic control.

[0051] When receiving the driving preparation notification, the driver 40 operates the seat operating unit 3 to bring the seat position of the driver's seat 31 closer to the manual driving position until the driving preparation notification is no longer transmitted.

[0052] Thus, in the automatic driving mode, the seat position of the driver's seat 31 is set within a range where the moving time of the driver's seat 31 to the manual driving position does not exceed the reference time. The driver's seat 31 moves to the manual driving position within the control transition time from notifying the driver 40 of the control transition request, and after the driver 40 performs an approval operation, the driver 40 can start driving the vehicle 10 in the manual driving mode.

[0053] Furthermore, in the automatic driving mode, the seat position of the driver's seat 31 can be set to a position where the driver 40 can take a relaxed posture within the range of postures allowed for the driver 40 when the vehicle 10 is driven by automatic control.

[0054] As described in detail above, when the monitoring device 12 of the present embodiment determines that the driver 40 is not sufficiently prepared to start manual control, the monitoring device 12 notifies the driver 40 to increase the degree of preparation to start manual control, thereby enabling the driver 30 to maintain a state in which it can return to a manually controllable state within a predetermined time.

[0055] Figure 2 1 is a hardware configuration diagram of a vehicle 10 equipped with a monitoring system 1 including a monitoring device 12 of the present embodiment. The vehicle 10 includes monitoring cameras 2, 2a, a seat operating unit 3, a seat driving unit 4, a seat position detection sensor 5, a user interface (UI) 6, an automatic control device 11, and a monitoring device 12. The monitoring system 1 includes at least the monitoring camera 2, the seat position detection sensor 5, the automatic control device 11, and the monitoring device 12.

[0056] The monitoring cameras 2 and 2 a , the seat operating unit 3 , the seat driving unit 4 , the seat position detection sensor 5 , the UI 6 , the automatic control device 11 , and the monitoring device 12 are communicatively connected via an in-vehicle network 14 conforming to a standard such as a controller area network.

[0057] The monitoring cameras 2 and 2a are arranged in the vehicle cabin 30 so as to obtain monitoring images including the face of the driver 40 driving the vehicle 10. The monitoring camera 2 is arranged, for example, above the instrument panel. The monitoring camera 2a is arranged, for example, on the ceiling at the rear of the vehicle cabin 30.

[0058] The monitoring cameras 2 and 2a are examples of image acquisition units. The monitoring camera 2 acquires a monitoring image showing the periphery of the driver's seat 31 at a monitoring image acquisition time with a predetermined period, for example. The monitoring camera 2a acquires a monitoring image showing the interior of the vehicle cabin 30 at the monitoring image acquisition time in synchronization with the monitoring camera 2. The monitoring camera 2a can acquire a monitoring image of the driver's seat 31 when it is located at the rear.

[0059] The monitoring cameras 2 and 2a include a two-dimensional detector composed of an array of photoelectric conversion elements such as CCD or C-MOS that are sensitive to infrared rays, and a photographing optical system that forms an image of a region to be acquired on the two-dimensional detector.

[0060] Each time the monitoring cameras 2 and 2a acquire a monitoring image, they output the monitoring image and the image acquisition time to the monitoring device 12 via the in-vehicle network 14. The monitoring image is used in the monitoring device 12 to estimate the posture of the driver 40. The monitoring image is an example of posture information.

[0061] The driver's seat 31 has a seat back 31a, a seat cushion 31b and a headrest 31c. The seat back 31a can be displaced (changed position) discretely in 10 levels (divided into 10 stages) of inclination. The seat cushion 31b can be displaced discretely in 10 levels of position in the front-rear direction and in the up-down direction. The headrest 31c can be displaced discretely in 10 levels of inclination and in the up-down direction. In this specification, the seat position of the driver's seat 31 means including the inclination and / or position of each part of the driver's seat 31.

[0062] The seat operating unit 3 is operated by the driver 40, and outputs an operation signal indicating the position of the driver's seat 31 to the monitoring device 12. The monitoring device 12 generates a drive signal for displacing the seat position of the driver's seat 31 based on the operation signal, and outputs it to the seat drive unit 4. The seat drive unit 4 displaces the driver's seat 31 based on the drive signal. The seat operating unit 3 has switches for respectively operating the seat back 31a, the seat cushion 31b, and the headrest 31c. The monitoring device 12 stores the seat position of the driver's seat 31.

[0063] The seat drive unit 4 includes a drive device such as a motor for displacing the seat back 31a, the seat cushion 31b, and the headrest 31c. Figure 1 In (B) and the like, the seat drive unit 4 is disposed on the seat cushion 31b, but the seat drive unit 4 may be dispersedly disposed on each of the seat back 31a, the seat cushion 31b, and the headrest 31c. The seat drive unit 4 is controlled by the monitoring device 12 to vibrate the seat back 31a, the seat cushion 31b, and the headrest 31c.

[0064] The seat position detection sensor 5 detects the inclination or position of each part of the driver's seat 31. The seat position detection sensor 5 detects the inclination of the seat back 31a, and outputs seat position information indicating the inclination of the seat back 31a to the monitoring device 12. In addition, the seat position detection sensor 5 detects the position of the seat cushion 31b in the front-rear direction and the position in the up-down direction, and outputs seat position information indicating the position of the seat cushion 31b in the front-rear direction and the position in the up-down direction to the monitoring device 12. Furthermore, the seat position detection sensor 5 detects the inclination and the position in the up-down direction of the headrest 31c, and outputs seat position information indicating the inclination and the position in the up-down direction of the headrest 31c to the monitoring device 12. The seat position information is an example of posture information.

[0065] The seat position detection sensor 5 may include a sensor that detects the positions of the seat back 31a, the seat cushion 31b, and the headrest 31c mechanically, optically, or magnetically. Figure 1 In (B) and the like, the seat position detection sensor 5 is disposed on the seat cushion 31 b , but the seat position detection sensor 5 may be disposed in a dispersed manner on each of the seat cushion 31 b and the headrest 31 c .

[0066] UI6 is an example of a notification unit. UI6 is controlled by the automatic control device 11 or the monitoring device 12, and notifies the driver 40 of information related to the vehicle 10, such as a control change request. UI6 has a display device 6a such as a liquid crystal display or a touch panel to display the control change request, etc. In addition, UI6 may also have a sound output device (not shown) for notifying the driver 40 of the control change request, etc. In addition, UI6 has, for example, a touch panel or an operation button as an input device for the driver 40 to input operation information to the vehicle 10. UI6 outputs the input operation information to the automatic control device 11, etc. via the in-vehicle network 14.

[0067] The automatic control device 11 controls the movement of the vehicle 10. The automatic control device 11 has an automatic driving mode for driving the vehicle 10 by automatic driving and a manual driving mode for controlling the movement of the vehicle 10 based on the operation of the driver 40. In the automatic driving mode, the automatic control device 11 becomes the main body to drive the vehicle 10. In the automatic driving mode, the automatic control device 11 controls the movement such as steering, driving and braking based on the detection information of the sensor (not shown) mounted on the vehicle 10. When the automatic control device 11 starts driving in the automatic driving mode or the manual driving mode, it outputs information indicating the start of driving in the automatic driving mode or the manual driving mode to the monitoring device 12. In addition, when the automatic control device 11 ends driving in the automatic driving mode or the manual driving mode, it outputs information indicating the end of driving in the automatic driving mode or the manual driving mode to the monitoring device 12.

[0068] In addition, in the manual driving mode, the automatic control device 11 controls the movement of the vehicle 10 such as steering, driving, and braking based on the operation of the driver 40. In the manual driving mode, the driver 40 becomes the main body to drive the vehicle 10. In the manual driving mode, the automatic control device 11 controls the movement of the vehicle 10 based on the operation of at least one of the steering wheel 32, the brake pedal 33, and the accelerator pedal 34 by the driver 40.

[0069] The automatic control device 11 controls the steering, driving and braking actions without the participation of the driver. The driver 40 controls the steering, driving and braking actions.

[0070] The automatic control device 11 may also have multiple automatic driving levels in the automatic driving mode. In this specification, the multiple automatic driving levels may also include so-called automatic driving levels of level L3 to level 5. In this specification, the manual driving mode may also include so-called automatic driving levels of level L0 to level 2.

[0071] The automatic control device 11 detects objects such as other vehicles around the vehicle 10 based on the detection information of the sensor (not shown) mounted on the vehicle 10. In the automatic driving mode, the automatic control device 11 notifies the driver 40 via UI6 of a control change request to change the driving subject of the vehicle 10 from the automatic control device 11 to the driver 40 when the interval of more than a predetermined distance cannot be maintained between the vehicle 10 and other objects. In addition, in the automatic driving mode, before entering an area where driving under automatic control is not allowed, the automatic control device 11 notifies the driver 40 via UI6 of a control change request to change the driving subject from the automatic control device 11 to the driver 40. As an area where driving under automatic control is not allowed, an area where high-precision maps are not provided can be cited. Furthermore, in the automatic driving mode, the automatic control device 11 also notifies the driver 40 of a control change request via UI6 when it is determined that the driving of the vehicle 10 cannot be safely controlled due to an abnormality of a sensor or the like. The automatic control device 11 also outputs a control change request to the monitoring device 12 via the in-vehicle network 14.

[0072] The automatic control device 11 performs a control transition so as to hand over the control of the vehicle 10 to the driver within a predetermined control transition time from the time when the control transition request is notified to the driver 40. If the driver 40 cannot confirm the approval action of the control transition request within the control transition time, the automatic control device 11 stops the vehicle 10 safely (Minimal Risk Maneuver processing).

[0073] The automatic control device 11 changes from the automatic driving mode to the manual driving mode when, for example, it recognizes any of the following three driver approval actions: (1) the driver 40 holds the steering wheel 32 and operates the accelerator pedal 34; (2) the driver 40 holds the steering wheel 32 and operates the brake pedal 33; (3) the driver 40 holds the steering wheel 32 and operates the steering wheel 32.

[0074] When a control transition request is input from the automatic control device 11 , the monitoring device 12 moves the driver's seat 31 to the manual driving position.

[0075] The monitoring device 12 performs control processing and determination processing. To this end, the monitoring device 12 has a communication interface (I / F) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the monitoring device 12 to the in-vehicle network 14.

[0076] The memory 22 is an example of a storage unit, and includes, for example, a volatile semiconductor memory and a nonvolatile semiconductor memory. The memory 22 stores computer programs of applications used in information processing executed by the processor 23 and various data.

[0077] All or part of the functions of the monitoring device 12 are, for example, functional modules implemented by a computer program running on the processor 23. The processor 23 has a control unit 231 and a determination unit 232. Alternatively, the functional modules of the processor 23 may be dedicated arithmetic circuits provided in the processor 23. The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may also have other arithmetic circuits such as a logic operation unit, a numerical operation unit, or a graphics processing unit.

[0078] The control unit 231 moves the seat position of the driver's seat 31 according to the operation of the seat operating unit 3 by the driver 40. The control unit 231 stores the seat position of the driver's seat 31 set by the driver 40 in the manual driving mode as the manual driving position. The manual driving position can also be set to a seat position including a predetermined range. The details of the operation of the monitoring device 12 will be described later.

[0079] The automatic control device 11 and the monitoring device 12 are, for example, electronic control units (ECUs). Figure 3In the description, the automatic control device 11 and the monitoring device 12 are described as separate devices (for example, Electronic Control Unit: ECU), but all or part of these devices may be configured as one device.

[0080] Figure 3 FIG. 1 is an example of an operation flow chart related to monitoring processing of the monitoring device 12 of this embodiment. When the automatic control device 11 is in the automatic driving mode, the monitoring device 12 performs monitoring at a predetermined period according to the following steps: Figure 3 The monitoring process is performed according to the operation flow chart shown in the figure. The monitoring time period can be set to, for example, 10 to 60 seconds.

[0081] First, the control unit 231 obtains posture information, which is information indicating the posture of the driver 40 obtained when the vehicle 10 is driven by automatic control (step S101). The control unit 231 obtains seat position information as posture information via the in-vehicle network 14. In addition, the control unit 231 obtains a monitoring image as posture information via the in-vehicle network 14.

[0082] Next, the determination unit 232 determines whether the driver 40 is not ready to start manual control based on the posture information (step S102). The determination process of the determination unit 232 determining whether the driver 40 is not ready to start manual control will be described later.

[0083] When it is determined that the readiness to start manual control is low (step S102: Yes), the control unit 231 issues a driving readiness notification via the UI 6 (step S103), and ends a series of processing. The control unit 231 is an example of a notification control unit. By performing monitoring processing at a predetermined period, when the vehicle 10 is driven by automatic control, a driving readiness notification is issued to the driver 40 until the driver 40 reaches an allowed posture.

[0084] When it is determined based on the seat position information that the degree of readiness for starting manual control is low, the control unit 231 may notify the driver 40 to move the seat position of the driver's seat 31 forward through the UI 6 by display or voice.

[0085] Furthermore, when it is determined based on the monitoring image that the degree of readiness to start manual control is low, the control unit 231 may notify the driver 40 through the UI 6 by display or voice so that the driver 40 can concentrate on driving the vehicle 10 .

[0086] In addition, the control unit 231 can also issue a driving preparation notification to the driver 40 by changing the temperature of the air-conditioning device, changing the intensity of the air supply of the air-conditioning device, changing the air supply direction of the air-conditioning device, changing the intensity of the lighting in the vehicle interior 30, changing the color of the lighting in the vehicle interior 30, and vibrating the driver's seat 31.

[0087] The control unit 231 continues to issue a driving preparation notification to the driver 40 until the driver 40 is no longer judged to be low in the degree of preparation for starting manual control. The driver 40 can improve the degree of preparation for starting manual control by changing the seat position of the driver's seat 31 or improving the posture to a state where the driver can drive.

[0088] In addition, the control unit 231 may also input a schedule for notifying the driver 40 of the control change request from the automatic control device 11. In the case where the schedule for notifying the driver 40 of the control change request is within 10 minutes, the control unit 231 may also issue the driving preparation notification to the driver 40 more emphatically than usual. For example, the control unit 231 may also issue the driving preparation notification to the driver 40 together with an alarm sound. In addition, in the case where the driving preparation notification is issued using the air conditioning device, the control unit 231 may also make the intensity of the air supply stronger than usual. In addition, in the case where the driving preparation notification is issued using the lighting in the vehicle interior 30, the control unit 231 may also make the intensity of the lighting stronger than usual.

[0089] On the other hand, when it is not determined that the readiness level for starting manual control is low (step S102 : No), a series of processing is terminated.

[0090] Figure 4 FIG. 2 is an example of an operation flow chart related to the determination process of the monitoring device 12 of this embodiment. The determination unit 232 performs determination process based on the seat position information. The determination unit 232 performs determination process at a first determination time having a predetermined period according to Figure 4 The determination process is executed according to the operation flow chart shown in the figure. The cycle of the first determination time can be set to, for example, 1 to 10 seconds.

[0091] First, the determination unit 232 estimates a preparation transition time based on the seat position information (step S201). The preparation transition time is a time required to move the current seat position of the driver's seat 31 to the manual driving position.

[0092] First, the determination unit 232 estimates the moving time required to move the front-rear direction position of the seat cushion 31b to the manual driving position based on the seat position information indicating the current front-rear direction position of the seat cushion 31b.

[0093] Specifically, the determination unit 232 estimates the time required to move the front-to-back position of the seat cushion 31b to the manual driving position based on the weight (mass) of the driver 40, the mass of the driver's seat 31, the force of the seat drive unit 4 driving the seat cushion 31b, the current seat position information and the manual driving position.

[0094] The weight (mass) of the driver 40 can be input via the UI 6. Alternatively, a standard weight pre-stored in the memory 22 may be used as the weight (mass) of the driver 40.

[0095] Similarly, the determination unit 232 estimates the moving time required to move the vertical position of the seat cushion 31b to the manual driving position based on the seat position information indicating the current vertical position of the seat cushion 31b.

[0096] In addition, the determination unit 232 similarly estimates the movement time required to move the inclination of the seat back 31a to the manual driving position based on the seat position information indicating the current inclination of the seat back 31a. When estimating the movement time of the seat back 31a, the ratio of the weight of the driver 40 applied to the seat back 31a can be used.

[0097] In addition, the determination unit 232 similarly estimates the moving time required to move the inclination and the vertical position of the headrest 31c to the manual driving position based on the seat position information indicating the current inclination and the vertical position of the headrest 31c. When estimating the moving time of the headrest 31c, the ratio of the weight of the driver 40 applied to the headrest 31c can be used.

[0098] The determination unit 232 uses the maximum moving time among the moving times for moving the inclination or position of each part of the driver's seat 31 to the manual driving position as the current preparation transition time of the driver's seat 31 .

[0099] The fact that the preparatory transition time of the driver's seat 31 is long means that the seat cushion 31 b is located rearward, the inclination of the seat back 31 a is large, or the inclination of the headrest 31 c is large.

[0100] The rearward positioning of the seat cushion 31 b , the large inclination of the seat back 31 a , or the large inclination of the headrest 31 c means that the driver's 40 hands are off the steering wheel 32 , or the driver's 40 feet are off the brake pedal 33 and the accelerator pedal 34 .

[0101] On the other hand, it is estimated that the driver 40 is in a relaxed posture because the seat cushion 31 b is located rearward, the seat back 31 a has a large inclination, or the headrest 31 c has a large inclination.

[0102] Next, the determination unit 232 determines whether the preparation transition time exceeds a predetermined reference time (step S202). The reference time is preferably determined based on the control transition time, which is the time required for the driver 40 to start driving the vehicle 10 in the manual driving mode after notifying the driver 40 of the control transition request. Specifically, the reference time is the time obtained by subtracting the approval time required for the driver 40 to perform an operation to approve the control transition request from the control transition time. For example, when the control transition time is set to 10 seconds and the approval time is set to 2 seconds, the reference time is 8 seconds.

[0103] When the preparation transition time exceeds the reference time (step S202: Yes), the determination unit 232 determines that the driver 40 is less prepared to start manual control (step S203), and ends a series of processing. The driver's seat 31 is located at a position where the preparation transition time exceeds the reference time, which is an example of the driver 40 being prepared to start manual control at a third level that is not permitted when the vehicle 10 is driven by automatic control.

[0104] On the other hand, when the preparation transition time is zero or the preparation transition time does not exceed the reference time (step S202: No), the determination unit 232 determines that the driver 40 is not less prepared to start manual control (step S204), and ends a series of processing. The preparation transition time being zero means that the driver's seat 31 is located in the manual driving position. The driver's seat 31 being located in the manual driving position is an example of the driver 40 being prepared to start manual control to the first degree that allows manual control of the vehicle 10. In addition, the driver's seat 31 being located in a position where the preparation transition time does not exceed the reference time is an example of the driver 40 being prepared to start manual control to the second degree that is allowed when the vehicle 10 is driven by automatic control. That is, the determination unit 232 determines whether the driver 40 is prepared to start manual control to the first degree, the second degree, or the third degree.

[0105] Figure 5 This is another example of an operation flow chart related to the determination process of the monitoring device 12 of this embodiment. The determination unit 232 performs determination processing based on the monitoring image as posture information. The determination unit 232 performs determination processing at a second determination time having a predetermined period according to Figure 5 The determination process is executed according to the operation flow chart shown in the figure. The cycle of the second determination time may be set to, for example, 5 to 10 minutes.

[0106] First, the determination unit 232 estimates the posture of the driver 40 based on the monitoring image (step S301). The determination unit 232 estimates the posture of the driver 40 based on the monitoring image acquired by the monitoring cameras 2 and 2a.

[0107] The determination unit 232 has a recognizer that has been learned (trained) to recognize the posture of the driver 40. The determination unit 232 infers the posture of the driver 40 shown in the monitoring image by inputting the monitoring image into the recognizer. Preferably, the recognizer has been learned to recognize a relaxed posture such as a state of crossing arms and a state of crossing legs. In addition, preferably, the recognizer recognizes the posture of the driver 40 when operating a smartphone or a portable computer terminal.

[0108] Next, the determination unit 232 determines whether the driver 40 has taken a predetermined posture (step S302). The predetermined posture includes a relaxed posture such as a posture with arms crossed and a posture with legs crossed. In addition, the predetermined posture includes a posture of the driver 40 operating a smartphone or a portable computer terminal. These postures of the driver 40 are examples of the third degree that the driver 40 is ready to start manual control when the vehicle 10 is driven by automatic control.

[0109] When the driver 40 is in the predetermined posture (step S302: Yes), the determination unit 232 determines whether a predetermined reference time has passed (step S303). The reference time may be, for example, 5 minutes to 10 minutes. When the driver 40 is in the predetermined posture during the reference time, it is considered that the driver 40 is not in a state of being able to recognize the notified control change request, judge the situation, and immediately perform the operation of approving the control change request.

[0110] If the reference time has not passed (step S303: No), the process returns to before step S301. On the other hand, if the reference time has passed (step S303: Yes), the determination unit 232 determines that the driver 40 is not ready to start manual control (step S304), and ends a series of processes. This is because if the driver 40 is in a predetermined posture for a long time, the driver's 40 awareness level of driving decreases.

[0111] In addition, when the driver 40 is not in the predetermined posture (step S302: No), the determination unit 232 determines that the driver 40 is not ready to start manual control (step S305), and ends a series of processing. For example, when the driver 40 is in a posture without crossing his arms or legs, the determination unit 232 determines that the driver 40 is not ready to start manual control. The postures of the driver 40 are examples of the second degree of readiness of the driver 40 to start manual control that is allowed when the vehicle 10 is driven by automatic control.

[0112] In addition, when the driver 40 is in a posture of operating the steering wheel 32, or in a posture of operating the brake pedal 33 or the accelerator pedal 34, the determination unit 232 determines that the degree of readiness of the driver 40 to start manual control is not low (high). These postures of the driver 40 are examples of the degree of readiness of the driver 40 to start manual control being the first degree at which manual control of the vehicle 10 can be performed. That is, the determination unit 232 determines whether the degree of readiness of the driver 40 to start manual control is the first degree, the second degree, or the third degree.

[0113] In step S102, the determination unit 232 performs Figure 4 The determination process shown and Figure 5 In the determination processing shown, when it is determined in any of the processes that the driver 40 is prepared to start manual control at a low level, the process proceeds to step S103 .

[0114] As described in detail above, the monitoring device of this embodiment notifies the driver to improve the degree of preparation for starting manual control when it is determined that the driver is not ready to start manual control. As a result, the driver can maintain a state in which the manual control state can be returned within a predetermined time. The driver 40 can take a relaxed state within the range of postures allowed when the vehicle 10 is driven by automatic control.

[0115] Next, refer to Figure 6 A modification of the monitoring device according to the above-described present embodiment will be described. Figure 6 This is an example of an operation flowchart related to a modification of the monitoring process of the monitoring device according to the present embodiment.

[0116] In this variation, when it is determined based on the posture information of the driver 40 that the driver 40 is not ready to start manual control, and when it is determined based on the cognitive state indicating the degree to which the driver 40 is involved in driving that the driver 40 is not ready to start manual control, a driving preparation notification is issued to the driver 40.

[0117] exist Figure 6 In the example, step S403 and step S404 are newly added processes. Figure 6 In the process, step S401, step S402 and step S405 are the same as the above-mentioned steps S101 to S103.

[0118] When it is determined based on the posture information that the readiness to start manual control is low (step S402: Yes), the determination unit 232 estimates the cognitive state of the driver 40 (step S403). The estimation process of estimating the cognitive state of the driver 40 will be described later.

[0119] Next, the determination unit 232 determines whether the driver 40 is low in readiness to start manual control based on the cognitive state of the driver 40 (step S404). When the determination unit 232 estimates that the driver 40 is low in cognitive state, it determines that the driver 40 is low in readiness to start manual control.

[0120] For example, when the driver 40 feels sleepy and has a low degree of eye opening, it can be considered that the driver 40 is not at a level of consciousness capable of recognizing the notified control transition request, judging the situation, and immediately approving the control transition request.

[0121] On the other hand, when the determination unit 232 does not estimate that the cognitive state of the driver 40 is low, it does not determine that the driver 40 is low in terms of being ready to start manual control.

[0122] When the driver 40 is not ready to start manual control (step S404 : Yes), the control unit 231 issues a driving preparation notification to make the driver 40 increase the degree of readiness to start manual control (step S405 ), and ends a series of processing.

[0123] On the other hand, when the driver 40 is not ready to start manual control (step S404 : No), a series of processing ends.

[0124] Next, refer to Figure 7 The estimation process of the monitoring device 12 will be described. Figure 7 FIG. 1 is an example of an operation flow chart related to the estimation processing of the monitoring device 12 of this embodiment. The monitoring device 12 performs estimation processing according to the predetermined period at the estimation time. Figure 7 The operation flowchart shown executes the estimation process.

[0125] First, the determination unit 232 acquires monitoring information of the driver 40 (step S501). The monitoring information is a monitoring image acquired by the monitoring cameras 2 and 2a.

[0126] Next, the determination unit 232 estimates the cognitive state of the driver 40 based on the monitoring information (step S502), and ends a series of processing. The determination unit 232 obtains participation information indicating the degree of participation of the driver 40 in driving based on the monitoring image. The participation information includes the degree of eye opening, mouth opening, sight direction, frequency of sight directed to the rearview mirror (not shown), or blinking frequency of the driver 40. In addition, the determination unit 232 may also obtain any one of the degree of eye opening, mouth opening, sight direction, frequency of sight directed to the rearview mirror (not shown), or blinking frequency of the driver 40.

[0127] The degree of eye opening, mouth opening, sight direction, frequency of sight directed to the rearview mirror (not shown), and blinking frequency of the driver 40 are examples of participation information indicating the degree of participation of the driver 40 in driving. For example, when the driver 40 becomes more sleepy, the participation information changes. The determination unit 232 estimates the cognitive state indicating the degree of participation of the driver in driving based on the participation information.

[0128] The determination unit 232 estimates that the cognitive state of the driver 40 is low when the eye opening degree of the driver 40 continues to be lower than a predetermined reference eye opening degree for a predetermined time. The predetermined time may be, for example, 5 to 10 minutes.

[0129] The determination unit 232 estimates that the cognitive state of the driver 40 is low when the state in which the mouth opening degree of the driver 40 is higher than the predetermined reference mouth opening degree continues for a predetermined time. The predetermined time may be, for example, 5 to 10 minutes.

[0130] The determination unit 232 estimates that the cognitive state of the driver 40 is low when the sight line direction of the driver 40 is outside the predetermined range for a predetermined time. The predetermined range may be, for example, 30 degrees to the left and right and 30 degrees to the top and bottom relative to the traveling direction of the vehicle 10. The predetermined time may be, for example, 5 to 10 minutes.

[0131] The determination unit 232 infers that the cognitive state of the driver 40 is low when the frequency of looking at the rearview mirror (not shown) located in the vehicle cabin 30 is less than a predetermined reference frequency for a predetermined time. The predetermined reference frequency may be, for example, three times per minute. The predetermined time may be, for example, 5 to 10 minutes.

[0132] The determination unit 232 estimates that the cognitive state of the driver 40 is low when the blink frequency is less than a predetermined reference frequency for a predetermined time. The predetermined reference frequency may be, for example, 10 times per minute, and the predetermined time may be, for example, 5 to 10 minutes.

[0133] According to the above-described present modification, the monitoring device 12 can more accurately determine the degree to which the driver is ready to start manual control based on the posture information and the cognitive state.

[0134] In the present disclosure, the monitoring device, monitoring computer program, and monitoring method of the above-mentioned embodiments may be appropriately changed without departing from the purpose of the present disclosure. In addition, the technical scope of the present disclosure is not limited to these embodiments, but relates to the inventions described in the claims and their equivalents.

[0135] For example, in the above-mentioned embodiment, the posture information is obtained based on the seat position of the driver's seat and the monitoring image, but the posture information can also be obtained using other methods. In addition, the example of the driving preparation notification described in the above-mentioned embodiment is an example, and the driving preparation notification is not limited to this.

Claims

1. A monitoring device comprising: a determination unit that determines whether the driver's degree of readiness to start manual control is at a first degree, a second degree, or a third degree, based on posture information indicating the driver's posture acquired when the vehicle is driven by automatic control, the first degree being a degree at which manual control of the vehicle is possible, the second degree being a degree that is permitted when the vehicle is driven by automatic control, and the third degree being a degree that is not permitted when the vehicle is driven by automatic control; and The notification control unit issues a predetermined notification via the notification unit so as to cause the driver to increase the degree of readiness to start manual control when the determination unit determines that the degree of readiness to start manual control is the third degree.

2. The monitoring device according to claim 1, the determining unit estimates a transition time based on the posture information, the transition time being a time required for the driver to become capable of starting to drive the vehicle in the manual driving mode after notifying the driver of a transition request via the notifying unit, the transition request being a request for transition from an automatic driving mode in which the vehicle is driven by automatic control to a manual driving mode in which the vehicle is driven by manual control, The determination unit determines that the degree to which the driver is ready to start manual control is the first degree when the transition time is zero, determines that the degree to which the driver is ready to start manual control is the second degree when the transition time is determined to be no more than a predetermined reference time, and determines that the degree to which the driver is ready to start manual control is the third degree when the transition time is determined to be more than a predetermined reference time.

3. The monitoring device according to claim 2, The predetermined reference time is determined based on a time required for the driver to start driving the vehicle in the manual driving mode after the driver is notified of the transition request.

4. The monitoring device according to claim 2, The posture information includes seat position information indicating the position of a driver's seat where the driver is seated. The determination unit estimates the transition time based on the seat position information.

5. The monitoring device according to claim 1, The posture information includes an image showing the driver, The determination unit determines that the degree of readiness to start manual control is the third degree when the driver shown in the image is in a predetermined posture.

6. The monitoring device according to claim 5, The predetermined posture includes a posture of crossing arms and a posture of crossing legs.

7. The monitoring device according to any one of claims 1 to 6, A second determination unit is provided for determining whether the driver's level of readiness to start manual control is low based on a cognitive state indicating the driver's level of involvement in driving, The notification control unit issues the predetermined notification to the driver when the determination unit determines that the degree to which the driver is ready to start manual control based on the posture information is the third degree and the second determination unit determines that the degree to which the driver is ready to start manual control based on the cognitive state is low.

8. A computer program product, comprising a monitoring computer program, wherein the monitoring computer program causes a processor to execute: Based on posture information indicating the posture of the driver acquired when the vehicle is driven by automatic control, determining whether the driver is ready to start manual control at a first level, a second level, or a third level, the first level being a level at which manual control of the vehicle is possible, the second level being a level that is permitted when the vehicle is driven by automatic control, and the third level being a level that is not permitted when the vehicle is driven by automatic control; When it is determined that the degree of readiness for starting manual control is the third degree, a predetermined notification is issued via the notification unit so that the driver increases the degree of readiness for starting manual control.

9. A monitoring method, The monitoring device performs: determining, based on posture information indicating the posture of the driver obtained when the vehicle is driven by automatic control, whether the driver is ready to start manual control at a first level, a second level, or a third level, the first level being a level at which manual control of the vehicle is possible, the second level being a level that is permitted when the vehicle is driven by automatic control, and the third level being a level that is not permitted when the vehicle is driven by automatic control, When it is determined that the degree of readiness for starting manual control is the third degree, a predetermined notification is issued via the notification unit so that the driver increases the degree of readiness for starting manual control.

Images