Monitoring device, monitoring-purpose computer program, and monitoring method

Abstract

To provide a monitoring device that provides notification to a driver so as to increase his / her readiness to take manual control when the driver's readiness to take the manual control is determined to be low.SOLUTION: A monitoring device includes: a determination unit that determines whether a degree of readiness of a driver to initiate manual control is any one of a first degree capable of manual control of the vehicle, a second degree permitted when the vehicle is being driven under automatic control, or a third degree that is not permitted when the vehicle is being driven under automatic control based on attitude information representing a driver's attitude acquired while the vehicle was being driven under the automatic control; and a notification control unit that provides a predetermined notification so that the driver is more prepared to start the manual control via the notification control unit when the determination unit determines that the readiness to initiate the manual control is at the third degree.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a monitoring device, a monitoring computer program, and a monitoring method. [Background technology]

[0002] The automatic vehicle control system does not require the driver to always be ready to drive the vehicle when the vehicle is being driven under automatic control. When driving a vehicle under automatic control, the driver can, for example, move the seat position so that the driver can take a relaxed posture.

[0003] On the other hand, if the automatic control system determines that the vehicle cannot be driven safely under automatic control, it requests the driver to switch the driving of the vehicle from automatic control to manual control (TakeOverRequest: TOR request).

[0004] For example, Patent Document 1 proposes a driver condition detection device that accumulates a correspondence relationship between the deviation of a driver's physical information from a standard range and the time it takes to return to the standard range, estimates the recovery time from the acquired physical information to the standard range based on the correspondence relationship, and determines that the driver is in a state where he or she is unable to drive the vehicle if the estimated recovery time exceeds a predetermined allowable time. The standard range represents the physical information when the driver is able to drive the vehicle. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-210013 A Summary of the Invention [Problem to be solved by the invention]

[0006] In response to the TOR request, the driver must initiate manual control of the vehicle. The automatic control system executes the transfer of control so as to transfer control of the vehicle to the driver within a predetermined time (e.g., 10 seconds) from the time of notification of the TOR request.

[0007] Therefore, depending on the driver's posture, when the automatic control system notifies the driver of a TOR request, the time it takes for the driver to return to a state in which he or she can manually control the vehicle may exceed the time allowed for the transition.

[0008] On the other hand, from the viewpoint of allowing the driver to assume a relaxed posture as much as possible, it is preferable that a posture within a range that enables the driver to return to a state in which he or she can manually control the vehicle within a specified time period after the automatic control system notifies the driver of a TOR request is permitted.

[0009] As in Patent Document 1, if the driver is asked to immediately assume a position suitable for driving the vehicle once the driver is no longer in a position suitable for driving the vehicle, there is a risk that the driver will not be able to assume a relaxed position.

[0010] In other words, when driving a vehicle under automatic control, the driver is not required to always be in a state where the vehicle can be driven under manual control, but is required to be ready to start manual control to the extent that he or she can start driving the vehicle under manual control within a specified time.

[0011] Therefore, the present disclosure aims to provide a monitoring device that does not notify the driver when the degree of readiness to start manual control based on the driver's posture is either a degree that allows manual control of the vehicle or a degree that is acceptable when the vehicle is being driven under automatic control, but that notifies the driver to increase the degree of readiness to start manual control when the degree is a degree that is not acceptable when the vehicle is being driven under automatic control. [Means for solving the problem]

[0012] (1) According to one embodiment, a monitoring device is provided, which includes a determination unit that determines whether a degree of readiness of a driver to start manual control is a first degree at which manual control of the vehicle is possible, a second degree that is permissible when the vehicle is being driven under automatic control, or a third degree that is not permissible when the vehicle is being driven under automatic control, based on posture information indicating the posture of the driver acquired when the vehicle is being driven under automatic control, and a notification control unit that, when the determination unit determines that the degree of readiness to start manual control is the third degree, gives a predetermined notification via the notification unit to encourage the driver to increase the degree of readiness to start manual control.

[0013] (2) In the monitoring device of (1), it is preferable that the judgment unit estimates, based on the posture information, a transition time required for the driver to become able to start driving the vehicle in manual driving mode after a transition request requesting that the vehicle transition from an automatic driving mode, in which the vehicle is driven under automatic control, to a manual driving mode, in which the vehicle is driven under manual control, is notified to the driver via the notification unit, and that the judgment unit determines that the driver's degree of readiness to start manual control is a first degree if the transition time is zero, determines that the driver's degree of readiness to start manual control is a second degree if it is determined that the transition time does not exceed a predetermined reference time, and determines that the driver's degree of readiness to start manual control is a third degree if it is determined that the transition time exceeds the predetermined reference time.

[0014] (3) In the monitoring device of (2), it is preferable that the specified reference time is determined based on the time at which the driver is required to start driving the vehicle in manual driving mode after the transition request is notified to the driver.

[0015] (4) In the monitoring device of (2), it is preferable that the posture information includes seat position information that indicates a position of a driver's seat in which a driver sits, and the second determination unit estimates the transition time based on the seat position information.

[0016] (5) In the monitoring device of (1), it is preferable that the posture information includes an image showing the driver, and the judgment unit judges that the degree of readiness to start manual control is a third degree when the driver shown in the image is in a predetermined posture.

[0017] In the monitoring device of (6)(5), it is preferable that the predetermined postures include a posture with arms crossed and a posture with legs crossed.

[0018] (7) In any of the monitoring devices of (1) to (6), it is preferable to have a third judgment unit that judges whether the driver's level of readiness to start manual control is low or not based on the cognitive state indicating the driver's level of involvement in driving, and the notification control unit gives a predetermined notification to the driver when the judgment unit judges that the driver's level of readiness to start manual control is the third level based on the posture information and the third judgment unit judges that the driver's level of readiness to start manual control is low based on the cognitive state.

[0019] (8) According to another embodiment, there is provided a monitoring computer program, which causes a processor to execute a process including: determining whether the degree of readiness of the driver to start manual control is a first degree at which manual control of the vehicle is possible, a second degree that is permissible when the vehicle is being driven under automatic control, or a third degree that is not permissible when the vehicle is being driven under automatic control, based on posture information indicating the posture of the driver acquired when the vehicle is being driven under automatic control; and, if the degree of readiness to start manual control is determined to be the third degree, giving a predetermined notification via a notification unit to increase the degree of readiness of the driver to start manual control.

[0020] (9) According to yet another embodiment, there is provided a monitoring method, characterized in that the monitoring device determines whether the degree of readiness of the driver to start manual control is a first degree at which manual control of the vehicle is possible, a second degree that is permissible when the vehicle is being driven under automatic control, or a third degree that is not permissible when the vehicle is being driven under automatic control, based on posture information indicating the posture of the driver acquired when the vehicle is being driven under automatic control, and if the degree of readiness to start manual control is determined to be the third degree, the sound control device executes a predetermined notification via a notification unit to increase the degree of readiness of the driver to start manual control. Effect of the Invention

[0021] The monitoring device of the present disclosure does not notify the driver when the degree of readiness to begin manual control based on the driver's posture is either a degree that allows manual control of the vehicle or a degree that is acceptable when the vehicle is being driven under automatic control, but if it is determined that the degree is not acceptable when the vehicle is being driven under automatic control, a notification is given to the driver to increase the degree of readiness to begin manual control, thereby allowing the driver to remain relaxed and in a state in which manual control can be returned to within a specified time. [Brief description of the drawings]

[0022] [Figure 1] (A) to (C) explain an overview of the operation of the monitoring device of this embodiment, where (A) is a diagram showing a vehicle, (B) is a diagram showing the driver's seat in manual driving mode, and (C) is a diagram showing the driver's seat in automatic driving mode. [Diagram 2] 1 is a hardware configuration diagram of a vehicle in which a monitoring device according to an embodiment of the present invention is implemented. [Diagram 3] 5 is an example of an operational flowchart relating to a monitoring process of the monitoring device of the present embodiment. [Figure 4] 5 is an example of an operational flowchart relating to a determination process of the monitoring device of the present embodiment. [Diagram 5]10 is an example of another operational flowchart relating to the determination process of the monitoring device of the present embodiment. [Figure 6] 10 is an example of an operational flowchart relating to a modified example of a monitoring process of the monitoring device of the present embodiment. [Figure 7] 5 is an example of an operational flowchart relating to an estimation process of the monitoring device of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] 1(A) to 1(C) explain an overview of the operation of the monitoring device of this embodiment. Fig. 1(A) is a diagram showing a vehicle 10, Fig. 1(B) is a diagram showing a driver's seat 31 in a manual driving mode, and Fig. 1(C) is a diagram showing a driver's seat 31 in an automatic driving mode.

[0024] 1(A), a vehicle 10 has an automatic control device 11 and a monitoring device 12. The automatic control device 11 has an automatic driving mode (e.g., driving modes of levels 3 to 5) in which the automatic control device 11 mainly drives the vehicle 10, and a manual driving mode (e.g., driving modes of levels 0 to 2) in which a driver 40 mainly drives the vehicle 10.

[0025] In the manual driving mode, the monitoring device 12 moves the seat position of the driver's seat 31 in response to the operation of the seat operating unit 3 by the driver 40. In the manual driving mode, the monitoring device 12 stores the seat position of the driver's seat 31 set by the driver 40 (hereinafter also referred to as the manual driving position).

[0026] 1(B), in the manual driving mode, the automatic control device 11 controls the operation of the vehicle 10 based on the operation of the driver 40. The driver 40 sits in the driver's seat 31 in the vehicle compartment 30 so as to be able to operate the steering wheel 32, the brake pedal 33, and the accelerator pedal 34.

[0027] In the autonomous driving mode, the driver 40 is not required to be in a state in which he or she can drive the vehicle 10 at all times. In the autonomous driving mode, the monitoring device 12 moves the seat position of the driver's seat 31 in response to the operation of the seat operating unit 3 by the driver 40. The monitoring device 12 stores the seat position of the driver's seat 31. In the autonomous driving mode, the driver 40 can move the seat position of the driver's seat 31 so that the driver 40 can assume a relaxed posture, as shown in FIG. 1(C).

[0028] In the automatic driving mode, when the automatic control device 11 determines that the vehicle 10 cannot be driven safely under automatic control, it notifies the driver 40 via a user interface (UI) 6 of a control transfer request to transfer the subject of driving of the vehicle 10 from the automatic control device 11 to the driver 40. The automatic control device 11 executes the transfer of control so as to transfer the control of the vehicle 10 to the driver within a control transfer time from the time when the control transfer request is notified to the driver 40. The control transfer time is, for example, 10 seconds.

[0029] When the control transition request is notified to the driver 40, the monitoring device 12 moves the driver's seat 31 to the manual driving position. As a result, the driver 40 performs a predetermined operation to approve the control transition request, and then starts driving the vehicle 10 in the manual driving mode. The control transition time includes a movement time for moving the driver's seat 31 from the current seat position to the manual driving position, and an approval time required for the driver 40 to perform an operation to approve the control transition request.

[0030] The approval time can be considered to be a substantially constant time, but the movement time required for the driver's seat 31 to move to the manual driving position varies depending on the seat position of the driver's seat 31 in the automatic driving mode. The movement time for moving the driver's seat 31 from the current seat position to the manual driving position corresponds to the preparation transition time required for the driver 40 to be in a state where he or she can start driving the vehicle 10 in the manual driving mode.

[0031] Therefore, in the automatic driving mode, the monitoring device 12 determines whether the degree of preparation of the driver 40 to start manual control is low or not, based on the posture information indicating the posture of the driver 40.

[0032] The seat position of the driver's seat 31 being away from the manual driving position means that the driver's 40 hands are away from the steering wheel 32 and / or the driver's 40 feet are away from the brake pedal 33 and the accelerator pedal 34. In these postures, the driver 40 is less prepared to start manual control. The seat position of the driver's seat 31 is an example of posture information.

[0033] The monitoring device 12 determines the degree of readiness of the driver 40 to start manual control based on the seat position of the driver's seat 31. First, the monitoring device 12 determines, based on the seat position of the driver's seat 31, a preparation transition time required for the driver 40 to be in a state where he or she can start driving the vehicle 10 in the manual driving mode.

[0034] Specifically, based on the preparation transition time, the monitoring device 12 determines whether the driver 40 is at a first degree that allows manual control of the vehicle 10, a second degree that is acceptable when the vehicle is being driven under automatic control, or a third degree that is not acceptable when the vehicle is being driven under automatic control.

[0035] When the preparation transition time is zero, the monitoring device 12 determines that the degree of preparation to start manual control is high. The driver's seat 31 being in the manual driving position is an example of the degree of readiness of the driver 40 to start manual control being a first degree at which manual control of the vehicle 10 is possible.

[0036] When the preparation transition time exceeds a predetermined reference time, the monitoring device 12 determines that the degree of preparation for starting manual control of the driver 40 is low. The driver's seat 31 being in a position where the preparation transition time exceeds the reference time is an example of the degree of preparation for starting manual control of the driver 40 being a third degree that is not permissible when the vehicle 10 is being driven under automatic control.

[0037] When the preparation transition time is not zero but does not exceed a predetermined reference time, the monitoring device 12 determines that the degree of preparation of the driver 40 to start manual control is high. The driver's seat 31 being in a position where the preparation transition time does not exceed the reference time is an example of the degree of preparation of the driver 40 to start manual control being a second degree that is permissible when the vehicle 10 is driven under automatic control. Also, when the preparation transition time is zero, the monitoring device 12 determines that the degree of preparation of the driver 40 to start manual control is high. The preparation transition time being zero means that the driver's seat 31 is in a manual driving position. The driver's seat 31 being in a manual driving position is an example of the degree of preparation of the driver 40 to start manual control being a first degree that allows manual control of the vehicle 10.

[0038] When the driver 40 is less ready to begin manual control, the monitoring device 12 provides a driving readiness notification via the UI 6 until the driver 40 is in a position that is acceptable to the driver when the vehicle 10 is being driven under automatic control.

[0039] When the driving preparation notification is issued, the driver 40 operates the seat operation unit 3 to move the seat position of the driver's seat 31 closer to the manual driving position until the driving preparation notification is no longer issued.

[0040] As a result, in the automatic driving mode, the seat position of the driver's seat 31 is set within a range in which the movement time for the driver's seat 31 to move to the manual driving position does not exceed the reference time. After the driver's seat 31 moves to the manual driving position within the control transition time after the control transition request is notified to the driver 40 and the driver 40 performs an approval operation, the driver 40 can start driving the vehicle 10 in the manual driving mode.

[0041] In addition, in the automatic driving mode, the seat position of the driver's seat 31 can be set to a position that allows the driver 40 to assume a relaxed posture within the range of postures permitted for the driver 40 when the vehicle 10 is being driven under automatic control.

[0042] As described above in detail, when the monitoring device 12 of this embodiment determines that the driver 40 is not ready to start manual control, it notifies the driver 30 to increase his / her readiness to start manual control, so that the driver 30 can maintain a state in which he / she can return to a state in which manual control is possible within a specified time.

[0043] 2 is a hardware configuration diagram of a vehicle 10 in which a monitoring system 1 including a monitoring device 12 of this embodiment is implemented. The vehicle 10 has monitoring cameras 2, 2a, a seat operation unit 3, a seat drive unit 4, a seat position detection sensor 5, a user interface (UI) 6, an automatic control device 11, a monitoring device 12, etc. The monitoring system 1 includes at least the monitoring camera 2, the seat position detection sensor 5, the automatic control device 11, and the monitoring device 12.

[0044] The surveillance cameras 2, 2a, the seat operating unit 3, the seat driving unit 4, the seat position detection sensor 5, the UI 6, the automatic control device 11, and the monitoring device 12 are communicatively connected via an in-vehicle network 14 that complies with a standard such as a controller area network.

[0045] The surveillance cameras 2, 2a are disposed in the vehicle interior 30 so as to be able to obtain surveillance images including the face of the driver 40 who drives the vehicle 10. The surveillance camera 2 is disposed, for example, on the steering column. The surveillance camera 2a is disposed, for example, on the ceiling at the rear of the vehicle interior 30.

[0046] The surveillance cameras 2 and 2a are examples of an image acquisition unit. The surveillance camera 2 acquires surveillance images showing the periphery of the driver's seat 31 at a surveillance image acquisition time having a predetermined period, for example. The surveillance camera 2a acquires surveillance images showing the inside of the vehicle compartment 30 at a surveillance image acquisition time synchronized with the surveillance camera 2. The surveillance camera 2a can acquire surveillance images of the driver's seat 31 when it is located at the rear.

[0047] The surveillance cameras 2 and 2a each have a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to infrared light, such as a CCD or C-MOS, and an imaging optical system that forms an image of the area to be captured on the two-dimensional detector.

[0048] Each time the surveillance camera 2, 2a captures a surveillance image, it outputs the surveillance image and the image capture time when the surveillance image was captured to the surveillance device 12 via the in-vehicle network 14. The surveillance image is used by the surveillance device 12 to estimate the posture of the driver 40. The surveillance image is an example of posture information.

[0049] The driver's seat 31 has a seat back 31a, a seat cushion 31b, and a headrest 31c. The inclination of the seat back 31a can be changed discretely in 10 stages. The position of the seat cushion 31b in the front-rear direction and the position in the up-down direction can each be changed discretely in 10 stages. The inclination and the position in the up-down direction of the headrest 31c can each be changed discretely in 10 stages. In this specification, the seat position of the driver's seat 31 includes the inclination and / or position of each part of the driver's seat 31.

[0050] The seat operation unit 3 is operated by the driver 40 and outputs an operation signal indicating a position to which the driver's seat 31 should be displaced to the monitoring device 12. The monitoring device 12 generates a drive signal for displacing the seat position of the driver's seat 31 based on the operation signal and outputs the drive signal to the seat drive unit 4. The seat drive unit 4 displaces the driver's seat 31 based on the drive signal. The seat operation unit 3 has switches for operating each of the seat back 31a, the seat cushion 31b, and the headrest 31c. The monitoring device 12 stores the seat position of the driver's seat 31.

[0051] The seat drive unit 4 has a drive device such as a motor for displacing the seat back 31a, the seat cushion 31b, and the headrest 31c. In Fig. 1(B) and other drawings, the seat drive unit 4 is disposed in the seat cushion 31b, but the seat drive unit 4 may be disposed separately in each of the seat back 31a, the seat cushion 31b, and the headrest 31c. The seat drive unit 4 is controlled by the monitor device 12 to vibrate the seat back 31a, the seat cushion 31b, and the headrest 31c.

[0052] The seat position detection sensor 5 detects the inclination or position of each part of the driver's seat 31. The seat position detection sensor 5 detects the inclination of the seat back 31a, and outputs seat position information representing the inclination of the seat back 31a to the monitoring device 12. The seat position detection sensor 5 also detects the front-rear and up-down positions of the seat cushion 31b, and outputs seat position information representing the front-rear and up-down positions of the seat cushion 31b to the monitoring device 12. Furthermore, the seat position detection sensor 5 detects the inclination and up-down position of the headrest 31c, and outputs the inclination information representing the inclination of the headrest 31c and seat position information representing the up-down position to the monitoring device 12. The seat position information is an example of posture information.

[0053] The seat position detection sensor 5 may include sensors that mechanically, optically, or magnetically detect the positions of the seat back 31a, the seat cushion 31b, and the headrest 31c. In Fig. 1(B) and other figures, the seat position detection sensor 5 is disposed in the seat cushion 31b, but the seat position detection sensor 5 may be disposed separately in each of the seat cushion 31b and the headrest 31c.

[0054] The UI 6 is an example of a notification unit. The UI 6 is controlled by the automatic control device 11 or the monitoring device 12 to notify the driver 40 of information related to the vehicle 10, such as a control transition request. The UI 6 has a display device 6a, such as a liquid crystal display or a touch panel, for displaying the control transition request or the like. The UI 6 may also have an audio output device (not shown) for notifying the driver 40 of the control transition request or the like. The UI 6 also has, for example, a touch panel or an operation button as an input device for inputting operation information from the driver 40 to the vehicle 10. The UI 6 outputs the input operation information to the automatic control device 11 or the like via the in-vehicle network 14.

[0055] The automatic control device 11 controls the operation of the vehicle 10. The automatic control device 11 has an automatic driving mode in which the vehicle 10 is driven automatically, and a manual driving mode in which the operation of the vehicle 10 is controlled based on the operation of the driver 40. In the automatic driving mode, the automatic control device 11 mainly drives the vehicle 10. In the automatic driving mode, the automatic control device 11 controls operations such as steering, driving, and braking based on detection information of a sensor (not shown) mounted on the vehicle 10. When starting driving in the automatic driving mode or the manual driving mode, the automatic control device 11 outputs information indicating that driving in the automatic driving mode or the manual driving mode is started to the monitoring device 12. In addition, when ending driving in the automatic driving mode or the manual driving mode, the automatic control device 11 outputs information indicating that driving in the automatic driving mode or the manual driving mode is ended to the monitoring device 12.

[0056] In addition, in the manual driving mode, the automatic control device 11 controls the operation of the vehicle 10, such as steering, driving, and braking, based on the operation of the driver 40. In the manual driving mode, the driver 40 takes the lead in driving the vehicle 10. In the manual driving mode, the automatic control device 11 controls the operation of the vehicle 10 based on the operation of at least one of the steering wheel 32, the brake pedal 33, and the accelerator pedal 34 by the driver 40.

[0057] The automatic control device 11 mainly drives the vehicle 10 means that the automatic control device 11 controls the steering, driving, and braking operations without the driver's involvement. The driver 40 mainly drives the vehicle 10 means that the driver is involved in controlling at least one of the steering, driving, and braking operations.

[0058] The automatic control device 11 may have a plurality of automatic driving levels in the automatic driving mode. In this specification, the plurality of automatic driving levels may include so-called automatic driving levels of levels 3 to 5. Also, in this specification, the manual driving mode may include so-called automatic driving levels of levels 0 to 2.

[0059] The automatic control device 11 detects objects such as other vehicles around the vehicle 10 based on detection information from a sensor (not shown) mounted on the vehicle 10. In the automatic driving mode, when the automatic control device 11 cannot maintain a predetermined distance or more between the vehicle 10 and another object, the automatic control device 11 notifies the driver 40 via the UI 6 of a control transfer request that requests the subject of driving of the vehicle 10 to be transferred from the automatic control device 11 to the driver 40. In addition, in the automatic driving mode, the automatic control device 11 notifies the driver 40 via the UI 6 of a control transfer request that requests the driver 40 to transfer the driving from the automatic control device 11 to the driver 40 before entering an area where driving under automatic control is not permitted. An example of an area where driving under automatic control is not permitted is an area where a high-precision map is not provided. Furthermore, in the automatic driving mode, the automatic control device 11 notifies the driver 40 of the control transfer request via the UI 6 even when it is determined that the driving of the vehicle 10 cannot be safely controlled due to an abnormality in a sensor or the like. The automatic control device 11 also outputs the control transfer request to the monitoring device 12 via the in-vehicle network 14.

[0060] The automatic control device 11 executes control transfer so as to transfer control of the vehicle 10 to the driver within a predetermined control transfer time from the time when the control transfer request is notified to the driver 40. If the driver 40's approval action for the control transfer request cannot be confirmed within the control transfer time, the automatic control device 11 safely stops the vehicle 10 (Minimal Risk Maneuver processing).

[0061] The automatic control device 11 transitions from the automatic driving mode to the manual driving mode when it recognizes, for example, any one of the following three driver approval actions: (1) the driver 40 keeps the steering wheel 32 steered and operates the accelerator pedal 34, (2) the driver 40 keeps the steering wheel 32 steered and operates the brake pedal 33, or (3) the driver 40 keeps the steering wheel 32 steered and operates the steering wheel 32.

[0062] When a control transition request is input from the automatic control device 11, the monitoring device 12 moves the driver's seat 31 to the manual driving position.

[0063] The monitoring device 12 executes control processing and determination processing. To that end, the monitoring device 12 has a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the monitoring device 12 to the in-vehicle network 14.

[0064] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores computer programs of applications used in the information processing executed by the processor 23 and various data.

[0065] All or part of the functions of the monitoring device 12 are functional modules realized by, for example, a computer program running on the processor 23. The processor 23 has a control unit 231 and a determination unit 232. Alternatively, the functional modules of the processor 23 may be dedicated arithmetic circuits provided in the processor 23. The processor 23 has one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further have other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit.

[0066] The control unit 231 moves the seat position of the driver's seat 31 in response to the operation of the seat operation unit 3 by the driver 40. In the manual driving mode, the control unit 231 stores the seat position of the driver's seat 31 set by the driver 40 as a manual driving position. The manual driving position may be set to include a predetermined range of seat positions. The operation of the monitoring device 12 will be described in detail later.

[0067] The automatic control device 11 and the monitoring device 12 are, for example, electronic control devices (ECUs). In Fig. 3, the automatic control device 11 and the monitoring device 12 are described as separate devices (for example, ECUs), but all or part of these devices may be configured as a single device.

[0068] Fig. 3 is an example of an operation flowchart relating to the monitoring process of the monitoring device 12 of this embodiment. When the automatic control device 11 is in the automatic operation mode, the monitoring device 12 executes the monitoring process at a monitoring time having a predetermined cycle according to the operation flowchart shown in Fig. 3. The cycle of the monitoring time can be set to, for example, 10 to 60 seconds.

[0069] First, the control unit 231 acquires posture information indicating the posture of the driver 40 acquired when the vehicle 10 is being driven under automatic control (step S101). The control unit 231 acquires seat position information as the posture information via the in-vehicle network 14. The control unit 231 also acquires a monitoring image as the posture information via the in-vehicle network 14. The control unit 231 also acquires a monitoring image as the posture information via the in-vehicle network 14.

[0070] Next, the determination unit 232 determines whether or not the degree of preparation for starting manual control of the driver 40 is low based on the posture information (step S102). The determination process by the determination unit 232 to determine whether or not the degree of preparation for starting manual control of the driver 40 is low will be described later.

[0071] When it is determined that the degree of preparation for starting manual control is low (step S102-Yes), the control unit 231 gives a driving preparation notification via the UI 6 (step S103) and ends the series of processes. The control unit 231 is an example of a notification control unit. By executing the monitoring process at a predetermined cycle, when the vehicle 10 is being driven by automatic control, the driving preparation notification is given to the driver 40 until the driver 40 assumes a position acceptable to the driver 40.

[0072] If it is determined that the degree of preparation for starting manual control is low based on the seat position information, the control unit 231 may notify the driver 40 by display or audio via the UI 6 that the seat position of the driver's seat 31 should be moved forward.

[0073] In addition, if it is determined that the level of preparation for starting manual control is low based on the monitoring image, the control unit 231 may notify the driver 40 by display or audio via the UI 6 to pay attention to driving the vehicle 10.

[0074] In addition, the control unit 231 may notify the driver 40 of the driving preparation notification by changing the temperature of the air conditioning system, changing the strength of the air conditioning system's airflow, changing the direction of the air conditioning system's airflow, changing the intensity of the lighting in the vehicle compartment 30, changing the color of the lighting in the vehicle compartment 30, vibrating the driver's seat 31, etc.

[0075] The control unit 231 continues to notify the driver 40 of the driving preparation notification until it is no longer determined that the degree of preparation for starting manual control is low. The driver 40 can increase the degree of preparation for starting manual control by changing the seat position of the driver's seat 31, improving his / her posture to a state where he / she can drive, or the like.

[0076] Furthermore, the control unit 231 may input a schedule for notifying the driver 40 of a control transition request from the automatic control device 11. When the schedule for notifying the driver 40 of a control transition request is within 10 minutes, the control unit 231 may notify the driver 40 of the driving preparation notification with a stronger emphasis than usual. For example, the control unit 231 may notify the driver 40 of the driving preparation notification together with a warning sound. Furthermore, when notifying the driver 40 of the driving preparation notification using an air conditioner, the control unit 231 may increase the intensity of the transmission more than usual. Furthermore, when notifying the driver 40 of the driving preparation notification using lighting in the vehicle interior 30, the control unit 231 may increase the intensity of the lighting more than usual.

[0077] On the other hand, if it is not determined that the degree of preparation for starting manual control is low (step S102-No), the series of processes ends.

[0078] Fig. 4 is an example of an operational flowchart relating to the determination process of the monitoring device 12 of this embodiment. The determination unit 232 executes the determination process based on the seat position information. The determination unit 232 executes the determination process at a first determination time having a predetermined cycle according to the operational flowchart shown in Fig. 4. The cycle of the first determination time can be set to, for example, 1 to 10 seconds.

[0079] First, the determination unit 232 estimates a preparation transition time based on the seat position information (step S201). The preparation transition time is a time required to move the current seat position of the driver's seat 31 to the manual driving position.

[0080] First, the determination unit 232 estimates a moving time required to move the position of the seat cushion 31b in the front-rear direction to the manual driving position, based on the seat position information indicating the current position of the seat cushion 31b in the front-rear direction.

[0081] Specifically, the determination unit 232 estimates the movement time required to move the fore-and-aft position of the seat cushion 31b to the manual driving position based on the weight (mass) of the driver 40, the mass of the driver's seat 31, the force with which the seat drive unit 4 drives the seat cushion 31b, the current seat position information, and the manual driving position.

[0082] The weight (mass) of the driver 40 may be input via the UI 5. In addition, a standard weight stored in advance in the memory 22 may be used as the weight (mass) of the driver 40.

[0083] Similarly, the determination unit 232 estimates the moving time required to move the vertical position of the seat cushion 31b to the manual driving position, based on the seat position information indicating the current vertical position of the seat cushion 31b.

[0084] Similarly, the determination unit 232 estimates the moving time required to move the inclination of the seat back 31a to the manual driving position based on the seat position information indicating the current inclination of the seat back 31a. When estimating the moving time of the seat back 31a, the proportion of the weight of the driver 40 applied to the seat back 31a is used.

[0085] Similarly, the determination unit 232 estimates the moving time required to move the tilt and vertical position of the headrest 31c to the manual driving position based on the tilt information indicating the current tilt of the headrest 31c and the seat position information indicating the vertical position. When estimating the moving time of the headrest 31c, the proportion of the weight of the driver 40 applied to the headrest 31c is used.

[0086] The determination unit 232 determines the maximum movement time among the movement times for moving the inclination or position of each part of the driver's seat 31 to the manual driving position as the current preparation transition time of the driver's seat 31.

[0087] A long preparation transition time for the driver's seat 31 means that the seat cushion 31b is located further rearward, the inclination of the seat back 31a is greater, or the inclination representing the inclination of the headrest 31c is greater, etc.

[0088] The seat cushion 31b being positioned rearward, the seat back 31a being tilted significantly, or the headrest 31c being tilted significantly, means that the hands of the driver 40 are away from the steering wheel 32, or the feet of the driver 40 are away from the brake pedal 33 and the accelerator pedal 34.

[0089] On the other hand, if the seat cushion 31b is positioned rearward, the seat back 31a is inclined greatly, or the head rest 31c is inclined greatly, it is estimated that the driver 40 is in a relaxed position.

[0090] Next, the determination unit 232 determines whether the preparation transition time exceeds a predetermined reference time (step S202). The reference time is preferably determined based on a control transition time at which the driver 40 is required to start driving the vehicle 10 in the manual driving mode after the control transition request is notified to the driver 40. Specifically, the reference time is a time obtained by subtracting an approval time required for the driver 40 to perform an operation to approve the control transition request from the control transition time. For example, if the control transition time is 10 seconds and the approval time is 2 seconds, the reference time is 8 seconds.

[0091] When the preparation transition time exceeds the reference time (step S202-Yes), the determination unit 232 determines that the degree of preparation of the driver 40 to start manual control is low (step S203), and ends the series of processes. The driver's seat 31 being in a position where the preparation transition time exceeds the reference time is an example of the degree of preparation of the driver 40 to start manual control being a third degree that is not permissible when the vehicle 10 is being driven under automatic control.

[0092] On the other hand, if the preparation transition time is zero or does not exceed the reference time (step S202-No), the determination unit 232 determines that the degree of preparation for starting manual control of the driver 40 is not low (step S204), and ends the series of processes. The preparation transition time being zero means that the driver's seat 31 is in the manual driving position. The driver's seat 31 being in the manual driving position is an example of the degree of preparation for starting manual control of the driver 40 being a first degree that allows manual control of the vehicle 10. Also, the driver's seat 31 being in a position where the preparation transition time does not exceed the reference time is an example of the degree of preparation for starting manual control of the driver 40 being a second degree that is permitted when the vehicle 10 is being driven under automatic control. In other words, the determination unit 232 determines whether the degree of preparation for starting manual control of the driver 40 is the first degree, the second degree, or the third degree.

[0093] Fig. 5 is another example of an operation flowchart relating to the determination process of the monitoring device 12 of this embodiment. The determination unit 232 executes the determination process based on a monitoring image as posture information. The determination unit 232 executes the determination process at a second determination time having a predetermined cycle according to the operation flowchart shown in Fig. 5. The cycle of the second determination time can be, for example, 5 to 10 minutes.

[0094] First, the determination unit 232 estimates the posture of the driver 40 based on the monitoring images (step S301). The determination unit 232 estimates the posture of the driver 40 based on the monitoring images acquired by the monitoring cameras 2 and 2a.

[0095] The determination unit 232 has a classifier that has been trained to identify the posture of the driver 40. The determination unit 232 estimates the posture of the driver 40 depicted in the monitoring image by inputting the monitoring image to the classifier. It is preferable that the classifier has been trained to identify a relaxed posture such as a state in which the arms are folded and a state in which the legs are crossed. It is also preferable that the classifier identifies a posture in which the driver 40 is operating a terminal such as a smartphone or a laptop computer.

[0096] Next, the determination unit 232 determines whether the driver 40 is in a predetermined posture (step S302). The predetermined postures include relaxed postures such as a posture with folded arms and a posture with folded legs. The predetermined postures also include a posture in which the driver 40 is operating a smartphone or a laptop computer terminal. These postures of the driver 40 are examples in which the degree of readiness of the driver 40 to start manual control is a third degree that is not permitted when the vehicle 10 is being driven under automatic control.

[0097] When the driver 40 is in the predetermined posture (step S302-Yes), the determination unit 232 determines whether a predetermined reference time has elapsed (step S303). The reference time can be, for example, 5 to 10 minutes. When the driver 40 is in the predetermined posture for the reference time, the consciousness level of the driver 40 is considered to be not in a state where the driver 40 can immediately recognize the notified control transfer request, judge the situation, and perform an operation to approve the control transfer request.

[0098] If the reference time has not elapsed (step S303-No), the process returns to before step S301. On the other hand, if the reference time has elapsed (step S303-Yes), the determination unit 232 determines that the degree of preparation for starting manual control of the driver 40 is low (step S304), and ends the series of processes. This is because if the driver 40 is in a predetermined posture for a long time, the driver 40's awareness level regarding driving will decrease.

[0099] Furthermore, if the driver 40 is not in a predetermined posture (step S302-No), the determination unit 232 determines that the degree of readiness of the driver 40 to start manual control is not low (step S305), and ends the series of processes. For example, if the driver 40 is in a posture with his / her arms uncrossed or his / her legs uncrossed, the determination unit 232 determines that the degree of readiness of the driver 40 to start manual control is not low (high). These postures of the driver 40 are examples in which the degree of readiness of the driver 40 to start manual control is the second degree permitted when the vehicle 10 is being driven under automatic control.

[0100] Furthermore, when the driver 40 is in a posture where the driver 40 is operating the steering wheel 32, or in a posture where the driver 40 is operating the brake pedal 33 or the accelerator pedal 34, the determination unit 232 determines that the degree of readiness of the driver 40 to start manual control is not low (high). These postures of the driver 40 are examples where the degree of readiness of the driver 40 to start manual control is a first degree that allows manual control of the vehicle 10. In other words, the determination unit 232 determines whether the degree of readiness of the driver 40 to start manual control is a first degree, a second degree, or a third degree.

[0101] In step S102, the judgment unit 232 performs the judgment process shown in FIG. 4 and the judgment process shown in FIG. 5, and if it is judged in either process that the degree of preparation for starting manual control of the driver 40 is low, the process proceeds to step S103.

[0102] As described above in detail, when the monitoring device of the present embodiment determines that the driver's readiness to start manual control is low, the monitoring device notifies the driver to increase the driver's readiness to start manual control. This allows the driver to maintain a state in which the driver can return to a state in which manual control is possible within a predetermined time. The driver 10 can take a relaxed state within the range of postures permitted for the driver 40 when the vehicle 10 is being driven under automatic control.

[0103] Next, a modified example of the monitoring device of the present embodiment described above will be described below with reference to Fig. 6. Fig. 6 is an example of an operational flowchart relating to a modified example of the monitoring process of the monitoring device of the present embodiment.

[0104] In this modified example, when it is determined that the degree of readiness of driver 40 to start manual control is low based on the posture information of driver 40, and when it is determined that the degree of readiness of driver 40 to start manual control is low based on the cognitive state representing the degree of involvement of driver 40 in driving, a driving readiness notification is given to driver 40.

[0105] In Fig. 6, steps S403 and S404 are newly added processes. In Fig. 6, steps S401, S402, and S405 are similar to the above-mentioned steps S101 to S103.

[0106] When it is determined that the degree of preparation for starting manual control is low based on the posture information (step S402-Yes), the determination unit 232 estimates the cognitive state of the driver 40 (step S403). The estimation process for estimating the cognitive state of the driver 40 will be described later.

[0107] Next, the determination unit 232 determines whether the degree of readiness of the driver 40 to start manual control is low based on the cognitive state of the driver 40 (step S404). When the cognitive state of the driver 40 is estimated to be low, the determination unit 232 determines that the degree of readiness of the driver 40 to start manual control is low.

[0108] For example, if the driver 40 is feeling drowsy and has only a small degree of eye opening, it is considered that the driver 40 is not at a level of consciousness where he or she can immediately recognize the notified control transfer request, assess the situation, and accept the control transfer request.

[0109] On the other hand, when the cognitive state of the driver 40 is not estimated to be low, the determination unit 232 does not determine that the degree of preparation for starting manual control of the driver 40 is low.

[0110] If the degree of preparation of the driver 40 to start manual control is low (step S404-Yes), the control unit 231 provides a driving preparation notification to the driver 40 to increase the degree of preparation to start manual control (step S405), and ends the series of processes.

[0111] On the other hand, if the degree of preparation for starting manual control of the driver 40 is not low (step S404-No), the series of processes ends.

[0112] Next, the estimation process of the monitoring device 12 will be described below with reference to Fig. 7. Fig. 7 is an example of an operation flowchart related to the estimation process of the monitoring device 12 of this embodiment. The monitoring device 12 executes the estimation process at an estimation time having a predetermined period according to the operation flowchart shown in Fig. 7.

[0113] First, the determination unit 232 acquires monitoring information of the driver 40 (step S501). The monitoring information is monitoring images acquired by the monitoring cameras 2 and 2a.

[0114] Next, the determination unit 232 estimates the cognitive state of the driver 40 based on the monitoring information (step S502), and ends the series of processes. The determination unit 232 obtains involvement information indicating the degree of involvement of the driver 40 in driving based on the monitoring image. The involvement information includes the degree of eye opening, the degree of mouth opening, the direction of gaze, the frequency of directing the gaze to the rearview mirror (not shown), the frequency of blinking, etc. of the driver 40. Note that the determination unit 232 may obtain any one of the degree of eye opening, the degree of mouth opening, the direction of gaze, the frequency of directing the gaze to the rearview mirror (not shown), or the frequency of blinking of the driver 40.

[0115] The degree to which the driver 40's eyes are open, the degree to which the driver opens his / her eyes, the direction of his / her gaze, the frequency with which the driver 40 looks at the rearview mirror (not shown), and the frequency with which the driver blinks are examples of involvement information that indicates the degree of involvement of the driver 40 in driving. For example, when the driver 40 becomes more drowsy, the involvement information changes. The determination unit 232 estimates the cognitive state that indicates the degree of the driver's involvement in driving based on the involvement information.

[0116] When the state in which the eye opening degree of the driver 40 is lower than a predetermined reference eye opening degree continues for a predetermined time, the determination unit 232 estimates that the cognitive state of the driver 40 is low. The predetermined time can be, for example, 5 to 10 minutes.

[0117] When the state in which the driver 40's mouth opening degree is higher than a predetermined reference mouth opening degree continues for a predetermined time, the determination unit 232 estimates that the cognitive state of the driver 40 is low. The predetermined time can be, for example, 5 to 10 minutes.

[0118] The determination unit 232 estimates that the cognitive state of the driver 40 is low when the direction of the driver's line of sight is out of a predetermined range for a predetermined period of time. The predetermined range may be, for example, 30 degrees to the left and right and 30 degrees up and down with respect to the traveling direction of the vehicle 10. The predetermined period of time may be, for example, 5 to 10 minutes.

[0119] The determination unit 232 estimates that the cognitive state of the driver 40 is low when the frequency of directing the gaze to the rearview mirror (not shown) in the vehicle interior 30 is equal to or lower than a predetermined reference frequency for a predetermined period of time. The predetermined reference frequency may be, for example, three times per minute. The predetermined period of time may be, for example, five to ten minutes.

[0120] The determination unit 232 estimates that the cognitive state of the driver 40 is low when the blinking frequency is equal to or lower than a predetermined reference frequency for a predetermined period of time. The predetermined reference frequency may be, for example, 10 times per minute. The predetermined period of time may be, for example, 5 to 10 minutes.

[0121] According to this modification described above, the monitoring device 12 can more accurately determine the degree of readiness of the driver to initiate manual control based on the cognitive state together with the posture information.

[0122] In the present disclosure, the monitoring device, the monitoring computer program, and the monitoring method of the above-mentioned embodiments can be modified as appropriate without departing from the spirit of the present disclosure. Furthermore, the technical scope of the present disclosure is not limited to those embodiments, but extends to the inventions described in the claims and their equivalents.

[0123] For example, in the above-described embodiment, the posture information is acquired based on the seat position of the driver's seat and the monitored image, but the posture information may be acquired using other methods. Also, the example of the driving preparation notification described in the above-described embodiment is merely an example, and the driving preparation notification is not limited thereto. [Explanation of symbols]

[0124] 1. Surveillance System 2. Surveillance cameras 3 Seat operation section 4 Seat drive unit 5 Seat position detection sensor 6 User Interface (UI) 6a Display device 10 Vehicles 11 Automatic control device 12 Monitoring equipment 21 Communication Interface 22 Memory 23 Processors 231 Control Unit 232 Judgment section 14 In-vehicle network 30 Cabin 31 Driver's seat 31a Seat back 31b Seat cushion 31c Headrest 32 Steering wheel 33 Brake pedal 34 Accelerator pedal

Claims

1. a determination unit that determines whether the degree of readiness of the driver to start manual control is a first degree at which manual control of the vehicle is possible, a second degree that is permissible when the vehicle is being driven under automatic control, or a third degree that is not permissible when the vehicle is being driven under automatic control, based on posture information that indicates the posture of the driver acquired when the vehicle is being driven under automatic control; a notification control unit that, when the determination unit determines that the degree of preparation for starting manual control is the third degree, issues a predetermined notification via a notification unit to encourage a driver to increase the degree of preparation for starting manual control; A monitoring device comprising:

2. The determination unit estimates a transition time required for the driver to be able to start driving the vehicle in the manual driving mode after a transition request for transitioning from an automatic driving mode in which the vehicle is driven under automatic control to a manual driving mode in which the vehicle is driven under manual control is notified to the driver via the notification unit, based on the posture information; 2. The monitoring device of claim 1, wherein the determination unit determines that the degree of readiness to start manual control of the driver is the first degree when the transition time is zero, determines that the degree of readiness to start manual control of the driver is the second degree when it is determined that the transition time does not exceed a predetermined reference time, and determines that the degree of readiness to start manual control of the driver is the third degree when it is determined that the transition time exceeds a predetermined reference time.

3. The monitoring device according to claim 2 , wherein the predetermined reference time is determined based on a time at which the driver is required to start driving the vehicle in the manual driving mode after the transition request is notified to the driver.

4. The posture information includes seat position information indicating a position of a driver's seat in which a driver is seated, The monitoring device according to claim 2 , wherein the second determination unit estimates the transition time based on the seat position information.

5. 2. The monitoring device of claim 1, wherein the posture information includes an image depicting a driver, and the determination unit determines that the degree of readiness to begin manual control is the third degree when the driver depicted in the image is in a predetermined posture.

6. The monitoring device according to claim 5 , wherein the predetermined postures include a posture with arms crossed and a posture with legs crossed.

7. a third determination unit that determines whether the degree of preparation for starting manual control of the driver is low or not based on a cognitive state that indicates a degree of involvement of the driver in driving; The monitoring device according to any one of claims 1 to 6, wherein the notification control unit gives the specified notification to the driver when the determination unit determines that the degree of readiness to start manual control of the driver based on the posture information is the third degree, and when the third determination unit determines that the degree of readiness to start manual control of the driver is low based on the cognitive state.

8. determining whether the degree of readiness of the driver to start manual control is a first degree at which manual control of the vehicle is possible, a second degree that is permissible when the vehicle is being driven under automatic control, or a third degree that is not permissible when the vehicle is being driven under automatic control, based on posture information indicating the posture of the driver acquired when the vehicle is being driven under automatic control; when the degree of preparation for starting manual control is determined to be the third degree, providing a predetermined notification via a notification unit to encourage the driver to increase the degree of preparation for starting manual control. A monitoring computer program, comprising: a processor configured to execute a process including:

9. The monitoring device determines whether the degree of readiness of the driver to start manual control is among a first degree at which manual control of the vehicle is possible, a second degree that is permissible when the vehicle is being driven under automatic control, and a third degree that is not permissible when the vehicle is being driven under automatic control, based on posture information indicating the posture of the driver acquired when the vehicle is being driven under automatic control; A monitoring computer program characterized in that, when the degree of readiness to start manual control is determined to be the third degree, the sound control device executes a predetermined notification via a notification unit to encourage the driver to increase the degree of readiness to start manual control.

Images