Monitoring device

Abstract

To provide a monitoring device that is capable of promoting a user to see a vehicle traveling via remote traveling control if the user is not actually seeing the vehicle.SOLUTION: A monitoring device monitors a user in a case where remote traveling control that makes a vehicle travel in accordance with an instruction from a user present outside the vehicle is being executed. The monitoring device is configured to provide a visual notification for promoting the user to see the vehicle if a non-viewing condition holds in which a non-viewing state where the user is not seeing the vehicle has continued for a first predetermined time or more.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a monitoring device that monitors a user when remote driving control for driving a vehicle in accordance with an instruction from a user outside the vehicle is being executed.

Background Art

[0002] Conventionally, a remote driving control device that executes remote driving control has been known. For example, the remote driving control device described in Patent Document 1 executes remote driving control based on a command transmitted by a remote operation device operated by a user. The remote operation device transmits a command when the movement of the user's finger on the touch sensing portion satisfies a predetermined operation condition.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] When remote driving control is being executed and the vehicle is running, the user needs to actually watch the vehicle to monitor it. However, the remote driving control device and the remote operation device described in Patent Document 1 do not determine whether the user is actually looking at the vehicle. For this reason, the remote driving control device and the remote operation device described in Patent Document 1 cannot prompt a user who is not actually looking at the vehicle to look at the vehicle.

[0005] The present invention has been made to address the above-described problems. That is, one object of the present invention is to provide a monitoring device that can prompt a user to look at a vehicle when the user is not actually looking at the vehicle running under remote driving control.

[0006] The driving support device of the present invention (hereinafter referred to as "the device of the present invention") monitors the user when remote driving control for driving the vehicle in accordance with an instruction from a user outside the vehicle is being executed. When a non-visual condition that the user is in a non-visual state of not looking at the vehicle continues for a first predetermined time or more is satisfied (step 215 "Yes", step 235, step 240 "Yes", step 245 "No", step 250), the monitoring device is configured to perform a visual notification for prompting the user to look at the vehicle (step 415 "Yes", step 420).

[0007] According to the device of the present invention, since a visual notification is performed when the non-visual condition is satisfied, when the user is not actually looking at the vehicle traveling under remote driving control, the user can be prompted to look at the vehicle.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Modes for Carrying Out the Invention

[0009] As shown in FIG. 1, the vehicle control system according to the present embodiment includes "a vehicle control device 10 applied to vehicle VA" and a remote operation device 20. The vehicle control device 10 and the remote operation device 20 are communicably connected via a network NW.

[0010] The vehicle control device 10 includes the components shown in FIG. 1. In this specification, "ECU 30" is an electronic control device mainly comprising a microcomputer. ECU 30 is also referred to as a control unit, a controller, and a computer. The microcomputer includes a CPU (processor), a ROM, a RAM, an interface, and the like. The functions realized by ECU 30 may be realized by a plurality of ECUs.

[0011] The camera 32 acquires image data by photographing the scenery around the vehicle VA. The sonar 34 acquires sonar data regarding the position of an object existing around the vehicle VA with respect to the vehicle VA. The ECU 30 acquires the image data from the camera 32 and the sonar data from the sonar 34. Note that the ECU 30 recognizes an object located around the vehicle VA based on the image data and the sonar data.

[0012] The GNSS (Global Navigation Satellite System) receiver 36 receives signals from a plurality of artificial satellites and specifies the current position (latitude and longitude) of the vehicle VA based on the received signals. The communication interface (I / F) 38 is an interface for connecting the vehicle control device 10 to the network NW.

[0013] The power train actuator 40 changes the driving force generated by the drive device (for example, an internal combustion engine and / or an electric motor) of the vehicle VA. The brake actuator 42 changes the braking force applied to the vehicle VA. The steering motor 44 is incorporated in the steering mechanism 46. The steering mechanism 46 is a mechanism for steering the steered wheels in response to an operation of the steering wheel. The steering motor 44 generates an automatic steering torque for changing the steering angle of the steered wheels in the steering mechanism 46 in response to an instruction from the ECU 30.

[0014] The remote operation device 20 is a device that can be operated even when the user is outside the vehicle, and as an example, it is a smartphone. The remote operation device 20 includes the components shown in FIG. 1.

[0015] The control unit 50 includes a CPU (processor), a ROM, a RAM, an interface, etc. The camera 52 can acquire a face image by photographing the user's face. The azimuth sensor detects the direction in which the remote control device 20 (the central axis CA (refer to FIG. 3)) is facing. The GNSS receiver 56 and the communication I / F 58 are the same as the GNSS receiver 36 and the communication I / F 38 respectively. These descriptions are omitted.

[0016] The display device 60 is a touch panel type display that allows the user to input to the remote control device 20 by touching the display device 60. When the display device 60 is not of the touch panel type, the remote control device 20 is provided with an input device. The speaker 62 emits a warning sound. The vibration motor 64 is a motor for vibrating the remote control device 20.

[0017] (Overview of operation) When a user outside the vehicle operates the remote control device 20, the vehicle control device 10 executes remote driving control to automatically drive the vehicle VA according to the command transmitted by the remote control device 20. For example, as remote driving control, smart summon and reverse summon are known.

[0018] In smart summon, the vehicle VA automatically drives from the parking space where the vehicle VA is parked to the current position of the remote control device 20 or the position specified by the user. In reverse summon, when the user gets out of the vehicle at the entrance of the parking lot or the like and operates the remote control device 20, the vehicle VA automatically drives while searching for a parkable parking space, and parks in that parking space when a parking space is found.

[0019] While the remote driving control is being executed, the user performs a predetermined operation on the remote operation device 20. When the remote operation device 20 receives the predetermined operation, it transmits a command to the vehicle control device 10. When the vehicle control device 10 receives the above command, it automatically drives the vehicle VA. During the automatic driving, the vehicle control device 10 controls the power train actuator 40, the brake actuator 42, and the steering motor 44 so that the vehicle VA travels along the target route to the target position. Further, the vehicle control device 10 recognizes an object based on the image data and the sonar data, and controls the power train actuator 40, the brake actuator 42, and the steering motor 44 to avoid contact with the object.

[0020] While such remote driving control is being executed, the user needs to actually look at the vehicle VA and monitor it.

[0021] Therefore, when the remote driving control is being executed, the vehicle control device 10 or the remote operation device 20 determines whether the user is looking at the vehicle VA. When the non-visual condition that the non-visual state where the user is not looking at the vehicle VA continues for a first predetermined time or more is satisfied, the remote operation device 20 gives a visual notification to prompt the user to look at the vehicle VA.

[0022] As a result, when the remote driving control is being executed and the user is not looking at the vehicle VA, a visual notification is given, so the possibility that the user looks at the vehicle VA can be increased.

[0023] Among the vehicle control device 10 and the remote operation device 20, the "device that determines whether the user is looking at the vehicle VA" is referred to as a monitoring device. In the present embodiment, the case where the monitoring device is the vehicle control device 10 will be described as an example.

[0024] (Specific operation) The CPU of the ECU 30 executes the routine shown by the flowchart in FIGS. 2 to 4 every time a predetermined time elapses. Hereinafter, the CPU of the ECU 30 will be referred to as the "first CPU".

[0025] <User Monitoring Routine> When an appropriate time arrives, the first CPU starts processing from step 200 in FIG. 2 and determines whether the execution flag Xexe is "1" at step 205. The execution flag Xexe is set to "1" when starting remote driving control and set to "0" when ending remote driving control. Further, the execution flag Xexe is set to "0" in the initial routine. The initial routine is executed by the first CPU when an ignition key switch (not shown) of the vehicle VA is changed from the off position to the on position.

[0026] When the user touches a predetermined start button displayed on the remote operation device 20, the remote operation device 20 transmits a start command to the vehicle control device 10. When the first CPU receives the start command, it starts remote driving control. Remote driving control ends when the vehicle VA arrives at the space designated by the user, when the non-visual state continues for a second predetermined time longer than the first predetermined time, etc.

[0027] If the execution flag Xexe is "0", the first CPU determines "No" at step 205 and the process proceeds to step 295. At step 295, the first CPU temporarily ends this routine.

[0028] If the execution flag Xexe is "1", the first CPU determines "Yes" at step 205 and executes steps 210 and 215. Step 210: The first CPU executes a visual state determination subroutine described later. In the visual state determination subroutine, it is determined whether the user is in a visual state of looking at the vehicle VA or a non-visual state of not looking at the vehicle VA. Step 215: The first CPU determines whether it is determined that the user is in a non-visual state in the visual state determination subroutine.

[0029] When it is determined in the visual state determination subroutine that the user is in a non-visual state, the first CPU determines "Yes" in step 215, and the process proceeds to step 220. In step 220, the first CPU determines whether the non-visual flag Xnvs is "0". The non-visual flag Xnvs is set to "1" when the user is not looking at the vehicle VA, and is set to "0" when the user is looking at the vehicle VA. The non-visual flag Xnvs is set to "0" in the initial routine. Further, even when the remote driving control is started, the non-visual flag Xnvs is set to "0".

[0030] If the non-visual flag Xnvs is "0", the first CPU determines "Yes" in step 220 and executes steps 225 and 230. Step 225: The first CPU sets the non-visual flag Xnvs to "1". Step 230: The first CPU sets the timer T to "0". The timer T is a timer for counting the time when the user is in a non-visual state. Thereafter, the process proceeds to step 295, and the first CPU temporarily ends this routine.

[0031] When the non-visual flag Xnvs is "1" when the process proceeds to step 220, the first CPU determines "No" in step 220 and executes steps 235 and 240. Step 235: The first CPU adds "1" to the timer T. Step 240: The first CPU determines whether the timer T is greater than or equal to the first threshold value Tth1. The first threshold value Tth1 is set to a value such that when the timer T reaches the first threshold value Tth1, the non-visual state continues for a first predetermined time.

[0032] If the timer T is less than the first threshold value Tth1 (that is, if the non-visual state has not continued for more than the first predetermined time), the first CPU determines "No" in step 240, and the process proceeds to step 295. In step 295, the first CPU temporarily ends this routine.

[0033] When the timer T is equal to or greater than the first threshold value Tth1 (i.e., when the non-visual state continues for the first predetermined time or longer), the first CPU determines that the non-visual condition is satisfied. In this case, the first CPU determines "Yes" in step 240, and the process proceeds to step 245. In step 245, the first CPU determines whether the timer T is equal to or greater than "the second threshold value Tth2 set to a value larger than the first threshold value Tth1". The second threshold value Tth2 is set to a value such that when the timer T reaches the second threshold value Tth2, the non-visual state continues for the second predetermined time.

[0034] When the timer T is less than the second threshold value Tth2 (i.e., when the non-visual state has not continued for the second predetermined time or longer), the first CPU determines "No" in step 245, and the process proceeds to step 250. In step 250, the first CPU sets the notification flag Xnt to "1".

[0035] The notification flag Xnt is set to "1" when causing the remote operation device 20 to perform visual notification, and is set to "0" when not causing the remote operation device 20 to perform visual notification. In the initial routine, the notification flag Xnt is set to "0". Further, when the remote driving control is started, the notification flag Xnt is also set to "0".

[0036] When the timer T is equal to or greater than the second threshold value Tth2 (i.e., when the non-visual state continues for the second predetermined time or longer), the first CPU determines "Yes" in step 245, and the process proceeds to step 255. In step 255, the first CPU sets the stop flag Xst to "1". The stop flag Xst is set to "1" when stopping the vehicle VA, and is set to "0" when not stopping the vehicle VA. In the initial routine, the stop flag Xst is set to "0". Further, when the remote driving control is started, the stop flag Xst is also set to "0". Thereafter, the process proceeds to step 295, and the first CPU temporarily ends this routine.

[0037] When the stop flag Xst is set to "1", the first CPU sends a stop command to the remote operation device 20. When the remote operation device 20 receives the stop command, it causes the display device 60 to display a stop screen including a restart button. When the user operates the restart button, the remote operation device 20 sends a restart signal. When the first CPU receives the restart signal, it sets the notification flag Xnt and the stop flag Xst to "0" and resumes the remote driving control.

[0038] When the process proceeds to step 215 and it is determined in the visual state determination subroutine that the user is not in a non-visual state, the first CPU determines "No" in step 215, and the process proceeds to step 265.

[0039] In step 265, the first CPU determines whether the non-visual flag Xnvs is "1". If the non-visual flag Xnvs is "1", the first CPU determines "Yes" in step 265 and executes steps 270 to 280.

[0040] Step 270: The first CPU sets the non-visual flag Xnvs to "0". Step 275: The first CPU sets the notification flag Xnt to "0". Step 280: The first CPU sets the timer T to "0". Thereafter, the process proceeds to step 295, and the first CPU temporarily ends this routine.

[0041] When the non-visual flag Xnvs is "0" when the process proceeds to step 265, the CPU determines "No" in step 265, and the process proceeds to step 295. In step 295, the CPU temporarily ends this routine.

[0042] <Visual State Determination Subroutine> When the process proceeds to step 210, the first CPU starts the process from step 300 in FIG. 3 and executes steps 305 to 345.

[0043] Step 305: The first CPU acquires the face image captured by the camera 52 of the remote operation device 20. Step 310: The first CPU acquires a line-of-sight angle θeye representing the direction of the user's line of sight with respect to the central axis CA of the remote operation device 20 based on the face image. Note that it is a virtual line parallel to the longitudinal direction of the smartphone passing through the center of the smartphone which is the remote operation device 20.

[0044] Step 315: The first CPU identifies the current position of the vehicle VA based on the satellite signal received by the GNSS receiver 36. Step 320: The first CPU acquires the current position of the remote operation device 20. Specifically, the CPU of the control unit 50 of the remote operation device 20 (hereinafter referred to as the "second CPU") acquires the current position of the remote operation device 20 based on the satellite signal received by the GNSS receiver 56. The first CPU acquires the current position of the remote operation device 20 from the remote operation device 20.

[0045] Step 325: The first CPU identifies a vehicle azimuth Dvc at which the vehicle VA is located as seen from the remote operation device 20 based on the current position of the vehicle VA and the current position of the remote operation device 20. Note that the vehicle azimuth Dvc is a value that increases clockwise with 0 degrees being north. That is, the vehicle azimuth Dvc is 90 degrees when the vehicle VA is located east as seen from the remote operation device 20, 180 degrees when the vehicle VA is located south as seen from the remote operation device 20, and 270 degrees when the vehicle VA is located west as seen from the remote operation device 20. Step 330: The first CPU identifies an azimuth Dcen of the central axis CA of the remote operation device 20. The azimuth Dcen is a value that increases clockwise with 0 degrees being north, similar to the vehicle azimuth Dvc. The second CPU identifies the azimuth Dcen based on the detection value of the azimuth sensor 54. The first CPU acquires the azimuth Dcen from the remote operation device 20.

[0046] Step 335: The first CPU obtains the vehicle angle θvc, which is the angle with respect to the central axis CA of the direction in which the vehicle VA is located with respect to the remote control device 20, by subtracting the azimuth Dcen from the vehicle azimuth Dvc. Step 340: The first CPU obtains, as the angular difference Δθ, the magnitude of the value obtained by subtracting the vehicle angle θvc from the line-of-sight angle θeye.

[0047] Step 345: The first CPU determines whether or not the angular difference Δθ is less than or equal to a predetermined threshold value Δθth. When the angular difference Δθ is less than or equal to the threshold value Δθth, the user is in a visual state. In this case, the first CPU determines "Yes" in step 345, and the process proceeds to step 350. In step 350, the first CPU determines that the user is in a visual state. Thereafter, the process proceeds to step 395 to temporarily end this routine.

[0048] On the other hand, when the angular difference Δθ is greater than the threshold value Δθth, the user is in a non-visual state. In this case, the first CPU determines "No" in step 345, and the process proceeds to step 355. In step 355, the first CPU determines that the user is in a non-visual state. Thereafter, the process proceeds to step 395 to temporarily end this routine.

[0049] <Monitoring result control routine> When an appropriate time point arrives, the first CPU starts processing from step 400 in FIG. 4, and determines whether or not the execution flag Xexe is "1" in step 405.

[0050] When the execution flag Xexe is "0", the first CPU determines "No" in step 405, and the process proceeds to step 495 and the first CPU temporarily ends this routine. When the execution flag Xexe is "1", the first CPU determines "Yes" in step 405, and the process proceeds to step 410. In step 410, the first CPU determines whether or not the stop flag Xst is "1".

[0051] If the stop flag Xst is "0", the first CPU determines "No" in step 410, and the process proceeds to step 415. In step 415, the first CPU determines whether the notification flag Xnt is "1". If the notification flag Xnt is "0", the first CPU determines "No" in step 415, and the process proceeds to step 495. In step 495, the first CPU temporarily ends this routine.

[0052] On the other hand, if the notification flag Xnt is "1", the first CPU determines "Yes" in step 415, and the process proceeds to step 420. In step 420, the first CPU causes the remote operation device 20 to perform a visual notification.

[0053] Specifically, the first CPU transmits a visual command to the remote operation device 20. When the remote operation device 20 receives the visual command, it performs a visual notification. In the visual notification, the remote operation device 20 causes the display device 60 to display a message "Please look at the vehicle." Note that in the visual notification, the remote operation device 20 may cause the speaker 62 to emit the voice of the above message, or may vibrate the remote operation device 20 by operating the vibration motor 64. Thereafter, the process proceeds to step 495, and the first CPU temporarily ends this routine.

[0054] When the stop flag Xst is "1" when the process proceeds to step 410, the first CPU determines "Yes" in step 410 and executes steps 425 and 430. Step 425: The first CPU controls the power train actuator 40 and the brake actuator 42 to stop the vehicle VA. Step 430: The first CPU causes the remote operation device 20 to perform a stop notification. Specifically, the first CPU transmits a stop command to the remote operation device 20. The remote operation device 20 causes the display device 60 to display a stop screen. In the stop screen, the above restart button and a message "Since you were not looking at the vehicle, the vehicle has been stopped." are displayed. After that, the process proceeds to step 495, and the first CPU temporarily ends this routine.

[0055] According to the present embodiment, when the non-visual state continues for a first predetermined time, the monitoring device (vehicle control device 10) causes the remote operation device 20 to give a visual notification. As a result, when the remote driving control is being performed and the user is not looking at the vehicle VA, a visual notification is given, and as a result, the possibility that the user looks at the vehicle VA can be increased.

[0056] Furthermore, when the non-visual state continues for a second predetermined time, the monitoring device (vehicle control device 10) stops the vehicle VA. Thereby, the possibility that the vehicle VA contacts an obstacle when the user is not looking at the vehicle VA can be reduced.

[0057] Furthermore, the monitoring device (vehicle control device 10) acquires an angular difference Δθ which is the magnitude of the value obtained by subtracting the vehicle angle θvc from the line-of-sight angle θeye acquired based on the face image. That is, the monitoring device (vehicle control device 10) acquires the angular difference Δθ which is the magnitude of the angle formed between the user's line-of-sight direction and the direction in which the vehicle is located as seen from the remote operation device 20. Then, when the angular difference Δθ is larger than a predetermined threshold value Δθth, the monitoring device (vehicle control device 10) determines that it is a non-visual state. Thereby, it is possible to accurately determine whether the user is in a visual state or a non-visual state.

[0058] (First Modified Example) In this modified example, the first CPU determines whether the user is in a visual state or a non-visual state using the vehicle angle θvc without using the line-of-sight angle θeye. Specifically, when the magnitude of the vehicle angle θvc is equal to or less than the threshold angle θth, the first CPU determines that the user is in a visual state, and when the magnitude of the vehicle angle θvc is larger than the threshold angle θth, the first CPU determines that the user is in a non-visual state. Thereby, it becomes unnecessary for the remote operation device 20 to activate the camera 52 to acquire the face image of the user.

[0059] (Second Modified Example) The monitoring device may be the remote operation device 20. In this case, the second CPU executes the routines shown in FIGS. 2 to 4. When the visual state continues for the second predetermined time, the second CPU transmits a stop command to the vehicle control device 10. When the vehicle control device 10 receives the stop command, it stops the vehicle VA.

[0060] (Third Modified Example) Although the line-of-sight angle θeye and the vehicle angle θvc are acquired with reference to the central axis CA, the line-of-sight angle θeye and the vehicle angle θvc may be acquired with reference to a predetermined reference line of the remote operation device 20.

[0061] The vehicle control device 10 is applicable to vehicles such as engine vehicles, hybrid vehicles, plug-in hybrid vehicles, fuel cell vehicles, and electric vehicles. Furthermore, the vehicle control device 10 is applicable to autonomous driving vehicles.

Description of Reference Numerals

[0062] 10…Vehicle control device, 20…Remote operation device, 30…ECU, 36…GNSS receiver, 50…Control unit, 52…Camera, 54…Azimuth sensor, 56…GNSS receiver.

Claims

1. In a monitoring device that monitors a user when remote driving control for driving a vehicle in accordance with an instruction from a user outside the vehicle is being executed, when a non-visual condition that the non-visual state where the user is not looking at the vehicle continues for a first predetermined time or longer is satisfied, the monitoring device is configured to give a visual notification for prompting the user to look at the vehicle. Monitoring device.

2. In the monitoring device according to Claim 1, when the non-visual state continues for a second predetermined time or longer that is longer than the first predetermined time, it is configured to stop the vehicle. Monitoring device.

3. In the monitoring device according to Claim 1, the remote driving control is executed in accordance with an instruction transmitted by the remote operation device when the user operates the remote operation device, the monitoring device specifies a line-of-sight angle of the user's line of sight with respect to a predetermined reference line of the remote operation device based on the face image of the user, specifies a vehicle angle with respect to the reference line in the direction in which the vehicle is located with respect to the remote operation device, and when the magnitude of the difference between the line-of-sight angle and the vehicle angle is greater than a predetermined threshold value, it determines that the user is in the non-visual state. Monitoring device configured as such.

4. In the monitoring device according to Claim 1, the remote driving control is executed in accordance with an instruction transmitted by the remote operation device when the user operates the remote operation device, the monitoring device determines that the user is in the non-visual state when the vehicle angle with respect to a predetermined reference line of the remote operation device in the direction in which the vehicle is located with respect to the remote operation device is greater than a predetermined threshold value. Monitoring device configured as such.

5. In the monitoring device according to any one of Claims 3 and 4, the monitoring device is configured to be mounted on the remote operation device or the vehicle. Monitoring device.

Images