Vehicle control device, vehicle control method and computer program

The vehicle control system adjusts the driver's field of view based on speed and road conditions to enhance target recognition and prevent collisions by setting a variable cone angle and implementing safety measures.

JP2024162128APending Publication Date: 2024-11-21TOYOTA JIDOSHA KK +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2023077377
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-09
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

As vehicle speed increases, the driver's field of vision narrows, making it difficult to recognize targets outside the vehicle, posing a risk of missed recognition of critical objects like other vehicles or pedestrians.

Method used

A vehicle control system that adjusts the driver's field of view based on vehicle speed, road type, and road shape by setting a cone with a variable apex angle, determining target visibility, and taking safety measures if the field of view does not overlap the target's projected location.

Benefits of technology

Ensures the driver's field of view is appropriately set, enhancing target recognition and reducing the risk of collisions by providing notifications or automatic vehicle operations to avoid obstacles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024162128000001_ABST
    Figure 2024162128000001_ABST
Patent Text Reader

Abstract

To enable determination of whether a driver recognizes an object outside a vehicle to be made taking into account a change in driver's field of view.SOLUTION: A control device 4: identifies an object which exists around a vehicle 100 and requires attention based on surrounding data representing a surrounding situation of the vehicle 100; detects a position of a viewpoint and a sight direction of a driver based on driver data representing a state of the driver of the vehicle 100; sets a field of view of the driver in the sight direction of the driver based on a speed of the vehicle 100 and a type or a shape of a road; places a virtual plane on which the object is projected at a position of the object opposite to the vehicle; determines whether the field of view of the driver overlaps with a location on the virtual plane on which the object is projected; and takes a safety measure if the field of view of the driver does not overlap with the location on which the object is projected.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a vehicle control device, a vehicle control method, and a computer program. [Background technology]

[0002] Patent Document 1 discloses a conventional warning device capable of issuing an alert depending on the driver's gaze state. The warning device is configured to calculate the gaze point at which the driver is gazing based on gaze information indicating the direction of the driver's gaze and eye position information indicating the position of the driver's eyeball on a three-dimensional coordinate system, determine whether the driver is gazing at an object based on the object to be gazed at and the gaze point, and issue an alert if it is determined that the driver is not gazing at the object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-185218 A Summary of the Invention [Problem to be solved by the invention]

[0004] The driver's field of view tends to become narrower as the vehicle speed increases. Therefore, the driver tends to have difficulty recognizing targets outside the vehicle, such as other vehicles and pedestrians, as the vehicle speed increases. Therefore, if the driver does not determine whether he or she can recognize targets outside the vehicle while taking into consideration such changes in the field of view, there is a risk that the driver will determine that he or she can recognize targets outside the vehicle that the driver should be paying attention to, even though the driver cannot recognize the targets.

[0005] The present invention has been made in response to such problems, and aims to make it possible to determine whether a driver is aware of targets outside the vehicle, taking into account changes in the driver's field of vision. [Means for solving the problem]

[0006] The gist of the present disclosure is as follows.

[0007] (1) A vehicle control device configured to detect targets that should be watched around the vehicle based on surrounding data that represents the conditions around the vehicle, detect the driver's viewpoint position and line of sight based on driver data that represents the state of the vehicle's driver, set the driver's field of view toward the driver's line of sight based on the vehicle speed, road type, or road shape, set a virtual plane onto which the target is projected at the target position so as to face the vehicle, determine whether the driver's field of view overlaps the projection point of the target on the virtual plane, and take safety measures if the driver's field of view does not overlap the projection point of the target.

[0008] (2) A vehicle control device as described in (1) above, which sets a cone having a predetermined apex angle with the driver's viewpoint as its apex and a line-of-sight vector extending from the viewpoint toward the line of sight as its central axis as the driver's field of view, and changes the apex angle based on the vehicle speed, road type, or road shape.

[0009] (3) A vehicle control device as described in (2) above, in which when the vehicle speed is high, the apex angle is set to a smaller value than when the vehicle speed is low.

[0010] (4) The vehicle control device according to (2) above, wherein when the road type is a road with a high average vehicle speed, the apex angle is set to a smaller value than when the road type is a road with a low average vehicle speed.

[0011] (5) A vehicle control device as described in (2) above, in which when the road shape is such that driving at a relatively high speed is possible, the apex angle is set to a smaller value than when the road shape is such that driving at a relatively low speed is necessary.

[0012] (6) A vehicle control device as described in any one of (1) to (5) above, in which the vehicle has an output device for notifying the driver and is configured to notify the driver via the output device as a safety measure.

[0013] (7) A vehicle control device as described in any one of (1) to (6) above, configured to automatically perform driving operations of the vehicle so as to avoid targets as a safety measure.

[0014] (8) A method of controlling a vehicle using a control device, comprising: detecting targets to be watched around the vehicle based on surrounding data representing the conditions around the vehicle; detecting the driver's viewpoint position and line of sight based on driver data representing the state of the driver of the vehicle; setting the driver's field of view toward the driver's line of sight based on the vehicle speed, road type, or road shape; setting a virtual plane onto which the target is projected at the target position so as to face the vehicle; determining whether the driver's field of view overlaps the projection point of the target on the virtual plane; and taking safety measures if the driver's field of view does not overlap the projection point of the target.

[0015] (9) A computer program for a vehicle control device, which causes the control device to detect targets that should be watched around the vehicle based on surrounding data that represents the situation around the vehicle, detect the driver's viewpoint position and line of sight based on driver data that represents the driver's state, set the driver's field of view toward the driver's line of sight based on the vehicle speed, road type, or road shape, set a virtual plane onto which the target is projected at the target position so as to face the vehicle, determine whether the driver's field of view overlaps the projection point of the target on the virtual plane, and cause the vehicle to take safety measures if the driver's field of view does not overlap the projection point of the target. Effect of the Invention

[0016] According to these aspects of the present invention, the driver's field of view can be set to an appropriate range according to the vehicle speed, road type, or road shape, so that a determination of whether the driver is aware of targets outside the vehicle can be made taking into account changes in the driver's field of view. [Brief description of the drawings]

[0017] [Figure 1] 1 is a schematic system configuration diagram of a vehicle according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of a control device. [Diagram 3] 10 is a flowchart illustrating details of a notification process according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram illustrating a driver's field of view. [Diagram 5] FIG. 13 is a diagram showing an example of a field of view setting table for setting the apex angle of the viewing cone based on the vehicle speed. [Figure 6A] 1 is a diagram illustrating an example of a method for determining whether or not a driver can recognize a gaze target; [Figure 6B] 1 is a diagram illustrating an example of a method for determining whether or not a driver can recognize a gaze target; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, like components are designated by like reference numerals.

[0019] FIG. 1 is a system schematic diagram of a vehicle 100 in accordance with one embodiment of the present invention.

[0020] The vehicle 100 includes a surrounding sensor 1, a driver sensor 2, an output device 3, and a control device 4. The surrounding sensor 1, the driver sensor 2, the output device 3, and the control device 4 are communicatively connected via an in-vehicle network 5 that complies with a standard such as a controller area network. The vehicle 100 may be a vehicle with an automatic driving function or a driving assistance function, or may be a manually driven vehicle.

[0021] The surrounding sensor 1 is a sensor for generating surrounding data that indicates the situation around the vehicle 100. In this embodiment, the surrounding sensor 1 includes one or more external cameras for capturing images of the surroundings of the vehicle 100. The external cameras capture images of the surroundings of the vehicle 100 at a predetermined frame rate (for example, 10 [Hz] to 40 [Hz]) and generate surrounding images showing the surroundings of the vehicle 100. Each time the external cameras generate a surrounding image, they transmit the generated surrounding image to the control device 4 as surrounding data.

[0022] Instead of or in addition to the external camera, a distance measuring sensor that measures the distance to targets such as other vehicles, pedestrians, and bicycles present around the vehicle 100 may be provided as the surrounding sensor 1. Examples of distance measuring sensors include a LiDAR (Light Detection And Ranging) that irradiates radar light and measures distance based on the reflected light, and a millimeter wave radar sensor that irradiates radio waves and measures distance based on the reflected waves.

[0023] The driver sensor 2 is a sensor for generating driver data representing the state of the driver. In this embodiment, the driver sensor 2 includes a driver monitor camera for capturing an image of the driver's appearance including the driver's face. The driver monitor camera captures the driver's appearance at a predetermined frame rate (for example, 10 [Hz] to 40 [Hz]) and generates an appearance image showing the driver's appearance. Every time the driver monitor camera generates an appearance image of the driver, it transmits the generated appearance image to the control device 4 as driver data.

[0024] The output device 3 is a device for notifying the driver of the vehicle 100 through the driver's bodily senses (for example, vision, hearing, touch, etc.). In this embodiment, the output device 3 includes a display (for example, a meter display, a center display, or a head-up display) arranged at a position visible to the driver, and a speaker. The display displays information (for example, text information or image information) according to a display signal output from the control device 4. The speaker outputs sound according to an audio signal output from the control device 4.

[0025] The control device 4 is an ECU (Electronic Control Unit) including a communication unit 41, a storage unit 42, and a processing unit 43 (see FIG. 2), and is configured at least to identify a target outside the vehicle to be watched (a target to be watched) and to notify the driver according to the driver's recognition state of the target via the output device 3. In addition to the above-mentioned surrounding data and driver data, various types of data can be input to the control device 4, such as vehicle position data acquired by a positioning sensor and vehicle speed data acquired by a vehicle speed sensor.

[0026] FIG. 2 is a diagram showing a hardware configuration of the control device 4. As shown in FIG.

[0027] The control device 4 includes a communication unit 41, a storage unit 42, and a processing unit 43.

[0028] The communication unit 41 includes an interface circuit for connecting the control device 4 to the in-vehicle network 5. The communication unit 41 supplies data received from the outside (peripheral data, driver data, etc.) to the processing unit 43. The communication unit 41 also transmits the display signal and audio signal output from the processing unit 43 to the output device 3.

[0029] The storage unit 42 has a storage medium such as a hard disk drive (HDD), a solid disk drive (SSD), or a semiconductor memory, and stores various computer programs and data used for processing by the processing unit 43.

[0030] The processing unit 43 has one or more CPUs (Central Processing Units) and their peripheral circuits, and is, for example, a processor, which executes various computer programs stored in the storage unit 42. Hereinafter, an example of processing executed by the processing unit 43, and therefore the control device 4, will be described with reference to FIG.

[0031] FIG. 3 is a flowchart illustrating details of the notification process according to this embodiment.

[0032] In step S1, the control device 4 detects targets existing around the vehicle 100 based on the surrounding data received from the surrounding sensor 1, specifically, on a surrounding image received from an external camera.

[0033] In this embodiment, the control device 4 detects the area in which the target is shown in the surrounding image and the type of the target shown in the area by sequentially inputting the surrounding images received from the external camera to the classifier. The classifier can be, for example, a convolutional neural network (CNN) having a plurality of convolution layers connected in series from the input side to the output side. The control device 4 estimates the distance from the external camera to the target using the standard size of the target stored in the storage unit 42 for each type of target and the size of the target detected in the surrounding image, and calculates the position and speed of the target by tracking the target detected in the surrounding image in a time series. Note that the method of detecting the target is not limited to such a method, and it is possible to detect the target using various known methods.

[0034] In step S2, the control device 4 identifies an object to be watched (hereinafter referred to as a "gaze target object") among objects present around the vehicle 100. The gaze target object may be, for example, another vehicle approaching the vehicle 100, a pedestrian, a bicycle, or the like, that is, an object that may collide with the vehicle 100. However, the present invention is not limited to this, and a predetermined object such as a pedestrian or a bicycle may be identified as the gaze target object, or an object whose distance from the vehicle 100 in a direction perpendicular to the traveling direction of the vehicle 100 (lateral direction) is smaller than a predetermined threshold value may be identified as the gaze target object.

[0035] In step S3, the control device 4 detects the driver's viewpoint position and line of sight direction based on the driver data received from the driver sensor 2, specifically, based on the external image of the driver received from the driver monitor camera.

[0036] In this embodiment, the control device 4 detects the position of the pupil center, which is the moving point (i.e., the driver's viewpoint position), and the position of the Purkinje image (corneal reflection image), which is the reference point, by sequentially performing image processing on the face images received from the driver monitor camera 21, and detects the driver's line of sight direction based on the positional relationship between the Purkinje image and the pupil center (the position of the moving point relative to the reference point). Note that the method of detecting the driver's viewpoint position and line of sight direction is not limited to this method, and various known methods may be used for detection.

[0037] In step S4, the control device 4 sets the driver's field of view. As described above, the driver's field of view tends to become narrower as the vehicle speed increases. Therefore, in this embodiment, as shown in FIG. 4, the driver's field of view is assumed to be a viewing cone having a predetermined apex angle (angle of view) with the driver's viewpoint position as the apex and a line-of-sight vector extending from the viewpoint position toward the line-of-sight direction as the central axis, and the apex angle of the viewing cone is changed according to the vehicle speed. The apex of the field of view does not necessarily have to be the driver's viewpoint position, and may be, for example, the center position of the face.

[0038] In this embodiment, the control device 4 refers to a field of view setting table shown in Fig. 5, which is set in advance by experiments or the like, and sets the apex angle of the viewing cone based on the vehicle speed. As shown in Fig. 5, the apex angle of the viewing cone is basically set smaller when the vehicle speed is high than when it is low. As a result, the apex angle of the viewing cone, and therefore the radius of the base of the viewing cone, basically become smaller as the vehicle speed increases, so that the driver's field of view can be narrowed.

[0039] It should be noted that the shape of the field of view setting table shown in FIG. 5 is merely an example, and the apex angle may be gradually decreased as the vehicle speed increases as shown in FIG. 5, or may be decreased in stages as the vehicle speed increases.

[0040] In this embodiment, the shape of the field of view is a cone as shown in Fig. 4, but the shape of the field of view is not limited to a cone and may be other cones such as an elliptical cone or a polygonal cone. The field of view cone shown in Fig. 4 is a so-called right cone whose central axis passes through the center of gravity of the base, but it may be a so-called oblique cone whose central axis does not pass through the center of gravity of the base.

[0041] In this embodiment, the apex angle of the visual cone is set based on the vehicle speed, but this is not limiting, and the type of road on which the vehicle is currently traveling (general road, expressway, etc.) may be determined, and the apex angle of the visual cone may be set based on the road type. In this case, for example, the apex angle of the visual cone may be set to a predetermined value that is preset for each road type according to the average vehicle speed for each road type, and the higher the average vehicle speed of the road, the smaller the predetermined value may be.

[0042] Also, the shape of the road ahead (straight, curved, etc.) may be determined, and the apex angle of the viewing cone may be set based on the road shape. In this case, if the shape of the road ahead requires relatively slow driving (e.g. curved), the apex angle may be set larger than if the shape allows relatively high speed driving (e.g. straight).

[0043] In step S5, the control device 4 judges whether or not the driver can recognize the gaze target. If the driver can recognize the gaze target, the control device 4 ends this processing. On the other hand, if the driver cannot recognize the gaze target, the control device 4 proceeds to the processing of step S6.

[0044] In this embodiment, the control device 4 sets a virtual plane on which the gaze target is projected so as to be perpendicular to the vehicle travel direction, for example, as shown in Fig. 6A, and if the driver's field of view overlaps the projection location of the gaze target on the virtual plane, it is determined that the driver can recognize the gaze target, and on the other hand, if the driver's field of view does not overlap the projection location of the gaze target on the virtual plane, it is determined that the driver cannot recognize the gaze target. The speed of the vehicle in Fig. 6A is lower than the speed of the vehicle in Fig. 6B, and therefore the field of view in Fig. 6A is wider than the field of view in Fig. 6B. In Figs. 6A and 6B, the virtual plane is set at a position that is the shortest distance from the driver's viewpoint to the gaze target, but is not limited thereto, and can be set at a position that includes the gaze target, for example.

[0045] In step S6, the control device 4 takes a safety measure to avoid a collision with the gazed-up object. In this embodiment, as a safety measure, a notification is given to the driver via the output device 3. The notification method is not particularly limited, and for example, the driver may be notified by making a voice announcement that the gazed-up object cannot be recognized, by issuing a warning sound, by making full use of image information or text information, or by a combination of these.

[0046] In addition, if vehicle 100 is a vehicle capable of automatically performing at least some of the driving operations related to acceleration, steering, and braking (e.g., a vehicle with an automatic driving function or a vehicle with a driving assistance function), as a safety measure, the driving operations may be automatically performed to avoid a collision with the object being watched.

[0047] The vehicle control device 4 according to the present embodiment described above is configured to detect objects to be gazed upon that are present around the vehicle based on surrounding data representing the conditions around the vehicle, detect the driver's viewpoint position and line of sight based on driver data representing the driver's state, set the driver's field of view based on the vehicle speed, road type, or road shape, set a virtual plane onto which the object to be gazed upon is projected at the position of the object to be gazed upon so as to face the vehicle, determine whether the driver's field of view overlaps the projection point of the object to be gazed upon on the virtual plane, and take safety measures if the driver's field of view does not overlap the projection point of the object to be gazed upon.

[0048] Specifically, the control device 4 is configured to set a cone having a predetermined apex angle with the driver's viewpoint as the apex and a line-of-sight vector extending from the viewpoint toward the line-of-sight direction as the driver's field of view, and to change the apex angle based on the vehicle speed, road type, or road shape. For example, when the vehicle speed is high, the apex angle can be set to a smaller value than when the vehicle speed is low. For example, when the road type is a road with a high average vehicle speed, the apex angle can be set to a smaller value than when the road type is a road with a low average vehicle speed. For example, when the road shape is a shape that allows driving at a relatively high speed, the apex angle can be set to a smaller value than when the road shape is a shape that requires driving at a relatively low speed.

[0049] This allows the determination of whether the driver is aware of an object being gazed upon outside the vehicle to be made taking into account changes in the driver's field of vision, thereby improving the accuracy of determining whether the driver is aware of an object being gazed upon.

[0050] In addition, as a safety measure, the control device 4 can provide notification via an output device or automatically perform driving operations of the vehicle to avoid the watched object, thereby preventing a collision with the watched object.

[0051] Although the embodiments of the present invention have been described above, the above-mentioned embodiments merely show some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above-mentioned embodiments.

[0052] For example, in the above embodiment, the computer program (notification process) executed in the control device 4 may be provided in a form recorded on a computer-readable portable recording medium, such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. [Explanation of symbols]

[0053] 3 Output Device 4. Control device 100 vehicles

Claims

1. A control device for a vehicle, Detecting a target to be watched around the vehicle based on surrounding data representing a situation around the vehicle; Detecting a line of sight direction of the driver based on driver data representing a state of the driver of the vehicle; setting a field of view of the driver in a line of sight direction of the driver based on a speed of the vehicle, a road type, or a road shape; A virtual plane on which the target is projected is set at the position of the target so as to face the vehicle, and it is determined whether the driver's field of view overlaps with the projection location of the target on the virtual plane; configured to take a safety measure if the driver's field of view does not overlap the projection location of the target; Vehicle control device.

2. A cone having a predetermined apex angle with the driver's viewpoint as its apex and a line-of-sight vector extending from the driver's viewpoint toward the line of sight of the driver as its central axis is set as the driver's field of view, and the apex angle is changed based on the vehicle speed, road type, or road shape. The vehicle control device according to claim 1 .

3. When the vehicle speed is high, the apex angle is set to a smaller value than when the vehicle speed is low. The vehicle control device according to claim 2.

4. When the road type is a road with a high average vehicle speed, the apex angle is set to a smaller value than when the road type is a road with a low average vehicle speed. The vehicle control device according to claim 2.

5. When the road shape is a shape that allows travel at a relatively high speed, the apex angle is set to a smaller value than when the road shape is a shape that requires travel at a relatively low speed. The vehicle control device according to claim 2.

6. The vehicle has an output device for notifying the driver, As the safety measure, a notification is given via the output device. The vehicle control device according to claim 1 .

7. As the safety measure, the vehicle is automatically operated to avoid the target. The vehicle control device according to claim 1 .

8. A method for controlling a vehicle by a control device, comprising: Detecting a target to be watched around the vehicle based on surrounding data representing a situation around the vehicle; Detecting a line of sight direction of the driver based on driver data representing a state of the driver of the vehicle; setting a field of view of the driver in a line of sight direction of the driver based on a speed of the vehicle, a road type, or a road shape; A virtual plane on which the target is projected is set at the position of the target so as to face the vehicle, and it is determined whether the driver's field of view overlaps with the projection location of the target on the virtual plane; configured to take a safety measure if the driver's field of view does not overlap the projection location of the target; How to control a vehicle.

9. A computer program for a vehicle control device, comprising: The control device includes: Detecting a target to be watched that exists around the vehicle based on surrounding data that represents a situation around the vehicle; Detecting a viewpoint position and a line of sight direction of the driver based on driver data representing a state of the driver of the vehicle; A visual field of the driver is set in a direction of the driver's line of sight based on a speed of the vehicle, a road type, or a road shape; A virtual plane on which the target is projected is set at the position of the target so as to face the vehicle, and a determination is made as to whether the driver's field of view overlaps with the projection location of the target on the virtual plane; If the driver's field of view does not overlap the projection location of the target, a safety measure is taken by the vehicle. A computer program for making something happen.

Citation Information

Patent Citations

  • Driving support device

    JP2008257582A

  • Route guidance control device and route guidance control method

    JP2019200125A

  • Recognition device, recognition method, and program

    JP2020201670A

  • Map display control device and map display control method

    WO2021070238A1

  • Alarm device

    JP2019185218A