Vehicle control device, vehicle control method, and non-transitory computer recording medium

By detecting the target objects around the vehicle and the driver's viewpoint, setting a virtual plane according to the vehicle speed and road conditions, determining whether the driver can identify the target objects, and taking safety measures, solving the problem of identification difficulties caused by the narrowing of the field of vision during high-speed driving. Improves recognition accuracy and security.

CN118928418APending Publication Date: 2024-11-12TOYOTA JIDOSHA KK +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410550083.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-09
Filing Date
2024-05-06
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The driver's field of vision shrinks with the change of vehicle speed, making it difficult to identify other vehicles or pedestrians outside the vehicle when driving at high speed. The prior art has failed to effectively consider the change in the field of vision to make judgments.

Method used

By detecting the target object around the vehicle and the driver's viewpoint position and line of sight direction, setting the virtual plane according to the vehicle speed, road type or shape, and determining whether the driver's field of vision covers the target object. If it is not covered, safety measures will be taken. Or automatically avoid.

Benefits of technology

The changes in the driver's field of vision are effectively considered, the accuracy of identifying external target objects of the vehicle is improved, and the risk of collision during high-speed driving is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118928418A_ABST
    Figure CN118928418A_ABST
Patent Text Reader

Abstract

The present invention makes it possible to determine whether a driver recognizes a target outside a vehicle in consideration of a change in the field of view of the driver. The control device (4) is configured to detect a target object to be watched in the periphery of the vehicle (100) on the basis of periphery data indicating the condition of the periphery of the vehicle (100), to detect the viewpoint position and the line-of-sight direction of a driver on the basis of driver data indicating the state of the driver of the vehicle (100), and to control the vehicle (100) on the basis of the speed of the vehicle (100), the road type, or the road shape. The visual field of the driver is set in the sight line direction of the driver, a virtual plane on which the target object is projected is set at the position of the target object so as to face the vehicle, and whether the visual field of the driver overlaps the projection site of the target object on the virtual plane is determined, and if the visual field of the driver does not overlap the projection site of the target object, the visual field of the driver is determined. If yes, safety measures are taken.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a vehicle control device, a vehicle control method and a non-transitory computer recording medium. Background Art

[0002] Patent document 1 discloses an alarm device as a previous device capable of informing the driver based on the driver's gaze status. The alarm device is configured to calculate the gaze point that the driver is looking at based on line of sight information indicating the direction of the driver's line of sight and eye position information indicating the driver's eye position in three-dimensional coordinates, determine whether the driver is looking at the object based on the object to be looked at and the gaze point, and inform the driver if it is determined that the driver is not looking. Prior art literature Patent Literature

[0003] Patent Document 1: Japanese Patent Application Publication No. 2019-185218 Summary of the invention Problems to be solved by the invention

[0004] The driver's field of vision tends to become narrower as the vehicle speed increases. Therefore, the driver tends to have a harder time recognizing objects such as other vehicles and pedestrians outside the vehicle as the vehicle speed increases. Therefore, if the driver is judged to have successfully recognized an object outside the vehicle without considering such a change in the field of vision, it is possible that the driver may be judged to have successfully recognized an object outside the vehicle even though the driver has failed to recognize the object that should be observed.

[0005] The present invention has been made with attention paid to such a problem, and an object of the present invention is to enable determination of whether the driver has recognized an object outside the vehicle in consideration of changes in the driver's field of vision. Means for solving problems

[0006] The gist of the present disclosure is as follows.

[0007] (1) A vehicle control device, configured to detect a target object to be observed existing around the vehicle based on surrounding data indicating the condition of the surroundings of the vehicle, detect the viewpoint position and line of sight direction of the driver based on driver data indicating the status of the driver of the vehicle, set the driver's field of view in the direction of the driver's line of sight based on the speed of the vehicle, the road type or the road shape, set a virtual plane on which the target object is projected relative to the vehicle at the position of the target object, and determine whether the driver's field of view overlaps with the projection part of the target object on the virtual plane, and take safety measures if the driver's field of view does not overlap with the projection part of the target object.

[0008] (2) A control device for a vehicle according to (1) above, wherein a cone with a predetermined vertex angle having a viewpoint of the driver as the vertex and a line of sight vector extending from the viewpoint toward the line of sight as the central axis is set as the driver's field of view, and the vertex angle is changed based on the speed of the vehicle, the type of road, or the shape of the road.

[0009] (3) The vehicle control device according to (2) above, wherein when the speed of the vehicle is high, the vertex angle is set to a smaller value than when the speed of the vehicle is low.

[0010] (4) The vehicle control device according to (2) above, wherein when the road type is a road with a high average vehicle speed, the vertex angle is set to a smaller value than when the road type is a road with a low average vehicle speed.

[0011] (5) A vehicle control device according to (2) above, wherein when the road shape allows relatively high-speed travel, the vertex angle is set to a smaller value than when the road shape requires relatively low-speed travel.

[0012] (6) A vehicle control device according to any one of (1) to (5) above, wherein the vehicle has an output device for notifying the driver, and the control device is configured to make the notification via the output device as a safety measure.

[0013] (7) A vehicle control device according to any one of (1) to (6) above, wherein the control device is configured to automatically perform a driving operation of the vehicle in a manner avoiding a target object as a safety measure.

[0014] (8) A vehicle control method, which is a vehicle control method based on a control device, detecting a target object to be watched that exists around the vehicle based on surrounding data indicating the condition of the surroundings of the vehicle, detecting the viewpoint position and line of sight direction of the driver based on driver data indicating the state of the driver of the vehicle, setting the driver's field of view in the direction of the driver's line of sight based on the speed of the vehicle, the road type or the road shape, setting a virtual plane onto which the target object is projected in a manner relative to the vehicle at the position of the target object, and determining whether the driver's field of view overlaps with the projection part of the target object on the virtual plane, and taking safety measures if the driver's field of view does not overlap with the projection part of the target object.

[0015] (9) A non-transitory computer recording medium comprising a computer program for a vehicle control device, the computer program being used to cause the control device to execute the following operations: detecting a target object to be observed existing in the periphery of the vehicle based on periphery data indicating the condition of the periphery of the vehicle, detecting the viewpoint position and line of sight direction of the driver based on driver data indicating the state of the driver, setting the driver's field of view in the direction of the driver's line of sight based on the speed of the vehicle, the type of road or the shape of the road, setting a virtual plane onto which the target object is projected relative to the vehicle at the position of the target object, and determining whether the driver's field of view overlaps with the projection portion of the target object on the virtual plane, and causing the vehicle to take safety measures if the driver's field of view does not overlap with the projection portion of the target object. Effects of the Invention

[0016] According to these aspects of the present invention, the driver's field of view can be set to an appropriate range according to the vehicle speed, road type or road shape, so that the driver can determine whether he recognizes an object outside the vehicle in consideration of changes in the driver's field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic system configuration diagram of a vehicle according to one embodiment of the present invention. Figure 2 It is a diagram showing the hardware configuration of the control device. Figure 3 This is a flowchart for explaining the details of the notification process according to one embodiment of the present invention. Figure 4 This is a diagram for explaining the driver's field of view. Figure 5 This is a diagram showing an example of a visual field setting table for setting the vertex angle of the visual cone based on the speed of the vehicle. Fig. 6A 1 is a diagram for explaining an example of a method for determining whether the driver has successfully recognized the fixation target object. Figure 6B 1 is a diagram for explaining an example of a method for determining whether the driver has successfully recognized the fixation target object. DETAILED DESCRIPTION

[0018] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, the same reference numerals are given to the same components.

[0019] Figure 1 1 is a system schematic diagram of a vehicle 100 according to an embodiment of the present invention.

[0020] The vehicle 100 includes a peripheral sensor 1, a driver sensor 2, an output device 3, and a control device 4. The peripheral sensor 1, the driver sensor 2, the output device 3, and the control device 4 are communicatively connected via an in-vehicle network 5 conforming to the controller area network standard. The vehicle 100 may be a vehicle with an automatic driving function or a driving assistance function, or may be a manually driven vehicle.

[0021] The surrounding sensor 1 is a sensor for generating surrounding data indicating the state of the surroundings of the vehicle 100. In the present embodiment, one or more external cameras for photographing the surroundings of the vehicle 100 are provided as the surrounding sensor 1. The external camera photographs the surroundings of the vehicle 100 at a predetermined frame rate (e.g., 10 [Hz] to 40 [Hz]) and generates a surrounding image showing the surroundings of the vehicle 100. Each time a surrounding image is generated, the external camera transmits the generated surrounding image to the control device 4 as surrounding data.

[0022] In addition, instead of or in addition to the external camera, a distance measuring sensor for measuring the distance to other vehicles, pedestrians, bicycles, and other targets existing around the vehicle 100 may be provided as the surrounding sensor 1. Examples of the distance measuring sensor include a laser radar (LiDAR; Light Detection And Ranging) that irradiates laser light and measures the distance based on the reflected light, and a millimeter wave radar sensor that irradiates radio waves and measures the distance based on the reflected waves.

[0023] The driver sensor 2 is a sensor for generating driver data indicating the state of the driver. In the present embodiment, a driver monitoring camera for photographing the appearance of the driver including the driver's face is provided as the driver sensor 2. The driver monitoring camera photographs the appearance of the driver at a predetermined frame rate (e.g., 10 [Hz] to 40 [Hz]) to generate an appearance image in which the appearance of the driver is reflected. Each time an appearance image of the driver is generated, the driver monitoring camera sends the generated appearance image to the control device 4 as driver data.

[0024] The output device 3 is a device for notifying the driver of the vehicle 100 through the driver's physical senses (such as vision, hearing, and touch). In this embodiment, the output device 3 includes a display (such as an instrument display, a central display, or a head-up display, etc.) and a speaker arranged at a position visible to the driver. The display displays information (such as text information, image information) corresponding to the display signal output from the control device 4. The speaker outputs a voice corresponding to the voice signal output from the control device 4.

[0025] The control device 4 is an ECU (Electronic Control Unit) including a communication unit 41, a storage unit 42, and a processing unit 43 (see Figure 2 ), is at least configured to be able to determine a target object outside the vehicle that should be watched (watched target object), and notify the driver via the output device 3 based on the driver's recognition status of the target object. In addition to the aforementioned surrounding data and driver data, the control device 4 can also input various data such as vehicle position data obtained by a positioning sensor and vehicle speed data obtained by a vehicle speed sensor.

[0026] Figure 2 It is a diagram showing the hardware configuration of the control device 4 .

[0027] The control device 4 includes a communication unit 41 , a storage unit 42 , and a processing unit 43 .

[0028] The communication unit 41 includes an interface circuit for connecting the control device 4 to the in-vehicle network 5. The communication unit 41 provides data (peripheral data, driver data, etc.) received from the outside to the processing unit 43. In addition, the communication unit 41 sends the display signal and the audio signal output from the processing unit 43 to the output device 3.

[0029] The storage unit 42 has a storage medium such as a HDD (Hard Disk Drive), an SSD (Solid Disk Drive), or a semiconductor memory, and stores various computer programs and data used in the processing of the processing unit 43 .

[0030] The processing unit 43 has one or more CPUs (Central Processing Units) and their peripheral circuits, and executes various computer programs stored in the storage unit 42, such as a processor. Figure 3 , an example of processing executed in the processing unit 43 and further in the control device 4 is described.

[0031] Figure 3 This is a flowchart for explaining the details of the notification process according to this embodiment.

[0032] In step S1 , the control device 4 detects a target object existing around the vehicle 100 based on the surrounding data received from the surrounding sensor 1 , specifically, a surrounding image received from an external camera.

[0033] In the present embodiment, the control device 4 detects the area representing the target object and the type of the target object represented by the area in the surrounding image by sequentially inputting the surrounding image received from the external camera into the discriminator. The discriminator may be, for example, a convolutional neural network (CNN) having a plurality of convolutional layers connected in series from the input side to the output side. Then, the control device 4 estimates the distance from the external camera to the target object by using the standard size of the target object stored in the storage unit 42 according to each type of the target object and the size of the target object detected in the surrounding image, and tracks the target object detected in the surrounding image in a time series, thereby calculating the position and speed of the target object. In addition, the method for detecting the target object is not limited to this method, and detection may be performed using various known methods.

[0034] In step S2, the control device 4 determines the target object to be watched among the targets existing around the vehicle 100 (hereinafter referred to as the "watch target target"). The watch target target can be set to, for example, other vehicles, pedestrians, bicycles, etc. approaching the vehicle 100, that is, targets that may collide with the vehicle 100. However, it is not limited to this, for example, predetermined targets such as pedestrians and bicycles can be determined as the watch target targets, and targets whose interval with the vehicle 100 in the direction orthogonal to the traveling direction of the vehicle 100 (lateral direction) is less than a predetermined threshold can be determined as the watch target targets.

[0035] In step S3 , the control device 4 detects the viewpoint position and the line of sight direction of the driver based on the driver data received from the driver sensor 2 , specifically, the driver's appearance image received from the driver monitoring camera.

[0036] In this embodiment, the control device 4 detects the position of the pupil center as a moving point (i.e., the driver's viewpoint position) and the position of the Purkinje image (corneal reflection image) as a reference point by sequentially performing image processing on the facial image received from the driver monitoring camera 21, and detects the driver's line of sight direction based on the positional relationship between the Purkinje image and the pupil center (the position of the moving point relative to the reference point). In addition, the detection method of the driver's viewpoint position and line of sight direction is not limited to this method, and detection can be performed using various known methods.

[0037] In step S4, the control device 4 sets the driver's field of view. As described above, the driver's field of view tends to become narrower as the speed of the vehicle increases. Figure 4As shown, a visual cone having a predetermined vertex angle (field of view angle) with the driver's viewpoint position as the vertex and a sight vector extending from the viewpoint position toward the sight direction as the central axis is assumed as the driver's field of view, and the vertex angle of the visual cone is changed according to the speed of the vehicle. The vertex of the field of view does not have to be the driver's viewpoint position, and can also be set to, for example, the center position of the face.

[0038] In this embodiment, the control device 4 refers to the preset Figure 5 The field of view setting table shown in FIG. 1 sets the vertex angle of the vision cone based on the speed of the vehicle. Figure 5 As shown, basically when the vehicle speed is high, the vertex angle of the visual cone is smaller than when the vehicle speed is low. Therefore, basically, the higher the vehicle speed, the smaller the vertex angle of the visual cone and the radius of the bottom surface of the visual cone, so the driver's field of vision can be narrowed.

[0039] also, Figure 5 The shape of the field of view setting table shown is only an example, and it can also be Figure 5 As shown, the vertex angle gradually decreases as the speed of the vehicle increases, but the present invention is not limited thereto, and the vertex angle may be decreased in stages as the speed of the vehicle increases.

[0040] In addition, in this embodiment, if Figure 4 As shown, the shape of the field of view is a cone, but the shape of the field of view is not limited to a cone, and can also be an elliptical cone, a polygonal cone or other cones. In addition, Figure 4 The visual cone shown is a so-called right cone whose central axis passes through the centroid of the base, but may also be a so-called oblique cone whose central axis does not pass through the centroid of the base.

[0041] In addition, in the present embodiment, the vertex angle of the vision cone is set based on the speed of the vehicle, but the present invention is not limited thereto. The type of the road currently being traveled (general road, expressway, etc.) may be determined, and the vertex angle of the vision cone may be set based on the road type. In this case, for example, the vertex angle of the vision cone may be set to a predetermined value pre-set for each road type based on the average vehicle speed of each road type, and the predetermined value may be made smaller for roads with higher average vehicle speeds.

[0042] Alternatively, the shape of the road ahead (straight or curved, etc.) may be determined, and the vertex angle of the vision cone may be set based on the road shape. In this case, if the shape of the road ahead is a shape that requires relatively low speed driving (e.g., a curve), the vertex angle may be made larger than if the shape is a shape that allows relatively high speed driving (e.g., a straight line).

[0043] In step S5, the control device 4 determines whether the driver has successfully recognized the target object. If the driver has successfully recognized the target object, the control device 4 ends the current processing. If the driver has not recognized the target object, the control device 4 proceeds to step S6.

[0044] In this embodiment, the control device 4 is, for example, Fig. 6A As shown in FIG. 1 , a virtual plane onto which the gaze target object is projected is set in a manner orthogonal to the vehicle's traveling direction. If the driver's field of view overlaps with the projection portion of the gaze target object on the virtual plane, it is determined that the driver has successfully recognized the gaze target object. Figure 6B As shown, if the driver's field of view does not overlap with the projection portion of the gaze target object on the virtual plane, it is determined that the driver has failed to recognize the gaze target object. Fig. 6A The speed ratio of the vehicle Figure 6B The speed of the vehicle in is small, so Fig. 6A The field of view ratio Figure 6B In addition, Fig. 6A and Figure 6B In the example, the virtual plane is set at a position that is the shortest distance from the driver's viewpoint to the gaze target object, but the present invention is not limited thereto and may be set at a position that includes the gaze target object, for example.

[0045] In step S6, the control device 4 takes safety measures to avoid collision with the target object. In this embodiment, as a safety measure, the driver is notified via the output device 3. The notification method is not particularly limited, for example, the failure to recognize the target object can be notified by implementing a voice-based broadcast, or by issuing a warning sound, or by driving image information, text information, or a combination thereof.

[0046] In addition, if vehicle 100 is a vehicle that can automatically perform at least a portion of driving operations related to acceleration, steering and braking (for example, a vehicle with an automatic driving function, a vehicle with a driving assistance function), then as a safety measure, the driving operations can be automatically performed in a manner to avoid collision with the target object being watched.

[0047] The control device 4 of the vehicle based on the present embodiment described above is configured to detect a gaze target object to be gazed at that exists around the vehicle based on surrounding data indicating the condition of the surroundings of the vehicle, detect the viewpoint position and line of sight direction of the driver based on driver data indicating the state of the driver, set the driver's field of view based on the speed, road type or road shape of the vehicle, set a virtual plane onto which the gaze target is projected in a manner relative to the vehicle at the position of the gaze target, and determine whether the driver's field of view overlaps with the projection portion of the gaze target on the virtual plane; if the driver's field of view does not overlap with the projection portion of the gaze target, take safety measures.

[0048] Specifically, the control device 4 is configured to set a cone with a predetermined vertex angle with the driver's viewpoint position as the vertex and the sight vector extending from the viewpoint position toward the sight direction as the central axis as the driver's field of view, and change the vertex angle based on the speed, road type or road shape of the vehicle. For example, when the speed of the vehicle is high, the vertex angle can be set to a smaller value than when the speed of the vehicle is low. In addition, for example, when the road type is a road with a high average speed, the vertex angle can be set to a smaller value than when the road type is a road with a low average speed. In addition, for example, when the road shape is a shape that can be driven at a relatively high speed, the vertex angle can be set to a smaller value than when the road shape is a shape that needs to be driven at a relatively low speed.

[0049] This makes it possible to determine whether the driver has recognized the attention target object outside the vehicle in consideration of changes in the driver's field of vision, thereby improving the accuracy of determining whether the attention target object has been recognized.

[0050] Furthermore, as a safety measure, the control device 4 issues a notification via an output device or automatically performs a driving operation of the vehicle so as to avoid the attention target object, so that a collision with the attention target object can be suppressed.

[0051] Although the embodiments of the present invention have been described above, the above embodiments merely illustrate a part of application examples of the present invention and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments.

[0052] For example, in the above-described embodiment, the computer program (notification processing) executed in the control device 4 may be provided in the form of being recorded in a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. Description of Reference Numerals

[0053] 3 Output Devices 4 Control device 100 vehicles

Claims

1. A vehicle control device, configured to: Based on the surrounding data indicating the surrounding conditions of the vehicle, detecting a target object to be watched that exists around the vehicle, detecting a sight line direction of the driver based on driver data representing a state of the driver of the vehicle, The driver's field of view is set in the direction of the driver's line of sight based on the speed of the vehicle, the road type or the road shape, At the position of the target object, a virtual plane onto which the target object is projected is set in a manner opposite to the vehicle, and it is determined whether the driver's field of vision overlaps with the projection portion of the target object on the virtual plane. If the driver's field of view does not overlap with the projection portion of the target object, safety measures are taken.

2. The vehicle control device according to claim 1, wherein: A cone with a specified vertex angle, with the driver's viewpoint as the vertex and a line of sight vector extending from the driver's viewpoint toward the driver's line of sight as the central axis, is set as the driver's field of view, and the vertex angle is changed based on the vehicle's speed, road type or road shape.

3. The vehicle control device according to claim 2, wherein: When the speed of the vehicle is high, the vertex angle is set to a smaller value than when the speed of the vehicle is low.

4. The vehicle control device according to claim 2, wherein: When the road type is a road with a high average vehicle speed, the vertex angle is set to a smaller value than when the road type is a road with a low average vehicle speed.

5. The vehicle control device according to claim 2, wherein: When the road shape is a shape that enables relatively high-speed travel, the vertex angle is set to a smaller value than when the road shape is a shape that requires relatively low-speed travel.

6. The vehicle control device according to any one of claims 1 to 5, wherein: The vehicle has an output device for notifying the driver, The control device is configured to perform a notification via the output device as the safety measure.

7. The vehicle control device according to any one of claims 1 to 6, wherein: As the safety measure, the driving operation of the vehicle is automatically performed so as to avoid the target object.

8. A vehicle control method, which is a vehicle control method based on a control device, comprising: Based on the surrounding data indicating the surrounding conditions of the vehicle, detecting a target object to be watched that exists around the vehicle, detecting a sight line direction of the driver based on driver data representing a state of the driver of the vehicle, The driver's field of view is set in the direction of the driver's line of sight based on the speed of the vehicle, the road type or the road shape, At the position of the target object, a virtual plane onto which the target object is projected is set in a manner opposite to the vehicle, and it is determined whether the driver's field of vision overlaps with the projection portion of the target object on the virtual plane. If the driver's field of view does not overlap with the projection portion of the target object, safety measures are taken.

9. A non-transitory computer recording medium comprising a computer program for a vehicle control device, The computer program is used to cause the control device to execute: Based on the surrounding data indicating the surrounding conditions of the vehicle, detecting a target object to be watched that exists around the vehicle, detecting a viewpoint position and a line of sight direction of the driver based on driver data indicating a state of the driver of the vehicle, The driver's field of view is set in the direction of the driver's line of sight based on the speed of the vehicle, the road type or the road shape, At the position of the target object, a virtual plane onto which the target object is projected is set in a manner opposite to the vehicle, and it is determined whether the driver's field of vision overlaps with the projection portion of the target object on the virtual plane. If the driver's field of view does not overlap with the projection portion of the target object, the vehicle is caused to take safety measures.