Attention arousal device and attention arousal method

Abstract

To provide an attention arousal device capable of arousing an attention of a driver by detecting a warning target without using position information and speed information of a vehicle and operation information of a turn signal switch, and an attention arousal method.SOLUTION: An attention arousal device determines a direction of travel of an own vehicle based on a direction of a face or a line of sight of and a steering wheel operation posture of a driver of which the images are recognized from an intra-cabin image acquired by a camera that images the inside of a cabin. The attention arousal device sets a region to perform an image recognition with respect to a vehicle outside image, which is acquired by a camera imaging a periphery of the own vehicle, in accordance with the direction of travel of the own vehicle. In a case where an image of an object of which the notification is required is recognized within the region, the attention arousal device notifies the driver of the presence of the object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The disclosed embodiments relate to an attention calling device and an attention calling method. [Background technology]

[0002] There is a pedestrian warning device that controls the orientation of an on-board camera to change the imaging range around the vehicle depending on the current position and driving state of the vehicle, and detects pedestrians approaching the vehicle by processing the images around the vehicle.

[0003] For example, when a vehicle is near the center of an intersection, traveling at or below a predetermined speed, and operation of the right-turn turn signal switch is detected, the pedestrian warning device sets the imaging range of the on-board camera to the range to the right rear side of the vehicle, detects pedestrians, and issues a warning (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-309797 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in order to detect pedestrians or other objects to be warned about, conventional pedestrian warning devices needed to obtain vehicle position information from a navigation device, as well as receive speed information and turn signal switch operation information from the vehicle.

[0006] One aspect of the embodiment has been made in consideration of the above, and aims to provide an attention warning device and an attention warning method that can detect a warning target and alert the driver without using vehicle position information, speed information, and turn signal switch operation information. [Means for solving the problem]

[0007] According to one aspect of the embodiment, the attention warning device determines the traveling direction of the vehicle based on the direction of the driver's face or line of sight and the steering wheel operation posture, which are image-recognized from an interior image acquired by a camera that captures the interior of the vehicle. The attention warning device sets an area for image recognition in an exterior image acquired by a camera that captures the surroundings of the vehicle, according to the traveling direction of the vehicle. When the attention warning device recognizes an object requiring notification within the area, it notifies the driver of the presence of the object. Effect of the Invention

[0008] The warning device and warning method according to one aspect of the embodiment determine the direction of travel of the vehicle based on the direction of the driver's face or gaze and the steering wheel operating posture, and set an area for image recognition, thereby achieving the effect of detecting a warning target and warning the driver without using vehicle position information, speed information, and turn signal switch operation information. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of an attention calling device according to an embodiment. [Figure 2A] FIG. 2A is an explanatory diagram of a method for determining a forward turning direction according to an embodiment. [Figure 2B] FIG. 2B is an explanatory diagram of a method for determining a forward turning direction according to the embodiment. [Figure 3A] FIG. 3A is an explanatory diagram of a method for setting an area for image recognition according to an embodiment. [Figure 3B] FIG. 3B is an explanatory diagram of a method for setting an area for image recognition according to the embodiment. [Figure 4] FIG. 4 is a flowchart illustrating an example of processing executed by the controller according to the embodiment. [Diagram 5] FIG. 5 is a flowchart illustrating an example of processing executed by the controller according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of an attention calling device and an attention calling method will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiment described below. The attention calling device according to the embodiment is mounted on a drive recorder for a vehicle. Note that the attention calling device according to the embodiment may be mounted on a vehicle by itself.

[0011] The attention calling device according to the embodiment is a device that detects a surrounding object that is present in the traveling direction of a vehicle (hereinafter, referred to as "own vehicle") and may collide with the own vehicle, and notifies the driver of the presence of the object. Fig. 1 is an explanatory diagram showing an example of the configuration of the attention calling device 1 according to the embodiment.

[0012] 1, the attention calling device 1 is connected to a front camera 51, a rear camera 52, an in-vehicle camera 53, and a notification device 54. The front camera 51 is an imaging device that captures an image in front of the vehicle. The front camera 51 outputs data of the captured image to the attention calling device 1.

[0013] The rear camera 52 is an imaging device that captures an image of the rear of the vehicle. The rear camera 52 outputs data of the image of the rear of the vehicle to the attention warning device 1. The interior camera 53 is an imaging device that captures an image of the interior of the vehicle. The interior camera 53 outputs data of the captured image to the attention warning device 1.

[0014] The notification device 54 includes a display that displays various information and a speaker that can output various information as audio. The notification device 54 notifies the driver of the various information input from the attention calling device 1 by video and audio. The notification device 54 may be configured to notify the driver of the various information by either video or audio.

[0015] The attention calling device 1 includes a controller 2 and a storage unit 3. The storage unit 3 is, for example, an information storage device such as a data flash, and stores recorded data 31. The recorded data 31 is data of images captured by a front camera 51, a rear camera 52, and an in-vehicle camera 53.

[0016] The controller 2 includes a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and various circuits. The CPU executes a program stored in the ROM using the RAM as a working area, whereby the controller 2 detects target objects present around the vehicle and notifies the driver.

[0017] The controller 2 may be configured partially or entirely with hardware such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0018] The controller 2 performs image recognition of the driver's face and line of sight direction from the vehicle interior image input from the vehicle interior camera 53. Furthermore, the controller 2 performs image recognition of the driver's steering wheel operating posture from the vehicle interior image.

[0019] Then, the controller 2 determines the traveling direction of the vehicle based on the direction of the driver's face or line of sight and the steering wheel operation posture that are recognized by image recognition. A specific example of a method for determining the traveling direction will be described later with reference to Figures 2A and 2B.

[0020] Next, the controller 2 sets an area for image recognition according to the determined traveling direction of the host vehicle for the outside images input from the front camera 51 and the rear camera 52. A specific example of a method for setting the area for image recognition will be described later with reference to Figs. 3A and 3B.

[0021] When the controller 2 recognizes an object requiring notification within the set area, the controller 2 notifies the driver of the presence of the object by the notification device 54. This allows the controller 2 to detect the object to be warned from the video inside and outside the vehicle by image recognition and alert the driver without acquiring vehicle information from the vehicle, such as vehicle position information, speed information, and turn signal switch operation information.

[0022] In addition, the controller 2 predicts the traveling direction of the vehicle and the traveling direction of the target object from the outside-of-vehicle image. Then, when the controller 2 determines that the traveling direction of the vehicle and the traveling direction of the target object intersect, it notifies the driver of the presence of the target object.

[0023] Furthermore, if the controller 2 determines that the traveling direction of the vehicle and the traveling direction of the target object do not intersect, the controller 2 does not notify the driver of the presence of the target object. As a result, even if the controller 2 detects a target object, if there is no risk of collision with the vehicle, the controller 2 does not issue an unnecessary notification, thereby reducing the annoyance of notifications.

[0024] Furthermore, the controller 2 estimates the vehicle speed of the host vehicle from the outside-of-vehicle image, and when the vehicle speed is equal to or lower than a predetermined vehicle speed, the controller 2 notifies the driver of the presence of a target object.

[0025] Furthermore, when the vehicle speed is not equal to or lower than a predetermined vehicle speed, that is, when the vehicle speed is faster than the predetermined vehicle speed, the controller 2 does not notify the driver of the presence of the target object. As a result, the controller 2 notifies the driver of the presence of the target object only when the vehicle speed is slow and the driver can reliably take risk avoidance action by being notified of the presence of the target object, so that it is possible to provide a notification that is useful to the driver.

[0026] Next, a specific example of a method for determining a traveling direction will be described with reference to Fig. 2A and Fig. 2B. Fig. 2A and Fig. 2B are explanatory diagrams of a method for determining a traveling direction according to an embodiment. In Fig. 2A and Fig. 2B, the skeleton of the driver D is indicated by a thick line.

[0027] 2A, when the driver D drives the vehicle straight, the face and gaze of the driver D are directed forward. At this time, a straight line L1 connecting the hands gripping the steering wheel is parallel to a horizontal straight line L2.

[0028] In contrast, for example, when the driver D turns the vehicle to the left, the driver D turns his face and gaze to the left front. At this time, the driver D turns the steering wheel counterclockwise. Therefore, the angle between the straight line L1 connecting the hands gripping the steering wheel and the horizontal straight line L2, that is, the inclination of the straight line L1 with respect to the horizontal plane, is θ.

[0029] Therefore, the controller 2 performs image recognition of the hand or arm position of the driver D, and estimates the steering wheel operation posture from the skeletal position of the hand or arm. This allows the controller 2 to estimate the steering wheel operation amount from the inclination θ of the straight line L1 recognized from the image of the interior of the vehicle with respect to the horizontal plane, without acquiring information indicating the steering wheel operation amount from the steering angle sensor of the vehicle.

[0030] Moreover, if the face or line of sight of the driver D, which is image-recognized by the controller 2, faces leftward, the controller 2 can estimate that the vehicle will turn left. If the face or line of sight of the driver D, which is image-recognized by the controller 2, faces rightward, the controller 2 can estimate that the vehicle will turn right.

[0031] Next, a specific example of a method for setting an area for image recognition will be described with reference to Figures 3A and 3B. Figures 3A and 3B are explanatory diagrams of a method for setting an area for image recognition according to an embodiment.

[0032] As shown in FIG. 3A, when the host vehicle C is turning right, the controller 2 sets the right 1 / 3 area A3 of the imaging area A of the forward camera 51 as the area to be recognized, and does not set areas A1 and A2 other than area A3 as the area to be recognized.

[0033] The controller 2 then performs image recognition of the target object only in the right 1 / 3 area P1 of the image P in front of the vehicle. This allows the controller 2 to perform image recognition of a person T present even in an area that is a blind spot caused by the right front pillar of the vehicle turning right. Moreover, the controller 2 does not need to perform image recognition for areas that are not a blind spot caused by the right front pillar of the vehicle, thereby reducing the processing load of the image recognition process.

[0034] Similarly, when the vehicle C turns left, the controller 2 sets the area A1, which is 1 / 3 of the side on which the vehicle C turns, as the area for image recognition, and does not set areas A2 and A3 other than the area A1 as the areas for image recognition.The controller 2 then performs image recognition of the target object only in the left 1 / 3 of the image P in front of the vehicle.

[0035] Thereafter, when the vehicle C continues turning right and starts to enter the road to which it will turn, the controller 2 sets the area to be recognized as an image to be expanded to 2 / 3 of the area A2, A3 on the side where the vehicle C is turning, as shown in FIG. 3B.

[0036] Then, the controller 2 performs image recognition of the target object only in the central and right area P2 of the image P in front of the vehicle. After that, after the vehicle C has completely entered the road to which it is turning right, the driver's face or line of sight faces forward and the amount of steering wheel operation becomes zero, so the controller 2 sets the imaging area A2 of the front camera 51 as the area to be recognized.

[0037] Furthermore, the controller 2 may be configured to change the area to be recognized in accordance with the traveling direction of the vehicle C and the traveling direction of the target object. In this case, for example, if the person T detected in the state shown in Fig. 3A is moving along the road toward the vehicle C, the controller 2 sets only the area A3 as the area to be recognized in the image, without expanding the area to be recognized in the image, even if the state shown in Fig. 3B is reached.

[0038] In addition, for example, if the person T detected in the state shown in FIG. 3A is moving in the direction of crossing the road, when the state shown in FIG. 3B is reached, the controller 2 expands the image recognition area to include area A2 in addition to area A3.

[0039] This allows the controller 2 to set an appropriate area as the area for image recognition in accordance with the moving direction of the person T, who is the detected target object, while suppressing an increase in the processing load of image processing.

[0040] Hereinafter, the process executed by the controller 2 of the attention calling device 1 will be specifically described with reference to Fig. 4 and Fig. 5. Fig. 4 and Fig. 5 are flowcharts showing an example of the process executed by the controller 2 according to the embodiment. When the ignition switch of the vehicle is turned on, the controller 2 starts the process shown in Fig. 4 and Fig. 5.

[0041] When the ignition switch is turned on, the controller 2 first detects the driver's face by image recognition from the vehicle interior image captured by the vehicle interior camera 53 (step S101). At this time, the controller 2 detects the driver's face and the direction of his / her line of sight.

[0042] Next, the controller 2 determines whether the driver's face is facing left or right (step S102). At this time, the controller 2 determines whether the driver's face is facing left or right based on the direction of the driver's face or line of sight that is recognized by the image recognition.

[0043] When the controller 2 determines that the driver's face is not facing either left or right (step S102, No), that is, when the controller 2 determines that the driver is facing forward, the controller 2 starts the process again from step S101.

[0044] Furthermore, when it is determined that the driver's face is facing either left or right (Yes at step S102), the controller 2 detects the driver's posture by image recognition from the vehicle interior image (step S103).

[0045] Next, the controller 2 judges whether or not the line connecting the driver's hands gripping the steering wheel is horizontal (step S104). When the controller 2 judges that the line connecting the driver's hands gripping the steering wheel is horizontal (step S104, Yes), the controller 2 starts the process again from step S101.

[0046] Furthermore, when the controller 2 determines that the straight line connecting the driver's hands gripping the steering wheel is not horizontal (step S104, No), it determines whether the inclination of the straight line connecting the driver's hands gripping the steering wheel with respect to the horizontal direction is equal to or smaller than a certain value (step S105). The certain value here is, for example, the maximum value of the inclination of the straight line connecting the driver's hands gripping the steering wheel when the vehicle is traveling around a curve.

[0047] When the controller 2 determines that the inclination of the line connecting both hands gripping the steering wheel with respect to the horizontal direction is not equal to or less than a certain value (step S105, No), the controller 2 proceeds to step S108 shown in Fig. 5. When the controller 2 determines that the inclination of the line connecting both hands gripping the steering wheel with respect to the horizontal direction is equal to or less than a certain value (step S105, Yes), the controller 2 detects the speed of the host vehicle (step S106).

[0048] At this time, the controller 2 detects the vehicle speed of the host vehicle from the image ahead of the host vehicle captured by the front camera 51. For example, the controller 2 performs image recognition of stationary objects such as utility poles and signs included in the image ahead, and detects the vehicle speed based on how much the stationary objects move in the image per unit time.

[0049] Then, the controller 2 judges whether the vehicle speed is equal to or lower than a certain value (step S107). The certain value here is the lower limit of the vehicle speed when a general vehicle travels around a curve. If the controller 2 judges that the vehicle speed is not equal to or lower than the certain value (step S107, No), it starts the process again from step S101.

[0050] Moreover, when the controller 2 determines that the vehicle speed is equal to or lower than the certain value (Yes at step S107), the controller 2 shifts the process to step S108 shown in FIG.

[0051] In step S108, the controller 2 determines the steering wheel operation amount and the traveling direction of the host vehicle from the direction of the driver's face or line of sight detected from the vehicle interior image and the steering wheel operation posture.

[0052] Next, the controller 2 performs image recognition on the 1 / 3 of the camera images from the front camera 51 and the rear camera 52 on the side where the vehicle turns, and detects a target object of which the driver should be notified (step S109). That is, the controller 2 sets an area for image recognition on the outside image according to the traveling direction of the vehicle, and detects a target object of which the driver should be notified from the set area.

[0053] After that, the controller 2 judges whether or not a target object is detected (step S110). If it is judged that a target object is not detected (step S110, No), the controller 2 judges whether or not the operation amount of the steering wheel has returned to zero based on the driver's posture recognized by the image (step S111).

[0054] When the controller 2 determines that the steering wheel operation amount has returned to zero (step S111, Yes), it ends the process and restarts the process from step S101. When the controller 2 determines that the steering wheel operation amount has not returned to zero (step S111, No), it changes the image recognition range according to the steering wheel operation amount (step S112).

[0055] Specifically, the controller 2 changes the image recognition range so as to widen the area to be recognized in the image according to the steering operation amount that increases as the vehicle advances in the right or left turn direction. After that, the controller 2 moves the process to step S110. This makes it easier for the controller 2 to detect the target object that could not be detected in step S110 in the next step S110.

[0056] Furthermore, when it is determined that a target object has been detected (step S110, Yes), the controller 2 predicts the traveling direction of the target object (step S113). At this time, when the target object is a person T, the controller 2 predicts the moving direction of the target object from the movement of the skeleton of the person T that has been image-recognized as the target object.

[0057] Furthermore, the controller 2 estimates the surrounding conditions of the person T, which is a target object, from the image of the surroundings of the vehicle, and predicts the moving direction of the person T based on the surrounding conditions of the person T. For example, when the person T is walking on a sidewalk, the controller 2 predicts the extension direction of the sidewalk as the moving direction of the person T. Also, when the person T is standing in front of a pedestrian crossing, the controller 2 predicts the extension direction of the pedestrian crossing as the moving direction of the person T. This allows the controller 2 to more accurately predict the direction in which the target object will move when the target object is person T.

[0058] Furthermore, when the target object is a car or a motorcycle, the controller 2 predicts the travel direction of the target object from the orientation of the body of the car or motorcycle recognized by image recognition. Furthermore, when the target object, a car or motorcycle, is moving on a roadway, the controller 2 predicts the extension direction of the roadway as the travel direction of the target object. Furthermore, when the target object, a car or motorcycle, is located in front of a pedestrian crossing, the controller 2 predicts the direction of crossing the pedestrian crossing as the travel direction of the target object. This allows the controller 2 to more accurately predict the direction in which the target object will travel when the target object is a car or a motorcycle.

[0059] The controller 2 predicts the traveling direction of the host vehicle from the outside-of-vehicle image, and determines whether or not the traveling direction of the target object predicted in step S113 intersects with the traveling direction of the host vehicle (step S114).

[0060] When the controller 2 determines that the traveling direction of the target object intersects with the traveling direction of the vehicle (step S114, Yes), it notifies the driver of the danger by voice and image (step S115). Specifically, the controller 2 notifies the driver of the presence of the target object by voice and image. After that, the controller 2 moves the process to step S111.

[0061] In addition, if the controller 2 determines that the predicted traveling direction of the target object does not intersect with the traveling direction of the vehicle (step S114, No), it ends the processing without notifying the driver of the presence of the target object, and starts the processing again from step S101.

[0062] This allows the controller 2 to warn the driver when there is a risk of collision between the target object and the vehicle. Also, even if the controller 2 detects a target object, when there is no risk of collision between the target object and the vehicle, it can suppress unnecessary warnings to the driver.

[0063] Further advantages and modifications may readily occur to those skilled in the art. Thus, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and equivalents thereof. [Explanation of symbols]

[0064] 1 Alert device 2. Controller 3 Storage section 31 Recording data 51 Front Camera 52 Rear camera 53 In-vehicle camera 54 Notification device

Claims

1. A direction of travel of the vehicle is determined based on the direction of the driver's face or line of sight and the steering wheel operation posture, which are image-recognized from an interior image of the vehicle acquired by a camera that captures the interior of the vehicle; setting an area for image recognition in an outside image acquired by a camera that captures an image of the surroundings of the vehicle according to a traveling direction of the vehicle; When an object requiring notification is image-recognized within the area, the presence of the object is notified to the driver. Alert device.

2. predicting a traveling direction of the object from the outside image of the vehicle; When it is determined that the traveling direction of the vehicle and the traveling direction of the object intersect, the driver is notified of the presence of the object. The attention calling device according to claim 1 .

3. A surrounding situation of the object is estimated from the outside image of the vehicle, and a traveling direction of the object is predicted based on the surrounding situation. The attention calling device according to claim 2 .

4. The area is changed according to the traveling direction of the vehicle and the traveling direction of the object. The attention calling device according to claim 2 .

5. The steering wheel operation posture is estimated from the driver's hand position or arm position, which is image-recognized from the vehicle interior image. The attention calling device according to claim 1 .

6. A vehicle speed of the vehicle is estimated from the outside image, and when the vehicle speed is equal to or lower than a predetermined vehicle speed, the driver is notified of the presence of the object. The attention calling device according to claim 1 .

7. The area is expanded according to the movement of the host vehicle in the traveling direction. The attention calling device according to any one of claims 1 to 6.

8. A direction of travel of the vehicle is determined based on the direction of the driver's face or line of sight and the steering wheel operation posture, which are image-recognized from an interior image of the vehicle acquired by a camera that captures the interior of the vehicle; setting an area for image recognition in an outside image acquired by a camera that captures an image of the surroundings of the vehicle according to a traveling direction of the vehicle; When an object requiring notification is image-recognized within the area, the presence of the object is notified to the driver. An attention-warning method performed by an attention-warning device.

Images