Alarm system

Abstract

The invention relates to an alarm system. The present invention reduces the risk of deterioration of the traveling state of a vehicle caused by the driver seeing at a display unit of an in-vehicle device. This alarm system is mounted in a vehicle, and is provided with a display unit, an imaging unit, and a control unit. The display part is used for displaying information of the vehicle-mounted device. The imaging unit is capable of acquiring an image including the face of the driver. The control unit causes the display unit to display information pertaining to the traveling state of the vehicle when the control unit determines that the state in which the driver looks at the display unit continues for a predetermined time or longer on the basis of the image.

Description

Technical Field

[0001] The present disclosure relates to alarm systems. Background Art

[0002] A sight line monitoring device is disclosed that monitors the direction of a driver's sight line and issues a warning using an output device mounted on the vehicle when the sight line deviates from a preset range (for example, see Patent Document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2020 / 213022

[0006] According to the prior art, when the driver is distracted driving, the sight line monitoring device warns the driver by directing the sight line to a predetermined range. However, the sight line monitoring device of the prior art does not warn the driver in response to the change in the driving state of the vehicle caused by distracted driving. Therefore, the driver cannot immediately respond to the deterioration of the driving state. Summary of the invention

[0007] The present disclosure has been made in view of the above-mentioned facts and aims to reduce the risk of deterioration of the running state of the vehicle due to the driver looking at the display unit of the vehicle-mounted device.

[0008] An alarm system of one embodiment of the present disclosure that solves the above-mentioned problem is an alarm system installed in a vehicle, wherein the alarm system comprises: a display unit for displaying information from a vehicle-mounted device; a shooting unit capable of acquiring an image including the driver's face; and a control unit for displaying information related to the driving state of the vehicle on the display unit when it is determined based on the image that the driver has been looking at the display unit for more than a specified time.

[0009] Effects of the Invention

[0010] According to the present disclosure, it is possible to reduce the risk of deterioration of the driving state of the vehicle due to the driver looking at the display unit of the vehicle-mounted device. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a block diagram showing a schematic configuration of an alarm system according to an embodiment of the present disclosure.

[0012] Figure 2 Yes means Figure 1 A diagram showing an example of the arrangement of components of an alarm system within a vehicle.

[0013] Figure 3 yes Figure 1Functional block diagram of the control unit.

[0014] Figure 4 Yes means Figure 1 FIG. 1 is a diagram showing an example of an alarm display on a display unit of FIG.

[0015] Figure 5 Yes means Figure 1 A flowchart of an example of a process executed by a control unit.

[0016] Description of reference numerals:

[0017] 10: Alarm system;

[0018] 11: Photography Department;

[0019] 12: Alarm device;

[0020] 13: Display unit;

[0021] 14: Control unit;

[0022] 15: Storage unit;

[0023] 16: ECU;

[0024] 17: Vehicle speed sensor;

[0025] 18: Distance measuring device;

[0026] 19: Navigation device;

[0027] 20: Vehicle;

[0028] 21: Driver;

[0029] 31: image acquisition unit;

[0030] 32: Line of sight estimation unit;

[0031] 33: driving information acquisition unit;

[0032] 34: gaze state determination unit;

[0033] 35: warning information generation unit;

[0034] 36: Display control unit. DETAILED DESCRIPTION

[0035] Hereinafter, an embodiment of the present disclosure will be described with reference to the accompanying drawings. It should be noted that the drawings used in the following description are schematic. In addition, the dimensional ratios on the drawings are not necessarily consistent with the dimensional ratios in reality.

[0036] (Composition of the alarm system)

[0037] like Figure 1As shown, the alarm system 10 according to one embodiment of the present disclosure is configured to include a photographing unit 11, an alarm device 12, and a display unit 13. Figure 2 As shown in the example, each part of the alarm system 10 is mounted on a vehicle 20. In the present application, the vehicle 20 includes a passenger car, a truck, a bus, a large / small special vehicle, etc., but is not limited to these.

[0038] The imaging unit 11 is a camera disposed in the vehicle compartment of the vehicle 20 and captures an image including the face of the driver 21. The imaging unit 11 may be disposed near an interior mirror in the vehicle 20, on a dashboard, or in the center of a steering wheel. The imaging unit 11 is configured so that the face of the driver 21 is included in the field of view. For example, the optical axis O of the imaging unit 11 is directed toward the face of the driver 21. The imaging unit 11 outputs the captured image of the driver 21 to the alarm device 12.

[0039] The alarm device 12 includes a control unit 14 and a storage unit 15. The alarm device 12 is configured to acquire an image including a face of a driver 21 driving a vehicle 20 from the imaging unit 11. In addition, the alarm device 12 is configured to acquire information related to the driving state of the vehicle 20 from one or more ECUs (Electronic Control Units) 16 in the vehicle 20. The control unit 14 of the alarm device 12 can generate an alarm based on the information acquired from the imaging unit 11 and the ECU 16, and display the alarm on the display unit 13.

[0040] The information related to the driving state of the vehicle 20 includes information on the driving speed of the vehicle 20, information on the inter-vehicle distance to the vehicle traveling ahead, and information indicating changes in the driving speed and inter-vehicle distance. The information related to the driving state is not limited to information related to the driving speed and inter-vehicle distance. For example, the information related to the driving state may also include information on the steering angle, information related to the shaking of the vehicle 20 and the departure from the lane, etc.

[0041] The control unit 14 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU (central processing unit) or a GPU (graphics processing unit), or a dedicated processor specifically used for specific processing. A programmable circuit is, for example, an FPGA (field-programmable gate array). A dedicated circuit is, for example, an ASIC (application specific integrated circuit). The control unit 14 controls each part of the alarm device 12 while executing processing related to the operation of the alarm device 12.

[0042] The storage unit 15 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory or any combination thereof. The semiconductor memory is, for example, a RAM (random access memory) or a ROM (read only memory). The RAM is, for example, an SRAM (static random access memory) or a DRAM (dynamic random access memory). The ROM is, for example, an EEPROM (electrically erasable programmable read only memory). The storage unit 15 functions as, for example, a main storage device, an auxiliary storage device or a cache memory. The storage unit 15 stores programs executed by the control unit 14, data used in the operation of the alarm device 12, and data obtained through the operation of the alarm device 12.

[0043] The storage unit 15 may store the positions of the imaging unit 11 and the display unit 13. For example, the storage unit 15 may store the position of the imaging unit 11 and the range of the display surface of the display image of the display unit 13 by means of a coordinate system fixed to the vehicle 20. In addition, the storage unit 15 may store the standard eyeball positions of the driver 21 for one or more registered drivers 21. In this case, the storage unit 15 may store information for identifying the registered driver based on the driver's facial image.

[0044] ECU16 is an electronic control unit that controls various equipment in the vehicle 20. ECU16 includes ECUs for inter-vehicle distance control systems and advanced driver assistance systems (ADAS: Advanced driver assistance systems) such as ACC (Advanced Cruise Control System). Each ECU16 is connected to each other via a network such as CAN (Controller Area Network). ECU16 can obtain signals from a vehicle speed sensor 17 and a distance measuring device 18, wherein the vehicle speed sensor 17 detects the speed of the vehicle 20 and the distance measuring device 18 measures the inter-vehicle distance with the vehicle traveling in front of the vehicle 20. ECU16 can output information on the speed and / or inter-vehicle distance of the vehicle 20 to the alarm device 12. The signals of the vehicle speed sensor 17 and the distance measuring device 18 can also be output to the alarm device 12 without passing through ECU16. ECU16 can be configured in various places inside the vehicle 20.

[0045] The vehicle speed sensor 17 is a sensor for measuring the rotation speed of the tire, for example. The speed of the vehicle 20 is calculated from the rotation speed of the tire per unit time and the length of the outer circumference of the tire. As the vehicle speed sensor 17, a sensor used for displaying a speedometer of the vehicle 20 can be used.

[0046] The distance measuring device 18 includes LIDAR (Light Detection and Ranging), millimeter wave radar, and stereo camera. Any one or more of these devices can be used as the distance measuring device 18. LIDAR and millimeter wave radar include scanning devices that scan pulsed electromagnetic waves (infrared, millimeter waves, etc.) and measure scattered light. These distance measuring devices 18 can calculate the distance to each reflection point of the object based on the time (time-of-flight) taken until the irradiated electromagnetic waves are reflected by the object and detected. LIDAR and millimeter wave radar can be configured on the front side of the vehicle 20, for example, as shown in FIG. Figure 2 The distance measuring device 18 is arranged on the front bumper like the distance measuring device 18 of FIG. The stereo camera uses a plurality of cameras arranged in parallel and equidistantly to shoot an object, and calculates the distance according to the principle of triangulation based on the parallax of the images shot by the cameras. The distance measuring device 18 is not limited to the above-mentioned device, and various devices capable of measuring distance can be used. For example, the stereo camera can be arranged on the inner side of the windshield.

[0047] The display unit 13 is used to display information of the onboard device installed in the vehicle 20. The display unit 13 displays images based on image signals input from the onboard devices in the vehicle 20 including the navigation device 19. For example, various displays such as a liquid crystal display (LCD), an organic EL (Electro-Luminescence) display, and an inorganic EL display can be used as the display unit 13. The display unit 13 is sometimes embedded in the instrument panel and sometimes arranged on the instrument panel. When the driver 21 looks at the display unit 13, it is necessary to move the line of sight from the front of the vehicle 20 to the display unit 13.

[0048] The navigation device 19 displays a road map of the vicinity of the place where the vehicle 20 is traveling and a route leading to the destination of the vehicle 20. The navigation device 19 has various functions such as zooming in / out of the map, searching for the destination, and displaying surrounding facilities. The navigation device 19 always generates an image signal and outputs the image signal to the display unit 13. The display unit 13 usually displays an image from the navigation device 19.

[0049] (Configuration of the control unit)

[0050] like Figure 3 As shown, the control unit 14 includes an image acquisition unit 31, a sight estimation unit 32, a driving information acquisition unit 33, a gaze state determination unit 34, a warning information generation unit 35, and a display control unit 36. These units of the control unit 14 can be either hardware modules or software modules. The processing performed by each unit can be said to be the processing performed by the control unit 14.

[0051] The image acquisition unit 31 acquires an image signal of an image including the face of the driver 21 from the imaging unit 11 .

[0052] The sight line estimation unit 32 estimates the position and direction of the sight line of the driver 21 based on the image signal obtained from the shooting unit 11. For example, the sight line estimation unit 32 estimates the position of the eyeball of the driver 21 and the direction of the sight line of the driver 21 based on the image of the driver 21. A known method can be used to estimate the position of the eyeball of the driver 21 and the direction of the sight line. For example, the method of estimating the sight line direction includes a method using a Purkinje image, a method using the relative positional relationship between the corner of the eye and the iris, a method of approximating the outline of the pupil with an ellipse and estimating based on the ellipse parameters, etc. In addition, the position of the eyeball of the driver 21 can also use information pre-stored in the storage unit 15 of the alarm device 12 for each registered driver. In this case, the control unit 14 can also determine the driver 21 based on the image of the driver 21's face.

[0053] The driving information acquisition unit 33 acquires information on the driving speed of the vehicle 20 and information on the inter-vehicle distance to the vehicle traveling in front of the vehicle 20 from the ECU 16. Alternatively, the driving information acquisition unit 33 further acquires information indicating a change in the driving speed, for example, acquiring information on the acceleration of the vehicle 20. In addition, the driving information acquisition unit 33 may acquire information on the speed of the change in the inter-vehicle distance. Alternatively, the change in the speed of the vehicle 20 and the change in the inter-vehicle distance may be calculated by the control unit 14 based on the driving speed information and the inter-vehicle distance information, respectively.

[0054] When the line of sight of the driver 21 intersects with the display unit 13, the gaze state determination unit 34 recognizes that the driver 21 is looking at the display unit 13. Whether the line of sight of the driver 21 intersects with the display unit 13 can be determined based on the position and direction of the line of sight estimated by the line of sight estimation unit 32 and the position information of the display unit 13 stored in the storage unit 15.

[0055] When the state that the driver 21 is looking at the display unit 13 continues for more than a predetermined time, the gaze state determination unit 34 determines that the user is looking at the display unit 13. That is, in the present disclosure, "looking" means looking at the display unit 13 continuously for more than a predetermined time. The predetermined time can be set to, for example, 1 second or 2 seconds. The time that the driver 21 continues to look at the display unit 13 is preferably within 1 second. Depending on the surrounding environment during driving, if the driver 21 continues to look at the display unit 13 for more than 2 seconds, a dangerous situation may occur.

[0056] The warning information generating unit 35 can set the prescribed time to different values ​​according to the information related to the driving state acquired by the driving information acquiring unit 33. For example, the gaze state determining unit 34 can set the prescribed time such that the faster the driving speed, the shorter the prescribed time. In addition, for example, the gaze state determining unit 34 can set the prescribed time such that the shorter the distance between vehicles, the shorter the prescribed time. By setting in this way, the higher the possibility of the occurrence of danger caused by the driver 21 staring at the display unit 13, the shorter the time it can be issued to the user. In this way, the possibility of danger occurring during the driving of the vehicle 20 can be reduced.

[0057] The warning information generating unit 35 generates warning information to be displayed on the display unit 13 when the attention state determining unit 34 determines that the driver 21 is looking at the display unit 13 .

[0058] The warning information includes information related to the driving state. For example, the warning information may include the driving speed. In particular, when the driving speed is decreasing, the warning information generation unit 35 may generate a warning information to warn that the vehicle 20 is decelerating. If the driving speed decreases due to the driver 21's attention being directed toward the display unit 13, it may sometimes cause the driver 21 of the following vehicle to be irritated and sometimes cause the risk of being rear-ended. In this case, the warning information may include content reporting that the speed is too low.

[0059] In addition, the warning information generation unit 35 may generate a warning based on the information of the inter-vehicle distance. For example, the warning information generation unit 35 may generate a warning when the inter-vehicle distance is shorter than a prescribed distance or when the inter-vehicle distance is shortened faster than a prescribed speed. When the inter-vehicle distance with the vehicle traveling ahead is short or when the inter-vehicle distance is shortened rapidly, there may be a risk of rear-ending the vehicle ahead. In this case, the warning information may include content reporting the short inter-vehicle distance.

[0060] like Figure 4 As shown in the example, the display control unit 36 ​​causes the warning information generated by the warning information generating unit 35 to be displayed on the display unit 13. When the display unit 13 displays an image output from the navigation device 19, the display control unit 36 ​​causes the warning information to be inserted into the image displayed on the display unit 13 for display. The warning information may be displayed in a switching manner with the image from the navigation device 19. In addition, the warning information may be displayed overlapping the image from the navigation device 19. In addition, the control unit 14 may cause the speaker in the vehicle 20 to generate a warning sound in addition to causing the display unit 13 to display the warning information. Furthermore, the control unit 14 may be configured to reduce the sound output of an audio device such as a radio in the vehicle 20 when displaying the warning information and generating the warning sound.

[0061] Thus, according to the alarm system 10 of the present disclosure, by directly displaying the information notifying the deterioration of the driving state on the display unit 13 that the driver 21 is looking at, the driver 21 can immediately take measures against the deterioration of the driving state. The driver 21 can accelerate the vehicle 20 when the speed is decreasing. In addition, the driver 21 can release the accelerator or step on the brake when the distance between vehicles becomes shorter. Compared with simply urging the driver 21 to look forward, the alarm system of the present disclosure can enable the driver 21 to quickly respond to the deterioration of the driving state.

[0062] In addition, in the above-mentioned embodiment, the warning information is displayed in an overlapping manner on the display unit 13 used in the display of the normal navigation device 19. In order to read the display content, the driver 21 tends to stare at the screen of the navigation device 19 for a relatively long time, so it is highly effective to display the warning information to divert the driver's 21's attention to driving.

[0063] (Warning display method)

[0064] Below, use Figure 5 The flow of the process performed by the control unit 14 of the alarm device 12 is described. The method disclosed in this specification can be executed by a processor included in the control unit 14 according to a program. Such a program can be stored in a non-transitory computer-readable medium. Examples of non-transitory computer-readable media include hard disks, RAM, ROM, flash memory, CD-ROM, optical storage devices, magnetic storage devices, etc., but are not limited to these.

[0065] The control unit 14 acquires an image including the face of the driver 21 from the image acquisition unit 31 ( S1 ).

[0066] The control unit 14 estimates the line of sight of the driver 21 based on the image of the driver 21 ( S2 ).

[0067] The control unit 14 determines whether the line of sight of the driver 21 intersects with the display unit 13 ( S3 ). When the line of sight of the driver 21 intersects with the display unit 13 , it is recognized that the driver 21 is looking at the display unit 13 .

[0068] When the line of sight of the driver 21 crosses the display unit 13 in S3 (S3: Yes), the control unit 14 proceeds to S4. When the line of sight of the driver 21 does not cross the display unit 13 in S3 (S3: No), the driver 21 does not look at the display unit 13, so the control unit 14 ends the process without performing any special processing.

[0069] The control unit 14 acquires information related to the running state from the ECU 16 ( S4 ).

[0070] The control unit 14 determines whether the driver 21 is looking at the display unit 13 ( S5 ), and determines that the driver 21 is looking at the display unit 13 when the line of sight of the driver 21 crosses the display unit 13 for a predetermined time.

[0071] When the driver 21 is looking at the display unit 13 in S5 (S5: Yes), the control unit 14 proceeds to S6. When the driver 21 does not look at the display unit 13 because the time the driver 21 looks at the display unit 13 in S5 is less than the prescribed time (S5: No), the control unit 14 returns to S1 and continues to monitor whether the driver 21 is looking at the display unit 13.

[0072] The control unit 14 generates warning information using the information on the running state ( S6 ).

[0073] The control unit 14 inserts the warning information generated in S6 into the image of the navigation device 19 displayed on the display unit 13 and displays it (S7).

[0074] The control unit 14 repeatedly executes the processes from S1 to S7 to continuously monitor the driving of the driver 21 while the vehicle 20 is running, thereby reducing the risk of deterioration of the running state of the vehicle. Thus, the driver 21 can run the vehicle 20 stably.

[0075] It should be noted that the present invention is not limited to the above-mentioned embodiments, and various modifications or changes can be made. For example, the functions included in each unit, each step, etc. can be reconfigured in a logically non-contradictory manner, and multiple units or steps can be combined into one, or divided.

[0076] For example, in the above-mentioned embodiment, the photographing unit 11 photographs and outputs an image including the face of the driver 21, and the control unit 14 of the alarm unit 12 estimates the position and direction of the line of sight based on the image of the driver 21. However, the photographing unit 11 may be an intelligent camera having an image recognition function, and the processing of estimating the position and direction of the line of sight of the driver 21 may be performed by the photographing unit 11. For example, the photographing unit 11 may be equipped with a learned model implemented by machine learning, and estimate the position and direction of the line of sight based on the image of the driver. In this case, the alarm device 12 obtains the position and direction of the driver's line of sight as an output from the photographing unit 11. It is not necessary to provide the line of sight estimation unit 32 in the control unit 14.

[0077] In addition, the imaging unit 11 may be a stereo camera in which a plurality of cameras are arranged in parallel, rather than a monocular camera. In this case, the control unit 14 of the alarm device 12 may analyze the images output from the plurality of cameras to estimate the position and direction of the line of sight.

[0078] Furthermore, in the above-mentioned embodiment, the control unit 14 determines whether the driver 21 is looking at the display unit 13 by determining whether the line of sight of the driver 21 intersects with the display surface of the display image of the display unit 13 based on the image of the driver's 21 face. However, it is not necessary to estimate the position and direction of the line of sight based on the image of the face. For example, a large amount of information obtained by combining various images of the driver's 21 face and information on whether the driver is looking at the display unit may be used as data for machine learning, and through machine learning, it is possible to directly determine whether the driver 21 is looking at the display unit 13 based on the image of the driver's 21 face.

[0079] In the above embodiment, the navigation device 19 is exemplified as the vehicle-mounted device. However, the vehicle-mounted device is not limited to the navigation device 19. In particular, the vehicle-mounted device may include a vehicle-mounted multimedia device that integrates a music player, reception of various broadcasts, and communication functions in cooperation with a mobile phone in addition to a navigation function.

Claims

1. An alarm system, mounted on a vehicle, comprising: A display unit, used for displaying information of an on-vehicle device; a camera capable of acquiring an image including the driver's face; and The control unit displays information related to the driving state of the vehicle on the display unit when it is determined based on the image that the driver continues to look at the display unit for a predetermined time or longer.

2. The alarm system according to claim 1, wherein: The predetermined time is set to a time that varies according to the running state of the vehicle.

3. The alarm system according to claim 1, wherein: The control unit acquires information on the traveling speed of the vehicle, and displays at least one of the information on the traveling speed and information indicating a change in the traveling speed on the display unit as information related to the traveling state.

4. The alarm system according to claim 1, wherein: The information related to the driving state includes the inter-vehicle distance between the vehicle traveling in front of the vehicle and the vehicle, and the control unit displays the information of the inter-vehicle distance as information related to the driving state on the display unit when at least either the inter-vehicle distance is shorter than a prescribed distance or the inter-vehicle distance becomes shorter faster than a prescribed speed.

5. The alarm system according to claim 1, wherein: The in-vehicle device is a navigation device that guides a route that a vehicle should travel, and the control unit displays the information related to the travel state in an overlapping manner with the information of the navigation device displayed on the display unit.

Images