Warning device for vehicle

Abstract

The present invention provides a warning device for a vehicle, which gives consideration to both safety and convenience when an operation of a portable device by a driver while the vehicle is traveling is detected. A warning device for a vehicle, which warns a driver in accordance with the state of the driver and the use state of a portable device by the driver during the travel of the vehicle, is provided with: a near-infrared image information acquisition unit that acquires near-infrared image information on the basis of an image captured by a near-infrared camera provided in the cabin; a visible light image information acquisition unit that acquires visible light image information on the basis of an image captured by a visible light camera provided in the vehicle interior; and a warning unit that determines the use state of the portable device by the driver on the basis of the near-infrared image information and the visible light image information, and gives a warning to the driver in accordance with the determination result, the warning unit being configured to warn the driver when the function of the portable device in use is a permitted function that is permitted to be used in advance. The warning unit stops the warning or reduces the degree of warning.

Description

Technical Field

[0001] The present invention relates to a warning device for a vehicle, and more particularly to a warning device for a vehicle that issues a warning based on an operation of a portable device by a driver while the vehicle is traveling. Background Art

[0002] With the widespread use of portable devices such as smartphones and tablets, drivers are increasingly observed looking at or operating these devices while driving, a phenomenon known as "mobile phone use." Because such driver behavior can lead to accidents, technologies are being developed to restrict drivers' use of these devices.

[0003] For example, Patent Document 1 discloses an information processing device that encourages safe driving when it determines that the driver is operating a device such as a smartphone based on the difference between position information of a specific part of the driver and the reflectivity of light near the specific part.

[0004] Patent Document 2 discloses a warning device that warns a driver when it is determined that the driver is using a portable device while the vehicle is traveling, based on the heat distribution inside the vehicle measured by an infrared sensor and the shift position detected by a shift position sensor.

[0005] Prior art literature Patent Literature Patent Document 1: Japanese Patent Application Laid-Open No. 2017-111508 Patent Document 2: Japanese Patent Application Publication No. 2018-156539 Summary of the Invention

[0006] Technical issues However, in recent years, an increasing number of drivers have been using the navigation function installed in their portable devices to obtain information necessary for driving, such as confirming their route and / or travel time to their destination. While the operation of portable devices while the vehicle is in motion should be strictly restricted, the operation / use of specific functions, such as navigation, may sometimes be permitted as long as safety is ensured.

[0007] However, because conventional information processing devices or warning devices always issue warnings when they determine that the driver is operating a portable device, any operation or use of the portable device is prohibited without exception. Consequently, even specific functions for obtaining information necessary for driving may not be allowed to be operated or used, resulting in a lack of convenience.

[0008] An object of the present invention is to address such a situation. That is, it is an object to give a warning to restrict the operation of a portable device by a driver during vehicle travel, and to relax the restriction for operations related to specific functions, taking into account safety and convenience, etc.

[0009] Solution To solve such a problem, a warning device for a vehicle according to the present invention has the following configuration.

[0010] That is, one aspect of the present invention provides a warning device for a vehicle that gives a warning to a driver based on the driver's state and the usage state of a portable device by the driver during vehicle travel. The warning device for a vehicle includes: a near-infrared image information acquisition unit that acquires near-infrared image information based on an image captured by a near-infrared camera provided in the vehicle interior of the vehicle; a visible light image information acquisition unit that acquires visible light image information based on an image captured by a visible light camera provided in the vehicle interior; and a warning unit that determines the usage state of the portable device by the driver based on the near-infrared image information and the visible light image information, and gives a warning to the driver according to the determination result. When the function of the portable device in use is a permitted function that is permitted to be used in advance, the warning unit stops the warning or reduces the degree of the warning.

[0011] Advantageous Effects According to the warning device for a vehicle having such a feature, it is possible to give a warning when it is detected that the driver operates the portable device during vehicle travel, and to relax the operation restriction for a specific function when a predetermined condition is satisfied, taking into account safety and convenience. Brief Description of the Drawings

[0012] Figure 1 It is an explanatory diagram showing a schematic configuration of a vehicle control system including a warning device for a vehicle according to an embodiment of the present invention.

[0013] Figure 2 It is an explanatory diagram showing an example of an image obtained by photographing the interior of a vehicle equipped with a warning device for a vehicle according to an embodiment of the present invention.

[0014] Figure 3 It is an explanatory diagram showing an example of an image obtained by photographing the interior of a vehicle equipped with a warning device for a vehicle according to an embodiment of the present invention.

[0015] Figure 4 It is an explanatory diagram showing an example of a screen of a portable device used in the interior of a vehicle equipped with a warning device for a vehicle according to an embodiment of the present invention.

[0016] Figure 5It is a flowchart showing the process of warning processing in a vehicle warning device according to an embodiment of the present invention.

[0017] Symbol Explanation 1: Vehicle control system, 3: In-vehicle network, 4: CGW, 5: Steering wheel 10: Warning ECU, 11: CPU, 14: Storage unit 20: Occupant monitoring ECU, 21: Camera, 22: Microphone 30: Color camera ECU, 31: Color camera, 32: Meter 40: Near-infrared camera ECU, 41: Near-infrared camera, 43: Instrument panel 50: Reporting ECU, 51: Speaker, 52: Display device 80: Portable device, 81: Specific part (hand), 82: QR code (registered trademark) 100: Vehicle, 311: Color image information acquisition unit, 311A: Screen information acquisition unit 411: Near-infrared image information acquisition unit, 411A: Device information acquisition unit 411B: Motion information acquisition unit, 511: Warning unit, 511A: Warning level determination unit Detailed Embodiment

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same reference numerals in different drawings denote parts having the same function, and repeated descriptions in each drawing are appropriately omitted.

[0019] As Figure 1 shown, the vehicle warning device according to the embodiment of the present invention functions as a part of the vehicle control system 1 and is provided in the vehicle 100. The vehicle control system 1 includes various electronic devices required for the vehicle 100 to travel, and a plurality of in-vehicle ECUs (Electronic Control Units) that control these electronic devices. Each electronic device and each in-vehicle ECU are connected via an in-vehicle network 3 such as CAN (Controller Area Network) and LIN (Local Interconnect Network) so as to be able to communicate with each other, and are connected to a central gateway (CGW) 4 as a relay device to form the vehicle control system 1.

[0020] Each in-vehicle ECU can be configured to include a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and / or a circuit, and a storage unit including a RAM (Random Access Memory) and / or a ROM (Read Only Memory). In addition, part or all of the operations performed by the in-vehicle ECU can be implemented by hardware such as an ASIC (application specific integrated circuit), an FPGA (field-programmable gate array), or a GPU (Graphics Processing Unit).

[0021] In the following description, detailed descriptions and illustrations of electronic devices and the like that are not directly related to the vehicle warning device (warning ECU 10) of the present embodiment are omitted.

[0022] As Figure 1 shown, the vehicle control system 1 is configured to include a warning ECU 10, an occupant monitoring ECU 20, a color camera ECU 30, a near-infrared camera ECU 40, and a reporting ECU 50 as in-vehicle ECUs. Each in-vehicle ECU is connected to an electronic device that is the control target of each in-vehicle ECU, and controls the operation of the electronic device to which it is connected based on information (data) obtained from the in-vehicle network 3. In addition, each in-vehicle ECU outputs information related to the operation of the connected electronic device obtained from the connected electronic device to the in-vehicle network 3.

[0023] That is, in the present embodiment, the warning ECU 10, the occupant monitoring ECU 20, the color camera ECU 30, the near-infrared camera ECU 40, and the reporting ECU 50 communicate with each other to control the operation of each electronic device. In particular, in the warning ECU 10, the state of the driver and the usage state of the portable device by the driver are monitored based on the information received from the occupant monitoring ECU 20, the color camera ECU 30, and the near-infrared camera ECU 40, and a warning is issued to the driver by sending a control signal related to the warning corresponding to the monitoring result to the reporting ECU 50. Details of the warning ECU 10 will be described later.

[0024] The occupant monitoring ECU 20 forms part of an occupant monitoring device (DMS: Driver Monitoring System) that monitors occupants including the driver of the vehicle. It monitors the occupants by controlling electronic devices such as the camera 21, microphone 22, lighting devices (not shown) such as LEDs that illuminate the occupants, a seating sensor (not shown) provided on the seat cushion, and a door opening / closing sensor (not shown) included in the occupant monitoring device and connected to the occupant monitoring ECU 20.

[0025] The color camera ECU 30 is connected to the color camera 31 which is a visible light camera, and controls the color camera 31 to monitor the use status of the driver's seat and portable devices located around the driver's seat. The color camera 31 is installed at a position where it can capture the driver's seat and the driver from a position above the driver's seat such as the ceiling or the pillar. For example, it captures the driver sitting on the driver's seat and the surrounding area of the driver at a predetermined cycle, and outputs the captured image as color image information to the color camera ECU 30.

[0026] In the color camera ECU 30, the color image information is output to the in-vehicle network 3, or various information included in the color image information is output to the in-vehicle network 3. The color camera ECU 30 can, for example, extract screen information indicating the status of the screen of the portable device and / or the display content as various information included in the color image information, and output the extracted screen information to the in-vehicle network 3.

[0027] It should be noted that in this embodiment, although the case of using the color camera 31 as an example of the visible light camera has been described, as long as it is possible to extract image information (color image information, visible light image information) including the screen information of the portable device, a visible light camera other than the color camera can be applied. In addition, when obtaining visible light image information from a visible light camera other than the color camera, the visible light image information can be a color image or a monochrome image.

[0028] In the color camera ECU 30, when the image captured by the color camera 31 contains icons, QR codes (registered trademarks), etc. that identify the functions (applications) being used displayed on the screen of the portable device, the icons and QR codes are included in the screen information as function identification information and output to the in-vehicle network 3.

[0029] It should be noted that in the screen information, the lighting information may include information indicating the lighting or extinguishing of the portable device screen based on the lighting state of the backlight of the portable device, and information indicating the degree of brightness of the screen, such as bright lighting or dim lighting when lit. The degree of brightness depends on the type and / or setting of the portable device. The backlight of the portable device usually becomes a bright lighting state when continuously operating the portable device, turns to a dim lighting state when not operated for a certain period of time, and then becomes an extinguished state.

[0030] In addition, especially when the screen of the portable device is brightly lit, the driver's line of sight is likely to turn to the screen of the portable device, which may induce distraction. In addition, when driving at night, the brightness difference between the screen of the portable device and the surrounding area becomes larger, which may cause unpleasant glare. Therefore, by including information indicating such a degree of brightness in the lighting information, it becomes useful information for determining the usage state of the portable device.

[0031] The near-infrared camera ECU 40 is connected to the near-infrared camera 41, controls the near-infrared camera 41 to monitor the movement of specific parts such as the driver's hand and / or fingers, and monitors the driver's seat and the portable device located around the driver's seat. Similar to the color camera 31, the near-infrared camera 41 is set at a position where it can photograph the driver's seat and the driver from a position higher than the driver's seat, such as the ceiling, pillars, etc. For example, it photographs the driver sitting in the driver's seat and the surrounding area of the driver at a predetermined cycle, and outputs the photographed image as near-infrared image information to the near-infrared camera ECU 40.

[0032] In the near-infrared camera ECU 40, the near-infrared image information is output to the vehicle network 3, or various information included in the near-infrared image information is output to the vehicle network 3. The near-infrared camera ECU 40 can extract, as the information included in the near-infrared image information, for example, motion information indicating the position and movement of specific parts of the driver, and device information indicating the shape, size, and the position of the screen of the portable device, etc., and output the extracted motion information and device information to the vehicle network 3.

[0033] When extracting motion information, the near-infrared camera ECU 40 can calculate the position coordinates of specific parts such as the driver's hand and / or fingers from the near-infrared image information, and include the position coordinates and their displacements in the motion information of the specific parts. In addition, when extracting device information, the near-infrared camera ECU 40 can calculate the position coordinates and their displacements of the portable device, and the position coordinates and their displacements of the screen of the portable device from the near-infrared image information, and include them in the device information.

[0034] For the color camera 31 and the near-infrared camera 41, as long as they can photograph the driver's seat and the driver from a position above the driver's seat, they can be installed at positions other than the ceiling or the pillar, and are preferably installed at a position where the interior of the vehicle can be observed more widely. By obtaining an image that can comprehensively capture the entire interior of the vehicle, not only can the state of the driver and the usage state of the portable device during vehicle driving be grasped, but also the state of the passengers sitting in the rear seats can be grasped, or it can also be used to prevent lost items left in the vehicle after it stops, the abandonment of infants, etc. In addition, the color camera 31 and the near-infrared camera 41 can be arranged close to each other or separately.

[0035] In Figure 2 and Figure 3 An example of an image captured by the color camera 31 is shown.

[0036] In the color camera 31, since the driver's seat and the driver are photographed from a position above the driver's seat such as the ceiling or the pillar, it is possible to photograph an image that includes, for example, in the field of view, on-vehicle devices such as the steering wheel 5, the instrument 32 in front of the driver's seat, the display device 52, the instrument panel 43, specific parts 81 such as the driver's hand and / or finger, and the screen of the portable device 80 held by the driver.

[0037] In Figure 2 An example of an image in which the driver is holding the portable device 80 with a specific part (hand) 81 is shown. In Figure 3 An example of an image in which the portable device 80 is fixed at a predetermined position on the instrument panel 43 is shown. In Figure 2 and Figure 3 On the screen of the portable device 80, a QR code (registered trademark) 82 described later is displayed. By obtaining such an image as a moving image or continuously or periodically as a still image, it is possible to grasp the state of the screen of the portable device and / or the display content of the screen that changes moment by moment.

[0038] However, in the image obtained by irradiating light containing near-infrared rays on the driver and the periphery of the driver as the subject with the near-infrared camera 41, differences in the reflection and / or absorption of light caused by material differences of the subject are reflected. That is, in the near-infrared camera 41, regardless of the brightness inside the vehicle, an image that clearly reflects the shape of the portable device, the size of the portable device itself, the position of the screen of the portable device, the size of the screen, specific parts such as the driver's hand and / or fingers, and the position of the specific parts, etc., can be captured. That is, for example, even in a state where the driver's hand and / or fingers touch the screen of the portable device and partially cover the screen, an image that can identify the hand, fingers, and the screen of the portable device at the material level can be obtained.

[0039] Therefore, by obtaining a moving image with the near-infrared camera 41, or continuously or periodically obtaining a still image, it is possible to obtain motion information related to the movement of the driver's hand and / or fingers, position coordinates indicating the position of the portable device screen, and accurate device information related to the displacement of the position coordinates.

[0040] When the near-infrared camera 41 is configured to be close to the color camera 31, the layout of the image captured by the near-infrared camera 41 becomes substantially the same as the layout of the image captured by the color camera 31. Therefore, when the near-infrared camera 41 is configured to be close to the color camera 31, the image captured by the near-infrared camera 41 and Figure 2 and Figure 3 the image captured by the color camera 31 shown have the same layout, and become an image that clearly reflects the shape and size of the portable device, the position and size of the screen in the portable device, the driver's hand and / or fingers, and their positions, etc.

[0041] In Figure 4 an example of the screen of the portable device 80 is shown. In the example of Figure 4 , the screen in the case of using the navigation function in the portable device 80 is shown. In the example of Figure 4 , a QR code (registered trademark) 82 indicating that the navigation function is being used as function-specific information is displayed at the end of the screen. In the color camera 31, when such a QR code (registered trademark) is included in the obtained image, the QR code (registered trademark) can be included in the screen information as function-specific information. As function-specific information, a QR code (registered trademark) can be displayed on the screen of the portable device 80, and naturally, in addition to this, matrix-type or stacked two-dimensional codes, icons, watermarks, etc. can also be displayed.

[0042] The reporting ECU 50 is connected to, for example, a speaker 51 and a display device 52, and controls the speaker 51 or the display device 52 in such a way as to give a warning corresponding to the warning level to the driver of the vehicle 100 according to the control signal from the warning ECU 10.

[0043] The speaker 51 can output, as sound, a warning message (described later) determined in association with the warning level according to the reporting instruction, or output a predetermined warning sound or the like. The speaker 51 does not necessarily need to be set for reporting and / or warning purposes. For example, it can be shared with the speakers of the in-vehicle audio device and / or the navigation system.

[0044] The display device 52 is composed of, for example, a display panel provided in the instrument panel, a HUD (Head-Up Display) that projects onto the front window, etc., and displays an image that can be visually confirmed by the occupant. When the display device 52 is a HUD, a message and an image corresponding to the warning level can be displayed on the front window of the vehicle 100, thereby warning the occupants of the vehicle 100, especially the driver.

[0045] In addition, the reporting ECU 50 can, for example, also warn the driver by vibration according to the warning level by controlling a seat cushion vibrator that vibrates the seat cushion of the driver's seat where the driver is seated and a steering wheel vibrator that vibrates the steering wheel.

[0046] Next, the warning ECU 10 as a vehicle warning device will be described. As Figure 1 shown, the warning ECU 10 includes a CPU (Central Processing Unit) 11 and a storage unit 14. It should be noted that the storage unit 14 includes a volatile memory such as a RAM that temporarily processes data used by the CPU 11, and a non-volatile memory that can rewrite a ROM that stores programs executed by the CPU 11 and the like.

[0047] The CPU 11 executes various processes based on the programs stored in the storage unit 14. In the present embodiment, the CPU 11 functions as Figure 1 shown, a color image information acquisition unit 311, a near-infrared image information acquisition unit 411, and a warning unit 511 that are visible light image information acquisition units.

[0048] In addition, information necessary for the CPU 11 to act as the color image information acquisition unit 311, the near-infrared image information acquisition unit 411, and the warning unit 511 is stored in the storage unit 14. Hereinafter, the color image information acquisition unit 311, the near-infrared image information acquisition unit 411, and the warning unit 511 will be described.

[0049] The color image information acquisition unit 311 acquires color image information based on an image captured by the color camera 31 mounted in the vehicle interior of the vehicle 100. Specifically, the color image information acquisition unit 311 includes a screen information acquisition unit 311A that acquires the screen information included in the color image information via the in-vehicle network 3.

[0050] The screen information acquisition unit 311A acquires the state of the screen of the portable device extracted from the color image information and the screen information indicating the display content of the screen. The screen information at least includes lighting information indicating the lighting state of the screen of the portable device, and such function specific information in the case where an icon of a specific function (application) being used or a two-dimensional code such as a QR code (registered trademark) is displayed on the screen of the portable device.

[0051] When the driver uses the portable device during vehicle travel, there is a concern that it may lead to negligence of the front due to staring at the screen of the portable device or operating it, which may induce a vehicle accident. Therefore, when the driver is using the portable device during vehicle travel, it is necessary to warn (alert) the driver. Thus, in the screen information acquisition unit 311A, the screen information including the lighting information indicating the lighting state of the screen of the portable device is acquired as one of the information for determining the state of the driver's use of the portable device.

[0052] On the other hand, there is also a case where the navigation function mounted on the portable device is used to confirm the driving route and / or driving time to the destination, etc., and information required for driving is acquired from the portable device. In the case of using a specific function such as the navigation function and as long as safety is ensured, sometimes the same warning as when using other functions may not be given, and a certain degree of use may be permitted. Therefore, by including the function specific information in the screen information, in the warning level determination unit 511A described later, it is possible to determine whether the function being used in the portable device is a permitted function that is pre-permitted to be used.

[0053] In addition to being able to acquire the screen information extracted from the color image information from the color camera ECU 30 via the in-vehicle network 3, the screen information acquisition unit 311A can also acquire the color image information that is the image captured by the color camera 31, and obtain the screen information from the color image information through calculation in the screen information acquisition unit 311A.

[0054] The near-infrared image information acquisition unit 411 acquires near-infrared image information based on an image captured by the near-infrared camera 41 installed in the vehicle interior of the vehicle 100. Specifically, the near-infrared image information acquisition unit 411 includes a device information acquisition unit 411A and a motion information acquisition unit 411B, and acquires the device information and motion information included in the near-infrared image information via the in-vehicle network 3.

[0055] The device information acquisition unit 411A acquires device information indicating features of the appearance such as the shape and size of the portable device, and the position of the screen of the portable device. The device information at least includes information related to the position coordinates and displacement of the portable device, and the position coordinates and displacement of the screen of the portable device.

[0056] The motion information acquisition unit 411B acquires motion information indicating the position and motion of a specific part of the driver. The motion information at least includes information related to the position coordinates and displacement of a specific part such as the driver's hand and / or finger.

[0057] By acquiring the device information and motion information, in the warning level determination unit 511A described later, in combination with the screen information, it is possible to determine, for example, whether there is a motion such as the driver's finger and / or hand touching the screen of the portable device, which is a suspected operation of the portable device, and it is possible to determine the usage state of the portable device.

[0058] In addition to being able to directly acquire the motion information and device information extracted from the near-infrared image information from the near-infrared camera ECU 40 via the in-vehicle network 3, the device information acquisition unit 411A and the motion information acquisition unit 411B can also acquire the image captured by the near-infrared camera 41, and respectively acquire the motion information and screen information from the image through calculation.

[0059] The warning unit 511 determines the usage state of the portable device by the driver based on the near-infrared image information and the visible light image information, and gives a warning to the driver according to the determination result. More specifically, the warning unit 511 determines the state of the driver and the usage state of the portable device by the driver based on the screen information included in the color image information, the device information and the motion information included in the near-infrared image information, and gives a warning to the driver according to the state of the driver and the usage state of the portable device by the driver.

[0060] In the warning unit 511, a learned model that has learned screen information, device information, and motion information can be used to determine the usage state of the portable device. In this case, the learned model that has learned screen information, device information, and motion information is pre-stored in the storage unit 14. Then, the CPU 11 that functions as the warning unit 511 can input the screen information, device information, and motion information by reading the learned model stored in the storage unit 14 and output the usage state of the portable device.

[0061] In addition, a learned model that has learned color image information and near-infrared image information may be stored in the storage unit 14. In this case, the warning unit 511 can input the color image information and the near-infrared image information into the learned model, thereby obtaining the screen information included in the color image information, and the device information and motion information included in the near-infrared image information through the learned model, and determining the usage state of the portable device based on the extracted information.

[0062] In addition, in the warning unit 511, when the function of the portable device in use during warning is a permitted function that is pre-permitted to be used, the warning is stopped or the degree of warning is reduced.

[0063] Therefore, the warning unit 511 has a warning level determination unit 511A, and the warning level determination unit 511A determines the warning level corresponding to the state of the driver and the usage state of the portable device by the driver based on the device information, motion information, and screen information. It should be noted that the warning level determination unit 511A can determine warning levels at multiple stages corresponding to the state of the driver and the usage state of the portable device by the driver.

[0064] It should be noted that the warning level determination unit 511A can obtain, for example, monitoring information based on the result of monitoring the state of the occupant such as the seating state and line of sight of the occupant from the occupant monitoring ECU 20 via the vehicle-mounted network 3, and use this monitoring information for the determination of the warning level.

[0065] For example, by obtaining the line-of-sight information related to the driver's line of sight in the monitoring information, it is possible to grasp whether the driver's line of sight is in the front (whether distracted driving is not being performed), whether the driver is staring at the screen of the portable device, etc., and it is possible to give a more appropriate determination of the warning level. For the line-of-sight information obtained from the occupant monitoring ECU 20, the direction of the driver's line of sight and the position coordinates of the target in front of the line of sight can be calculated based on the image captured by the camera 21, and it can include at least any one of information related to the direction of the line of sight, the calculated position coordinates, and the displacement of the position coordinates.

[0066] In addition, the warning level determination unit 511A acquires control information indicating whether the vehicle 100 is traveling under the control of the advanced driving assistance mode, speed information indicating the traveling speed of the vehicle, and uses these pieces of information for the determination of the warning level. By reflecting the speed information such as the warning level being increased as the vehicle speed becomes faster when acquiring the speed information in the final determination result, a more accurate determination of the warning level is performed. It should be noted that the advanced driving assistance mode is a mode that assists driving by, for example, partially automating driving under certain conditions.

[0067] The warning unit 511 issues a warning corresponding to the warning level determined by the warning level determination unit 511A. Specifically, the warning unit 511 outputs a control signal of a warning corresponding to the above warning level to the reporting ECU 50 via the in-vehicle network 3, and warns the driver via the reporting ECU 50.

[0068] The warning unit 511 can warn the driver, for example, by any one or a combination of the output of the sound of the warning message, the display of the warning message, the ringing of the warning sound, the vibration of the seat cushion of the driver's seat, and the vibration of the steering wheel. Then, the warning unit 511 issues a warning corresponding to the warning level by changing the volume of the warning sound or the sound of the warning message, or by changing the intensity of the vibration of the seat cushion and the steering wheel according to the warning level.

[0069] The information received from the screen information acquisition unit 311A, the device information acquisition unit 411A, and the motion information acquisition unit 411B is stored in the storage unit 14. In addition, a table associating the state of the driver and the usage state of the portable device with the warning level corresponding to that state, and / or a table associating the warning level with the warning message corresponding to that warning level, etc. may also be stored in the storage unit 14.

[0070] Hereinafter, for the warning process in the warning ECU 10 configured as such, it will be described according to Figure 5 a flowchart showing the process flow of the warning process. It should be noted that in the following description, the case where the navigation function is used as the permitted function for the portable device will be described.

[0071] In the warning ECU 10, the screen information acquisition unit 311A acquires screen information indicating the state of the screen of the portable device and the display content (step S101), the device information acquisition unit 411A acquires device information indicating the shape and / or position of the portable device (step S102), the motion information acquisition unit 411B acquires motion information indicating the position and motion of a specific part such as the driver's hand and fingers (step S103), and outputs them to the warning level determination unit 511A respectively.

[0072] If the screen information includes function-specific information, the warning level determination unit 511A determines, based on the function-specific information, whether the function currently being used in the portable device is the navigation function (step S104). If the warning level determination unit 511A determines, based on the function-specific information, that the function currently being used in the portable device is not the navigation function, that is, not a permitted function (No in step S104), the warning level determination process is performed as if a function other than the permitted function is being used (step S105).

[0073] On the other hand, if it is determined that the function being used in the portable device is the navigation function (YES in step S104), a warning level determination process is performed for use of the permitted function (step S106). If a warning level is determined in the warning level determination process (steps S105 or S106), the warning unit 511 determines the warning content corresponding to the determined warning level (step S107). Next, a control signal indicating the warning content determined in step S107 is output to the notification ECU 50 (step S108), thereby providing a warning to the driver.

[0074] Warning level determination unit 511A determines the warning level based on the acquired screen information, device information, and motion information, such as whether the portable device's screen is lit, whether the driver's hand and / or finger are moving in a manner suggestive of operating the portable device, and whether the portable device is in a predetermined location. This determines the driver's status and the status of the portable device being used. If a permitted function is being used, a lower warning level than that used for normal functions can be set, thereby allowing the permitted function to be used while ensuring safety and improving convenience.

[0075] The warning level determination process (step S106) when the permitted function is used and the warning level determination process (step S105) when the normal function is used can be respectively performed as independent determination processes. In the warning level determination process (step S106) when the permitted function is used, the same determination process as the warning level determination process (step S105) when the normal function is used can be performed. However, if it is determined that the permitted function is being used, a warning level lower than the warning level when the normal function is used can be set.

[0076] Thus, in the warning level determination unit 511A of the vehicle warning device according to the present embodiment, when the function of the portable device in use is a permitted function that is pre-permitted to be used, and when a predetermined condition is satisfied, a warning level lower than that when using a normal function is selected. That is, during vehicle travel, when a portable device for a normal function other than the permitted function is used, a warning for restricting this situation is given. For use related to a specific permitted function, the restriction can be mitigated in a permitted manner as long as a predetermined condition is satisfied, thereby taking into account both safety and convenience.

[0077] As described above, the warning level determination unit 511A obtains the above-described screen information, device information, and motion information based on the color image information acquired by the color camera 31 and the near-infrared image information acquired by the near-infrared camera 41. Since the near-infrared image information reflects the differences in the reflection and / or absorption characteristics of light caused by the material differences of the subject, regardless of the brightness inside the vehicle, the shape of the portable device, the size of the portable device itself, the position of the screen of the portable device, the size of the screen, specific parts such as the driver's hand and / or finger, and the position of the specific parts can be accurately grasped.

[0078] Therefore, for example, even in a state where the driver's hand and / or finger touches the screen of the portable device and partially covers the touch screen, etc., the screen of the portable device, the hand, the finger, etc. can be recognized at the material level. On the other hand, the color image information can accurately grasp the information displayed on the screen of the portable device, etc., so that specific function information displayed on the screen can be obtained.

[0079] That is, in the vehicle warning device according to the present embodiment, by using the color image information acquired by the color camera 31 and the near-infrared image information acquired by the near-infrared camera 41, the state of the driver and the usage state of the portable device can be grasped more accurately, and the best warning can be given to the driver.

[0080] As Figure 1 shown, by mounting the vehicle control system 1 including such a vehicle warning device (warning ECU) in the vehicle 100, in the vehicle 100, it is possible to appropriately warn (give a warning) the driver about the use of the portable device during vehicle travel, and to mitigate the usage restrictions for predetermined permitted functions such as the navigation function. Therefore, both safety and convenience can be taken into account.

[0081] As described above, according to the vehicle warning device of the present embodiment, it is possible to give a warning for restricting the driver's operation of the portable device during vehicle travel, and to mitigate the restrictions for operations related to specific functions, taking into account both safety and convenience.

[0082] With reference to the accompanying drawings, the embodiments of the present invention have been described in detail. However, the specific configuration is not limited to these embodiments, and even design changes and the like within the scope of not departing from the gist of the present invention are included in the present invention. In addition, as long as there are no particular contradictions or problems in the purpose, configuration, etc. of the above-described embodiments, the techniques of each other can be combined.

Claims

1. A warning device for a vehicle, characterized in that, During the driving of a vehicle, a warning is issued to the driver according to the driver's state and the state of the driver's use of a portable device. The vehicle warning device includes: A near-infrared image information acquisition unit that acquires near-infrared image information based on an image captured by a near-infrared camera equipped in the vehicle interior of the vehicle. A visible light image information acquisition unit that acquires visible light image information based on an image captured by a visible light camera equipped in the vehicle interior. And A warning unit that determines the state of the driver's use of the portable device based on the near-infrared image information and the visible light image information, and issues a warning to the driver according to the determination result. When the function in use of the portable device is a permitted function that is permitted to be used in advance, the warning unit stops the warning or reduces the degree of the warning.

2. The vehicle warning device according to claim 1, wherein: The near-infrared image information includes: motion information indicating the position and motion of a specific part of the driver; and device information including the position and appearance characteristics of the portable device. The visible light image information includes screen information indicating the display content of the screen of the portable device.

3. The vehicle warning device according to claim 2, wherein: The warning unit determines whether the function in use of the portable device is the permitted function based on the screen information.

4. The vehicle warning device according to claim 3, wherein: The permitted function is a navigation function.

5. The vehicle warning device according to claim 2, wherein: The screen information includes lighting information indicating the lighting state of the screen of the portable device, and function specific information displayed on the screen during the use of the permitted function.

6. The vehicle warning device according to claim 5, wherein: The function specific information is an icon or a QR code.

7. The vehicle warning device according to claim 1, wherein: When the vehicle is driving in a highly automated driving assistance mode, the warning unit reduces the degree of the warning.

8. The vehicle warning device according to claim 1, wherein: The warning unit acquires speed information indicating the driving speed of the vehicle, and the higher the driving speed, the higher the degree of the warning.

9. The vehicle warning device according to claim 1, wherein: The near-infrared camera and the visible light camera are equipped on the ceiling or pillar in the vehicle interior.

Images