Vehicle warning device

Abstract

To achieve both of safety and convenience when detecting an operation of a portable device by a driver during traveling a vehicle.SOLUTION: A vehicle warning device issues warning to a driver in response to a state of the driver and a use state of a portable device by the driver. The vehicle warning device includes: a near-infrared image information acquisition section for acquiring near-infrared image information based on an image photographed by a near-infrared camera arranged in a cabin; a visible light image information acquisition section for acquiring visible light image information based on an image photographed by a visible light camera arranged in a cabin; and a warning section for determining a use state of the portable device by the driver based on the near-infrared image information and the visible light image information, so as to issue warning with respect to the driver in response to a determination result. The warning section stops the warning or reduces the level of the warning when a function being used in the portable device is a permission function whose use is pre-permitted.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a warning device for a vehicle, and more particularly to a warning device for a vehicle that issues a warning in response to an operation on a portable device by a driver while the vehicle is traveling. [Background technology]

[0002] With the widespread use of mobile devices such as smartphones and tablet computers, it has become common to see drivers staring at or operating their mobile devices while driving, a practice known as "smartphone use while driving." Because such driver behavior can lead to vehicle accidents, technologies have been proposed to restrict drivers from operating mobile devices.

[0003] For example, Patent Document 1 discloses an information processing device that urges a driver to drive safely if it determines that the driver is operating a device such as a smartphone based on the position information of a specific part of the driver's body and the difference in light reflectance near the specific part. Furthermore, Patent Document 2 discloses a warning device that warns the driver when it is determined that the driver is using a portable device while the vehicle is moving, based on the heat distribution inside the vehicle measured by an infrared sensor and the shift position detected by a shift sensor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-111508 [Patent Document 2] Japanese Patent Application Laid-Open No. 2018-156539 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, an increasing number of drivers are using their mobile devices to obtain information necessary for driving, such as using the navigation function built into the device to check the route to a destination and the driving time. Although the operation of mobile devices while driving a vehicle should be strictly restricted, there are cases where the operation and use of specific functions, such as the navigation function, may be permitted only when safety is ensured.

[0006] However, since the above-mentioned conventional information processing devices or warning devices uniformly issue a warning when it is determined that the driver is operating a mobile device, the operation or use of a mobile device is treated as a prohibited act without exception. Therefore, even if a specific function is used to obtain information necessary for driving, the operation or use of the mobile device is not permitted at all, which can be inconvenient.

[0007] The present invention addresses such a situation by providing a warning to restrict the driver's operation of a mobile device while the vehicle is in motion, while relaxing restrictions on operations related to specific functions, thereby achieving both safety and convenience. [Means for solving the problem]

[0008] In order to solve the above problems, the warning device for a vehicle according to the present invention has the following configuration. That is, one aspect of the present invention is a vehicle warning device that issues a warning to a driver while the vehicle is traveling depending on the driver's condition and the driver's use of a portable device, the vehicle warning device comprising: a near-infrared image information acquisition unit that acquires near-infrared image information based on an image captured by a near-infrared camera installed in the passenger compartment of the vehicle; a visible light image information acquisition unit that acquires visible light image information based on an image captured by a visible light camera installed in the passenger compartment; and a warning unit that determines the driver's use of the portable device based on the near-infrared image information and the visible light image information and issues a warning to the driver depending on the determination result, wherein the warning unit stops the warning or reduces the level of the warning when a function currently being used on the portable device is a permitted function that has been previously permitted to be used. [Effects of the Invention]

[0009] A vehicle warning device with these characteristics can issue a warning when it detects the driver operating a mobile device while the vehicle is in motion, and can relax operational restrictions on certain functions when certain conditions are met, thereby achieving both safety and convenience. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is an explanatory diagram showing a schematic configuration of a vehicle control system including a vehicle warning device according to an embodiment of the present invention; [Figure 2] 1 is an explanatory diagram showing an example of an image captured inside a vehicle cabin in which a warning device for a vehicle according to an embodiment of the present invention is installed; [Figure 3] 1 is an explanatory diagram showing an example of an image captured inside a vehicle cabin in which a warning device for a vehicle according to an embodiment of the present invention is installed; [Figure 4] 1 is an explanatory diagram showing an example of a screen of a mobile device used in a vehicle interior equipped with a vehicle warning device according to an embodiment of the present invention; [Figure 5] 3 is a flowchart showing a flow of a warning process in the vehicle warning device according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same reference numerals in different drawings denote parts with the same functions, and duplicated descriptions in each drawing will be omitted as appropriate.

[0012] 1, a vehicle warning device according to an embodiment of the present invention functions as a part of a vehicle control system 1 and is provided in a vehicle 100. The vehicle control system 1 includes various electronic devices necessary for driving the vehicle 100 and a plurality of in-vehicle ECUs (Electronic Control Units) that control these electronic devices. The electronic devices and in-vehicle ECUs are connected to each other via an in-vehicle network 3 such as a CAN (Controller Area Network) or a LIN (Local Interconnect Network) so that they can communicate with each other, and are also connected to a central gateway (CGW) 4 that serves as a relay device, thereby constituting the vehicle control system 1.

[0013] Each in-vehicle ECU may be configured to include, for example, a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), electrical circuits, and a storage unit including a RAM (Random Access Memory), a ROM (Read Only Memory), etc. Also, some or all of the operations performed by the in-vehicle ECU may be realized by hardware such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit). In the following description, detailed description and illustration of electronic devices and the like that are not directly involved in the vehicle warning device (warning ECU 10) according to this embodiment will be omitted.

[0014] 1, the vehicle control system 1 includes, as on-board ECUs, a warning ECU 10, an occupant monitoring ECU 20, a color camera ECU 30, a near-infrared camera ECU 40, and an annunciation ECU 50. Each on-board ECU is connected to an electronic device that is the control target of that on-board ECU, and controls the operation of the connected electronic device based on information (data) acquired from an on-board network 3. Furthermore, each on-board ECU outputs information related to the operation of each electronic device acquired from the connected electronic device to the on-board network 3.

[0015] That is, in this embodiment, the warning ECU 10, the occupant monitoring ECU 20, the color camera ECU 30, the near-infrared camera ECU 40, and the notification ECU 50 transmit and receive information to each other and control the operation of each electronic device. In particular, the warning ECU 10 monitors the driver's condition and the driver's use of a portable device based on information received from the occupant monitoring ECU 20, the color camera ECU 30, and the near-infrared camera ECU 40, and issues a warning to the driver by transmitting a control signal related to a warning based on the monitoring result to the notification ECU 50. The warning ECU 10 will be described in detail later.

[0016] The occupant monitoring ECU 20 forms part of a driver monitoring system (DMS) that monitors occupants including the driver of the vehicle, and monitors occupants by controlling electronic devices included in the occupant monitoring system and connected to the occupant monitoring ECU 20, such as a camera 21, a microphone 22, lighting devices (not shown) such as LEDs that illuminate occupants, seating sensors (not shown) installed on the seats, and door opening / closing sensors (not shown).

[0017] The color camera ECU 30 is connected to a color camera 31, which is a visible light camera, and controls the color camera 31 to monitor the usage status of portable devices at the driver's seat and around the driver's seat. The color camera 31 is installed in a position where it can capture an image of the driver's seat and the driver from a position above the driver's seat, such as on the ceiling or pillar, and captures images of the driver seated in the driver's seat and the area around the driver at a predetermined interval, and outputs the captured images to the color camera ECU 30 as color image information.

[0018] The color camera ECU 30 outputs the color image information to the in-vehicle network 3, or outputs various information included in the color image information to the in-vehicle network 3. The color camera ECU 30 can extract, for example, screen information indicating the screen status and display content of the mobile device as information included in the color image information, and output the extracted screen information to the in-vehicle network 3.

[0019] In this embodiment, a case where color camera 31 is used as an example of a visible light camera is described, but any visible light camera other than a color camera can be used as long as it can extract image information (color image information, visible light image information) including screen information of a mobile device. Furthermore, when obtaining visible light image information from a visible light camera other than a color camera, the visible light image information may be a color image or a monochrome image.

[0020] In the color camera ECU 30, if the image captured by the color camera 31 includes an icon or a two-dimensional code such as a QR code (registered trademark) that identifies the function (application) currently in use and is displayed on the screen of the mobile device, the color camera ECU 30 includes this as function identification information in the screen information and outputs it to the in-vehicle network 3.

[0021] The screen information may include, as lighting information, information indicating whether the screen of the portable device is on or off based on the lighting state of the backlight of the portable device, and information indicating the degree of brightness of the screen, such as whether it is brightly lit or dimly lit if it is on. The degree of brightness depends on the type and settings of the portable device. Generally, the backlight of a portable device stays brightly lit when an operation on the portable device continues, and if no operation is performed for a certain period of time, it transitions to a dimly lit state and then turns off.

[0022] In particular, when the screen of a mobile device is brightly lit, the driver's line of sight tends to be directed toward the screen, which may induce inattention. In addition, when driving at night, the difference in brightness between the screen of the mobile device and the surroundings becomes large, which may cause unpleasant glare. Therefore, including information indicating the degree of brightness in the lighting information provides useful information for determining the use state of the mobile device.

[0023] The near-infrared camera ECU 40 is connected to the near-infrared camera 41 and controls the near-infrared camera 41 to monitor the movements of specific parts of the driver, such as the hands and fingers, as well as monitor the driver's seat and portable devices located around the driver's seat. Similar to the color camera 31, the near-infrared camera 41 is installed in a position where it can capture an image of the driver's seat and the driver from a position above the driver's seat, such as on the ceiling or pillar, and, for example, captures an image of the driver seated in the driver's seat and the area around the driver at a predetermined interval, and outputs the captured image to the near-infrared camera ECU 40 as near-infrared image information.

[0024] The near-infrared camera ECU 40 outputs the near-infrared image information to the in-vehicle network 3, or outputs various information included in the near-infrared image information to the in-vehicle network 3. The near-infrared camera ECU 40 can extract, from the information included in the near-infrared image information, for example, movement information indicating the position and movement of specific parts of the driver and device information indicating external characteristics of the mobile device, such as the shape, size, and screen position of the mobile device, and output the extracted movement information and device information to the in-vehicle network 3.

[0025] When extracting movement information, the near-infrared camera ECU 40 can calculate the position coordinates of specific parts such as the driver's hands and fingers from the near-infrared image information and include the position coordinates and their displacement in the movement information of the specific parts. When extracting device information, the near-infrared camera ECU 40 can calculate the position coordinates and their displacement of the mobile device and the position coordinates and displacement of the mobile device screen from the near-infrared image information and include these in the device information.

[0026] Both the color camera 31 and the near-infrared camera 41 may be installed in a location other than the ceiling or pillar as long as they are able to capture images of the driver's seat and the driver from a position above the driver's seat, and are preferably installed in a location that provides a wider view of the interior of the vehicle. By acquiring images that provide a broad view of the entire interior of the vehicle, it is possible to not only ascertain the driver's condition and the use of mobile devices while the vehicle is in motion, but also to ascertain the condition of passengers seated in the backseats and to prevent items being left behind in the vehicle or infants being left behind after the vehicle has stopped. The color camera 31 and the near-infrared camera 41 may be installed close to each other or at a distance from each other.

[0027] 2 and 3 show examples of images captured by the color camera 31. FIG. The color camera 31 captures images of the driver's seat and the driver from a position above the driver's seat, such as on the ceiling or pillar, and can therefore capture images that include, for example, the steering wheel 5, the meter 32 in front of the driver's seat, on-board equipment such as the display device 52, the dashboard 43, specific parts 81 such as the driver's hands and fingers, and the screen of the mobile device 80 held by the driver.

[0028] Fig. 2 shows an example of an image in which the driver is holding the mobile device 80 in a specific part (hand) 81. Fig. 3 shows an example of an image in which the mobile device 80 is fixed at a predetermined position on the dashboard 43. In Figs. 2 and 3, a QR code (registered trademark) 82, which will be described later, is displayed on the screen of the mobile device 80. By acquiring such images as moving images or continuously or periodically acquiring still images, it is possible to grasp the state of the mobile device screen and the display content that change from moment to moment.

[0029] Incidentally, images obtained by irradiating the driver and the area around the driver with light including near-infrared rays using near-infrared camera 41 reflect differences in the light reflection and absorption characteristics due to differences in the material of the subject. That is, regardless of the brightness of the interior of the vehicle, near-infrared camera 41 can capture images that clearly reflect the shape of the mobile device, the size of the mobile device itself, the position of the mobile device screen, the size of the screen, specific parts and positions of specific parts such as the driver's hands and fingers, etc. That is, even if the driver's hands or fingers are touching the screen of the mobile device and partially covering the screen, for example, it is possible to obtain images in which the screen of the mobile device, hands, fingers, etc. can be identified at the material level.

[0030] Therefore, by capturing video or continuously or periodically capturing still images using the near-infrared camera 41, it is possible to obtain movement information about the movement of the driver's hands and fingers, position coordinates indicating the position of the screen of the mobile device, and accurate device information about the displacement of the position coordinates.

[0031] When near-infrared camera 41 is placed close to color camera 31, the layout of the image captured by near-infrared camera 41 will be substantially the same as the layout of the image captured by color camera 31. Therefore, when near-infrared camera 41 is placed close to color camera 31, the image captured by near-infrared camera 41 will have the same layout as the image captured by color camera 31 shown in Figures 2 and 3, but will clearly reflect the shape and size of the portable device, the position and size of the screen on the portable device, the driver's hands and fingers and their positions, etc.

[0032] FIG. 4 shows an example of the screen of the mobile device 80. The example of FIG. 4 shows a screen when the navigation function is being used on the mobile device 80. In the example of FIG. 4, a QR code (registered trademark) 82 is displayed at the edge of the screen as function identification information indicating that the navigation function is being used. When such a QR code (registered trademark) is included in the acquired image, the color camera 31 can include the QR code (registered trademark) in the screen information as function identification information. The screen of the mobile device 80 can display not only a QR code (registered trademark), but also other matrix or stack type two-dimensional codes, icons, watermarks, and the like as function identification information.

[0033] The notification ECU 50 is connected to, for example, a speaker 51 and a display device 52, and controls the speaker 51 or the display device 52 in accordance with a control signal from the warning ECU 10 to give a warning to the driver of the vehicle 100 according to the warning level.

[0034] In response to a notification instruction, speaker 51 can output a warning message (described later) determined in association with a warning level as sound, or can output a predetermined warning sound, etc. Speaker 51 does not necessarily have to be provided for notifications or warnings, and can also be used as a speaker for an in-vehicle audio device or a navigation system, for example.

[0035] The display device 52 is configured, for example, by a display panel provided on the dashboard or a HUD (Head-Up Display) projected onto the windshield, and displays an image visible to the occupants. When the display device 52 is a HUD, it can display a message or image according to the warning level on the windshield of the vehicle 100 to warn the occupants of the vehicle 100, particularly the driver.

[0036] In addition to this, the notification ECU 50 can also control, for example, a seat vibrator that vibrates the driver's seat in which the driver sits, or a steering vibrator that vibrates the steering wheel, to issue a warning to the driver by vibration depending on the warning level.

[0037] Next, a warning ECU 10 serving as a warning device for a vehicle will be described. As shown in Fig. 1, the warning ECU 10 includes a CPU (Central Processing Unit) 11 and a storage unit 14. The storage unit 14 includes a volatile memory such as a RAM that temporarily processes data used by the CPU 11, and a rewritable non-volatile memory such as a ROM that stores programs executed by the CPU 11. The CPU 11 executes various processes based on the programs stored in the storage unit 14. In this embodiment, the CPU 11 executes the programs stored in the storage unit 14, thereby functioning as a color image information acquisition unit 311, a near-infrared image information acquisition unit 411, and a warning unit 511, all of which are visible light image information acquisition units, as shown in FIG.

[0038] The storage unit 14 also stores information necessary for the CPU 11 to operate as the color image information acquisition unit 311, the near-infrared image information acquisition unit 411, and the warning unit 511. The color image information acquisition unit 311, the near-infrared image information acquisition unit 411, and the warning unit 511 will be described below.

[0039] The color image information acquisition unit 311 acquires color image information based on an image captured by the color camera 31 installed in the cabin of the vehicle 100. Specifically, the color image information acquisition unit 311 has a screen information acquisition unit 311A that acquires screen information included in the color image information via the in-vehicle network 3.

[0040] The screen information acquisition unit 311A acquires screen information indicating the screen status and display content of the mobile device extracted from the color image information. The screen information includes at least lighting information indicating the lighting status of the mobile device screen, and function identification information when an icon identifying a function (application) in use or a two-dimensional code such as a QR code (registered trademark) is displayed on the mobile device screen.

[0041] When a driver uses a mobile device while the vehicle is moving, staring at or operating the screen of the mobile device can cause the driver to lose focus on the road ahead, which could lead to a vehicle accident. For this reason, if the driver is using a mobile device while the vehicle is moving, a warning (attention call) must be issued to the driver. For this reason, the screen information acquisition unit 311A acquires screen information including lighting information indicating the lighting state of the screen of the mobile device as one piece of information for determining the state of use of the mobile device by the driver.

[0042] On the other hand, there are cases where information necessary for driving is obtained from the mobile device, such as checking the route and driving time to a destination using a navigation function installed in the mobile device. When using a specific function such as a navigation function, and only when safety is ensured, it may be acceptable to allow the use to a certain extent rather than issuing a warning like the use of other functions. Therefore, by including function specification information in the screen information, it is possible for the warning level determination unit 511A, which will be described later, to determine whether the function being used on the mobile device is an authorized function whose use has been permitted in advance.

[0043] The screen information acquisition unit 311A can acquire screen information extracted from color image information from the color camera ECU 30 via the in-vehicle network 3, or it can acquire color image information as an image captured by the color camera 31 and acquire screen information from the color image information by calculation in the screen information acquisition unit 311A.

[0044] The near-infrared image information acquisition unit 411 acquires near-infrared image information based on an image captured by a near-infrared camera 41 installed in the cabin of the vehicle 100. Specifically, the near-infrared image information acquisition unit 411 has a device information acquisition unit 411A and a movement information acquisition unit 411B, and acquires the device information and movement information contained in the near-infrared image information via the in-vehicle network 3.

[0045] The device information acquisition unit 411A acquires device information indicating external characteristics of the mobile device such as the shape, size, and screen position of the mobile device. The device information includes at least information related to the position coordinates and displacement of the mobile device, and the position coordinates and displacement of the screen of the mobile device. The movement information acquisition unit 411B acquires movement information indicating the positions and movements of specific body parts of the driver. The movement information includes at least the position coordinates of the specific body parts of the driver, such as the hands and fingers, and information regarding their displacement.

[0046] By acquiring the device information and movement information, the warning level determination unit 511A described later can determine, in addition to the screen information, whether there is any movement that suggests the operation of the mobile device, such as the driver's fingers or hands touching the screen of the mobile device, and can determine the usage status of the mobile device.

[0047] The device information acquisition unit 411A and the movement information acquisition unit 411B can directly acquire movement information and device information extracted from near-infrared image information from the near-infrared camera ECU 40 via the in-vehicle network 3, and may also acquire an image captured by the near-infrared camera 41 and obtain movement information and screen information from the image by calculation.

[0048] The warning unit 511 determines the driver's use state of the portable device based on the near-infrared image information and the visible light image information, and issues a warning to the driver according to the determination result. More specifically, the warning unit 511 determines the driver's state and the driver's use state of the portable device based on the screen information included in the color image information and the device information and movement information included in the near-infrared image information, and issues a warning to the driver according to the driver's state and the driver's use state of the portable device.

[0049] The warning unit 511 may determine the usage status of the portable device using a trained model that has learned screen information, device information, and movement information. In this case, the trained model that has learned screen information, device information, and movement information is stored in advance in the storage unit 14. Then, the CPU 11 functioning as the warning unit 511 can read the trained model stored in the storage unit 14 and input the screen information, device information, and movement information to output the usage status of the portable device.

[0050] Alternatively, a trained model that has learned color image information and near-infrared image information may be stored in the storage unit 14. In this case, the warning unit 511 inputs the color image information and near-infrared image information into the trained model, and the trained model extracts screen information contained in the color image information and device information and movement information contained in the near-infrared image information, and can determine the usage state of the portable device from the extracted information.

[0051] Furthermore, when issuing a warning, warning unit 511 stops the warning or reduces the level of the warning if the function currently being used in the portable device is a permitted function whose use has been permitted in advance.

[0052] For this reason, warning unit 511 has warning level determination unit 511A that determines a warning level according to the driver's state and the state of use of the portable device by the driver, based on device information, movement information, and screen information. Note that warning level determination unit 511A can determine multiple stages of warning levels according to the driver's state and the state of use of the portable device by the driver.

[0053] In addition, the warning level determination unit 511A may, for example, obtain monitoring information based on the results of monitoring the occupant's status, such as their seating position and line of sight, from the occupant monitoring ECU 20 via the in-vehicle network 3, and use this information to determine the warning level.

[0054] For example, by acquiring gaze information on the driver's line of sight from among the monitoring information, it is possible to determine whether the driver is looking ahead (whether the driver is not looking away from the road) or whether the driver is staring at the screen of a mobile device, which can contribute to determining a more appropriate warning level. The gaze information acquired from the occupant monitoring ECU 20 calculates the direction of the driver's line of sight and the position coordinates of a target object ahead of the line of sight from an image captured by the camera 21, and includes at least one of information on the direction of the line of sight, the calculated position coordinates, and the displacement of the position coordinates.

[0055] Furthermore, the warning level determination unit 511A can acquire control information indicating whether the vehicle 100 is traveling under control in the advanced driving assistance mode and speed information indicating the vehicle's traveling speed, and use this information to determine the warning level. When acquiring the speed information, the speed information can be reflected in the final determination result, for example, by increasing the warning level as the vehicle speed increases, thereby making it possible to more accurately determine the warning level. Note that the advanced driving assistance mode is, for example, a mode that assists driving by partially automating driving under certain conditions.

[0056] The warning unit 511 issues a warning according to the warning level determined by the warning level determination unit 511 A. Specifically, the warning unit 511 outputs a control signal related to the warning according to the above-mentioned warning level to the notification ECU 50 via the in-vehicle network 3, and issues a warning to the driver via the notification ECU 50.

[0057] The warning unit 511 can warn the driver by, for example, outputting a warning message by voice, displaying a warning message, sounding a warning sound, vibrating the driver's seat, vibrating the steering wheel, etc. The warning unit 511 then changes the volume of the warning sound or the voice of the warning message, or changes the strength of the vibration of the seat or steering wheel, depending on the warning level, to issue a warning according to the warning level.

[0058] The storage unit 14 stores information received from the screen information acquisition unit 311A, the device information acquisition unit 411A, and the movement information acquisition unit 411B. The storage unit 14 may also store a table in which the driver's state and the state of use of the mobile device by the driver are associated with warning levels corresponding to the states, or a table in which the warning levels are associated with warning messages corresponding to the warning levels.

[0059] The warning process in the warning ECU 10 configured as above will be described below with reference to the flowchart showing the flow of the warning process in Fig. 5. In the following description, it is assumed that the mobile device uses the navigation function as an allowed function.

[0060] In the warning ECU 10, the screen information acquisition unit 311A acquires screen information indicating the screen status and display content of the portable device (step S101), the device information acquisition unit 411A acquires device information indicating the shape and position of the portable device (step S102), and the movement information acquisition unit 411B acquires movement information indicating the position and movement of specific parts of the driver's body such as hands and fingers (step S103), and outputs each to the warning level determination unit 511A.

[0061] If the screen information includes function specification information, warning level determination unit 511A determines whether the function being used on the portable device is a navigation function based on the function specification information (step S104). If warning level determination unit 511A determines based on the function specification information that the function being used on the portable device is not a navigation function, that is, not an permitted function (No in step S104), it determines that a function other than a permitted function is being used, and performs warning level determination processing for when a general function is being used (step S105).

[0062] On the other hand, if it is determined that the function being used in the portable device is the navigation function (Yes in step S104), a warning level determination process for when a permitted function is used (step S106) is performed. When the warning level is determined by the warning level determination process (step S105 or step S106), the warning unit 511 determines the warning content according to the determined warning level (step S107). Subsequently, a control signal indicating the warning content determined in step S107 is output to the notification ECU 50 (step S108), and a warning is issued to the driver.

[0063] The warning level determination unit 511A determines the warning level by grasping the driver's condition and the usage state of the mobile device based on the acquired screen information, device information, and movement information, for example, whether the screen of the mobile device is on, whether the driver's hands or fingers are moving in a way that suggests the mobile device is being operated, whether the mobile device is in a predetermined position, etc. At this time, if a permitted function is being used, the warning level is set lower than the warning level when a general function is being used, thereby allowing the use of the permitted function within a range that ensures safety and improving convenience.

[0064] The warning level determination process when an authorized function is used (step S106) and the warning level determination process when a general function is used (step S105) may be separate and independent determination processes, or in the warning level determination process when an authorized function is used (step S106), a determination process similar to the warning level determination process when a general function is used (step S105) may be performed, and if it is determined that an authorized function is being used, a warning level lower than the warning level when a general function is used may be set.

[0065] In this way, in the warning level determination unit 511A of the vehicle warning device according to this embodiment, if the function currently being used on the portable device is a permitted function that has been permitted to be used in advance, and if a predetermined condition is met, a warning level lower than the warning level when a general function is being used is selected. In other words, if a portable device is used for a general function other than a permitted function while the vehicle is running, a warning is issued to restrict this, and the restriction on the use of a specific permitted function is relaxed so that it is permitted only when a predetermined condition is met, thereby achieving both safety and convenience.

[0066] As described above, warning level determination unit 511A acquires the above-mentioned screen information, device information, and movement information from the color image information acquired by color camera 31 and the near-infrared image information acquired by near-infrared camera 41. The near-infrared image information reflects differences in the characteristics of light reflection and absorption due to differences in the material of the subject, so it is possible to accurately grasp the shape of the portable device, the size of the portable device itself, the position of the screen of the portable device, the size of the screen, specific parts of the driver's hands and fingers, and the positions of the specific parts, regardless of the brightness of the interior of the vehicle.

[0067] Therefore, for example, even if the driver's hand or finger touches the screen of the portable device and partially covers the screen, it is possible to identify the screen, hand, finger, etc. of the portable device at a material level. On the other hand, color image information can accurately grasp information displayed on the screen of the portable device, so it is possible to obtain specific function information displayed on the screen.

[0068] In other words, in the vehicle warning device of this embodiment, by using color image information acquired by color camera 31 and near-infrared image information acquired by near-infrared camera 41, it is possible to more accurately grasp the driver's condition and the usage status of the mobile device, and to provide the driver with an optimal warning.

[0069] As shown in FIG. 1, by installing a vehicle control system 1 including such a vehicle warning device (warning ECU) in a vehicle 100, the vehicle 100 can provide appropriate warnings (attention) to the driver when using a portable device while the vehicle is traveling, while relaxing usage restrictions on predetermined permitted functions such as navigation functions, thereby achieving both safety and convenience.

[0070] As described above, the vehicle warning device according to this embodiment can warn the driver to restrict the operation of a mobile device while the vehicle is in motion, while relaxing restrictions on operations related to specific functions, thereby achieving both safety and convenience.

[0071] Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configurations are not limited to these embodiments, and the present invention also includes design changes within the scope of the present invention. Furthermore, the above-described embodiments can be combined by utilizing each other's technologies as long as there are no particular contradictions or problems in their purposes, configurations, etc. [Explanation of symbols]

[0072] 1: Vehicle control system, 3: In-vehicle network, 4: CGW, 5: Steering 10: Warning ECU, 11: CPU, 14: Storage section 20: Occupant monitoring ECU, 21: Camera, 22: Microphone 30: Color camera ECU, 31: Color camera, 32: Meter 40: Near-infrared camera ECU, 41: Near-infrared camera, 43: Dashboard 50: Notification ECU, 51: Speaker, 52: Display device 80: Mobile device, 81: Specific body part (hand), 82: QR code (registered trademark) 100: vehicle, 311: color image information acquisition unit, 311A: screen information acquisition unit 411: Near-infrared image information acquisition unit, 411A: Device information acquisition unit 411B: movement information acquisition unit, 511: warning unit, 511A: warning level determination unit

Claims

1. A warning device for a vehicle that issues a warning to a driver according to the driver's state and the driver's use of a portable device while the vehicle is running, a near-infrared image information acquisition unit that acquires near-infrared image information based on an image captured by a near-infrared camera installed in a passenger compartment of the vehicle; a visible light image information acquisition unit that acquires visible light image information based on an image captured by a visible light camera installed in the vehicle interior; a warning unit that determines a use state of the portable device by the driver based on the near-infrared image information and the visible light image information, and issues a warning to the driver in accordance with the determination result; The warning unit stops or reduces the level of the warning when the function currently being used in the portable device is a permitted function that has been permitted to be used in advance.

2. the near-infrared image information includes movement information indicating the position and movement of a specific part of the driver, and device information including the position and appearance characteristics of a mobile device; 2. The warning device for a vehicle according to claim 1, wherein the visible light image information includes screen information indicating a display content on a screen of the portable device.

3. The vehicle warning device according to claim 2 , wherein the warning unit determines whether a function currently being used on the portable device is an allowed function based on the screen information.

4. 4. The vehicle warning device according to claim 3, wherein the permission function is a navigation function.

5. 3. The warning device for a vehicle according to claim 2, wherein the screen information includes lighting information indicating a lighting state of the screen of the portable device and function specification information that is displayed on the screen while the permitted function is being used.

6. The warning device for a vehicle according to claim 5, wherein the function specifying information is an icon or a two-dimensional code.

7. The vehicle warning device according to claim 1 , wherein the warning unit reduces the level of the warning when the vehicle is traveling in an advanced driving assistance mode.

8. The vehicle warning device according to claim 1 , wherein the warning unit acquires speed information indicating a traveling speed of the vehicle, and raises the level of the warning as the traveling speed becomes faster.

9. The warning device for a vehicle according to claim 1 , wherein the near-infrared camera and the visible light camera are mounted on a ceiling or a pillar inside the vehicle compartment.

Images